Introduction

The hdar R package provides seamless access to the WEkEO Harmonised Data Access (HDA) API, enabling users to programmatically query and download data from within R.

Accessing the HDA Service

To utilize the HDA service and library, you must first register for a WEkEO account. Copernicus data and the HDA service are available at no cost to all WEkEO users. Creating an account allows you full access to our services, ensuring you can leverage the full capabilities of HDA seamlessly. Registration is straightforward and can be completed through the following link: Register for WEkEO. Once your account is set up, you will be able to access the HDA services immediately.

Installation

To start using the hdar package, you first need to install and load it in your R environment.

# Install stable hdar from CRAN
if(!require("hdar")){install.packages("hdar")}
# or develop version from GitHub
# devtools::install_github("eea/hdar@develop")

# Load the hdar package
library(hdar)

Authentication

To interact with the HDA service, you need to authenticate by providing your username and password. The Client class allows you to pass these credentials directly and optionally save them to a configuration file for future use. If credentials are not specified as parameters, the client will read them from the ~/.hdarc file.

Creating and Authenticating the Client

You can create an instance of the Client class by passing your username and password directly. TThe initialization method has an optional parameter save_credentials that specifies whether the provided credentials should be saved in the ~/.hdarc configuration file. By default, save_credentials is set to FALSE.

Example: Pass User/Password at Initialization

Here is an example of how to authenticate by passing the user and password, and optionally saving these credentials:

# Define your username and password
username <- "your_username"
password <- "your_password"

# Create an instance of the Client class and save credentials to a config file
# The save_credentials parameter is optional and defaults to FALSE
client <- Client$new(username, password, save_credentials = TRUE)

If the save_credentials parameter is set to TRUE, the credentials will be saved in the ~/.hdarc file, making it easier to authenticate in future sessions without passing the credentials again.

Example: Using Saved Credentials

Once the credentials are saved, you can initialize the Client class without passing the credentials. The client will read the credentials from the ~/.hdarc file:

# Create an instance of the Client class without passing credentials
client <- Client$new()

Checking Authentication

Once the client is created, you can check if it has been authenticated properly by calling a method token() that verifies authentication. For example:

# Example method to check authentication by getting the auth token
client$get_token()

Copernicus Terms and Conditions (T&C)

Copernicus data is free to use and modify, still T&Cs must be accepted in order to download the data. hdarc offers a confortable functionality to read and accept/reject T&C of the individual Copernicus service:

client$show_terms()

Will open a browser where you can read all the available T&Cs. To accept/reject individual T&Cs or all at once use:

client$terms_and_conditions()

                                               term_id accepted
1                           Copernicus_General_License    FALSE
2                          Copernicus_Sentinel_License    FALSE
3                       EUMETSAT_Core_Products_Licence    FALSE
4                     EUMETSAT_Copernicus_Data_Licence    FALSE
5  Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m    FALSE
6  Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m    FALSE
7                             Copernicus_ECMWF_License    FALSE
8       Copernicus_Land_Monitoring_Service_Data_Policy    FALSE
9            Copernicus_Marine_Service_Product_License    FALSE
10                        CNES_Open_2.0_ETALAB_Licence    FALSE

client$terms_and_conditions(term_id = 'all')
                                              term_id accepted
1                           Copernicus_General_License     TRUE
2                          Copernicus_Sentinel_License     TRUE
3                       EUMETSAT_Core_Products_Licence     TRUE
4                     EUMETSAT_Copernicus_Data_Licence     TRUE
5  Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m     TRUE
6  Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m     TRUE
7                             Copernicus_ECMWF_License     TRUE
8       Copernicus_Land_Monitoring_Service_Data_Policy     TRUE
9            Copernicus_Marine_Service_Product_License     TRUE
10                        CNES_Open_2.0_ETALAB_Licence     TRUE

Finding Datasets

WEkEO offers a vast amount of different products. To find what you need the Client class provides a method called datasets that lists available datasets, optionally filtered by a text pattern.

Basic Usage

The basic usage of the datasets method is straightforward. You can retrieve a list of all datasets available on WEkEO by calling the datasets method on an instance of the Client class.

# retrieving the full list
# This takes about 2 minutes!
all_datasets <- client$datasets()

# list all datasets IDs on WEkEO
sapply(all_datasets,FUN = function(x){x$dataset_id})

Filtering Datasets

You can also filter the datasets by providing a text pattern. This is useful when you are looking for datasets that match a specific keyword or phrase.

filtered_datasets <- client$datasets("Seasonal Trajectories")

# list dataset IDs
sapply(filtered_datasets,FUN = function(x){x$dataset_id})
[1] "EO:EEA:DAT:CLMS_HRVPP_VPP-LAEA" "EO:EEA:DAT:CLMS_HRVPP_ST"       "EO:EEA:DAT:CLMS_HRVPP_ST-LAEA"
[4] "EO:EEA:DAT:CLMS_HRVPP_VPP"


filtered_datasets <- client$datasets("Baltic")

# list dataset IDs
sapply(filtered_datasets,FUN = function(x){x$dataset_id})
 [1] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_BGC_003_007:cmems_mod_bal_bgc-pp_anfc_P1D-i_202311"
 [2] "EO:MO:DAT:NWSHELF_MULTIYEAR_PHY_004_009:cmems_mod_nws_phy-sst_my_7km-2D_PT1H-i_202112"
 [3] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L4_MY_009_134:cmems_obs-oc_bal_bgc-plankton_my_l4-multi-1km_P1M_202211"
 [4] "EO:MO:DAT:SST_BAL_PHY_SUBSKIN_L4_NRT_010_034:cmems_obs-sst_bal_phy-subskin_nrt_l4_PT1H-m_202211"
 [5] "EO:MO:DAT:BALTICSEA_MULTIYEAR_PHY_003_011:cmems_mod_bal_phy_my_P1Y-m_202303"
 [6] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L3_NRT_009_131:cmems_obs-oc_bal_bgc-transp_nrt_l3-olci-300m_P1D_202207"
 [7] "EO:MO:DAT:BALTICSEA_MULTIYEAR_BGC_003_012:cmems_mod_bal_bgc_my_P1Y-m_202303"
 [8] "EO:MO:DAT:SST_BAL_SST_L4_REP_OBSERVATIONS_010_016:DMI_BAL_SST_L4_REP_OBSERVATIONS_010_016_202012"
 [9] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_PHY_003_006:cmems_mod_bal_phy_anfc_PT15M-i_202311"
[10] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L3_MY_009_133:cmems_obs-oc_bal_bgc-plankton_my_l3-multi-1km_P1D_202207"
[11] "EO:MO:DAT:SST_BAL_PHY_L3S_MY_010_040:cmems_obs-sst_bal_phy_my_l3s_P1D-m_202211"
[12] "EO:MO:DAT:SEAICE_BAL_SEAICE_L4_NRT_OBSERVATIONS_011_004:FMI-BAL-SEAICE_THICK-L4-NRT-OBS"
[13] "EO:MO:DAT:SEAICE_BAL_PHY_L4_MY_011_019:cmems_obs-si_bal_seaice-conc_my_1km_202112"
[14] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_WAV_003_010:cmems_mod_bal_wav_anfc_PT1H-i_202311"
[15] "EO:MO:DAT:BALTICSEA_REANALYSIS_WAV_003_015:dataset-bal-reanalysis-wav-hourly_202003"
[16] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L4_NRT_009_132:cmems_obs-oc_bal_bgc-plankton_nrt_l4-olci-300m_P1M_202207"
[17] "EO:MO:DAT:SST_BAL_SST_L3S_NRT_OBSERVATIONS_010_032:DMI-BALTIC-SST-L3S-NRT-OBS_FULL_TIME_SERIE_201904"

Understanding the Results

The datasets method returns a list containing datasets and associated information. This information may include dataset names, descriptions, and other metadata.

client$datasets("EO:ECMWF:DAT:DERIVED_NEAR_SURFACE_METEOROLOGICAL_VARIABLES")
[[1]]
[[1]]$terms
[[1]]$terms[[1]]
[1] "Copernicus_ECMWF_License"

[[1]]$dataset_id
[1] "EO:ECMWF:DAT:DERIVED_NEAR_SURFACE_METEOROLOGICAL_VARIABLES"

[[1]]$title
[1] "Near surface meteorological variables from 1979 to 2019 derived from bias-corrected reanalysis"

[[1]]$abstract
[1] "This dataset provides bias-corrected reconstruction of near-surface meteorological variables derived from the fifth generation of the European Centre for Medium-Range Weather Forecasts  (ECMWF) atmospheric reanalyses (ERA5). It is intended to be used as a meteorological forcing dataset for land surface and hydrological models. \nThe dataset has been obtained using the same methodology used to derive the widely used water, energy and climate change (WATCH) forcing data, and is thus also referred to as WATCH Forcing Data methodology applied to ERA5 (WFDE5). The data are derived from the ERA5 reanalysis product that have been re-gridded to a half-degree resolution. Data have been adjusted using an elevation correction and monthly-scale bias corrections based on Climatic Research Unit (CRU) data (for temperature, diurnal temperature range, cloud-cover, wet days number and precipitation fields) and Global Precipitation Climatology Centre (GPCC) data (for precipitation fields only). Additional corrections are included for varying atmospheric aerosol-loading and separate precipitation gauge observations. For full details please refer to the product user-guide.\nThis dataset was produced on behalf of Copernicus Climate Change Service (C3S) and was generated entirely within the Climate Data Store (CDS) Toolbox. The toolbox source code is provided in the documentation tab.\n\nVariables in the dataset/application are:\nGrid-point altitude, Near-surface air temperature, Near-surface specific humidity, Near-surface wind speed, Rainfall flux, Snowfall flux, Surface air pressure, Surface downwelling longwave radiation, Surface downwelling shortwave radiation"

[[1]]$doi
NULL

[[1]]$thumbnails
[1] "https://datastore.copernicus-climate.eu/c3s/published-forms-v2/c3sprod/derived-near-surface-meteorological-variables/overview.jpg"

Creating a Query Template

To search for a specific product, you need to create a query template. You can either use the WEkEO viewer and copy paste the JSON query:

A Guide to the hdar package