Add a Provider#
eodag provides a set of plugins which don’t know anything about the
providers themselves, they just implement generic methods required to talk to different kinds of data catalog. For instance:
QueryStringSearch: Search plugin that implements a search protocol that relies on query stringsHTTPHeaderAuth: Authentication plugin that implements HTTP authentication using headersHTTPDownload: Download plugin that implements download over HTTP protocol
Dynamically add a new provider#
You can dynamically add a new provider, from your python code using add_provider()
or update_providers_config() methods.
Check Python API User Guide / Add-or-update-a-provider for guidelines.
Configure a new provider#
The simplest way to add a new provider is to use existing plugins. This approach requires to
provide the new provider’s configuration in a YAML format. The following example shows how to add a new STAC provider:
another_earth_search:
search:
type: StacSearch
api_endpoint: https://earth-search.aws.element84.com/v1/search
need_auth: false
products:
S2_MSI_L1C:
productType: sentinel-2-l1c
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download:
type: AwsDownload
auth:
type: AwsAuth
credentials:
aws_access_key_id: PLEASE_CHANGE_ME
aws_secret_access_key: PLEASE_CHANGE_ME
It configures the following existing plugins: StacSearch (search),
AwsAuth (authentication) and AwsDownload (download).
Each plugin configuration is inserted following the appropriate plugin topic key:
searchfor search pluginsdownloadfor download pluginsauth,search_auth, ordownload_authfor authentication pluginsapifor api plugins
Of course, it is also necessary to know how to configure these plugins (which parameters they take, what values they can have, etc.).
You can get some inspiration from the Providers pre-configuration section by analysing how eodag configures the providers it comes installed with.
Add more plugins#
eodag is a plugin-oriented framework which means it can be easily extended. If the plugins it offers are not sufficient for your
own needs (i.e. getting data from a provider not supported by eodag), you should then write your own plugins (possibly by extending one of provided by eodag)
and configure them. What you are the most likely to be willing to do is either to develop a new Search plugin or an
Api plugin (e.g. to create an interface with another program).
eodag-sentinelsat (archived) was a good example of an Api plugin. It created an interface with the sentinalsat library to search and download products from SciHub.
See more details about how to create a new plugin in this dedicated section.
Providers pre-configuration#
All the providers are pre-configured in eodag in a YAML file.
Click on the link below to display its full content.
providers.yml
---
!provider # MARK: usgs
name: usgs
priority: 0
description: U.S geological survey catalog for Landsat products
roles:
- host
url: https://earthexplorer.usgs.gov/
api: !plugin
type: UsgsApi
need_auth: true
pagination:
max_items_per_page: 5000
total_items_nb_key_path: '$.totalHits'
metadata_mapping:
id:
- '_ID'
- '$.displayId'
geometry: '$.spatialBounds'
productType: '$.productType'
title: '$.displayId'
abstract: '$.summary'
cloudCover: '$.cloudCover'
startTimeFromAscendingNode: '$.temporalCoverage.startDate'
completionTimeFromAscendingNode: '$.temporalCoverage.endDate'
publicationDate: '$.publishDate'
thumbnail: '$.browse[0].thumbnailPath'
quicklook: '$.browse[0].browsePath'
storageStatus: '{$.available#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
downloadLink: 'https://earthexplorer.usgs.gov/download/external/options/{productType}/{entityId}/M2M/'
# metadata needed for download
entityId: '$.entityId'
productId: '$.id'
extract: True
order_enabled: true
products:
# datasets list http://kapadia.github.io/usgs/_sources/reference/catalog/ee.txt may be outdated
# see also https://dds.cr.usgs.gov/ee-data/coveragemaps/shp/ee/
LANDSAT_C2L1:
dataset: landsat_ot_c2_l1
LANDSAT_C2L2:
dataset: landsat_ot_c2_l2
LANDSAT_TM_C1:
dataset: landsat_tm_c1
LANDSAT_TM_C2L1:
dataset: landsat_tm_c2_l1
LANDSAT_TM_C2L2:
dataset: landsat_tm_c2_l2
LANDSAT_ETM_C1:
dataset: landsat_etm_c1
LANDSAT_ETM_C2L1:
dataset: landsat_etm_c2_l1
LANDSAT_ETM_C2L2:
dataset: landsat_etm_c2_l2
S2_MSI_L1C:
dataset: SENTINEL_2A
GENERIC_PRODUCT_TYPE:
dataset: '{productType}'
---
!provider # MARK: aws_eos
name: aws_eos
priority: 0
description: EOS search for Amazon public datasets
roles:
- host
url: https://developers.eos.com/datasets_description.html
search: !plugin
type: PostJsonSearch
api_endpoint: 'https://gate.eos.com/api/lms/search/v2/{collection}'
need_auth: true
auth_error_code:
- 402
- 403
results_entry: 'results'
ssl_verify: true
pagination:
next_page_query_obj: '{{"limit":{items_per_page},"page":{page}}}'
total_items_nb_key_path: '$.meta.found'
# 2021/04/28: aws_eos doesn't specify a limit in its docs. It says that the default
# value is 500 (https://doc.eos.com/search.api/#single-dataset-search).
# Let's set it to this value for now
max_items_per_page: 500
discover_metadata:
auto_discovery: true
metadata_pattern: '^[a-zA-Z0-9_]+$'
search_param: '{{{{"search":{{{{"{metadata}":"{{{metadata}}}" }}}} }}}}'
metadata_path: '$.*'
metadata_mapping:
# landsat8_downloadLink : 's3://landsat-pds/c{storedInCollection}/L8/{path}/{row}/{productID}'
geometry:
- '{{"search":{{"shape": {geometry#to_geojson} }} }}'
- '$.dataGeometry'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
products:
CBERS4_PAN10M_L2:
instrument: PAN10M
collection: cbers4
processingLevel: 2
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
platformSerialIdentifier:
- '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
- '$.satelliteName'
instrument:
- '{{"search":{{"sensor":"{instrument}" }} }}'
- '$.sensor'
processingLevel:
- '{{"search":{{"processingLevel":"{processingLevel}" }} }}'
- '$.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"sceneID":"{title}" }} }}'
- '$.sceneID'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.date'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.date'
illuminationAzimuthAngle:
- '{{"search":{{"sunAzimuth":"{illuminationAzimuthAngle}" }} }}'
- '$.sunAzimuth'
illuminationElevationAngle:
- '{{"search":{{"sunElevation":"{illuminationElevationAngle}" }} }}'
- '$.sunElevation'
# Custom parameters (not defined in the base document referenced above)
awsPath:
- '{{"search":{{"downloadUrl":"{awsPath}" }} }}'
- '$.downloadUrl'
downloadLink: 's3://cbers-pds/{awsPath}'
metaDownloadLink: 's3://cbers-meta-pds/{awsPath}'
previewBaseName: '{$.sceneID#replace_str("_L2","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
id:
- '{{"search":{{"sceneID":"{id}" }} }}'
- '{title}'
CBERS4_PAN10M_L4:
instrument: PAN10M
collection: cbers4
processingLevel: 4
metadata_mapping_from_product: CBERS4_PAN10M_L2
metadata_mapping:
# Custom parameters (not defined in the base document referenced above)
previewBaseName: '{$.sceneID#replace_str("_L4","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
CBERS4_PAN5M_L2:
instrument: PAN5M
collection: cbers4
processingLevel: 2
metadata_mapping_from_product: CBERS4_PAN10M_L2
CBERS4_PAN5M_L4:
instrument: PAN5M
collection: cbers4
processingLevel: 4
metadata_mapping_from_product: CBERS4_PAN10M_L2
metadata_mapping:
# Custom parameters (not defined in the base document referenced above)
previewBaseName: '{$.sceneID#replace_str("_L4","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
CBERS4_MUX_L2:
instrument: MUX
collection: cbers4
processingLevel: 2
metadata_mapping_from_product: CBERS4_PAN10M_L2
CBERS4_MUX_L4:
instrument: MUX
collection: cbers4
processingLevel: 4
metadata_mapping_from_product: CBERS4_PAN10M_L2
metadata_mapping:
# Custom parameters (not defined in the base document referenced above)
previewBaseName: '{$.sceneID#replace_str("_L4","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
CBERS4_AWFI_L2:
instrument: AWFI
collection: cbers4
processingLevel: 2
metadata_mapping_from_product: CBERS4_PAN10M_L2
CBERS4_AWFI_L4:
instrument: AWFI
collection: cbers4
processingLevel: 4
metadata_mapping_from_product: CBERS4_PAN10M_L2
metadata_mapping:
# Custom parameters (not defined in the base document referenced above)
previewBaseName: '{$.sceneID#replace_str("_L4","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'
quicklook: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}.jpg'
L8_OLI_TIRS_C1L1:
collection: landsat8
onAmazon: true
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
platform:
- '{{"search":{{"satelliteName":"{platform}" }} }}'
- '$.satelliteName'
instrument:
- '{{"search":{{"sensor":"{instrument}" }} }}'
- '$.sensor'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"productID":"{title}" }} }}'
- '$.productID'
# OpenSearch Parameters for Product Search (Table 5)
cloudCover:
- '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
- '$.cloudCoverage'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
availabilityTime:
- '{{"search":{{"IndexingTime":{{"from":"{availabilityTime}"}} }} }}'
- '$.IndexingTime'
acquisitionStation:
- '{{"search":{{"receivingStation":{{"from":"{acquisitionStation}"}} }} }}'
- '$.receivingStation'
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.sceneStartTime'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.sceneStopTime'
illuminationAzimuthAngle:
- '{{"search":{{"sunAzimuth":"{illuminationAzimuthAngle}" }} }}'
- '$.sunAzimuth'
illuminationElevationAngle:
- '{{"search":{{"sunElevation":"{illuminationElevationAngle}" }} }}'
- '$.sunElevation'
# Custom parameters (not defined in the base document referenced above)
onAmazon:
- '{{"search":{{"onAmazon":"{onAmazon}" }} }}'
- '$.onAmazon'
path:
- '{{"search":{{"path":"{path}" }} }}'
- '$.path'
row:
- '{{"search":{{"row":"{row}" }} }}'
- '$.row'
downloadLink: 's3://landsat-pds/c1/L8/{path:03.0f}/{row:03.0f}/{title}/'
thumbnail: '$.thumbnail'
quicklook: 'https://landsat-pds.s3.amazonaws.com/c1/L8/{path:03.0f}/{row:03.0f}/{title}/{title}_thumb_large.jpg'
id:
- '{{"search":{{"productID":"{id}" }} }}'
- '{title}'
MODIS_MCD43A4:
collection: modis
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
instrument:
- '{{"search":{{"satelliteName":"{instrument}" }} }}'
- '$.satelliteName'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"sceneID":"{title}" }} }}'
- '$.sceneID'
# OpenSearch Parameters for Product Search (Table 5)
cloudCover:
- '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
- '$.cloudCoverage'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.BeginningDateTime'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.EndingDateTime'
# Custom parameters (not defined in the base document referenced above)
verticalTileNumber:
- '{{"search":{{"verticalTileNumber":"{verticalTileNumber}" }} }}'
- '$.verticalTileNumber'
horizontalTileNumber:
- '{{"search":{{"horizontalTileNumber":"{horizontalTileNumber}" }} }}'
- '$.horizontalTileNumber'
doyDate: '{$.sceneID#slice_str(9,16,1)}'
downloadLink: 's3://modis-pds/MCD43A4.006/{horizontalTileNumber:02.0f}/{verticalTileNumber:02.0f}/{doyDate}/'
thumbnail: '$.thumbnail'
quicklook: '$.thumbnail'
id:
- '{{"search":{{"sceneID":"{id}" }} }}'
- '{title}'
NAIP:
collection: naip
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
platformSerialIdentifier:
- '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
- '$.satelliteName'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"sceneID":"{title}" }} }}'
- '$.sceneID'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.date'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.date'
# Custom parameters (not defined in the base document referenced above)
awsPath:
- '{{"search":{{"awsPath":"{awsPath}" }} }}'
- '$.awsPath'
downloadLink: 's3://naip-analytic/{awsPath}'
id:
- '{{"search":{{"sceneID":"{id}" }} }}'
- '{title}'
S1_SAR_GRD:
productType: GRD
collection: sentinel1
metadata_mapping:
cloudCover: '$.null'
# OpenSearch Parameters for Collection Search (Table 3)
platformSerialIdentifier:
- '{{"search":{{"missionId":"{platformSerialIdentifier}" }} }}'
- '$.missionId'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"sceneID":"{title}" }} }}'
- '$.sceneID'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- '{{"search":{{"absoluteOrbitNumber":"{orbitNumber}" }} }}'
- '$.absoluteOrbitNumber'
orbitDirection:
- '{{"search":{{"passDirection":"{orbitDirection}" }} }}'
- '{$.passDirection#to_lower}'
sensorMode:
- '{{"search":{{"mode":"{sensorMode}" }} }}'
- '$.mode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.date'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.date'
polarizationChannels:
- '{{"search":{{"polarization":"{polarizationChannels#replace_str_tuple((("HH","SH"),("VV","SV"),("HH+HV", "DH"),("VV+VH","DV")))}" }} }}'
- '{$.polarization#replace_str_tuple((("SH","HH"),("SV","VV"),("DH","HH+HV"),("DV","VV+VH")))}'
# Custom parameters (not defined in the base document referenced above)
awsPath:
- '{{"search":{{"awsPath":"{awsPath}" }} }}'
- '$.awsPath'
downloadLink: 's3://sentinel-s1-l1c/{awsPath}'
thumbnail: 'https://render.eosda.com/S1/thumb/{title}.png'
quicklook: 'https://render.eosda.com/S1/thumb/{title}.png'
id:
- '{{"search":{{"sceneID":"{id}" }} }}'
- '{title}'
S2_MSI_L1C:
collection: sentinel2
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
platformSerialIdentifier:
- '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
- '$.satelliteName'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title:
- '{{"search":{{"productName":"{title}" }} }}'
- '$.productName'
# OpenSearch Parameters for Product Search (Table 5)
cloudCover:
- '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
- '$.cloudCoverage'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.timestamp'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.timestamp'
illuminationAzimuthAngle:
- '{{"search":{{"azimuthAngle":"{illuminationAzimuthAngle}" }} }}'
- '$.azimuthAngle'
illuminationZenithAngle:
- '{{"search":{{"zenithAngle":"{illuminationZenithAngle}" }} }}'
- '$.zenithAngle'
# Custom parameters (not defined in the base document referenced above)
originalTileMeta: '{$.originalTileMeta#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
thumbnail: '{$.thumbnail#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
awsPath:
- '{{"search":{{"awsPath":"{awsPath}" }} }}'
- '$.awsPath'
downloadLink: 's3://sentinel-s2-l1c/{awsPath}'
productMetaLink: '{$.productMetaLink#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
productPath:
- '{{"search":{{"productPath":"{productPath}" }} }}'
- '$.productPath'
id:
- '{{"search":{{"productName":"{id}" }} }}'
- '{title}'
processedL2A: '$.null'
awsPathL2A: '$.null'
S2_MSI_L2A:
collection: sentinel2
processedL2A: true
# specific QueryStringSearch usage for these parameters (replaces current query)
specific_qssearch:
parameters:
- title
- id
results_entry: '$'
collection:
- tileInfo
- productInfo
merge_responses: true
metadata_mapping:
title:
- 'title'
- '$.name'
id:
- 'title'
- '{title}'
awsPathL2A: '$.tiles[0].path'
downloadLink: 's3://sentinel-s2-l2a/{awsPathL2A}'
productPath: '$.path'
startTimeFromAscendingNode: '$.timestamp'
completionTimeFromAscendingNode: '$.timestamp'
geometry: '$.tileDataGeometry'
productInfo: 'https://roda.sentinel-hub.com/sentinel-s2-l2a/{awsPathL2A}/productInfo.json'
originalSceneID: '$.tiles[0].datastrip.id'
metadata_mapping:
# OpenSearch Parameters for Collection Search (Table 3)
productType: '$.null'
platformSerialIdentifier:
- '{{"search":{{"satelliteName":"{platformSerialIdentifier}" }} }}'
- '$.satelliteName'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '{$.productName#fake_l2a_title_from_l1c}'
# OpenSearch Parameters for Product Search (Table 5)
cloudCover:
- '{{"search":{{"cloudCoverage":"{cloudCover}" }} }}'
- '$.cloudCoverage'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"search":{{"date":{{"from":"{startTimeFromAscendingNode}"}} }} }}'
- '$.timestamp'
completionTimeFromAscendingNode:
- '{{"search":{{"date":{{"to":"{completionTimeFromAscendingNode}"}} }} }}'
- '$.timestamp'
illuminationAzimuthAngle:
- '{{"search":{{"azimuthAngle":"{illuminationAzimuthAngle}" }} }}'
- '$.azimuthAngle'
illuminationZenithAngle:
- '{{"search":{{"zenithAngle":"{illuminationZenithAngle}" }} }}'
- '$.zenithAngle'
# Custom parameters (not defined in the base document referenced above)
originalTileMeta: '{$.originalTileMeta#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l2a")}'
thumbnail: '{$.thumbnail#replace_str("sentinel-s2-l1c.s3.amazonaws.com","roda.sentinel-hub.com/sentinel-s2-l1c")}'
quicklook: '{thumbnail}'
downloadLink: 's3://sentinel-s2-l2a/{awsPathL2A}'
awsPath: '$.null'
productMetaLink: '$.null'
productPath: '$.null'
productInfo: 'https://roda.sentinel-hub.com/sentinel-s2-l2a/{awsPathL2A}/productInfo.json'
id:
- '{id#s2msil2a_title_to_aws_productinfo}'
- '{title}'
processedL2A:
- '{{"search":{{"processedL2A":"{processedL2A}" }} }}'
- '$.processedL2A'
awsPathL2A:
- '{{"search":{{"awsPathL2A":"{awsPathL2A}" }} }}'
- '$.awsPathL2A'
download: !plugin
type: AwsDownload
ssl_verify: true
products:
CBERS4_MUX_L2:
default_bucket: 'cbers-pds'
complementary_url_key:
- metaDownloadLink
CBERS4_AWFI_L2:
complementary_url_key:
- metaDownloadLink
CBERS4_PAN5M_L2:
complementary_url_key:
- metaDownloadLink
CBERS4_PAN10M_L2:
complementary_url_key:
- metaDownloadLink
S1_SAR_GRD:
default_bucket: 'sentinel-s1-l1c'
build_safe: true
S2_MSI_L1C:
default_bucket: 'sentinel-s2-l1c'
build_safe: true
complementary_url_key:
- productPath
S2_MSI_L2A:
default_bucket: 'sentinel-s2-l2a'
build_safe: true
fetch_metadata:
fetch_url: '{productInfo}'
fetch_format: json
update_metadata:
title: '$.name'
id: '{title}'
productPath: '$.path'
complementary_url_key:
- productPath
download_auth: !plugin
type: AwsAuth
matching_url: s3://
requester_pays: True
search_auth: !plugin
type: HttpQueryStringAuth
matching_url: https://gate.eos.com
ssl_verify: true
---
!provider # MARK: peps
name: peps
priority: 1
description: |
The PEPS platform, the French "mirror site", redistributes the products of Sentinel satellites,
S1A, S1B, S2A and S2B, S3A and S3B from COPERNICUS, the European system for the Earth monitoring.
roles:
- host
url: https://peps.cnes.fr
search: !plugin
type: QueryStringSearch
api_endpoint: 'https://peps.cnes.fr/resto/api/collections/{collection}/search.json'
need_auth: false
ssl_verify: true
pagination:
next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
total_items_nb_key_path: '$.properties.totalResults'
# 2021/03/19: 500 is the max, no error if greater
max_items_per_page: 500
discover_metadata:
auto_discovery: true
metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
search_param: '{metadata}={{{metadata}}}'
metadata_path: '$.properties.*'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
uid: '$.id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- productType
- '$.properties.productType'
platform: '$.properties.collection'
platformSerialIdentifier:
- platform
- '$.properties.platform'
instrument:
- instrument
- '$.properties.instrument'
processingLevel:
- processingLevel
- '$.properties.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '$.properties.title'
topicCategory: '$.properties.topicCategory'
keyword: '$.properties.keywords'
abstract: '$.properties.description'
resolution:
- 'resolution'
- '$.properties.resolution'
organisationName:
- 'organisationName'
- '$.properties.organisationName'
publicationDate: '$.properties.published'
accessConstraint: '$.properties.license'
# OpenSearch Parameters for Product Search (Table 5)
parentIdentifier:
- 'parentIdentifier'
- '$.properties.parentIdentifier'
acquisitionType: '$.properties.acquisitionType'
orbitNumber:
- 'orbitNumber'
- '$.properties.orbitNumber'
orbitDirection:
- 'orbitDirection'
- '$.properties.orbitDirection'
swathIdentifier:
- 'swath'
- '$.properties.swath'
cloudCover:
- 'cloudCover=[0,{cloudCover}]'
- '$.properties.cloudCover'
snowCover:
- 'snowCover=[0,{snowCover}]'
- '$.properties.snowCover'
productVersion: '$.properties.version'
productQualityStatus: '$.properties.onlineQualityCheck'
processorName: '$.properties.processingName'
processingCenter: '$.properties.processingCenter'
creationDate: '$.properties.dhusIngestDate'
modificationDate: '$.properties.updated'
sensorMode:
- 'sensorMode'
- '$.properties.sensorMode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- startDate
- '$.properties.startDate'
completionTimeFromAscendingNode:
- completionDate
- '$.properties.completionDate'
polarizationChannels:
- 'polarisation'
- '{$.properties.polarisation#replace_str(" ","+")}'
# Custom parameters (not defined in the base document referenced above)
id:
- 'productIdentifier={id#remove_extension}'
- '$.properties.productIdentifier'
tileIdentifier:
- 'tileid'
- '$.properties.mgrs'
# The geographic extent of the product
geometry:
- 'geometry={geometry#to_rounded_wkt}'
- '$.geometry'
# The url of the quicklook
quicklook: '$.properties.quicklook'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: '$.properties.services.download.url'
# storageStatus: must be one of ONLINE, STAGING, OFFLINE
storageStatus: '{$.properties.storage.mode#get_group_name((?P<ONLINE>disk|tier2)|(?P<STAGING>staging)|(?P<OFFLINE>unknown|tape|tier3))}'
# Additional metadata provided by the providers but that don't appear in the reference spec
thumbnail: '$.properties.thumbnail'
links: $.null
services: $.null
products:
S1_SAR_OCN:
productType: OCN
collection: S1
metadata_mapping:
cloudCover: '$.null'
S1_SAR_GRD:
productType: GRD
collection: S1
metadata_mapping:
cloudCover: '$.null'
S1_SAR_SLC:
productType: SLC
collection: S1
metadata_mapping:
cloudCover: '$.null'
S2_MSI_L1C:
collection: S2ST
productType: S2MSI1C
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
download: !plugin
type: HTTPDownload
extract: true
archive_depth: 2
order_enabled: true
auth_error_code: 401
ssl_verify: true
dl_url_params:
issuerId: peps
auth: !plugin
type: GenericAuth
matching_url: https://peps.cnes.fr
---
!provider # MARK: creodias
name: creodias
priority: 0
description: CloudFerro DIAS
roles:
- host
url: https://creodias.eu/
search: !plugin
type: ODataV4Search
api_endpoint: 'https://datahub.creodias.eu/odata/v1/Products'
need_auth: false
timeout: 120
ssl_verify: true
dont_quote:
- '['
- ']'
- '$'
- '='
- '&'
- ':'
- '%'
pagination:
next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Attributes&$expand=Assets'
count_tpl: '&$count=True'
total_items_nb_key_path: '$."@odata.count"'
max_items_per_page: 1_000
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_by_tpl: '&$orderby={sort_param} {sort_order}'
sort_param_mapping:
startTimeFromAscendingNode: ContentDate/Start
completionTimeFromAscendingNode: ContentDate/End
publicationDate: PublicationDate
modificationDate: ModificationDate
sort_order_mapping:
ascending: asc
descending: desc
max_sort_params: 1
results_entry: 'value'
free_text_search_operations:
$filter:
union: ' or '
wrapper: '{}'
operations:
and:
- "Collection/Name eq '{collection}'"
- "OData.CSC.Intersects(area=geography'{geometry#to_ewkt}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{productType}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformShortName' and att/OData.CSC.StringAttribute/Value eq '{platform}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platformSerialIdentifier#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{resolution}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{organisationName}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{orbitNumber}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{orbitDirection#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {cloudCover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'operationalMode' and att/OData.CSC.StringAttribute/Value eq '{sensorMode}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'tileId' and att/OData.CSC.StringAttribute/Value eq '{tileIdentifier}')"
- "ContentDate/Start lt {completionTimeFromAscendingNode#to_iso_utc_datetime}"
- "ContentDate/End gt {startTimeFromAscendingNode#to_iso_utc_datetime}"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'polarisationChannels' and att/OData.CSC.StringAttribute/Value eq '{polarizationChannels#replace_str('\\\\+','%26')}')"
- contains(Name,'{id}')
discover_metadata:
auto_discovery: true
metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
search_param:
free_text_search_operations:
$filter:
operations:
and:
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq '{metadata}' and att/OData.CSC.StringAttribute/Value eq '{{{metadata}}}')"
metadata_path: '$.Attributes.*'
discover_product_types:
fetch_url: null
per_product_metadata_query: false
metadata_pre_mapping:
metadata_path: '$.Attributes'
metadata_path_id: 'Name'
metadata_path_value: 'Value'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
# Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
# but `free_text_search_operations.$filter.operations.and` entries are then used instead.
uid: '$.Id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- null
- '$.Attributes.productType'
platform:
- null
- '$.Attributes.platformShortName'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier'
instrument:
- null
- '$.Attributes.instrumentShortName'
processingLevel:
- null
- '$.Attributes.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '{$.Name#remove_extension}'
resolution:
- null
- '$.Attributes.spatialResolution'
organisationName:
- null
- '$.Attributes.authority'
publicationDate: '$.PublicationDate'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- null
- '$.Attributes.orbitNumber'
orbitDirection:
- null
- '{$.Attributes.orbitDirection#to_lower}'
cloudCover:
- null
- '$.Attributes.cloudCover'
modificationDate: '$.ModificationDate'
sensorMode:
- null
- '$.Attributes.operationalMode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- null
- '$.ContentDate.Start'
completionTimeFromAscendingNode:
- null
- '$.ContentDate.End'
polarizationChannels:
- null
- '{$.Attributes.polarisationChannels#replace_str("&","+")}'
# Custom parameters (not defined in the base document referenced above)
id:
- null
- '{$.Name#remove_extension}'
tileIdentifier:
- null
- '$.Attributes.tileId'
# The geographic extent of the product
geometry:
- null
- '{$.Footprint#from_ewkt}'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: 'https://zipper.creodias.eu/odata/v1/Products({uid})/$value'
# storageStatus: must be one of ONLINE, STAGING, OFFLINE
storageStatus: '{$.Online#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
collection:
- null
- $.null
quicklook: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
thumbnail: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
# Additional metadata provided by the providers but that don't appear in the reference spec
productIdentifier: '$.S3Path'
download: !plugin
type: HTTPDownload
extract: true
order_enabled: false
archive_depth: 2
ssl_verify: true
auth: !plugin
type: KeycloakOIDCPasswordAuth
matching_url: https://[-\w]+.creodias.eu
oidc_config_url: https://identity.cloudferro.com/auth/realms/Creodias-new/.well-known/openid-configuration
client_id: 'CLOUDFERRO_PUBLIC'
client_secret: 'dc0aca03-2dc6-4798-a5de-fc5aeb6c8ee1'
token_provision: qs
token_qs_key: 'token'
auth_error_code: 401
ssl_verify: true
allowed_audiences: ["CLOUDFERRO_PUBLIC"]
products:
# S1
S1_AUX_GNSSRD:
productType: AUX_GNSSRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_MOEORB:
productType: AUX_MOEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_POEORB:
productType: AUX_POEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PREORB:
productType: AUX_PREORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PROQUA:
productType: AUX_PROQUA
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_RESORB:
productType: AUX_RESORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_RAW:
productType: RAW
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_GRD:
productType: GRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_SLC:
productType: SLC
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_OCN:
productType: OCN
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_IW_MCM:
productType: S1SAR_L3_IW_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_DH_MCM:
productType: S1SAR_L3_DH_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
# S2
S2_MSI_L1C:
collection: SENTINEL-2
productType: S2MSI1C
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
S2_MSI_L2A:
collection: SENTINEL-2
productType: S2MSI2A
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
# S3 SRAL
S3_SRA:
productType: SR_1_SRA___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_A:
productType: SR_1_SRA_A_
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_BS:
productType: SR_1_SRA_BS
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN:
productType: SR_2_LAN___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_HY:
productType: SR_2_LAN_HY
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_SI:
productType: SR_2_LAN_SI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_LI:
productType: SR_2_LAN_LI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_WAT:
productType: SR_2_WAT___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 OLCI
S3_EFR:
productType: OL_1_EFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_ERR:
productType: OL_1_ERR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LRR:
productType: OL_2_LRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LFR:
productType: OL_2_LFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WRR:
productType: OL_2_WRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WFR:
productType: OL_2_WFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SLSTR
S3_SLSTR_L1RBT:
productType: SL_1_RBT___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2LST:
productType: SL_2_LST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2WST:
productType: SL_2_WST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2AOD:
productType: SL_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2FRP:
productType: SL_2_FRP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SY
S3_SY_AOD:
productType: SY_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_SYN:
productType: SY_2_SYN___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_V10:
productType: SY_2_V10___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VG1:
productType: SY_2_VG1___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VGP:
productType: SY_2_VGP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S5P L1
S5P_L1B_IR_SIR:
productType: L1B_IR_SIR
collection: SENTINEL-5P
S5P_L1B_IR_UVN:
productType: L1B_IR_UVN
collection: SENTINEL-5P
S5P_L1B_RA_BD1:
productType: L1B_RA_BD1
collection: SENTINEL-5P
S5P_L1B_RA_BD2:
productType: L1B_RA_BD2
collection: SENTINEL-5P
S5P_L1B_RA_BD3:
productType: L1B_RA_BD3
collection: SENTINEL-5P
S5P_L1B_RA_BD4:
productType: L1B_RA_BD4
collection: SENTINEL-5P
S5P_L1B_RA_BD5:
productType: L1B_RA_BD5
collection: SENTINEL-5P
S5P_L1B_RA_BD6:
productType: L1B_RA_BD6
collection: SENTINEL-5P
S5P_L1B_RA_BD7:
productType: L1B_RA_BD7
collection: SENTINEL-5P
S5P_L1B_RA_BD8:
productType: L1B_RA_BD8
collection: SENTINEL-5P
# S5P L2
S5P_L2_NO2:
productType: L2__NO2___
collection: SENTINEL-5P
S5P_L2_CLOUD:
productType: L2__CLOUD_
collection: SENTINEL-5P
S5P_L2_O3:
productType: L2__O3____
collection: SENTINEL-5P
S5P_L2_CO:
productType: L2__CO____
collection: SENTINEL-5P
S5P_L2_AER_AI:
productType: L2__AER_AI
collection: SENTINEL-5P
S5P_L2_O3_PR:
productType: L2__O3__PR
collection: SENTINEL-5P
S5P_L2_O3_TCL:
productType: L2__O3_TCL
collection: SENTINEL-5P
S5P_L2_AER_LH:
productType: L2__AER_LH
collection: SENTINEL-5P
S5P_L2_HCHO:
productType: L2__HCHO__
collection: SENTINEL-5P
S5P_L2_CH4:
productType: L2__CH4___
collection: SENTINEL-5P
S5P_L2_NP_BD3:
productType: L2__NP_BD3
collection: SENTINEL-5P
S5P_L2_NP_BD6:
productType: L2__NP_BD6
collection: SENTINEL-5P
S5P_L2_NP_BD7:
productType: L2__NP_BD7
collection: SENTINEL-5P
S5P_L2_SO2:
productType: L2__SO2___
collection: SENTINEL-5P
# COP DEM
COP_DEM_GLO30_DGED:
productType: DGE_30
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO30_DTED:
productType: DTE_30
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO90_DGED:
productType: DGE_90
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO90_DTED:
productType: DTE_90
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
---
!provider # MARK: usgs_satapi_aws
name: usgs_satapi_aws
priority: 0
roles:
- host
description: USGS Landsatlook SAT API
url: https://landsatlook.usgs.gov/stac-server
search: !plugin
type: StacSearch
api_endpoint: https://landsatlook.usgs.gov/stac-server/search
need_auth: false
ssl_verify: true
discover_queryables:
fetch_url: null
product_type_fetch_url: null
pagination:
# 2021/03/19: no more than 10_000 (if greater, returns a 500 error code)
# but in practive if an Internal Server Error is returned for more than
# about 500 products.
max_items_per_page: 500
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: id
startTimeFromAscendingNode: properties.datetime
creationDate: properties.created
modificationDate: properties.updated
platformSerialIdentifier: properties.platform
illuminationElevationAngle: properties.view:sun_elevation
illuminationAzimuthAngle: properties.view:sun_azimuth
cloudCover: properties.eo:cloud_cover
metadata_mapping:
assets: '{$.assets#from_alternate(s3)}'
awsProductId: '{$.assets.thumbnail.href#replace_str(r".+/([A-Z0-9_]+)/[\w.]+$",r"\1")}'
products:
LANDSAT_C2L1:
productType: landsat-c2l1
LANDSAT_C2L2_SR:
productType: landsat-c2l2-sr
LANDSAT_C2L2_ST:
productType: landsat-c2l2-st
LANDSAT_C2L2ALB_BT:
productType: landsat-c2l2alb-bt
LANDSAT_C2L2ALB_SR:
productType: landsat-c2l2alb-sr
LANDSAT_C2L2ALB_ST:
productType: landsat-c2l2alb-st
LANDSAT_C2L2ALB_TA:
productType: landsat-c2l2alb-ta
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: AwsDownload
ssl_verify: true
auth: !plugin
type: AwsAuth
matching_url: s3://
requester_pays: True
---
!provider # MARK: earth_search
name: earth_search
priority: 0
roles:
- host
description: Earth Search
url: https://www.element84.com/earth-search/
search: !plugin
type: StacSearch
api_endpoint: https://earth-search.aws.element84.com/v1/search
need_auth: false
ssl_verify: true
discover_product_types:
results_entry: '$.collections[?id!="sentinel-s2-l2a-cogs"]'
discover_queryables:
fetch_url: null
product_type_fetch_url: null
pagination:
# Override the default next page url key path of StacSearch because the next link returned
# by Earth Search is invalid (as of 2021/04/29). Entry set to null (None) to avoid using the
# next page retrieval mechanism, `next_page_url_tpl` will be used instead (inherited from StacSearch)
# Remove that entry if Earth Search updates that and returns a valid link.
next_page_url_key_path: null
# 2021/04/28: Earth-Search relies on Sat-API whose docs (http://sat-utils.github.io/sat-api/#search-stac-items-by-simple-filtering-)
# say the max is 10_000. In practice a too high number (e.g. 5_000) returns a 502 error ({"message": "Internal server error"}).
# Let's set it to a more robust number: 500
max_items_per_page: 500
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: id
startTimeFromAscendingNode: properties.datetime
creationDate: properties.created
modificationDate: properties.updated
platformSerialIdentifier: properties.platform
resolution: properties.gsd
cloudCover: properties.eo:cloud_cover
metadata_mapping:
utmZone:
- '{{"query":{{"mgrs:utm_zone":{{"eq":"{utmZone}"}}}}}}'
- '$.properties."mgrs:utm_zone"'
latitudeBand:
- '{{"query":{{"mgrs:latitude_band":{{"eq":"{latitudeBand}"}}}}}}'
- '$.properties."mgrs:latitude_band"'
gridSquare:
- '{{"query":{{"mgrs:grid_square":{{"eq":"{gridSquare}"}}}}}}'
- '$.properties."mgrs:grid_square"'
tileIdentifier:
- '{{"query":{{"mgrs:utm_zone":{{"eq":"{tileIdentifier#slice_str(0,2,1)}"}},"mgrs:latitude_band":{{"eq":"{tileIdentifier#slice_str(2,3,1)}"}},"mgrs:grid_square":{{"eq":"{tileIdentifier#slice_str(3,5,1)}"}}}}}}'
- '{utmZone}{latitudeBand}{gridSquare}'
assets: '{$.assets#dict_filter($[?(href=~"^s3.*")])}'
products:
S1_SAR_GRD:
productType: sentinel-1-grd
S2_MSI_L1C:
productType: sentinel-2-l1c
metadata_mapping:
id:
- '{{"query":{{"s2:product_uri":{{"eq":"{id}.SAFE"}}}}}}'
- '{$.properties."s2:product_uri"#remove_extension}'
title: '{$.properties."s2:product_uri"#remove_extension}'
platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
productPath: |
$.properties."s2:product_uri".`sub(/([S2AB]{3})_MSIL1C_([0-9]{4})([0-9]{2})([0-9]{2})(T.*).SAFE/, products!\\2!\\3!\\4!\\1_MSIL1C_\\2\\3\\4\\5)`.`sub(/!0*/, /)`
tilePath: |
$.assets.tileinfo_metadata.href.`sub(/.*/sentinel-s2-l1c\/(tiles\/.*)\/tileInfo\.json/, \\1)`
utmZone:
- '{{"query":{{"mgrs:utm_zone":{{"eq":"{utmZone}"}}}}}}'
- '$.properties."mgrs:utm_zone"'
latitudeBand:
- '{{"query":{{"mgrs:latitude_band":{{"eq":"{latitudeBand}"}}}}}}'
- '$.properties."mgrs:latitude_band"'
gridSquare:
- '{{"query":{{"mgrs:grid_square":{{"eq":"{gridSquare}"}}}}}}'
- '$.properties."mgrs:grid_square"'
tileIdentifier:
- '{{"query":{{"mgrs:utm_zone":{{"eq":"{tileIdentifier#slice_str(0,2,1)}"}},"mgrs:latitude_band":{{"eq":"{tileIdentifier#slice_str(2,3,1)}"}},"mgrs:grid_square":{{"eq":"{tileIdentifier#slice_str(3,5,1)}"}}}}}}'
- '{utmZone}{latitudeBand}{gridSquare}'
S2_MSI_L2A_COG:
productType: sentinel-2-l2a
metadata_mapping:
assets: '{$.assets#dict_filter_and_sub("$[?(@.href =~ \"^https://sentinel-cogs.*/\")]","https\u003A//[a-z0-9-\.]+/","s3\u003A//sentinel-cogs/")}'
LANDSAT_C2L2:
productType: landsat-c2-l2
NAIP:
productType: naip
COP_DEM_GLO30_DGED:
productType: cop-dem-glo-30
COP_DEM_GLO90_DGED:
productType: cop-dem-glo-90
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: AwsDownload
ssl_verify: true
products:
S2_MSI_L1C:
default_bucket: 'sentinel-s2-l1c'
build_safe: true
complementary_url_key:
- productPath
- tilePath
auth: !plugin
type: AwsAuth
matching_url: s3://
requester_pays: True
---
!provider # MARK: earth_search_gcs
name: earth_search_gcs
priority: 0
roles:
- host
description: Google Cloud Storage through Earth Search
url: https://www.element84.com/earth-search/
search: !plugin
type: StacSearch
api_endpoint: https://earth-search.aws.element84.com/v0/search
need_auth: false
ssl_verify: true
discover_product_types:
fetch_url: null
discover_queryables:
fetch_url: null
product_type_fetch_url: null
pagination:
# Override the default next page url key path of StacSearch because the next link returned
# by Earth Search is invalid (as of 2021/04/29). Entry set to null (None) to avoid using the
# next page retrieval mechanism, `next_page_url_tpl` will be used instead (inherited from StacSearch)
# Remove that entry if Earth Search updates that and returns a valid link.
next_page_url_key_path: null
# 2021/04/28: Earth-Search relies on Sat-API whose docs (http://sat-utils.github.io/sat-api/#search-stac-items-by-simple-filtering-)
# say the max is 10_000. In practice a too high number (e.g. 5_000) returns a 502 error ({"message": "Internal server error"}).
# Let's set it to a more robust number: 500
max_items_per_page: 500
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: id
startTimeFromAscendingNode: properties.datetime
creationDate: properties.created
modificationDate: properties.updated
platformSerialIdentifier: properties.platform
resolution: properties.gsd
cloudCover: properties.eo:cloud_cover
products:
S2_MSI_L1C:
productType: sentinel-s2-l1c
metadata_mapping:
title: '$.properties."sentinel:product_id"'
platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
utmZone: '$.properties."sentinel:utm_zone"'
latitudeBand: '$.properties."sentinel:latitude_band"'
gridSquare: '$.properties."sentinel:grid_square"'
downloadLink: 's3://gcp-public-data-sentinel-2/tiles/{utmZone}/{latitudeBand}/{gridSquare}/{title}.SAFE'
L8_OLI_TIRS_C1L1:
productType: landsat-8-l1-c1
metadata_mapping:
wrsPath: '$.properties."landsat:wrs_path"'
wrsRow: '$.properties."landsat:wrs_row"'
downloadLink: 's3://gcp-public-data-landsat/LC08/01/{wrsPath:03d}/{wrsRow:03d}/{title}'
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: AwsDownload
s3_endpoint: https://storage.googleapis.com
ignore_assets: True
ssl_verify: true
products:
S2_MSI_L1C:
default_bucket: 'gcp-public-data-sentinel-2'
auth: !plugin
type: AwsAuth
matching_url: s3://
s3_endpoint: https://storage.googleapis.com
matching_conf:
s3_endpoint: https://storage.googleapis.com
---
!provider # MARK: ecmwf
name: ecmwf
priority: 0
description: ECMWF archive products
roles:
- host
url: https://www.ecmwf.int
api: !plugin
type: EcmwfApi
auth_endpoint: https://api.ecmwf.int/v1
metadata_mapping:
productType: $.productType
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- 'date={startTimeFromAscendingNode#to_iso_date}/to/{completionTimeFromAscendingNode#to_iso_date(-1,)}'
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
# The geographic extent of the product
geometry:
- 'area={geometry#to_nwse_bounds_str(/)}'
- '$.geometry'
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
qs: $.qs
downloadLink: 'https://apps.ecmwf.int/datasets/data/{dataset}?{qs#to_geojson}'
products:
# See Archive Catalog in https://apps.ecmwf.int/archive-catalogue/
# See available Public Datasets in https://apps.ecmwf.int/datasets/
TIGGE_CF_SFC:
dataset: tigge
GENERIC_PRODUCT_TYPE:
dataset: '{productType}'
---
!provider # MARK: cop_ads
name: cop_ads
priority: 0
description: Copernicus Atmosphere Data Store
roles:
- host
url: https://ads.atmosphere.copernicus.eu/
anchor_month_year: &month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
auth: !plugin
type: HTTPHeaderAuth
ssl_verify: true
headers:
PRIVATE-TOKEN: "{apikey}"
download: !plugin
type: HTTPDownload
timeout: 30
ssl_verify: true
auth_error_code: 401
order_enabled: true
order_method: POST
order_on_response:
metadata_mapping:
orderId: $.json.jobID
orderStatusLink: https://ads.atmosphere.copernicus.eu/api/retrieve/v1/jobs/{orderId}?request=true
searchLink: https://ads.atmosphere.copernicus.eu/api/retrieve/v1/jobs/{orderId}/results
order_status:
request:
method: GET
metadata_mapping:
status: $.json.status
order:date: $.json.created
creationDate: $.json.created
publicationDate: $.json.finished
modificationDate: $.json.updated
ecmwf:dataset: $.json.processID
request_params: $.json.metadata.request.ids
error:
status: failed
success:
status: successful
on_success:
need_search: true
metadata_mapping:
downloadLink: $.json.asset.value.href
search: !plugin
type: ECMWFSearch
need_auth: true
ssl_verify: true
timeout: 30
end_date_excluded: false
dates_required: true
remove_from_query:
- dataset
discover_queryables:
fetch_url: null
product_type_fetch_url: null
constraints_url: "https://ads.atmosphere.copernicus.eu/api/catalogue/v1/collections/{dataset}/constraints.json"
form_url: https://ads.atmosphere.copernicus.eu/api/catalogue/v1/collections/{dataset}/form.json
metadata_mapping:
productType: $.productType
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- 'date={startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date}'
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
# The geographic extent of the product
geometry:
- '{{"area": {geometry#to_nwse_bounds}}}'
- $.geometry
qs: $.qs
orderLink: 'https://ads.atmosphere.copernicus.eu/api/retrieve/v1/processes/{dataset}/execution?{{"inputs": {qs#to_geojson}}}'
products:
# See available Public Datasets in https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset
CAMS_GAC_FORECAST:
dataset: cams-global-atmospheric-composition-forecasts
CAMS_GFE_GFAS:
dataset: cams-global-fire-emissions-gfas
CAMS_EU_AIR_QUALITY_FORECAST:
dataset: cams-europe-air-quality-forecasts
CAMS_EU_AIR_QUALITY_RE:
dataset: cams-europe-air-quality-reanalyses
metadata_mapping:
<<: *month_year
CAMS_GRF:
dataset: cams-global-radiative-forcings
metadata_mapping:
<<: *month_year
CAMS_GRF_AUX:
dataset: cams-global-radiative-forcing-auxilliary-variables
metadata_mapping:
<<: *month_year
CAMS_SOLAR_RADIATION:
dataset: cams-solar-radiation-timeseries
metadata_mapping:
latitude:
- '{{"location": {{"latitude": {"latitude"}, "longitude": {"longitude"}}}}}'
- $."latitude"
longitude:
- '{{"location": {{"latitude": {"latitude"}, "longitude": {"longitude"}}}}}'
- $."longitude"
CAMS_GREENHOUSE_EGG4_MONTHLY:
dataset: cams-global-ghg-reanalysis-egg4-monthly
metadata_mapping:
<<: *month_year
CAMS_GREENHOUSE_EGG4:
dataset: cams-global-ghg-reanalysis-egg4
CAMS_GREENHOUSE_INVERSION:
dataset: cams-global-greenhouse-gas-inversion
metadata_mapping:
<<: *month_year
CAMS_GLOBAL_EMISSIONS:
dataset: cams-global-emission-inventories
metadata_mapping:
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
CAMS_EAC4:
dataset: cams-global-reanalysis-eac4
CAMS_EAC4_MONTHLY:
dataset: cams-global-reanalysis-eac4-monthly
metadata_mapping:
<<: *month_year
GENERIC_PRODUCT_TYPE:
dataset: '{productType}'
---
!provider # MARK: cop_cds
name: cop_cds
priority: 0
description: Copernicus Climate Data Store
roles:
- host
url: https://cds.climate.copernicus.eu
# anchors to avoid duplications
anchor_nday_month_year: &nday_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"nominal_day": {_date#interval_to_datetime_dict}["day"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_day_month_year: &day_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_time_day_month_year: &time_day_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_month_year: &month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_time_month_year: &time_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
auth: !plugin
type: HTTPHeaderAuth
ssl_verify: true
headers:
PRIVATE-TOKEN: "{apikey}"
download: !plugin
type: HTTPDownload
timeout: 30
ssl_verify: true
order_enabled: true
auth_error_code: 401
order_method: POST
order_on_response:
metadata_mapping:
orderId: $.json.jobID
orderStatusLink: https://cds.climate.copernicus.eu/api/retrieve/v1/jobs/{orderId}?request=true
searchLink: https://cds.climate.copernicus.eu/api/retrieve/v1/jobs/{orderId}/results
order_status:
request:
method: GET
metadata_mapping:
status: $.json.status
order:date: $.json.created
creationDate: $.json.created
publicationDate: $.json.finished
modificationDate: $.json.updated
ecmwf:dataset: $.json.processID
request_params: $.json.metadata.request.ids
error:
status: failed
success:
status: successful
on_success:
need_search: true
metadata_mapping:
downloadLink: $.json.asset.value.href
search: !plugin
type: ECMWFSearch
need_auth: true
ssl_verify: true
timeout: 30
end_date_excluded: false
dates_required: true
remove_from_query:
- dataset
- date
discover_queryables:
fetch_url: null
product_type_fetch_url: null
constraints_url: https://cds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset}/constraints.json
form_url: https://cds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset}/form.json
metadata_mapping:
productType: $.productType
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- date={startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date}
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
# The geographic extent of the product
geometry:
- '{{"area": {geometry#to_nwse_bounds}}}'
- $.geometry
qs: $.qs
orderLink: 'https://cds.climate.copernicus.eu/api/retrieve/v1/processes/{dataset}/execution?{{"inputs": {qs#to_geojson}}}'
products:
# See available Public Datasets in https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset
AG_ERA5:
dataset: sis-agrometeorological-indicators
metadata_mapping:
<<: *day_month_year
ERA5_SL:
dataset: reanalysis-era5-single-levels
metadata_mapping:
<<: *time_day_month_year
ERA5_PL:
dataset: reanalysis-era5-pressure-levels
metadata_mapping:
<<: *time_day_month_year
ERA5_PL_MONTHLY:
dataset: reanalysis-era5-pressure-levels-monthly-means
metadata_mapping:
<<: *time_month_year
ERA5_LAND:
dataset: reanalysis-era5-land
metadata_mapping:
<<: *time_day_month_year
ERA5_LAND_MONTHLY:
dataset: reanalysis-era5-land-monthly-means
metadata_mapping:
<<: *time_month_year
ERA5_SL_MONTHLY:
dataset: reanalysis-era5-single-levels-monthly-means
metadata_mapping:
<<: *time_month_year
UERRA_EUROPE_SL:
dataset: reanalysis-uerra-europe-single-levels
metadata_mapping:
<<: *day_month_year
GLACIERS_DIST_RANDOLPH:
dataset: insitu-glaciers-extent
GRIDDED_GLACIERS_MASS_CHANGE:
dataset: derived-gridded-glacier-mass-change
metadata_mapping:
completionTimeFromAscendingNode:
- |
{{
"hydrological_year": {completionTimeFromAscendingNode#get_hydrological_year}
}}
- '{$.completionTimeFromAscendingNode#get_hydrological_year}'
SATELLITE_CARBON_DIOXIDE:
dataset: satellite-carbon-dioxide
metadata_mapping:
<<: *day_month_year
SATELLITE_FIRE_BURNED_AREA:
dataset: satellite-fire-burned-area
metadata_mapping:
<<: *nday_month_year
SATELLITE_METHANE:
dataset: satellite-methane
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_ICE_EDGE_TYPE:
dataset: satellite-sea-ice-edge-type
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_ICE_THICKNESS:
dataset: satellite-sea-ice-thickness
metadata_mapping:
<<: *month_year
SATELLITE_SEA_ICE_CONCENTRATION:
dataset: satellite-sea-ice-concentration
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_LEVEL_GLOBAL:
dataset: satellite-sea-level-global
metadata_mapping:
<<: *day_month_year
SEASONAL_POSTPROCESSED_PL:
dataset: seasonal-postprocessed-pressure-levels
metadata_mapping:
<<: *month_year
SEASONAL_POSTPROCESSED_SL:
dataset: seasonal-postprocessed-single-levels
metadata_mapping:
<<: *month_year
SEASONAL_ORIGINAL_SL:
dataset: seasonal-original-single-levels
metadata_mapping:
<<: *day_month_year
SEASONAL_ORIGINAL_PL:
dataset: seasonal-original-pressure-levels
metadata_mapping:
<<: *day_month_year
SEASONAL_MONTHLY_PL:
dataset: seasonal-monthly-pressure-levels
metadata_mapping:
<<: *month_year
SEASONAL_MONTHLY_SL:
dataset: seasonal-monthly-single-levels
metadata_mapping:
<<: *month_year
SIS_HYDRO_MET_PROJ:
dataset: sis-hydrology-meteorology-derived-projections
data_format: zip
CMIP6_CLIMATE_PROJECTIONS:
dataset: projections-cmip6
GENERIC_PRODUCT_TYPE:
dataset: '{productType}'
---
!provider # MARK: sara
name: sara
priority: 0
description: Sentinel Australasia Regional Access
roles:
- host
url: https://www.copernicus.gov.au/
search: !plugin
type: QueryStringSearch
# The endpoint is based off of the collection. There is a generic endpoint,
# but can be very slow if not enough metdata is provided.
api_endpoint: 'https://copernicus.nci.org.au/sara.server/1.0/api/collections/{collection}/search.json'
need_auth: false
ssl_verify: true
pagination:
next_page_url_tpl: '{url}?{search}&maxRecords={items_per_page}&page={page}'
total_items_nb_key_path: '$.properties.totalResults'
# 2021/03/19: 500 is the max, no error if greater
max_items_per_page: 500
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_by_tpl: '&sortParam={sort_param}&sortOrder={sort_order}'
sort_param_mapping:
startTimeFromAscendingNode: startDate
completionTimeFromAscendingNode: completionDate
sensorMode: sensorMode
sort_order_mapping:
ascending: asc
descending: desc
max_sort_params: 1
free_text_search_operations:
orbitDirection:
union: '&'
wrapper: '{}'
operations:
and:
- '{orbitDirection#to_title}'
polarisation:
union: '&'
wrapper: '{}'
operations:
and:
- '{polarizationChannels#replace_str("\\+",",")}'
discover_metadata:
auto_discovery: true
metadata_pattern: '^(?!collection)[a-zA-Z0-9_]+$'
search_param: '{metadata}={{{metadata}}}'
metadata_path: '$.properties.*'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
uid: '$.id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- productType
- '$.properties.productType'
platform: '$.properties.collection'
platformSerialIdentifier:
- platform
- '$.properties.platform'
instrument:
- instrument
- '$.properties.instrument'
processingLevel:
- processingLevel
- '$.properties.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '$.properties.title'
topicCategory: '$.properties.topicCategory'
keyword: '$.properties.keywords'
abstract: '$.properties.description'
resolution:
- 'resolution'
- '$.properties.resolution'
organisationName:
- 'organisationName'
- '$.properties.organisationName'
publicationDate: '$.properties.published'
accessConstraint: '$.properties.license'
# OpenSearch Parameters for Product Search (Table 5)
parentIdentifier:
- 'parentIdentifier'
- '$.properties.parentIdentifier'
acquisitionType: '$.properties.acquisitionType'
orbitNumber:
- 'orbitNumber'
- '$.properties.orbitNumber'
orbitDirection:
- null
- '{$.properties.orbitDirection#to_lower}'
swathIdentifier:
- 'swath'
- '$.properties.swath'
cloudCover:
- 'cloudCover=[0,{cloudCover}]'
- '$.properties.cloudCover'
snowCover:
- 'snowCover=[0,{snowCover}]'
- '$.properties.snowCover'
productVersion: '$.properties.version'
productQualityStatus: '$.properties.onlineQualityCheck'
processorName: '$.properties.processingName'
processingCenter: '$.properties.processingCenter'
creationDate: '$.properties.dhusIngestDate'
modificationDate: '$.properties.updated'
sensorMode:
- 'sensorMode'
- '$.properties.sensorMode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- startDate
- '$.properties.startDate'
completionTimeFromAscendingNode:
- completionDate
- '$.properties.completionDate'
polarizationChannels:
- null
- '{$.properties.polarisation#replace_str(",","+")}'
# Custom parameters (not defined in the base document referenced above)
id:
- 'productIdentifier={id#remove_extension}'
- '$.properties.productIdentifier'
# The geographic extent of the product
geometry:
- 'geometry={geometry#to_rounded_wkt}'
- '$.geometry'
# The url of the quicklook
quicklook: '$.properties.quicklook'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: '$.properties.services.download.url'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
# Additional metadata provided by the providers but that don't appear in the reference spec
thumbnail: '$.properties.thumbnail'
products:
# Sentinel 1
S1_SAR_OCN:
productType: OCN
collection: S1
instrument: C-SAR
S1_SAR_GRD:
productType: GRD
collection: S1
instrument: C-SAR
S1_SAR_SLC:
productType: SLC
collection: S1
instrument: C-SAR
# Sentinel 2
S2_MSI_L1C:
collection: S2
productType: S2MSIL1C
instrument: MSI
processingLevel: L1C
S2_MSI_L2A:
collection: S2
productType: S2MSIL2A
instrument: MSI
processingLevel: L2A
# OLCI products
# L1
S3_EFR:
productType: OL_1_EFR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-1
S3_ERR:
productType: OL_1_ERR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-1
# output during radiometric calibration mode
S3_RAC:
productType: OL_1_RAC___
collection: S3
instrument: OLCI
processingLevel: LEVEL-1
# L2
S3_OLCI_L2LRR:
productType: OL_2_LRR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-2
S3_OLCI_L2LFR:
productType: OL_2_LFR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-2
S3_OLCI_L2WRR:
productType: OL_2_WRR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-2
S3_OLCI_L2WFR:
productType: OL_2_WFR___
collection: S3
instrument: OLCI
processingLevel: LEVEL-2
# SLSTR products
S3_SLSTR_L1RBT:
productType: SL_1_RBT___
collection: S3
instrument: SLSTR
processingLevel: LEVEL-1
S3_SLSTR_L2LST:
productType: SL_2_LST___
collection: S3
instrument: SLSTR
processingLevel: LEVEL-2
S3_SLSTR_L2WST:
productType: SL_2_WST___
collection: S3
instrument: SLSTR
processingLevel: LEVEL-2
S3_SLSTR_L2AOD:
productType: SL_2_AOD___
collection: S3
instrument: SLSTR
processingLevel: LEVEL-2
S3_SLSTR_L2FRP:
productType: SL_2_FRP___
collection: S3
instrument: SLSTR
processingLevel: LEVEL-2
# SRAL
# L1
S3_SRA:
productType: SR_1_SRA___
collection: S3
instrument: SRAL
processingLevel: LEVEL-1
S3_SRA_A:
productType: SR_1_SRA_A_
collection: S3
instrument: SRAL
processingLevel: LEVEL-1
S3_SRA_BS:
productType: SR_1_SRA_BS
collection: S3
instrument: SRAL
processingLevel: LEVEL-1
# L2
S3_LAN:
productType: SR_2_LAN___
collection: S3
instrument: SRAL
processingLevel: LEVEL-2
S3_WAT:
productType: SR_2_WAT___
collection: S3
instrument: SRAL
processingLevel: LEVEL-2
# Synergy products
S3_SY_AOD:
productType: SY_2_AOD___
collection: S3
instrument: SYNERGY
processingLevel: LEVEL-2
S3_SY_SYN:
productType: SY_2_SYN___
collection: S3
instrument: SYNERGY
processingLevel: LEVEL-2
S3_SY_V10:
productType: SY_2_V10___
collection: S3
instrument: SYNERGY
processingLevel: LEVEL-2W
S3_SY_VG1:
productType: SY_2_VG1___
collection: S3
instrument: SYNERGY
processingLevel: LEVEL-2
S3_SY_VGP:
productType: SY_2_VGP___
collection: S3
instrument: SYNERGY
processingLevel: LEVEL-2
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
download: !plugin
type: HTTPDownload
extract: true
archive_depth: 2
order_enabled: true
auth_error_code: 403
ssl_verify: true
auth: !plugin
type: GenericAuth
matching_url: https://copernicus.nci.org.au
method: basic
---
!provider # MARK: meteoblue
name: meteoblue
priority: 0
roles:
- host
description: Meteoblue
url: https://www.meteoblue.com
search: !plugin
type: MeteoblueSearch
api_endpoint: 'https://my.meteoblue.com/dataset/query'
need_auth: true
ssl_verify: true
pagination:
next_page_query_obj: '{{"checkOnly":true}}'
discover_metadata:
auto_discovery: true
metadata_pattern: '^[a-zA-Z0-9_]+$'
search_param: '{{{{"{metadata}":{{{metadata}#to_geojson}} }}}}'
metadata_path: '$.*'
metadata_mapping:
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- '{{"timeIntervals": [ "{startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date(-1,)}" ] }}'
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
geometry:
- '{{"geometry": {geometry#to_geojson} }}'
- '$.geometry'
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
productType: '$.queries[0].domain'
storageStatus: '{$.requiresJobQueue#get_group_name((?P<ONLINE>False)|(?P<OFFLINE>True))}'
qs: $.qs
downloadLink: https://my.meteoblue.com/dataset/query?{qs#to_geojson}
# Meteoblue specific parameters
datapoints: '$.datapoints'
requiresJobQueue: '$.requiresJobQueue'
requiresComplexJobQueue: '$.requiresComplexJobQueue'
units:
- '{{"units": {units#to_geojson} }}'
- '$.units'
queries:
- '{{"queries": {queries#to_geojson} }}'
- '$.queries'
format:
- '{{"format": {format#to_geojson} }}'
- '$.format'
timeIntervalsAlignment:
- '{{"timeIntervalsAlignment": {timeIntervalsAlignment#to_geojson} }}'
- '$.timeIntervalsAlignment'
orderLink: https://my.meteoblue.com/dataset/query?{qs#replace_str(r"^(.*)(\")(queries\")(.)",r"\1\2runOnJobQueue\2\4 true, \2\3\4")}
# orderLink: '{$.downloadLink#replace_str(r"^(.*)(\")(queries\")(.)",r"\1\2runOnJobQueue\2\4 true, \2\3\4")}'
products:
NEMSGLOBAL_TCDC:
productType: NEMSGLOBAL
queries: [{'domain':'NEMSGLOBAL','gapFillDomain':null,'timeResolution':'daily','codes':[{'code':71,'level':'sfc','aggregation':'mean'}]}]
format: netCDF
units: {'temperature':'C','velocity':'km/h','length':'metric','energy':'watts'}
timeIntervalsAlignment:
geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
NEMSAUTO_TCDC:
productType: NEMSAUTO
queries: [{'domain':'NEMSAUTO','gapFillDomain':null,'timeResolution':'daily','codes':[{'code':71,'level':'sfc','aggregation':'mean'}]}]
format: netCDF
units: {'temperature':'C','velocity':'km/h','length':'metric','energy':'watts'}
timeIntervalsAlignment:
geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
geometry: {"type": "Polygon", "coordinates": [[[180, -90], [180, 90], [-180, 90], [-180, -90], [180, -90]]]}
download: !plugin
type: HTTPDownload
ssl_verify: true
method: POST
extract: False
order_enabled: true
order_method: POST
order_on_response:
metadata_mapping:
orderId: '$.json.id'
orderStatusLink: 'http://my.meteoblue.com/queue/status/{orderId}'
order_status:
request:
method: GET
metadata_mapping:
percent: $.json.percentCompleted
status: $.json.status
error:
status: error
on_success:
metadata_mapping:
orderId: '$.json.id'
downloadLink: 'http://queueresults.meteoblue.com/{orderId}'
downloadMethod: '{$.null#replace_str("Not Available","GET")}'
auth: !plugin
type: HttpQueryStringAuth
matching_url: https?://[a-z]+.meteoblue.com
auth_uri: 'http://my.meteoblue.com/dataset/meta?dataset=NEMSAUTO'
ssl_verify: true
---
!provider # MARK: cop_dataspace
name: cop_dataspace
priority: 0
description: Copernicus Data Space Ecosystem
roles:
- host
url: https://dataspace.copernicus.eu/
search: !plugin
type: ODataV4Search
api_endpoint: 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products'
need_auth: false
timeout: 120
ssl_verify: true
dont_quote:
- '['
- ']'
- '$'
- '='
- '&'
- ':'
- '%'
pagination:
next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Attributes&$expand=Assets'
count_tpl: '&$count=True'
total_items_nb_key_path: '$."@odata.count"'
max_items_per_page: 1_000
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_by_tpl: '&$orderby={sort_param} {sort_order}'
sort_param_mapping:
startTimeFromAscendingNode: ContentDate/Start
completionTimeFromAscendingNode: ContentDate/End
publicationDate: PublicationDate
modificationDate: ModificationDate
sort_order_mapping:
ascending: asc
descending: desc
max_sort_params: 1
results_entry: 'value'
free_text_search_operations:
$filter:
union: ' or '
wrapper: '{}'
operations:
and:
- "Collection/Name eq '{collection}'"
- "OData.CSC.Intersects(area=geography'{geometry#to_ewkt}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{productType}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformShortName' and att/OData.CSC.StringAttribute/Value eq '{platform}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platformSerialIdentifier#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{resolution}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{organisationName}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.IntegerAttribute/Value eq {orbitNumber})"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'relativeOrbitNumber' and att/OData.CSC.IntegerAttribute/Value eq {relativeOrbitNumber})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{orbitDirection#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {cloudCover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'operationalMode' and att/OData.CSC.StringAttribute/Value eq '{sensorMode}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'polarisationChannels' and att/OData.CSC.StringAttribute/Value eq '{polarizationChannels#replace_str('\\\\+','%26')}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'tileId' and att/OData.CSC.StringAttribute/Value eq '{tileIdentifier}')"
- "ModificationDate gt {modifiedAfter#to_iso_utc_datetime}"
- "ModificationDate lt {modifiedBefore#to_iso_utc_datetime}"
- "PublicationDate gt {publishedAfter#to_iso_utc_datetime}"
- "PublicationDate lt {publishedBefore#to_iso_utc_datetime}"
- "ContentDate/Start lt {completionTimeFromAscendingNode#to_iso_utc_datetime}"
- "ContentDate/End gt {startTimeFromAscendingNode#to_iso_utc_datetime}"
- contains(Name,'{id}')
discover_metadata:
auto_discovery: true
metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
search_param:
free_text_search_operations:
$filter:
operations:
and:
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq '{metadata}' and att/OData.CSC.StringAttribute/Value eq '{{{metadata}}}')"
metadata_path: '$.Attributes.*'
per_product_metadata_query: false
metadata_pre_mapping:
metadata_path: '$.Attributes'
metadata_path_id: 'Name'
metadata_path_value: 'Value'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
# Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
# but `free_text_search_operations.$filter.operations.and` entries are then used instead.
uid: '$.Id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- null
- '$.Attributes.productType'
platform:
- null
- '$.Attributes.platformShortName'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier'
instrument:
- null
- '$.Attributes.instrumentShortName'
processingLevel:
- null
- '$.Attributes.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '{$.Name#remove_extension}'
resolution:
- null
- '$.Attributes.spatialResolution'
organisationName:
- null
- '$.Attributes.authority'
publishedAfter:
- null
- '$.PublicationDate'
publishedBefore:
- null
- '$.PublicationDate'
publicationDate: '$.PublicationDate'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- null
- '$.Attributes.orbitNumber'
orbitDirection:
- null
- '{$.Attributes.orbitDirection#to_lower}'
cloudCover:
- null
- '$.Attributes.cloudCover'
modifiedAfter:
- null
- '$.ModificationDate'
modifiedBefore:
- null
- '$.ModificationDate'
modificationDate:
- null
- '$.ModificationDate'
relativeOrbitNumber:
- null
- '$.Attributes.relativeOrbitNumber'
sensorMode:
- null
- '$.Attributes.operationalMode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- null
- '$.ContentDate.Start'
completionTimeFromAscendingNode:
- null
- '$.ContentDate.End'
polarizationChannels:
- null
- '{$.Attributes.polarisationChannels#replace_str("&","+")}'
# Custom parameters (not defined in the base document referenced above)
id:
- null
- '{$.Name#remove_extension}'
tileIdentifier:
- null
- '$.Attributes.tileId'
# The geographic extent of the product
geometry:
- null
- '{$.Footprint#from_ewkt}'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products({uid})/$value'
# storageStatus: must be one of ONLINE, STAGING, OFFLINE
storageStatus: '{$.Online#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
collection:
- null
- $.null
quicklook: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
thumbnail: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
# Additional metadata provided by the providers but that don't appear in the reference spec
productIdentifier: '$.S3Path'
download: !plugin
type: HTTPDownload
extract: true
order_enabled: false
archive_depth: 2
ssl_verify: true
auth: !plugin
type: KeycloakOIDCPasswordAuth
matching_url: https://catalogue.dataspace.copernicus.eu
oidc_config_url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/.well-known/openid-configuration
client_id: 'cdse-public'
client_secret: null
token_provision: qs
token_qs_key: 'token'
auth_error_code: 401
ssl_verify: true
allowed_audiences: ["CLOUDFERRO_PUBLIC"]
products:
# S2
S2_MSI_L1C:
collection: SENTINEL-2
productType: S2MSI1C
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
S2_MSI_L2A:
collection: SENTINEL-2
productType: S2MSI2A
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
# S1
S1_AUX_GNSSRD:
productType: AUX_GNSSRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_MOEORB:
productType: AUX_MOEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_POEORB:
productType: AUX_POEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PREORB:
productType: AUX_PREORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PROQUA:
productType: AUX_PROQUA
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_RESORB:
productType: AUX_RESORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_RAW:
productType: RAW
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_GRD:
productType: GRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_GRD_COG:
productType: GRD-COG
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_SLC:
productType: SLC
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_OCN:
productType: OCN
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_IW_MCM:
productType: S1SAR_L3_IW_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_DH_MCM:
productType: S1SAR_L3_DH_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
# S3 SRAL
S3_SRA:
productType: SR_1_SRA___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_A:
productType: SR_1_SRA_A_
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_BS:
productType: SR_1_SRA_BS
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN:
productType: SR_2_LAN___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_HY:
productType: SR_2_LAN_HY
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_SI:
productType: SR_2_LAN_SI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_LI:
productType: SR_2_LAN_LI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_WAT:
productType: SR_2_WAT___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 OLCI
S3_EFR:
productType: OL_1_EFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_ERR:
productType: OL_1_ERR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LRR:
productType: OL_2_LRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LFR:
productType: OL_2_LFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WRR:
productType: OL_2_WRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WFR:
productType: OL_2_WFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SLSTR
S3_SLSTR_L1RBT:
productType: SL_1_RBT___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2LST:
productType: SL_2_LST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2WST:
productType: SL_2_WST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2AOD:
productType: SL_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2FRP:
productType: SL_2_FRP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SY
S3_SY_AOD:
productType: SY_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_SYN:
productType: SY_2_SYN___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_V10:
productType: SY_2_V10___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VG1:
productType: SY_2_VG1___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VGP:
productType: SY_2_VGP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S5P L1
S5P_L1B_IR_SIR:
productType: L1B_IR_SIR
collection: Sentinel-5P
S5P_L1B_IR_UVN:
productType: L1B_IR_UVN
collection: Sentinel-5P
S5P_L1B_RA_BD1:
productType: L1B_RA_BD1
collection: SENTINEL-5P
S5P_L1B_RA_BD2:
productType: L1B_RA_BD2
collection: SENTINEL-5P
S5P_L1B_RA_BD3:
productType: L1B_RA_BD3
collection: SENTINEL-5P
S5P_L1B_RA_BD4:
productType: L1B_RA_BD4
collection: SENTINEL-5P
S5P_L1B_RA_BD5:
productType: L1B_RA_BD5
collection: SENTINEL-5P
S5P_L1B_RA_BD6:
productType: L1B_RA_BD6
collection: SENTINEL-5P
S5P_L1B_RA_BD7:
productType: L1B_RA_BD7
collection: SENTINEL-5P
S5P_L1B_RA_BD8:
productType: L1B_RA_BD8
collection: SENTINEL-5P
# S5P L2
S5P_L2_NO2:
productType: L2__NO2___
collection: SENTINEL-5P
S5P_L2_CLOUD:
productType: L2__CLOUD_
collection: SENTINEL-5P
S5P_L2_O3:
productType: L2__O3____
collection: SENTINEL-5P
S5P_L2_CO:
productType: L2__CO____
collection: SENTINEL-5P
S5P_L2_AER_AI:
productType: L2__AER_AI
collection: SENTINEL-5P
S5P_L2_O3_PR:
productType: L2__O3__PR
collection: SENTINEL-5P
S5P_L2_O3_TCL:
productType: L2__O3_TCL
collection: Sentinel-5P
S5P_L2_AER_LH:
productType: L2__AER_LH
collection: SENTINEL-5P
S5P_L2_HCHO:
productType: L2__HCHO__
collection: SENTINEL-5P
S5P_L2_CH4:
productType: L2__CH4___
collection: SENTINEL-5P
S5P_L2_NP_BD3:
productType: L2__NP_BD3
collection: SENTINEL-5P
S5P_L2_NP_BD6:
productType: L2__NP_BD6
collection: SENTINEL-5P
S5P_L2_NP_BD7:
productType: L2__NP_BD7
collection: SENTINEL-5P
S5P_L2_SO2:
productType: L2__SO2___
collection: SENTINEL-5P
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
---
!provider # MARK: planetary_computer
name: planetary_computer
priority: 0
roles:
- host
description: Microsoft Planetary Computer
url: https://planetarycomputer.microsoft.com
search: !plugin
type: StacSearch
api_endpoint: https://planetarycomputer.microsoft.com/api/stac/v1/search
need_auth: false
ssl_verify: true
pagination:
max_items_per_page: 1000
sort:
sort_param_mapping:
id: id
startTimeFromAscendingNode: properties.datetime
platformSerialIdentifier: properties.platform
metadata_mapping:
tileIdentifier:
- '{{"query":{{"s2:mgrs_tile":{{"eq":"{tileIdentifier}"}}}}}}'
- '$.properties."s2:mgrs_tile"'
products:
S1_SAR_GRD:
productType: sentinel-1-grd
metadata_mapping:
processingLevel:
- '{{"query":{{"s1:processing_level":{{"eq":"{processingLevel}"}}}}}}'
- '$.properties."s1:processing_level"'
S2_MSI_L2A:
productType: sentinel-2-l2a
LANDSAT_C2L1:
productType: landsat-c2-l1
LANDSAT_C2L2:
productType: landsat-c2-l2
MODIS_MCD43A4:
productType: modis-43A4-061
NAIP:
productType: naip
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: HTTPDownload
auth_error_code: 403
ssl_verify: true
auth: !plugin
type: SASAuth
matching_url: https://[-\w\.]+.blob.core.windows.net
auth_uri: 'https://planetarycomputer.microsoft.com/api/sas/v1/sign?href={url}'
signed_url_key: href
ssl_verify: true
headers:
Ocp-Apim-Subscription-Key: "{apikey}"
---
!provider # MARK: hydroweb_next
name: hydroweb_next
priority: 0
roles:
- host
description: hydroweb.next thematic hub for hydrology data access
url: https://hydroweb.next.theia-land.fr
search: !plugin
type: StacSearch
api_endpoint: https://hydroweb.next.theia-land.fr/api/v1/rs-catalog/stac/search
need_auth: true
auth_error_code: 401
ssl_verify: true
asset_key_from_href: false
discover_queryables:
fetch_url: null
product_type_fetch_url: null
pagination:
max_items_per_page: 10_000
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: id
startTimeFromAscendingNode: properties.start_datetime
completionTimeFromAscendingNode: properties.end_datetime
productVersion: properties.version
processingLevel: processing:level
metadata_mapping:
startTimeFromAscendingNode:
- '{{"query":{{"end_datetime":{{"gte":"{startTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.start_datetime'
completionTimeFromAscendingNode:
- '{{"query":{{"start_datetime":{{"lte":"{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.end_datetime'
products:
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: HTTPDownload
auth_error_code: 401
ssl_verify: true
auth: !plugin
type: HTTPHeaderAuth
matching_url: https://hydroweb.next.theia-land.fr
headers:
X-API-Key: "{apikey}"
---
!provider # MARK: wekeo_main
name: wekeo_main # wekeo_main
group: wekeo
priority: 0
roles:
- host
description: WEkEO - Sentinel and some various Copernicus data
url: https://www.wekeo.eu/
# anchors to avoid duplications
anchor_s1_sar: &s1_sar_params
processingLevel:
- '{{"processingLevel": "{processingLevel}"}}'
- '$.id.`sub(/^[^_]([^_]+)_([^_]+)_([^_]+)_([0-4]+).*/, LEVEL\\4)`'
sensorMode:
- '{{"sensorMode": "{sensorMode}"}}'
- '$.id.`sub(/^[^_]([^_]+)_([^_]+)_.*/, \\2)`'
swath:
- '{{"swath": "{swath}"}}'
- '$.null'
polarisation:
- '{{"polarisation": "{polarisation}"}}'
- '$.null'
relativeOrbitNumber:
- '{{"relativeOrbitNumber": "{relativeOrbitNumber}"}}'
- '$.null'
missionTakeId:
- '{{"missionTakeId": "{missionTakeId}"}}'
- '$.null'
anchor_orbit_cycle: &orbit_cycle
relativeOrbitNumber:
- '{{"relativeOrbitNumber": "{relativeOrbitNumber}"}}'
- '$.null'
orbitNumber:
- '{{"orbit": "{orbitNumber}"}}'
- '$.null'
cycleNumber:
- '{{"cycle": "{cycleNumber}"}}'
- '$.null'
anchor_id_from_date: &id_from_date
id:
- |
{{
"startdate": "{id#replace_str(r'^.*_([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])_.*$',r'\1-\2-\3T\4%3A\5%3A00Z')}",
"enddate": "{id#replace_str(r'^.*_([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])([0-9][0-9])_.*$',r'\1-\2-\3T\4%3A\5%3A00Z')}"
}}
- '$.id'
search: !plugin
type: WekeoSearch
api_endpoint: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/search
timeout: 60
need_auth: true
auth_error_code: 401
results_entry: 'features'
pagination:
total_items_nb_key_path: '$.properties.totalResults'
next_page_query_obj: '{{"itemsPerPage":{items_per_page},"startIndex":{skip}}}'
max_items_per_page: 200
discover_product_types:
fetch_url: null
discover_queryables:
fetch_url: null
product_type_fetch_url: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/queryable/{provider_product_type}'
result_type: json
results_entry: '$.properties[*]'
metadata_mapping:
productType:
- '{{"dataset_id": "{productType}"}}'
- '$.null'
geometry:
- '{{"bbox": {geometry#to_bounds}}}'
- '$.geometry'
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
id:
- '{{"productIdentifier": "{id}"}}'
- '{$.id#remove_extension}'
startTimeFromAscendingNode:
- '{{"startdate": "{startTimeFromAscendingNode}"}}'
- '$.properties.startdate'
completionTimeFromAscendingNode:
- '{{"enddate": "{completionTimeFromAscendingNode}"}}'
- '$.properties.enddate'
downloadLink: '$.properties.location'
title: '$.id'
storageStatus: 'OFFLINE'
processingLevel:
- '{{"processingLevel": "{processingLevel}"}}'
- '$.null'
providerProductType:
- '{{"productType": "{providerProductType}"}}'
- '$.null'
timeliness:
- '{{"timeliness": "{timeliness}"}}'
- '$.null'
orbitDirection:
- '{{"orbitDirection": "{orbitDirection}"}}'
- '$.null'
variable:
- '{{"variable": "{variable}"}}'
- '$.null'
system:
- '{{"system": "{system}"}}'
- '$.null'
version:
- '{{"version": {version}}}'
- '$.null'
region:
- '{{"region": {region}}}'
- '$.null'
type:
- '{{"type": "{type}"}}'
- '$.null'
source:
- '{{"source": {source}}}'
- '$.null'
model:
- '{{"model": {model}}}'
- '$.null'
level:
- '{{"level": {level}}}'
- '$.null'
step:
- '{{"step": {step}}}'
- '$.null'
satellite:
- '{{"satellite": {satellite}}}'
- '$.null'
products:
S1_SAR_GRD:
productType: EO:ESA:DAT:SENTINEL-1
providerProductType: GRD
metadata_mapping:
<<: *s1_sar_params
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:ESA:DAT:SENTINEL-1"}}'
S1_SAR_RAW:
productType: EO:ESA:DAT:SENTINEL-1
providerProductType: RAW
metadata_mapping_from_product: S1_SAR_GRD
S1_SAR_OCN:
productType: EO:ESA:DAT:SENTINEL-1
providerProductType: OCN
metadata_mapping_from_product: S1_SAR_GRD
S1_SAR_SLC:
productType: EO:ESA:DAT:SENTINEL-1
providerProductType: SLC
metadata_mapping_from_product: S1_SAR_GRD
S2_MSI_L1C:
productType: EO:ESA:DAT:SENTINEL-2
providerProductType: S2MSI1C
processingLevel: S2MSI1C
metadata_mapping:
processingLevel:
- '{{"processingLevel": "{processingLevel}"}}'
- '$.id.`sub(/^[^_]([^_]+)_([^_]+)_.*/, S2\\2)`'
cloudCover:
- '{{"cloudCover": "{cloudCover}"}}'
- '$.null'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:ESA:DAT:SENTINEL-2"}}'
S2_MSI_L2A:
productType: EO:ESA:DAT:SENTINEL-2
providerProductType: S2MSI2A
processingLevel: S2MSI2A
metadata_mapping_from_product: S2_MSI_L1C
S3_LAN_HY:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: SR_2_LAN_HY
processingLevel: 2
metadata_mapping:
processingLevel:
- '{{"processingLevel": "{processingLevel}"}}'
- '2'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:ESA:DAT:SENTINEL-3"}}'
S3_LAN_SI:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: SR_2_LAN_SI
processingLevel: 2
metadata_mapping_from_product: S3_LAN_HY
S3_LAN_LI:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: SR_2_LAN_LI
processingLevel: 2
metadata_mapping_from_product: S3_LAN_HY
S3_OLCI_L2LFR:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: OL_2_LFR___
processingLevel: 2
metadata_mapping_from_product: S3_LAN_HY
S3_OLCI_L2LRR:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: OL_2_LRR___
processingLevel: 2
metadata_mapping_from_product: S3_LAN_HY
S3_SLSTR_L2:
productType: EO:ESA:DAT:SENTINEL-3
providerProductType: SL_2_LST___
processingLevel: 2
metadata_mapping_from_product: S3_LAN_HY
S3_EFR:
productType: EO:EUM:DAT:SENTINEL-3:OL_1_EFR___
metadata_mapping:
id:
- '{{"timeliness": {id#split_id_into_s3_params}["timeliness"], "sat": {id#split_id_into_s3_params}["sat"], "startdate": {id#split_id_into_s3_params}["startDate"], "enddate": {id#split_id_into_s3_params}["endDate"]}}'
- '{$.id#remove_extension}'
platform:
- '{{"sat": "{platform}"}}'
- '$.id.`sub(/^[^_]([^_]+)_.*/, Sentinel-\\1)`'
platformSerialIdentifier: '$.id.`sub(/^[^_]([^_]+)_.*/, S\\1)`'
<<: *orbit_cycle
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_1_EFR___"}}'
S3_ERR:
productType: EO:EUM:DAT:SENTINEL-3:OL_1_ERR___
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_1_ERR___"}}'
S3_OLCI_L2WFR:
productType: EO:EUM:DAT:SENTINEL-3:OL_2_WFR___
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_2_WFR___"}}'
S3_OLCI_L2WRR:
productType: EO:EUM:DAT:SENTINEL-3:OL_2_WRR___
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_2_WRR___"}}'
S3_SRA:
productType: EO:EUM:DAT:SENTINEL-3:SR_1_SRA___
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:SR_1_SRA___"}}'
S3_SRA_A:
productType: EO:EUM:DAT:SENTINEL-3:SR_1_SRA_A_
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:SR_1_SRA_A_"}}'
S3_SRA_BS:
productType: EO:EUM:DAT:SENTINEL-3:SR_1_SRA_BS
metadata_mapping_from_product: S3_EFR
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:SR_1_SRA_BS"}}'
S3_SLSTR_L1RBT:
productType: EO:EUM:DAT:SENTINEL-3:SL_1_RBT___
providerProductType: SL_1_RBT___
metadata_mapping:
id:
- '{{"productType": "SL_1_RBT___", "timeliness": {id#split_id_into_s3_params}["timeliness"], "sat": {id#split_id_into_s3_params}["sat"], "startdate": {id#split_id_into_s3_params}["startDate"], "enddate": {id#split_id_into_s3_params}["endDate"]}}'
- '{$.id#remove_extension}'
platform:
- '{{"sat": "{platform}"}}'
- '$.id.`sub(/^[^_]([^_]+)_.*/, Sentinel-\\1)`'
platformSerialIdentifier: '$.id.`sub(/^[^_]([^_]+)_.*/, S\\1)`'
<<: *orbit_cycle
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:SL_1_RBT___"}}'
S3_WAT:
productType: EO:EUM:DAT:SENTINEL-3:SR_2_WAT___
metadata_mapping:
id:
- '{{"type": "SR_2_WAT___", "timeliness": {id#split_id_into_s3_params}["timeliness"], "sat": {id#split_id_into_s3_params}["sat"], "startdate": {id#split_id_into_s3_params}["startDate"], "enddate": {id#split_id_into_s3_params}["endDate"]}}'
- '{$.id#remove_extension}'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EUM:DAT:SENTINEL-3:SR_2_WAT___"}}'
S5P_L1B_IR_ALL:
productType: EO:ESA:DAT:SENTINEL-5P
processingLevel: L1B
metadata_mapping:
processingLevel:
- '{{"processingLevel": "{processingLevel}"}}'
- '$.id.`sub(/^[^_]([^_]+)_([^_]+)_([^_]+)_.*/, \\3)`'
processingMode:
- '{{"processingMode": "{processingMode}"}}'
- '$.null'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:ESA:DAT:SENTINEL-5P"}}'
S5P_L2_IR_ALL:
productType: EO:ESA:DAT:SENTINEL-5P
processingLevel: L2
metadata_mapping_from_product: S5P_L1B_IR_ALL
EEA_DAILY_VI:
productType: EO:EEA:DAT:CLMS_HRVPP_VI
metadata_mapping:
id:
- '{{"uid": "{id}"}}'
- '$.id'
relativeOrbitNumber:
- '{{"relativeOrbitNumber": "{relativeOrbitNumber}"}}'
- '$.null'
version:
- '{{"productVersion": "{version}"}}'
- '$.null'
platformSerialIdentifier:
- '{{"platformSerialIdentifier": "{platformSerialIdentifier}"}}'
- '$.id.`sub(/^[^_]+_[^_]+_([^_]+)_.*/, \\1)`'
tileIdentifier:
- '{{"tileId": "{tileIdentifier}"}}'
- '$.null'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EEA:DAT:CLMS_HRVPP_VI"}}'
EEA_HRL_TCF:
productType: EO:EEA:DAT:HRL:TCF
metadata_mapping:
startTimeFromAscendingNode:
- '{{"year": {startTimeFromAscendingNode#to_datetime_dict("string")}["year"]}}'
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
resolution:
- '{{"resolution": "{resolution}"}}'
- '{$.id#replace_str(r"^.*_R([0-9]+m)_.*$",r"\1")}'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EEA:DAT:HRL:TCF"}}'
COP_DEM_GLO30_DGED:
productType: EO:ESA:DAT:COP-DEM
providerProductType: DGE_30
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:ESA:DAT:COP-DEM"}}'
COP_DEM_GLO30_DTED:
productType: EO:ESA:DAT:COP-DEM
providerProductType: DTE_30
metadata_mapping_from_product: COP_DEM_GLO30_DGED
COP_DEM_GLO90_DGED:
productType: EO:ESA:DAT:COP-DEM
providerProductType: DGE_90
metadata_mapping_from_product: COP_DEM_GLO30_DGED
COP_DEM_GLO90_DTED:
productType: EO:ESA:DAT:COP-DEM
providerProductType: DTE_90
metadata_mapping_from_product: COP_DEM_GLO30_DGED
CLMS_GLO_NDVI_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_NDVI_300M_V1_10DAILY_NETCDF
providerProductType: NDVI300
metadata_mapping:
<<: *id_from_date
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_NDVI_300M_V1_10DAILY_NETCDF"}}'
CLMS_GLO_NDVI_1KM_LTS:
productType: EO:CLMS:DAT:CLMS_GLOBAL_NDVI_1KM_V2_10DAILY_NETCDF
providerProductType: NDVI
metadata_mapping:
<<: *id_from_date
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_NDVI_1KM_V2_10DAILY_NETCDF"}}'
CLMS_CORINE:
productType: EO:EEA:DAT:CORINE
providerProductType: Corine Land Cover 2018
discover_metadata:
raise_mtd_discovery_error: true
metadata_mapping:
id:
- '{{"format": "{id#get_group_name((?P<GeoPackage>geoPackage)|(?P<ESRI fgdb>fgdb)|(?P<GeoTiff100mt>raster100m))}"}}'
- '$.id'
startTimeFromAscendingNode: '$.properties.startdate'
completionTimeFromAscendingNode: '$.properties.enddate'
format:
- '{{"format": "{format}"}}'
- '{$.id#get_group_name((?P<GeoPackage>geoPackage)|(?P<ESRI fgdb>fgdb)|(?P<GeoTiff100mt>raster100m))}'
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:EEA:DAT:CORINE"}}'
CLMS_GLO_FCOVER_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_FCOVER_300M_V1_10DAILY_NETCDF
providerProductType: FCOVER300
metadata_mapping:
<<: *id_from_date
productGroupId:
- '{{"productGroupId": "{productGroupId}"}}'
- '$.null'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_FCOVER_300M_V1_10DAILY_NETCDF"}}'
CLMS_GLO_DMP_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_DMP_300M_V1_10DAILY_NETCDF
providerProductType: DMP300
metadata_mapping_from_product: CLMS_GLO_FCOVER_333M
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_DMP_300M_V1_10DAILY_NETCDF"}}'
CLMS_GLO_GDMP_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_GDMP_300M_V1_10DAILY_NETCDF
providerProductType: GDMP300
metadata_mapping_from_product: CLMS_GLO_FCOVER_333M
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_GDMP_300M_V1_10DAILY_NETCDF"}}'
CLMS_GLO_FAPAR_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_FAPAR_300M_V1_10DAILY_NETCDF
providerProductType: FAPAR300
metadata_mapping_from_product: CLMS_GLO_FCOVER_333M
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_FAPAR_300M_V1_10DAILY_NETCDF"}}'
CLMS_GLO_LAI_333M:
productType: EO:CLMS:DAT:CLMS_GLOBAL_LAI_300M_V1_10DAILY_NETCDF
providerProductType: LAI300
metadata_mapping_from_product: CLMS_GLO_FCOVER_333M
metadata_mapping:
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "EO:CLMS:DAT:CLMS_GLOBAL_LAI_300M_V1_10DAILY_NETCDF"}}'
auth: !plugin
type: TokenAuth
matching_url: https://[-\w\.]+.wekeo2.eu
auth_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/gettoken'
refresh_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/refreshtoken'
token_type: json
token_key: access_token
refresh_token_key: refresh_token
token_expiration_key: expires_in
download: !plugin
type: HTTPDownload
base_uri: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess
flatten_top_dirs: true
auth_error_code: 401
order_enabled: true
order_method: 'POST'
order_on_response:
metadata_mapping:
orderId: '$.json.download_id'
orderStatusLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/downloads?download_id={orderId}'
downloadLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/{orderId}'
order_status:
request:
method: GET
metadata_mapping:
status: $.json.features[0].status
message: $.json.features[0].message
error:
status: Error
success:
status: Done
ordered:
http_code: 202
on_success:
metadata_mapping:
download_id: $.json.features[0]._id
downloadLink: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/{download_id}
---
!provider # MARK: wekeo_ecmwf
name: wekeo_ecmwf
group: wekeo
priority: 0
roles:
- host
description: WEkEO - ECMWF data
url: https://www.wekeo.eu/
# anchors to avoid duplications
anchor_time_day_month_year: &time_day_month_year
startTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
anchor_day_month_year: &day_month_year
startTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"]
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
anchor_month_year: &month_year
startTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"]
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
search: !plugin
type: WekeoECMWFSearch
api_endpoint: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/search
need_auth: true
timeout: 60
need_auth: true
auth_error_code: 401
dates_required: true
results_entry: 'features'
pagination:
total_items_nb_key_path: '$.properties.totalResults'
next_page_query_obj: '{{"itemsPerPage":{items_per_page},"startIndex":{skip}}}'
max_items_per_page: 200
discover_product_types:
fetch_url: null
available_values_url: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/queryable/{dataset}'
metadata_mapping:
geometry:
- '{{"bbox": {geometry#to_bounds}}}'
- $.geometry
startTimeFromAscendingNode:
- '{{"startdate": "{startTimeFromAscendingNode#to_iso_utc_datetime}"}}'
- $.properties.startdate
completionTimeFromAscendingNode:
- '{{"enddate": "{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}'
- $.properties.enddate
productType:
- dataset_id
- $.dataset
downloadLink: $.properties.location
dataset:
- dataset_id
- $.dataset
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "dataset_id": "{dataset}"}}'
products:
SATELLITE_CARBON_DIOXIDE:
dataset: EO:ECMWF:DAT:SATELLITE_CARBON_DIOXIDE
metadata_mapping:
<<: *day_month_year
SATELLITE_FIRE_BURNED_AREA:
dataset: EO:ECMWF:DAT:SATELLITE_FIRE_BURNED_AREA
metadata_mapping:
startTimeFromAscendingNode:
- |
{{
"year": {startTimeFromAscendingNode#to_datetime_dict(list)}["year"],
"month": {startTimeFromAscendingNode#to_datetime_dict(list)}["month"],
"nominal_day": {startTimeFromAscendingNode#to_datetime_dict(list)}["day"]
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
SATELLITE_METHANE:
dataset: EO:ECMWF:DAT:SATELLITE_METHANE
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_ICE_EDGE_TYPE:
dataset: EO:ECMWF:DAT:SATELLITE_SEA_ICE_EDGE_TYPE
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_ICE_THICKNESS:
dataset: EO:ECMWF:DAT:SATELLITE_SEA_ICE_THICKNESS
metadata_mapping:
<<: *month_year
SATELLITE_SEA_ICE_CONCENTRATION:
dataset: EO:ECMWF:DAT:SATELLITE_SEA_ICE_CONCENTRATION
metadata_mapping:
<<: *day_month_year
SATELLITE_SEA_LEVEL_GLOBAL:
dataset: EO:ECMWF:DAT:SATELLITE_SEA_LEVEL_GLOBAL
metadata_mapping:
<<: *day_month_year
SEASONAL_ORIGINAL_SL:
dataset: EO:ECMWF:DAT:SEASONAL_ORIGINAL_SINGLE_LEVELS
metadata_mapping:
<<: *day_month_year
SEASONAL_ORIGINAL_PL:
dataset: EO:ECMWF:DAT:SEASONAL_ORIGINAL_PRESSURE_LEVELS
metadata_mapping:
<<: *day_month_year
SEASONAL_POSTPROCESSED_SL:
dataset: EO:ECMWF:DAT:SEASONAL_POSTPROCESSED_SINGLE_LEVELS
metadata_mapping:
<<: *month_year
SEASONAL_POSTPROCESSED_PL:
dataset: EO:ECMWF:DAT:SEASONAL_POSTPROCESSED_PRESSURE_LEVELS
metadata_mapping:
<<: *month_year
SEASONAL_MONTHLY_SL:
dataset: EO:ECMWF:DAT:SEASONAL_MONTHLY_SINGLE_LEVELS
metadata_mapping:
<<: *month_year
SEASONAL_MONTHLY_PL:
dataset: EO:ECMWF:DAT:SEASONAL_MONTHLY_PRESSURE_LEVELS
metadata_mapping:
<<: *month_year
GLACIERS_DIST_RANDOLPH:
dataset: EO:ECMWF:DAT:INSITU_GLACIERS_EXTENT
metadata_mapping:
<<: *day_month_year
FIRE_HISTORICAL:
dataset: EO:ECMWF:DAT:CEMS_FIRE_HISTORICAL_V1
metadata_mapping:
<<: *day_month_year
GRIDDED_GLACIERS_MASS_CHANGE:
dataset: EO:ECMWF:DAT:DERIVED_GRIDDED_GLACIER_MASS_CHANGE
metadata_mapping:
startTimeFromAscendingNode:
- |
{{
"hydrological_year": {startTimeFromAscendingNode#get_hydrological_year}
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
UERRA_EUROPE_SL:
dataset: EO:ECMWF:DAT:REANALYSIS_UERRA_EUROPE_SINGLE_LEVELS
metadata_mapping:
<<: *time_day_month_year
AG_ERA5:
dataset: EO:ECMWF:DAT:SIS_AGROMETEOROLOGICAL_INDICATORS
metadata_mapping:
<<: *day_month_year
ERA5_SL:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_SINGLE_LEVELS
metadata_mapping:
<<: *time_day_month_year
ERA5_PL:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_PRESSURE_LEVELS
metadata_mapping:
<<: *time_day_month_year
ERA5_SL_MONTHLY:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_SINGLE_LEVELS_MONTHLY_MEANS
metadata_mapping:
startTimeFromAscendingNode:
- |
{{
"year": {startTimeFromAscendingNode#to_datetime_dict(list)}["year"],
"month": {startTimeFromAscendingNode#to_datetime_dict(list)}["month"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
ERA5_PL_MONTHLY:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_PRESSURE_LEVELS_MONTHLY_MEANS
metadata_mapping_from_product: ERA5_SL_MONTHLY
ERA5_LAND:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_LAND
metadata_mapping:
<<: *time_day_month_year
ERA5_LAND_MONTHLY:
dataset: EO:ECMWF:DAT:REANALYSIS_ERA5_LAND_MONTHLY_MEANS
metadata_mapping_from_product: ERA5_SL_MONTHLY
CAMS_EAC4:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_REANALYSIS_EAC4
CAMS_GLOBAL_EMISSIONS:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_EMISSION_INVENTORIES
metadata_mapping:
startTimeFromAscendingNode:
- |
{{
"year": {startTimeFromAscendingNode#to_datetime_dict(list)}["year"]
}}
- $.properties.startdate
completionTimeFromAscendingNode: $.properties.enddate
CAMS_EAC4_MONTHLY:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_REANALYSIS_EAC4_MONTHLY
metadata_mapping:
<<: *month_year
CAMS_GREENHOUSE_INVERSION:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_GREENHOUSE_GAS_INVERSION
metadata_mapping:
<<: *month_year
CAMS_EU_AIR_QUALITY_RE:
dataset: EO:ECMWF:DAT:CAMS_EUROPE_AIR_QUALITY_REANALYSES
metadata_mapping:
<<: *month_year
CAMS_GRF:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_RADIATIVE_FORCINGS
metadata_mapping:
<<: *month_year
CAMS_GRF_AUX:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_RADIATIVE_FORCING_AUXILLIARY_VARIABLES
metadata_mapping:
<<: *month_year
CAMS_GREENHOUSE_EGG4:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_GHG_REANALYSIS_EGG4
CAMS_GREENHOUSE_EGG4_MONTHLY:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_GHG_REANALYSIS_EGG4_MONTHLY
metadata_mapping:
<<: *month_year
CAMS_EU_AIR_QUALITY_FORECAST:
dataset: EO:ECMWF:DAT:CAMS_EUROPE_AIR_QUALITY_FORECASTS
CAMS_GAC_FORECAST:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_ATMOSPHERIC_COMPOSITION_FORECASTS
CAMS_GFE_GFAS:
dataset: EO:ECMWF:DAT:CAMS_GLOBAL_FIRE_EMISSIONS_GFAS
CAMS_SOLAR_RADIATION:
dataset: EO:ECMWF:DAT:CAMS_SOLAR_RADIATION_TIMESERIES
metadata_mapping:
geometry:
- '{{"longitude": {geometry#to_longitude_latitude}["lon"], "latitude": {geometry#to_longitude_latitude}["lat"]}}'
- '$.null'
auth: !plugin
type: TokenAuth
auth_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/gettoken'
refresh_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/refreshtoken'
token_type: json
token_key: access_token
refresh_token_key: refresh_token
token_expiration_key: expires_in
download: !plugin
type: HTTPDownload
auth_error_code: 401
order_enabled: true
order_method: 'POST'
order_on_response:
metadata_mapping:
orderId: '$.json.download_id'
orderStatusLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/downloads?download_id={orderId}'
order_status:
request:
method: GET
metadata_mapping:
status: $.json.features[0].status
order:date: $.json.features[0].started_at
creationDate: $.json.features[0].started_at
message: $.json.features[0].message
orderId: $.json.features[0]._id
ecmwf:dataset: $.json.features[0].dataset_id
error:
status: Error
success:
status: Done
ordered:
http_code: 202
on_success:
metadata_mapping:
download_id: $.json.features[0]._id
downloadLink: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/{download_id}
products:
FIRE_HISTORICAL:
output_extension: .grib
---
!provider # MARK: wekeo_cmems
name: wekeo_cmems
group: wekeo
priority: 0
roles:
- host
description: WEkEO - Copernicus Marine Service
url: https://www.wekeo.eu/
search: !plugin
type: PostJsonSearch
api_endpoint: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/search'
need_auth: true
timeout: 60
auth_error_code: 401
results_entry: 'features'
pagination:
total_items_nb_key_path: '$.properties.totalResults'
next_page_query_obj: '{{"itemsPerPage":{items_per_page},"startIndex":{skip}}}'
discover_product_types:
fetch_url: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/datasets?itemsPerPage=1400&q=EO:MO
single_collection_fetch_qs: q={productType}
single_collection_fetch_url: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/datasets/{productType}
result_type: json
results_entry: 'features'
generic_product_type_id: '$.dataset_id'
generic_product_type_parsable_properties:
collection: '$.dataset_id'
generic_product_type_parsable_metadata:
abstract: '$.metadata.description'
license: '$.terms'
instrument: '$.null'
platform: '$.null'
platformSerialIdentifier: '$.null'
single_product_type_parsable_metadata:
title: '$.metadata._source.datasetTitle'
missionStartDate: '$.metadata._source.tempextent_begin'
missionEndDate: '$.metadata._source.tempextent_end'
processingLevel: '$.null'
keywords: '$.metadata._source.keywords'
discover_queryables:
fetch_url: null
product_type_fetch_url: null
constraints_url: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/queryable/{dataset}'
constraints_entry: constraints
metadata_mapping:
productType:
- '{{"dataset_id": "{productType}"}}'
- '$.null'
id:
- '{{"min_date": {id#dates_from_cmems_id}["min_date"], "max_date": {id#dates_from_cmems_id}["max_date"]}}'
- '$.id'
geometry:
- '{{"bbox": {geometry#to_bounds}}}'
- '$.geometry'
startTimeFromAscendingNode:
- '{{"min_date": "{startTimeFromAscendingNode#to_iso_utc_datetime}"}}'
- '$.properties.startdate'
completionTimeFromAscendingNode:
- '{{"max_date": "{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}'
- '$.properties.enddate'
variable:
- '{{"variable": "{variable}"}}'
- '{$.properties.location#get_variables_from_path}'
downloadLink: '$.properties.location'
title: '$.id'
storageStatus: 'OFFLINE'
orderLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download?{{"location": "{downloadLink}","product_id":"{id}", "cacheable": "true", "dataset_id": "productType"}}'
products:
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
auth: !plugin
type: TokenAuth
matching_url: https://[-\w\.]+.wekeo2.eu
auth_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/gettoken'
refresh_uri: 'https://gateway.prod.wekeo2.eu/hda-broker/refreshtoken'
token_type: json
token_key: access_token
refresh_token_key: refresh_token
download: !plugin
type: HTTPDownload
auth_error_code: 401
order_enabled: true
order_method: 'POST'
order_on_response:
metadata_mapping:
orderId: '$.json.download_id'
orderStatusLink: 'https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/downloads?download_id={orderId}'
order_status:
request:
method: GET
metadata_mapping:
status: $.json.features[0].status
message: $.json.features[0].message
error:
status: Error
success:
status: Done
ordered:
http_code: 202
on_success:
metadata_mapping:
download_id: $.json.features[0]._id
downloadLink: https://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/{download_id}
---
!provider # MARK: creodias_s3
name: creodias_s3
priority: 0
description: CloudFerro DIAS data through S3 protocol
roles:
- host
url: https://creodias.eu/
search: !plugin
type: CreodiasS3Search
api_endpoint: 'https://datahub.creodias.eu/odata/v1/Products'
s3_endpoint: 'https://eodata.cloudferro.com'
need_auth: true
timeout: 120
ssl_verify: true
dont_quote:
- '['
- ']'
- '$'
- '='
- '&'
- ':'
- '%'
pagination:
next_page_url_tpl: '{url}?{search}&$top={items_per_page}&$skip={skip}&$expand=Attributes&$expand=Assets'
count_tpl: '&$count=True'
total_items_nb_key_path: '$."@odata.count"'
max_items_per_page: 1_000
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_by_tpl: '&$orderby={sort_param} {sort_order}'
sort_param_mapping:
startTimeFromAscendingNode: ContentDate/Start
completionTimeFromAscendingNode: ContentDate/End
publicationDate: PublicationDate
modificationDate: ModificationDate
sort_order_mapping:
ascending: asc
descending: desc
max_sort_params: 1
results_entry: 'value'
free_text_search_operations:
$filter:
union: ' or '
wrapper: '{}'
operations:
and:
- "Collection/Name eq '{collection}'"
- "OData.CSC.Intersects(area=geography'{geometry#to_ewkt}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{productType}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformShortName' and att/OData.CSC.StringAttribute/Value eq '{platform}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'platformSerialIdentifier' and att/OData.CSC.StringAttribute/Value eq '{platformSerialIdentifier#replace_str(\"^S[1-3]\", \"\")}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'spatialResolution' and att/OData.CSC.StringAttribute/Value eq '{resolution}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'authority' and att/OData.CSC.StringAttribute/Value eq '{organisationName}')"
- "Attributes/OData.CSC.IntegerAttribute/any(att:att/Name eq 'orbitNumber' and att/OData.CSC.StringAttribute/Value eq '{orbitNumber}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq '{orbitDirection#to_upper}')"
- "Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le {cloudCover})"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'operationalMode' and att/OData.CSC.StringAttribute/Value eq '{sensorMode}')"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'tileId' and att/OData.CSC.StringAttribute/Value eq '{tileIdentifier}')"
- "ContentDate/Start lt {completionTimeFromAscendingNode#to_iso_utc_datetime}"
- "ContentDate/End gt {startTimeFromAscendingNode#to_iso_utc_datetime}"
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'polarisationChannels' and att/OData.CSC.StringAttribute/Value eq '{polarizationChannels#replace_str('\\\\+','%26')}')"
- contains(Name,'{id}')
discover_metadata:
auto_discovery: true
metadata_pattern: '^(?!collection)[a-zA-Z0-9]+$'
search_param:
free_text_search_operations:
$filter:
operations:
and:
- "Attributes/OData.CSC.StringAttribute/any(att:att/Name eq '{metadata}' and att/OData.CSC.StringAttribute/Value eq '{{{metadata}}}')"
metadata_path: '$.Attributes.*'
discover_product_types:
fetch_url: null
per_product_metadata_query: false
metadata_pre_mapping:
metadata_path: '$.Attributes'
metadata_path_id: 'Name'
metadata_path_value: 'Value'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
# Queryable parameters are set with null as 1st configuration list value to mark them as queryable,
# but `free_text_search_operations.$filter.operations.and` entries are then used instead.
uid: '$.Id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- null
- '$.Attributes.productType'
platform:
- null
- '$.Attributes.platformShortName'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier'
instrument:
- null
- '$.Attributes.instrumentShortName'
processingLevel:
- null
- '$.Attributes.processingLevel'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '{$.Name#remove_extension}'
resolution:
- null
- '$.Attributes.spatialResolution'
organisationName:
- null
- '$.Attributes.authority'
publicationDate: '$.PublicationDate'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- null
- '$.Attributes.orbitNumber'
orbitDirection:
- null
- '{$.Attributes.orbitDirection#to_lower}'
cloudCover:
- null
- '$.Attributes.cloudCover'
modificationDate: '$.ModificationDate'
sensorMode:
- null
- '$.Attributes.operationalMode'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- null
- '$.ContentDate.Start'
completionTimeFromAscendingNode:
- null
- '$.ContentDate.End'
polarizationChannels:
- null
- '{$.Attributes.polarisationChannels#replace_str("&","+")}'
# Custom parameters (not defined in the base document referenced above)
id:
- null
- '{$.Name#remove_extension}'
tileIdentifier:
- null
- '$.Attributes.tileId'
# The geographic extent of the product
geometry:
- null
- '{$.Footprint#from_ewkt}'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: '$.S3Path.`sub(/^(.*)$/, s3:/\\1)`'
# storageStatus: must be one of ONLINE, STAGING, OFFLINE
storageStatus: '{$.Online#get_group_name((?P<ONLINE>True)|(?P<OFFLINE>False))}'
collection:
- null
- $.null
quicklook: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
thumbnail: '$.Assets[?(@.Type="QUICKLOOK")].DownloadLink'
download: !plugin
type: CreodiasS3Download
s3_endpoint: 'https://eodata.cloudferro.com'
s3_bucket: 'eodata'
ssl_verify: true
auth: !plugin
type: AwsAuth
auth_error_code: 403
s3_endpoint: 'https://eodata.cloudferro.com'
matching_conf:
s3_endpoint: 'https://eodata.cloudferro.com'
products:
# S1
S1_AUX_GNSSRD:
productType: AUX_GNSSRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_MOEORB:
productType: AUX_MOEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_POEORB:
productType: AUX_POEORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PREORB:
productType: AUX_PREORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_PROQUA:
productType: AUX_PROQUA
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_AUX_RESORB:
productType: AUX_RESORB
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_RAW:
productType: RAW
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_GRD:
productType: GRD
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_SLC:
productType: SLC
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_OCN:
productType: OCN
collection: SENTINEL-1
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_IW_MCM:
productType: S1SAR_L3_IW_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
S1_SAR_L3_DH_MCM:
productType: S1SAR_L3_DH_MCM
collection: GLOBAL-MOSAICS
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S1)`'
# S2
S2_MSI_L1C:
collection: SENTINEL-2
productType: S2MSI1C
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
S2_MSI_L2A:
collection: SENTINEL-2
productType: S2MSI2A
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S2)`'
# S3 SRAL
S3_SRA:
productType: SR_1_SRA___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_A:
productType: SR_1_SRA_A_
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SRA_BS:
productType: SR_1_SRA_BS
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN:
productType: SR_2_LAN___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_HY:
productType: SR_2_LAN_HY
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_SI:
productType: SR_2_LAN_SI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_LAN_LI:
productType: SR_2_LAN_LI
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_WAT:
productType: SR_2_WAT___
collection: SENTINEL-3
metadata_mapping:
cloudCover: '$.null'
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 OLCI
S3_EFR:
productType: OL_1_EFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_ERR:
productType: OL_1_ERR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LRR:
productType: OL_2_LRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2LFR:
productType: OL_2_LFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WRR:
productType: OL_2_WRR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_OLCI_L2WFR:
productType: OL_2_WFR___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SLSTR
S3_SLSTR_L1RBT:
productType: SL_1_RBT___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2LST:
productType: SL_2_LST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2WST:
productType: SL_2_WST___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2AOD:
productType: SL_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SLSTR_L2FRP:
productType: SL_2_FRP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S3 SY
S3_SY_AOD:
productType: SY_2_AOD___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_SYN:
productType: SY_2_SYN___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_V10:
productType: SY_2_V10___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VG1:
productType: SY_2_VG1___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
S3_SY_VGP:
productType: SY_2_VGP___
collection: SENTINEL-3
metadata_mapping:
platformSerialIdentifier:
- null
- '$.Attributes.platformSerialIdentifier.`sub(/^/, S3)`'
# S5P L1
S5P_L1B_IR_SIR:
productType: L1B_IR_SIR
collection: SENTINEL-5P
S5P_L1B_IR_UVN:
productType: L1B_IR_UVN
collection: SENTINEL-5P
S5P_L1B_RA_BD1:
productType: L1B_RA_BD1
collection: SENTINEL-5P
S5P_L1B_RA_BD2:
productType: L1B_RA_BD2
collection: SENTINEL-5P
S5P_L1B_RA_BD3:
productType: L1B_RA_BD3
collection: SENTINEL-5P
S5P_L1B_RA_BD4:
productType: L1B_RA_BD4
collection: SENTINEL-5P
S5P_L1B_RA_BD5:
productType: L1B_RA_BD5
collection: SENTINEL-5P
S5P_L1B_RA_BD6:
productType: L1B_RA_BD6
collection: SENTINEL-5P
S5P_L1B_RA_BD7:
productType: L1B_RA_BD7
collection: SENTINEL-5P
S5P_L1B_RA_BD8:
productType: L1B_RA_BD8
collection: SENTINEL-5P
# S5P L2
S5P_L2_NO2:
productType: L2__NO2___
collection: SENTINEL-5P
S5P_L2_CLOUD:
productType: L2__CLOUD_
collection: SENTINEL-5P
S5P_L2_O3:
productType: L2__O3____
collection: SENTINEL-5P
S5P_L2_CO:
productType: L2__CO____
collection: SENTINEL-5P
S5P_L2_AER_AI:
productType: L2__AER_AI
collection: SENTINEL-5P
S5P_L2_O3_PR:
productType: L2__O3__PR
collection: SENTINEL-5P
S5P_L2_O3_TCL:
productType: L2__O3_TCL
collection: SENTINEL-5P
S5P_L2_AER_LH:
productType: L2__AER_LH
collection: SENTINEL-5P
S5P_L2_HCHO:
productType: L2__HCHO__
collection: SENTINEL-5P
S5P_L2_CH4:
productType: L2__CH4___
collection: SENTINEL-5P
S5P_L2_NP_BD3:
productType: L2__NP_BD3
collection: SENTINEL-5P
S5P_L2_NP_BD6:
productType: L2__NP_BD6
collection: SENTINEL-5P
S5P_L2_NP_BD7:
productType: L2__NP_BD7
collection: SENTINEL-5P
S5P_L2_SO2:
productType: L2__SO2___
collection: SENTINEL-5P
# COP DEM
COP_DEM_GLO30_DGED:
productType: DGE_30
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO30_DTED:
productType: DTE_30
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO90_DGED:
productType: DGE_90
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
COP_DEM_GLO90_DTED:
productType: DTE_90
collection: COP-DEM
metadata_mapping:
cloudCover: '$.null'
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
---
!provider # MARK: dedt_lumi
name: dedt_lumi
priority: 0
roles:
- host
description: Destination Earth Digital Twin Outputs from LUMI through Polytope API
url: https://polytope.lumi.apps.dte.destination-earth.eu/openapi
search: !plugin
type: ECMWFSearch
need_auth: true
ssl_verify: true
dates_required: True
discover_queryables:
fetch_url: null
product_type_fetch_url: null
constraints_url: "https://s3.central.data.destination-earth.eu/swift/v1/dedt-lumi-constraints/{dataset}.json"
metadata_mapping:
geometry:
- '{{"feature": {geometry#to_geojson_polytope}}}'
- "$.geometry"
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- '{{"date": "{startTimeFromAscendingNode#to_non_separated_date}/to/{completionTimeFromAscendingNode#to_non_separated_date}"}}'
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
productType:
- dataset
- $.dataset
qs: $.qs
orderLink: 'https://polytope.lumi.apps.dte.destination-earth.eu/api/v1/requests/destination-earth?{{"verb": "retrieve", "request": {qs#to_geojson} }}'
products:
DT_EXTREMES:
dataset: extremes-dt
DT_CLIMATE_ADAPTATION:
dataset: climate-dt
download: !plugin
type: HTTPDownload
ssl_verify: true
auth_error_code: 401
order_enabled: True
order_method: POST
order_on_response:
metadata_mapping:
orderId: '{$.headers.Location#slice_str(-36,,1)}'
_previousOrderId: '{$.json.orderId#replace_str("Not Available","")}'
combinedOrderId: '{orderId#replace_str("Not Available","")}{_previousOrderId}'
orderStatusLink: "https://polytope.lumi.apps.dte.destination-earth.eu/api/v1/requests/{combinedOrderId#replace_str(r'^$','Not Available')}"
order_status:
request:
method: GET
headers:
metadata_mapping:
status: $.json.status
message: $.json.message
error_message: $.null
success:
http_code: 303
error:
status: failed
on_success:
result_type: json
metadata_mapping:
downloadLink: $.headers.Location
no_auth_download: True
products:
DT_EXTREMES:
DT_CLIMATE_ADAPTATION:
auth: !plugin
type: OIDCAuthorizationCodeFlowAuth
oidc_config_url: https://auth.destine.eu/realms/desp/.well-known/openid-configuration
redirect_uri: https://polytope.lumi.apps.dte.destination-earth.eu/
client_id: polytope-api-public
user_consent_needed: false
token_exchange_post_data_method: data
token_provision: header
login_form_xpath: //form[@id='kc-form-login']
authentication_uri_source: login-form
---
!provider # MARK: dedl
name: dedl
priority: 0
roles:
- host
description: DEDL STAC
url: https://hda.data.destination-earth.eu/stac/
# anchors to avoid duplications
anchor_orderable_mm: &orderable_mm
orderLink: '$.assets.downloadLink.href'
downloadLink: '$.null'
assets: '$.null'
search: !plugin
type: StacSearch
api_endpoint: https://hda.data.destination-earth.eu/stac/search
need_auth: true
timeout: 60
asset_key_from_href: false
metadata_mapping:
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
quicklook: '{thumbnail}'
storageStatus: '{$.properties."order:status"#get_group_name((?P<ONLINE>succeeded)|(?P<STAGING>shipping)|(?P<OFFLINE>orderable))}'
downloadLink: '$.assets.downloadLink.href'
assets: '$.null'
discover_product_types:
fetch_url: 'https://hda.data.destination-earth.eu/stac/collections'
result_type: json
results_entry: '$.collections[*]'
generic_product_type_id: '$.id'
generic_product_type_parsable_properties:
productType: '$.id'
generic_product_type_parsable_metadata:
abstract: '$.description'
keywords: '{$.keywords#csv_list}'
license: '$.license'
title: '$.title'
missionStartDate: '$.extent.temporal.interval[0][0]'
pagination:
next_page_url_key_path: null
max_items_per_page: 100
sort:
sort_param_mapping:
id: id
startTimeFromAscendingNode: datetime
creationDate: created
modificationDate: updated
platformSerialIdentifier: platform
resolution: gsd
cloudCover: eo:cloud_cover
download: !plugin
type: HTTPDownload
auth_error_code: 403
timeout: 20
ssl_verify: true
extract: true
archive_depth: 2
order_enabled: True
order_method: GET
order_on_response:
metadata_mapping:
orderStatusLink: $.json.location
order_status:
request:
method: GET
metadata_mapping:
status: $.json.status
message: $.json.description
success:
http_code: 200
on_success:
metadata_mapping:
downloadLink: '{orderStatusLink}'
auth: !plugin
type: OIDCTokenExchangeAuth
matching_url: https://[-\w\.]+.data.destination-earth.eu
subject:
oidc_config_url: https://auth.destine.eu/realms/desp/.well-known/openid-configuration
redirect_uri: https://hda.data.destination-earth.eu/stac
client_id: dedl-hda
user_consent_needed: false
exchange_url_error_pattern:
TERMS_AND_CONDITIONS: Terms and conditions are not accepted
token_exchange_post_data_method: data
token_key: access_token
token_provision: header
login_form_xpath: //form[@id='kc-form-login']
authentication_uri_source: login-form
subject_issuer: desp-oidc
token_uri: https://identity.data.destination-earth.eu/auth/realms/dedl/protocol/openid-connect/token
client_id: hda-public
audience: hda-public
token_key: access_token
products:
# Sentinel 1
S1_SAR_GRD:
productType: EO.ESA.DAT.SENTINEL-1.L1_GRD
S1_SAR_SLC:
productType: EO.ESA.DAT.SENTINEL-1.L1_SLC
# Sentinel 2
S2_MSI_L1C:
productType: EO.ESA.DAT.SENTINEL-2.MSI.L1C
S2_MSI_L2A:
productType: EO.ESA.DAT.SENTINEL-2.MSI.L2A
# Sentinel 3 - S3 OLCI L1
S3_EFR:
productType: EO.EUM.DAT.SENTINEL-3.OL_1_EFR___
S3_ERR:
productType: EO.EUM.DAT.SENTINEL-3.OL_1_ERR___
# Sentinel 3 - S3 OLCI L2
S3_OLCI_L2LRR:
productType: EO.ESA.DAT.SENTINEL-3.OL_2_LRR___
S3_OLCI_L2LFR:
productType: EO.ESA.DAT.SENTINEL-3.OL_2_LFR___
S3_OLCI_L2WRR:
productType: EO.EUM.DAT.SENTINEL-3.OL_2_WRR___
S3_OLCI_L2WFR:
productType: EO.EUM.DAT.SENTINEL-3.OL_2_WFR___
# Sentinel 3 - S3 SLSTR
S3_SLSTR_L1RBT:
productType: EO.EUM.DAT.SENTINEL-3.SL_1_RBT___
S3_SLSTR_L2LST:
productType: EO.ESA.DAT.SENTINEL-3.SL_2_LST___
S3_SLSTR_L2WST:
productType: EO.EUM.DAT.SENTINEL-3.SL_2_WST___
S3_SLSTR_L2AOD:
productType: EO.EUM.DAT.SENTINEL-3.AOD
S3_SLSTR_L2FRP:
productType: EO.EUM.DAT.SENTINEL-3.FRP
# Sentinel 3 - S3 SRAL
S3_SRA:
productType: EO.EUM.DAT.SENTINEL-3.SR_1_SRA___
S3_SRA_A:
productType: EO.EUM.DAT.SENTINEL-3.SR_1_SRA_A_
S3_SRA_BS:
productType: EO.EUM.DAT.SENTINEL-3.SR_1_SRA_BS
S3_LAN:
productType: EO.ESA.DAT.SENTINEL-3.SR_2_LAN___
S3_WAT:
productType: EO.EUM.DAT.SENTINEL-3.SR_2_WAT___
# S5
S5P_L1B_IR_ALL:
productType: EO.ESA.DAT.SENTINEL-5P.TROPOMI.L1
S5P_L2_IR_ALL:
productType: EO.ESA.DAT.SENTINEL-5P.TROPOMI.L2
# ECMWF - COPERNICUS CDS
ERA5_SL:
productType: EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS
metadata_mapping:
<<: *orderable_mm
ERA5_SL_MONTHLY:
productType: EO.ECMWF.DAT.REANALYSIS_ERA5_SINGLE_LEVELS_MONTHLY_MEANS
metadata_mapping:
<<: *orderable_mm
ERA5_PL:
productType: EO.ECMWF.DAT.ERA5_HOURLY_VARIABLES_ON_PRESSURE_LEVELS
metadata_mapping:
<<: *orderable_mm
ERA5_PL_MONTHLY:
productType: EO.ECMWF.DAT.ERA5_MONTHLY_MEANS_VARIABLES_ON_PRESSURE_LEVELS
metadata_mapping:
<<: *orderable_mm
ERA5_LAND:
productType: EO.ECMWF.DAT.ERA5_LAND_HOURLY
metadata_mapping:
<<: *orderable_mm
ERA5_LAND_MONTHLY:
productType: EO.ECMWF.DAT.ERA5_LAND_MONTHLY
metadata_mapping:
<<: *orderable_mm
UERRA_EUROPE_SL:
productType: EO.ECMWF.DAT.REANALYSIS_UERRA_EUROPE_SINGLE_LEVELS
metadata_mapping:
<<: *orderable_mm
GRIDDED_GLACIERS_MASS_CHANGE:
productType: EO.ECMWF.DAT.DERIVED_GRIDDED_GLACIER_MASS_CHANGE
metadata_mapping:
<<: *orderable_mm
GLACIERS_DIST_RANDOLPH:
productType: EO.ECMWF.DAT.GLACIERS_DISTRIBUTION_DATA_FROM_RANDOLPH_GLACIER_INVENTORY_2000
metadata_mapping:
<<: *orderable_mm
SATELLITE_CARBON_DIOXIDE:
productType: EO.ECMWF.DAT.CO2_DATA_FROM_SATELLITE_SENSORS_2002_PRESENT
metadata_mapping:
<<: *orderable_mm
SATELLITE_METHANE:
productType: EO.ECMWF.DAT.METHANE_DATA_SATELLITE_SENSORS_2002_PRESENT
metadata_mapping:
<<: *orderable_mm
SEASONAL_POSTPROCESSED_PL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_ANOMALIES_ON_PRESSURE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SATELLITE_SEA_LEVEL_GLOBAL:
productType: EO.ECMWF.DAT.SEA_LEVEL_DAILY_GRIDDED_DATA_FOR_GLOBAL_OCEAN_1993_PRESENT
metadata_mapping:
<<: *orderable_mm
SATELLITE_SEA_ICE_EDGE_TYPE:
productType: EO.ECMWF.DAT.SATELLITE_SEA_ICE_EDGE_TYPE
metadata_mapping:
<<: *orderable_mm
SATELLITE_SEA_ICE_THICKNESS:
productType: EO.ECMWF.DAT.SATELLITE_SEA_ICE_THICKNESS
metadata_mapping:
<<: *orderable_mm
SATELLITE_SEA_ICE_CONCENTRATION:
productType: EO.ECMWF.DAT.SATELLITE_SEA_ICE_CONCENTRATION
metadata_mapping:
<<: *orderable_mm
SEASONAL_POSTPROCESSED_SL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_ANOMALIES_ON_SINGLE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SEASONAL_ORIGINAL_SL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_DAILY_DATA_ON_SINGLE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SEASONAL_ORIGINAL_PL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_DAILY_DATA_ON_PRESSURE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SEASONAL_MONTHLY_PL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_MONTHLY_STATISTICS_ON_PRESSURE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SEASONAL_MONTHLY_SL:
productType: EO.ECMWF.DAT.SEASONAL_FORECAST_MONTHLY_STATISTICS_ON_SINGLE_LEVELS_2017_PRESENT
metadata_mapping:
<<: *orderable_mm
SIS_HYDRO_MET_PROJ:
productType: EO.ECMWF.DAT.SIS_HYDROLOGY_METEOROLOGY_DERIVED_PROJECTIONS
metadata_mapping:
<<: *orderable_mm
# ECMWF - CEMS
FIRE_HISTORICAL:
productType: EO.ECMWF.DAT.CEMS_FIRE_HISTORICAL
GLOFAS_FORECAST:
productType: EO.ECMWF.DAT.CEMS_GLOFAS_FORECAST
GLOFAS_HISTORICAL:
productType: EO.ECMWF.DAT.CEMS_GLOFAS_HISTORICAL
GLOFAS_REFORECAST:
productType: EO.ECMWF.DAT.CEMS_GLOFAS_REFORECAST
GLOFAS_SEASONAL:
productType: EO.ECMWF.DAT.CEMS_GLOFAS_SEASONAL
GLOFAS_SEASONAL_REFORECAST:
productType: EO.ECMWF.DAT.CEMS_GLOFAS_SEASONAL_REFORECAST
EFAS_FORECAST:
productType: EO.ECMWF.DAT.EFAS_FORECAST
EFAS_HISTORICAL:
productType: EO.ECMWF.DAT.EFAS_HISTORICAL
EFAS_REFORECAST:
productType: EO.ECMWF.DAT.EFAS_REFORECAST
EFAS_SEASONAL:
productType: EO.ECMWF.DAT.EFAS_SEASONAL
EFAS_SEASONAL_REFORECAST:
productType: EO.ECMWF.DAT.EFAS_SEASONAL_REFORECAST
# COPERNICUS ADS
CAMS_GAC_FORECAST:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_ATMOSHERIC_COMPO_FORECAST
metadata_mapping:
<<: *orderable_mm
CAMS_EU_AIR_QUALITY_FORECAST:
productType: EO.ECMWF.DAT.CAMS_EUROPE_AIR_QUALITY_FORECASTS
metadata_mapping:
<<: *orderable_mm
CAMS_GFE_GFAS:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_FIRE_EMISSIONS_GFAS
metadata_mapping:
<<: *orderable_mm
CAMS_SOLAR_RADIATION:
productType: EO.ECMWF.DAT.CAMS_SOLAR_RADIATION_TIMESERIES
metadata_mapping:
<<: *orderable_mm
CAMS_GREENHOUSE_INVERSION:
productType: EO.ECMWF.DAT.CAMS_GREENHOUSE_GAS_FLUXES
metadata_mapping:
<<: *orderable_mm
CAMS_EAC4_MONTHLY:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_REANALYSIS_EAC4_MONTHLY_AV_FIELDS
metadata_mapping:
<<: *orderable_mm
CAMS_EU_AIR_QUALITY_RE:
productType: EO.ECMWF.DAT.CAMS_EUROPE_AIR_QUALITY_REANALYSES
metadata_mapping:
<<: *orderable_mm
CAMS_EAC4:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_REANALYSIS_EAC4
metadata_mapping:
<<: *orderable_mm
CAMS_GRF_AUX:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_RADIATIVE_FORCING_AUX
metadata_mapping:
<<: *orderable_mm
CAMS_GRF:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_RADIATIVE_FORCING
metadata_mapping:
<<: *orderable_mm
CAMS_GREENHOUSE_EGG4_MONTHLY:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS
metadata_mapping:
<<: *orderable_mm
CAMS_GREENHOUSE_EGG4:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS
metadata_mapping:
<<: *orderable_mm
CAMS_GLOBAL_EMISSIONS:
productType: EO.ECMWF.DAT.CAMS_GLOBAL_EMISSION_INVENTORIES
metadata_mapping:
<<: *orderable_mm
# COPERNICUS ADS - Digital Elevation Model
COP_DEM_GLO30_DGED:
productType: EO.DEM.DAT.COP-DEM_GLO-30-DGED
COP_DEM_GLO30_DTED:
productType: EO.DEM.DAT.COP-DEM_GLO-30-DTED
COP_DEM_GLO90_DGED:
title: Copernicus DEM GLO-90 DGED
productType: EO.DEM.DAT.COP-DEM_GLO-90-DGED
COP_DEM_GLO90_DTED:
productType: EO.DEM.DAT.COP-DEM_GLO-90-DTED
# COPERNICUS Marine
MO_GLOBAL_ANALYSISFORECAST_PHY_001_024:
productType: EO.MO.DAT.GLOBAL_ANALYSISFORECAST_PHY_001_024
MO_GLOBAL_ANALYSISFORECAST_WAV_001_027:
productType: EO.MO.DAT.GLOBAL_ANALYSISFORECAST_WAV_001_027
MO_GLOBAL_ANALYSISFORECAST_BGC_001_028:
productType: EO.MO.DAT.GLOBAL_ANALYSISFORECAST_BGC_001_028
MO_GLOBAL_MULTIYEAR_PHY_ENS_001_031:
productType: EO.MO.DAT.GLOBAL_MULTIYEAR_PHY_ENS_001_031
MO_GLOBAL_MULTIYEAR_WAV_001_032:
productType: EO.MO.DAT.GLOBAL_MULTIYEAR_WAV_001_032
MO_GLOBAL_MULTIYEAR_BGC_001_033:
productType: EO.MO.DAT.GLOBAL_MULTIYEAR_BGC_001_033
MO_INSITU_GLO_PHY_TS_OA_NRT_013_002:
productType: EO.MO.DAT.INSITU_GLO_PHY_TS_OA_NRT_013_002
MO_INSITU_GLO_PHY_TS_OA_MY_013_052:
productType: EO.MO.DAT.INSITU_GLO_PHY_TS_OA_MY_013_052
MO_INSITU_GLO_PHY_UV_DISCRETE_NRT_013_048:
productType: EO.MO.DAT.INSITU_GLO_PHY_UV_DISCRETE_NRT_013_048
MO_MULTIOBS_GLO_BIO_BGC_3D_REP_015_010:
productType: EO.MO.DAT.MULTIOBS_GLO_BIO_BGC_3D_REP_015_010
MO_MULTIOBS_GLO_BIO_CARBON_SURFACE_MYNRT_015_008:
productType: EO.MO.DAT.MULTIOBS_GLO_BIO_CARBON_SURFACE_MYNRT_015_008
MO_MULTIOBS_GLO_BGC_NUTRIENTS_CARBON_PROFILES_MYNRT_015_009:
productType: EO.MO.DAT.MULTIOBS_GLO_BGC_NUTRIENTS_CARBON_PROFILES_MYNRT_015_009
MO_MULTIOBS_GLO_PHY_MYNRT_015_003:
productType: EO.MO.DAT.MULTIOBS_GLO_PHY_MYNRT_015_003
MO_MULTIOBS_GLO_PHY_S_SURFACE_MYNRT_015_013:
productType: EO.MO.DAT.MULTIOBS_GLO_PHY_S_SURFACE_MYNRT_015_013
MO_MULTIOBS_GLO_PHY_TSUV_3D_MYNRT_015_012:
productType: EO.MO.DAT.MULTIOBS_GLO_PHY_TSUV_3D_MYNRT_015_012
MO_MULTIOBS_GLO_PHY_W_3D_REP_015_007:
productType: EO.MO.DAT.MULTIOBS_GLO_PHY_W_3D_REP_015_007
MO_OCEANCOLOUR_GLO_BGC_L3_MY_009_103:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L3_MY_009_103
MO_OCEANCOLOUR_GLO_BGC_L3_MY_009_107:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L3_MY_009_107
MO_OCEANCOLOUR_GLO_BGC_L3_NRT_009_101:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L3_NRT_009_101
MO_OCEANCOLOUR_GLO_BGC_L4_MY_009_104:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L4_MY_009_104
MO_OCEANCOLOUR_GLO_BGC_L4_MY_009_108:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L4_MY_009_108
MO_OCEANCOLOUR_GLO_BGC_L4_NRT_009_102:
productType: EO.MO.DAT.OCEANCOLOUR_GLO_BGC_L4_NRT_009_102
MO_SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_001:
productType: EO.MO.DAT.SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_001
MO_SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_006:
productType: EO.MO.DAT.SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_006
MO_SEAICE_GLO_SEAICE_L4_REP_OBSERVATIONS_011_009:
productType: EO.MO.DAT.SEAICE_GLO_SEAICE_L4_REP_OBSERVATIONS_011_009
MO_SEALEVEL_GLO_PHY_L4_NRT_008_046:
productType: EO.MO.DAT.SEALEVEL_GLO_PHY_L4_NRT_008_046
MO_SEALEVEL_GLO_PHY_MDT_008_063:
productType: EO.MO.DAT.SEALEVEL_GLO_PHY_MDT_008_063
MO_SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010:
productType: EO.MO.DAT.SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010
MO_SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001:
productType: EO.MO.DAT.SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001
MO_SST_GLO_SST_L4_REP_OBSERVATIONS_010_011:
productType: EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_011
MO_SST_GLO_SST_L4_REP_OBSERVATIONS_010_024:
productType: EO.MO.DAT.SST_GLO_SST_L4_REP_OBSERVATIONS_010_024
MO_WIND_GLO_PHY_CLIMATE_L4_MY_012_003:
productType: EO.MO.DAT.WIND_GLO_PHY_CLIMATE_L4_MY_012_003
MO_WIND_GLO_PHY_L3_MY_012_005:
productType: EO.MO.DAT.WIND_GLO_PHY_L3_MY_012_005
MO_WIND_GLO_PHY_L3_NRT_012_002:
productType: EO.MO.DAT.WIND_GLO_PHY_L3_NRT_012_002
MO_WIND_GLO_PHY_L4_MY_012_006:
productType: EO.MO.DAT.WIND_GLO_PHY_L4_MY_012_006
MO_WIND_GLO_PHY_L4_NRT_012_004:
productType: EO.MO.DAT.WIND_GLO_PHY_L4_NRT_012_004
MO_WAVE_GLO_PHY_SWH_L3_NRT_014_001:
productType: EO.MO.DAT.WAVE_GLO_PHY_SWH_L3_NRT_014_001
MO_WAVE_GLO_PHY_SWH_L4_NRT_014_003:
productType: EO.MO.DAT.WAVE_GLO_PHY_SWH_L4_NRT_014_003
MO_WAVE_GLO_PHY_SPC_FWK_L3_NRT_014_002:
productType: EO.MO.DAT.WAVE_GLO_PHY_SPC_FWK_L3_NRT_014_002
# CLMS
CLMS_CORINE:
productType: EO.CLMS.DAT.CORINE
CLMS_GLO_DMP_333M:
productType: EO.CLMS.DAT.GLO.DMP300_V1
CLMS_GLO_FAPAR_333M:
productType: EO.CLMS.DAT.GLO.FAPAR300_V1
CLMS_GLO_FCOVER_333M:
productType: EO.CLMS.DAT.GLO.FCOVER300_V1
CLMS_GLO_GDMP_333M:
productType: EO.CLMS.DAT.GLO.GDMP300_V1
CLMS_GLO_LAI_333M:
productType: EO.CLMS.DAT.GLO.LAI300_V1
CLMS_GLO_NDVI_1KM_LTS:
productType: EO.CLMS.DAT.GLO.NDVI_1KM_V2
CLMS_GLO_NDVI_333M:
productType: EO.CLMS.DAT.GLO.NDVI300_V1
EEA_DAILY_VI:
productType: EO.CLMS.DAT.SENTINEL-2.HRVPP.VI
# Landsat data
LANDSAT_C2L1:
productType: EO.NASA.DAT.LANDSAT.C2_L1
LANDSAT_C2L2:
productType: EO.NASA.DAT.LANDSAT.C2_L2
# DT Output
DT_EXTREMES:
productType: EO.ECMWF.DAT.DT_EXTREMES
metadata_mapping:
storageStatus: '{$.null#replace_str("Not Available","OFFLINE")}'
<<: *orderable_mm
DT_CLIMATE_ADAPTATION:
productType: EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION
metadata_mapping:
storageStatus: '{$.null#replace_str("Not Available","OFFLINE")}'
<<: *orderable_mm
# AERIS
AERIS_IAGOS:
productType: EO.AERIS.DAT.IAGOS
# METOP
METOP_ASCSZFR02:
productType: EO.EUM.CM.METOP.ASCSZFR02
METOP_ASCSZOR02:
productType: EO.EUM.CM.METOP.ASCSZOR02
METOP_ASCSZRR02:
productType: EO.EUM.CM.METOP.ASCSZRR02
METOP_AMSU_L1:
productType: EO.EUM.DAT.METOP.AMSUL1
METOP_ASCSZF1B:
productType: EO.EUM.DAT.METOP.ASCSZF1B
METOP_ASCSZR1B:
productType: EO.EUM.DAT.METOP.ASCSZR1B
METOP_ASCSZO1B:
productType: EO.EUM.DAT.METOP.ASCSZO1B
METOP_AVHRRGACR02:
productType: EO.EUM.DAT.METOP.AVHRRGACR02
METOP_AVHRRL1:
productType: EO.EUM.DAT.METOP.AVHRRL1
METOP_GLB_SST_NC:
productType: EO.EUM.DAT.METOP.GLB-SST-NC
METOP_GOMEL1:
productType: EO.EUM.DAT.METOP.GOMEL1
METOP_GOMEL1R03:
productType: EO.EUM.DAT.METOP.GOMEL1R03
METOP_HIRSL1:
productType: EO.EUM.DAT.MULT.HIRSL1
METOP_IASIL1C_ALL:
productType: EO.EUM.DAT.METOP.IASIL1C-ALL
METOP_IASSND02:
productType: EO.EUM.DAT.METOP.IASSND02
METOP_IASTHR011:
productType: EO.EUM.DAT.METOP.IASTHR011
METOP_LSA_002:
productType: EO.EUM.DAT.METOP.LSA-002
METOP_MHSL1:
productType: EO.EUM.DAT.METOP.MHSL1
METOP_OSI_104:
productType: EO.EUM.DAT.METOP.OSI-104
METOP_OSI_150A:
productType: EO.EUM.DAT.METOP.OSI-150-A
METOP_OSI_150B:
productType: EO.EUM.DAT.METOP.OSI-150-B
METOP_SOMO12:
productType: EO.EUM.DAT.METOP.SOMO12
METOP_SOMO25:
productType: EO.EUM.DAT.METOP.SOMO25
# MSG
MSG_AMVR02:
productType: EO.EUM.DAT.AMVR02
MSG_GSAL2R02:
productType: EO.EUM.DAT.GSAL2R02
MSG_CLM:
productType: EO.EUM.DAT.MSG.CLM
MSG_CLM_IODC:
productType: EO.EUM.DAT.MSG.CLM-IODC
MSG_HRSEVIRI:
productType: EO.EUM.DAT.MSG.HRSEVIRI
MSG_HRSEVIRI_IODC:
productType: EO.EUM.DAT.MSG.HRSEVIRI-IODC
MSG_LSA_FRM:
productType: EO.EUM.DAT.MSG.LSA-FRM
MSG_LSA_LST_CDR:
productType: EO.EUM.DAT.MSG.LSA-LST-CDR
MSG_LSA_LSTDE:
productType: EO.EUM.DAT.MSG.LSA-LSTDE
MSG_RSS_CLM:
productType: EO.EUM.DAT.MSG.RSS-CLM
MSG_MSG15_RSS:
productType: EO.EUM.DAT.MSG.MSG15-RSS
# GSW
GSW_CHANGE:
productType: EO.GSW.DAT.CHANGE
GSW_EXTENT:
productType: EO.GSW.DAT.EXTENT
GSW_OCCURRENCE:
productType: EO.GSW.DAT.OCCURRENCE
GSW_RECURRENCE:
productType: EO.GSW.DAT.RECURRENCE
GSW_SEASONALITY:
productType: EO.GSW.DAT.SEASONALITY
GSW_TRANSITIONS:
productType: EO.GSW.DAT.TRANSITIONS
# Eurostat
EUSTAT_GREENHOUSE_GAS_EMISSION_AGRICULTURE:
productType: STAT.EUSTAT.DAT.GREENHOUSE_GAS_EMISSION_AGRICULTURE
EUSTAT_POP_AGE_GROUP_SEX_NUTS3:
productType: STAT.EUSTAT.DAT.POP_AGE_GROUP_SEX_NUTS3
EUSTAT_POP_AGE_SEX_NUTS2:
productType: STAT.EUSTAT.DAT.POP_AGE_SEX_NUTS2
EUSTAT_POP_CHANGE_DEMO_BALANCE_CRUDE_RATES_NUTS3:
productType: STAT.EUSTAT.DAT.POP_CHANGE_DEMO_BALANCE_CRUDE_RATES_NUTS3
EUSTAT_SHARE_ENERGY_FROM_RENEWABLE:
productType: STAT.EUSTAT.DAT.SHARE_ENERGY_FROM_RENEWABLE
EUSTAT_AVAILABLE_BEDS_HOSPITALS_NUTS2:
productType: STAT.EUSTAT.DAT.AVAILABLE_BEDS_HOSPITALS_NUTS2
EUSTAT_BATHING_SITES_WATER_QUALITY:
productType: STAT.EUSTAT.DAT.BATHING_SITES_WATER_QUALITY
EUSTAT_SOIL_SEALING_INDEX:
productType: STAT.EUSTAT.DAT.SOIL_SEALING_INDEX
EUSTAT_SURFACE_TERRESTRIAL_PROTECTED_AREAS:
productType: STAT.EUSTAT.DAT.SURFACE_TERRESTRIAL_PROTECTED_AREAS
EUSTAT_POP_DENSITY_NUTS3:
productType: STAT.EUSTAT.DAT.POP_DENSITY_NUTS3
# ISIMIP
ISIMIP_CLIMATE_FORCING_ISIMIP3B:
productType: EO.ISIMIP.DAT.CLIMATE-FORCING_ISIMIP3b
ISIMIP_SOCIO_ECONOMIC_FORCING_ISIMIP3B:
productType: EO.ISIMIP.DAT.SOCIO-ECONOMIC-FORCING_ISIMIP3b
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
---
!provider # MARK: eumetsat_ds
name: eumetsat_ds
priority: 0
description: EUMETSAT Data Store
roles:
- host
url: https://data.eumetsat.int
anchor_sentinel: &sentinel_params
orbitDirection: '{$.properties.acquisitionInformation[0].acquisitionParameters.orbitDirection#to_lower}'
relativeOrbitNumber: '$.properties.acquisitionInformation[0].acquisitionParameters.relativeOrbitNumber'
timeliness: '$.properties.productInformation.timeliness'
cycleNumber: '$.properties.acquisitionInformation[0].acquisitionParameters.cycleNumber'
anchor_orbit_zone_tile: &orbit_zone_tile
orbitNumber:
- orbit
- '$.properties.acquisitionInformation[0].acquisitionParameters.orbitNumber'
search: !plugin
type: QueryStringSearch
api_endpoint: 'https://api.eumetsat.int/data/search-products/1.0.0/os'
need_auth: false
ssl_verify: true
dont_quote:
- '='
- '&'
asset_key_from_href: false
pagination:
next_page_url_tpl: '{url}?{search}&c={items_per_page}&pw={page}'
start_page: 0
total_items_nb_key_path: '$.totalResults'
# 2024/02/01: 500 is the max, no error if greater
max_items_per_page: 500
sort:
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_by_tpl: '&sort={sort_param},{sort_order}'
sort_param_mapping:
startTimeFromAscendingNode: start,time
publicationDate: publicationDate,
sort_order_mapping:
ascending: '1'
descending: '0'
max_sort_params: 1
literal_search_params:
format: json
discover_metadata:
auto_discovery: true
metadata_pattern: '^[a-zA-Z0-9_]+$'
search_param: '{metadata}={{{metadata}}}'
metadata_path: '$.properties.*'
discover_product_types:
fetch_url: https://api.eumetsat.int/data/browse/1.0.0/collections?format=json
result_type: json
results_entry: '$.links[*]'
generic_product_type_id: '$.title'
generic_product_type_parsable_properties:
parentIdentifier: '$.title'
generic_product_type_parsable_metadata:
abstract: '$.null'
instrument: '$.null'
platform: '$.null'
platformSerialIdentifier: '$.null'
processingLevel: '$.null'
keywords: '$.null'
license: '$.null'
title: '$.title'
missionStartDate: '$.null'
metadata_mapping:
# Opensearch resource identifier within the search engine context (in our case
# within the context of the data provider)
uid: '$.id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- type
- '$.properties.productInformation.productType'
platform: '$.properties.acquisitionInformation[0].platform.platformShortName'
instrument: '$.properties.acquisitionInformation[0].instrument.instrumentShortName'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '{$.properties.title#sanitize}'
# OpenSearch Parameters for Product Search (Table 5)
parentIdentifier:
- pi
- '$.properties.parentIdentifier'
modificationDate: '$.properties.updated'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- dtstart
- '{$.properties.date#replace_str(r"\/.*","")}'
completionTimeFromAscendingNode:
- dtend
- '{$.properties.date#replace_str(r".*\/","")}'
# Custom parameters (not defined in the base document referenced above)
id:
- id
- $.properties.identifier
# The geographic extent of the product
geometry:
- 'geo={geometry#to_rounded_wkt}'
- '$.geometry'
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
# The url of the quicklook
quicklook: '$.properties.links.previews[?(@.title="Quicklook")].href'
# The url to download the product "as is" (literal or as a template to be completed either after the search result
# is obtained from the provider or during the eodag download phase)
downloadLink: '$.properties.links.data[?(@.title="Product download")].href'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
assets: '{$.properties.links.sip-entries#assets_list_to_dict}'
# Additional metadata provided by the providers but that don't appear in the reference spec
size: '$.properties.productInformation.size'
type: '$.null'
# set duplicate metadata due to metadata discovery to null
acquisitionInformation: '$.null'
productInformation: '$.null'
extraInformation: '$.null'
products:
# S3 SRAL
S3_SRA:
productType: SR_1_SRA___
parentIdentifier: EO:EUM:DAT:0406
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_A:
productType: SR_1_SRA_A_
parentIdentifier: EO:EUM:DAT:0413
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_1A_BC004:
productType: SR_1_SRA_A_
parentIdentifier: EO:EUM:DAT:0583
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_1A_BC005:
productType: SR_1_SRA_A_
parentIdentifier: EO:EUM:DAT:0836
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_1B_BC004:
productType: SR_1_SRA___
parentIdentifier: EO:EUM:DAT:0584
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_1B_BC005:
productType: SR_1_SRA___
parentIdentifier: EO:EUM:DAT:0833
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_BS:
productType: SR_1_SRA_BS
parentIdentifier: EO:EUM:DAT:0414
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_BS_BC004:
productType: SR_1_SRA_BS
parentIdentifier: EO:EUM:DAT:0585
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SRA_BS_BC005:
productType: SR_1_SRA_BS
parentIdentifier: EO:EUM:DAT:0835
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_WAT:
productType: SR_2_WAT___
parentIdentifier: EO:EUM:DAT:0415
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_WAT_BC004:
productType: SR_2_WAT___
parentIdentifier: EO:EUM:DAT:0586
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_WAT_BC005:
productType: SR_2_WAT___
parentIdentifier: EO:EUM:DAT:0834
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
# S3 OLCI
S3_EFR:
productType: OL_1_EFR___
parentIdentifier: EO:EUM:DAT:0409
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_EFR_BC002:
productType: OL_1_EFR___
parentIdentifier: EO:EUM:DAT:0577
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_ERR:
productType: OL_1_ERR___
parentIdentifier: EO:EUM:DAT:0410
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_ERR_BC002:
productType: OL_1_ERR___
parentIdentifier: EO:EUM:DAT:0578
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_OLCI_L2WRR:
productType: OL_2_WRR___
parentIdentifier: EO:EUM:DAT:0408
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_OLCI_L2WRR_BC003:
productType: OL_2_WRR___
parentIdentifier: EO:EUM:DAT:0557
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_OLCI_L2WFR:
productType: OL_2_WFR___
parentIdentifier: EO:EUM:DAT:0407
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_OLCI_L2WFR_BC003:
productType: OL_2_WFR___
parentIdentifier: EO:EUM:DAT:0556
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
# S3 SLSTR
S3_SLSTR_L1RBT:
productType: SL_1_RBT___
parentIdentifier: EO:EUM:DAT:0411
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L1RBT_BC003:
productType: SL_1_RBT___
parentIdentifier: EO:EUM:DAT:0581
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L1RBT_BC004:
productType: SL_1_RBT___
parentIdentifier: EO:EUM:DAT:0615
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L2WST:
productType: SL_2_WST___
parentIdentifier: EO:EUM:DAT:0412
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L2WST_BC003:
productType: SL_2_WST___
parentIdentifier: EO:EUM:DAT:0582
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L2AOD:
productType: SL_2_AOD___
parentIdentifier: EO:EUM:DAT:0416
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
S3_SLSTR_L2FRP:
productType: SL_2_FRP___
parentIdentifier: EO:EUM:DAT:0417
metadata_mapping:
<<: *orbit_zone_tile
<<: *sentinel_params
fire:
- fire
- '$.properties.extraInformation.fireDetected'
# METOP
METOP_AMSU_L1:
parentIdentifier: EO:EUM:DAT:METOP:AMSUL1
metadata_mapping:
<<: *orbit_zone_tile
METOP_OSI_104:
parentIdentifier: EO:EUM:DAT:METOP:OSI-104
metadata_mapping:
<<: *orbit_zone_tile
METOP_OSI_150A:
parentIdentifier: EO:EUM:DAT:METOP:OSI-150-A
metadata_mapping:
<<: *orbit_zone_tile
METOP_OSI_150B:
parentIdentifier: EO:EUM:DAT:METOP:OSI-150-B
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZF1B:
parentIdentifier: EO:EUM:DAT:METOP:ASCSZF1B
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZR1B:
parentIdentifier: EO:EUM:DAT:METOP:ASCSZR1B
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZO1B:
parentIdentifier: EO:EUM:DAT:METOP:ASCSZO1B
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZFR02:
parentIdentifier: EO:EUM:CM:METOP:ASCSZFR02
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZOR02:
parentIdentifier: EO:EUM:CM:METOP:ASCSZOR02
metadata_mapping:
<<: *orbit_zone_tile
METOP_ASCSZRR02:
parentIdentifier: EO:EUM:CM:METOP:ASCSZRR02
metadata_mapping:
<<: *orbit_zone_tile
METOP_AVHRRL1:
parentIdentifier: EO:EUM:DAT:METOP:AVHRRL1
metadata_mapping:
<<: *orbit_zone_tile
METOP_SOMO12:
parentIdentifier: EO:EUM:DAT:METOP:SOMO12
metadata_mapping:
<<: *orbit_zone_tile
METOP_SOMO25:
parentIdentifier: EO:EUM:DAT:METOP:SOMO25
metadata_mapping:
<<: *orbit_zone_tile
METOP_AVHRRGACR02:
parentIdentifier: EO:EUM:DAT:0558
METOP_LSA_002:
parentIdentifier: EO:EUM:DAT:METOP:LSA-002
METOP_GLB_SST_NC:
parentIdentifier: EO:EUM:DAT:METOP:GLB-SST-NC
METOP_GOMEL1:
parentIdentifier: EO:EUM:DAT:METOP:GOMEL1
metadata_mapping:
<<: *orbit_zone_tile
METOP_GOMEL1R03:
parentIdentifier: EO:EUM:DAT:0533
metadata_mapping:
<<: *orbit_zone_tile
METOP_IASTHR011:
parentIdentifier: EO:EUM:DAT:0576
metadata_mapping:
<<: *orbit_zone_tile
METOP_IASSND02:
parentIdentifier: EO:EUM:DAT:METOP:IASSND02
metadata_mapping:
<<: *orbit_zone_tile
METOP_IASIL1C_ALL:
parentIdentifier: EO:EUM:DAT:METOP:IASIL1C-ALL
metadata_mapping:
<<: *orbit_zone_tile
METOP_MHSL1:
parentIdentifier: EO:EUM:DAT:METOP:MHSL1
metadata_mapping:
<<: *orbit_zone_tile
METOP_HIRSL1:
parentIdentifier: EO:EUM:DAT:MULT:HIRSL1
metadata_mapping:
<<: *orbit_zone_tile
# MSG
MSG_CLM:
parentIdentifier: EO:EUM:DAT:MSG:CLM
MSG_CLM_IODC:
parentIdentifier: EO:EUM:DAT:MSG:CLM-IODC
MSG_GSAL2R02:
parentIdentifier: EO:EUM:DAT:0300
MSG_HRSEVIRI:
parentIdentifier: EO:EUM:DAT:MSG:HRSEVIRI
MSG_HRSEVIRI_IODC:
parentIdentifier: EO:EUM:DAT:MSG:HRSEVIRI-IODC
MSG_RSS_CLM:
parentIdentifier: EO:EUM:DAT:MSG:RSS-CLM
MSG_MSG15_RSS:
parentIdentifier: EO:EUM:DAT:MSG:MSG15-RSS
MSG_LSA_FRM:
parentIdentifier: EO:EUM:DAT:0398
MSG_LSA_LST_CDR:
parentIdentifier: EO:EUM:DAT:0088
MSG_LSA_LSTDE:
parentIdentifier: EO:EUM:DAT:0394
MSG_AMVR02:
parentIdentifier: EO:EUM:DAT:0405
MFG_GSA_57:
parentIdentifier: EO:EUM:DAT:0301
MFG_GSA_63:
parentIdentifier: EO:EUM:DAT:0302
MSG_MFG_GSA_0:
parentIdentifier: EO:EUM:DAT:0300
MSG_CTH:
parentIdentifier: EO:EUM:DAT:MSG:CTH
MSG_CTH_IODC:
parentIdentifier: EO:EUM:DAT:MSG:CTH-IODC
HIRS_FDR_1_MULTI:
parentIdentifier: EO:EUM:DAT:0647
MSG_OCA_CDR:
parentIdentifier: EO:EUM:DAT:0617
S6_RADIO_OCCULTATION:
parentIdentifier: EO:EUM:DAT:0853
MTG_LI_AF:
parentIdentifier: EO:EUM:DAT:0686
MTG_LI_LFL:
parentIdentifier: EO:EUM:DAT:0691
MTG_LI_LGR:
parentIdentifier: EO:EUM:DAT:0782
MTG_LI_AFA:
parentIdentifier: EO:EUM:DAT:0687
MTG_LI_AFR:
parentIdentifier: EO:EUM:DAT:0688
MTG_LI_LEF:
parentIdentifier: EO:EUM:DAT:0690
MTG_FCI_FDHSI:
parentIdentifier: EO:EUM:DAT:0662
MTG_FCI_HRFI:
parentIdentifier: EO:EUM:DAT:0665
MTG_FCI_ASR_BUFR:
parentIdentifier: EO:EUM:DAT:0799
MTG_FCI_ASR_NETCDF:
parentIdentifier: EO:EUM:DAT:0677
MTG_FCI_AMV_BUFR:
parentIdentifier: EO:EUM:DAT:0998
MTG_FCI_AMV_NETCDF:
parentIdentifier: EO:EUM:DAT:0676
MTG_FCI_CLM:
parentIdentifier: EO:EUM:DAT:0678
MTG_FCI_GII:
parentIdentifier: EO:EUM:DAT:0683
MTG_FCI_OCA:
parentIdentifier: EO:EUM:DAT:0684
MTG_FCI_OLR:
parentIdentifier: EO:EUM:DAT:0685
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
parentIdentifier: '{parentIdentifier}'
download: !plugin
type: HTTPDownload
extract: true
ignore_assets: True
ssl_verify: true
auth: !plugin
type: TokenAuth
matching_url: https://api.eumetsat.int
auth_uri: 'https://api.eumetsat.int/token'
auth_error_code: 401
req_data:
grant_type: client_credentials
token_type: json
token_key: access_token
token_expiration_key: expires_in
ssl_verify: true
---
!provider # MARK: cop_marine
name: cop_marine
priority: 0
description: Copernicus Marine Data Store
roles:
- host
url: https://marine.copernicus.eu/
search: !plugin
type: CopMarineSearch
api_endpoint: 'https://stac.marine.copernicus.eu/metadata/{collection}/product.stac.json'
need_auth: false
ssl_verify: true
asset_key_from_href: false
timeout: 20
results_entry: links
discover_product_types:
fetch_url: 'https://stac.marine.copernicus.eu/metadata/catalog.stac.json'
result_type: json
results_entry: '$.collections[*]'
generic_product_type_id: '$.id'
generic_product_type_parsable_properties:
collection: '$.id'
generic_product_type_parsable_metadata:
instrument: '$.null'
platform: '$.null'
platformSerialIdentifier: '$.null'
processingLevel: '$.properties.processingLevel'
keywords: '$.keywords'
license: '$.license'
title: '$.title'
missionStartDate: '$.extent.temporal.interval[0][0]'
abstract: '$.description'
providers: '$.providers'
doi: '$.sci:doi'
discover_metadata:
auto_discovery: false
discover_queryables:
fetch_url: null
metadata_mapping:
defaultGeometry: 'POLYGON((180 -90, 180 90, -180 90, -180 -90, 180 -90))'
products:
MO_GLOBAL_ANALYSISFORECAST_PHY_001_024:
productType: GLOBAL_ANALYSISFORECAST_PHY_001_024
MO_GLOBAL_ANALYSISFORECAST_BGC_001_028:
productType: GLOBAL_ANALYSISFORECAST_BGC_001_028
MO_GLOBAL_ANALYSISFORECAST_WAV_001_027:
productType: GLOBAL_ANALYSISFORECAST_WAV_001_027
MO_GLOBAL_MULTIYEAR_BGC_001_033:
productType: GLOBAL_MULTIYEAR_BGC_001_033
MO_GLOBAL_MULTIYEAR_WAV_001_032:
productType: GLOBAL_MULTIYEAR_WAV_001_032
MO_GLOBAL_MULTIYEAR_PHY_ENS_001_031:
productType: GLOBAL_MULTIYEAR_PHY_ENS_001_031
MO_INSITU_GLO_PHY_UV_DISCRETE_NRT_013_048:
productType: INSITU_GLO_PHY_UV_DISCRETE_NRT_013_048
code_mapping:
param: platformSerialIdentifier
index: 3
MO_INSITU_GLO_PHY_TS_OA_NRT_013_002:
productType: INSITU_GLO_PHY_TS_OA_NRT_013_002
code_mapping:
param: analysis_name
index: 1
MO_INSITU_GLO_PHY_TS_OA_MY_013_052:
productType: INSITU_GLO_PHY_TS_OA_MY_013_052
code_mapping:
param: analysis_name
index: 1
MO_MULTIOBS_GLO_BIO_BGC_3D_REP_015_010:
productType: MULTIOBS_GLO_BIO_BGC_3D_REP_015_010
MO_MULTIOBS_GLO_BIO_CARBON_SURFACE_MYNRT_015_008:
productType: MULTIOBS_GLO_BIO_CARBON_SURFACE_MYNRT_015_008
MO_MULTIOBS_GLO_BGC_NUTRIENTS_CARBON_PROFILES_MYNRT_015_009:
productType: MULTIOBS_GLO_BGC_NUTRIENTS_CARBON_PROFILES_MYNRT_015_009
MO_MULTIOBS_GLO_PHY_MYNRT_015_003:
productType: MULTIOBS_GLO_PHY_MYNRT_015_003
MO_MULTIOBS_GLO_PHY_S_SURFACE_MYNRT_015_013:
productType: MULTIOBS_GLO_PHY_S_SURFACE_MYNRT_015_013
MO_MULTIOBS_GLO_PHY_TSUV_3D_MYNRT_015_012:
productType: MULTIOBS_GLO_PHY_TSUV_3D_MYNRT_015_012
MO_MULTIOBS_GLO_PHY_W_3D_REP_015_007:
productType: MULTIOBS_GLO_PHY_W_3D_REP_015_007
MO_SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_001:
productType: SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_001
MO_SEAICE_GLO_SEAICE_L4_REP_OBSERVATIONS_011_009:
productType: SEAICE_GLO_SEAICE_L4_REP_OBSERVATIONS_011_009
MO_SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_006:
productType: SEAICE_GLO_SEAICE_L4_NRT_OBSERVATIONS_011_006
MO_SEALEVEL_GLO_PHY_L4_NRT_008_046:
productType: SEALEVEL_GLO_PHY_L4_NRT_008_046
MO_SEALEVEL_GLO_PHY_MDT_008_063:
productType: SEALEVEL_GLO_PHY_MDT_008_063
MO_SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010:
productType: SST_GLO_SST_L3S_NRT_OBSERVATIONS_010_010
code_mapping:
param: sensorType
index: 3
pattern: '[A-Z]{3}'
MO_SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001:
productType: SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001
MO_SST_GLO_SST_L4_REP_OBSERVATIONS_010_011:
productType: SST_GLO_SST_L4_REP_OBSERVATIONS_010_011
MO_SST_GLO_SST_L4_REP_OBSERVATIONS_010_024:
productType: SST_GLO_SST_L4_REP_OBSERVATIONS_010_024
MO_WAVE_GLO_PHY_SPC_FWK_L3_NRT_014_002:
productType: WAVE_GLO_PHY_SPC-FWK_L3_NRT_014_002
code_mapping:
param: platformSerialIdentifier
index: 0
pattern: 's1a|s1b|cfo'
MO_WAVE_GLO_PHY_SWH_L3_NRT_014_001:
productType: WAVE_GLO_PHY_SWH_L3_NRT_014_001
code_mapping:
param: platformSerialIdentifier
index: 4
MO_WAVE_GLO_PHY_SWH_L4_NRT_014_003:
productType: WAVE_GLO_PHY_SWH_L4_NRT_014_003
MO_WIND_GLO_PHY_CLIMATE_L4_MY_012_003:
productType: WIND_GLO_PHY_CLIMATE_L4_MY_012_003
MO_WIND_GLO_PHY_L3_NRT_012_002:
productType: WIND_GLO_PHY_L3_NRT_012_002
MO_WIND_GLO_PHY_L3_MY_012_005:
productType: WIND_GLO_PHY_L3_MY_012_005
MO_WIND_GLO_PHY_L4_NRT_012_004:
productType: WIND_GLO_PHY_L4_NRT_012_004
MO_WIND_GLO_PHY_L4_MY_012_006:
productType: WIND_GLO_PHY_L4_MY_012_006
MO_OCEANCOLOUR_GLO_BGC_L3_MY_009_107:
productType: OCEANCOLOUR_GLO_BGC_L3_MY_009_107
MO_OCEANCOLOUR_GLO_BGC_L3_NRT_009_101:
productType: OCEANCOLOUR_GLO_BGC_L3_NRT_009_101
code_mapping:
param: sensorType
index: 6
pattern: 'olci|gapfree-multi|multi-climatology|multi'
MO_OCEANCOLOUR_GLO_BGC_L3_MY_009_103:
productType: OCEANCOLOUR_GLO_BGC_L3_MY_009_103
code_mapping:
param: sensorType
index: 6
pattern: 'olci|gapfree-multi|multi-climatology|multi'
MO_OCEANCOLOUR_GLO_BGC_L4_NRT_009_102:
productType: OCEANCOLOUR_GLO_BGC_L4_NRT_009_102
code_mapping:
param: sensorType
index: 6
pattern: 'olci|gapfree-multi|multi-climatology|multi'
MO_OCEANCOLOUR_GLO_BGC_L4_MY_009_104:
productType: OCEANCOLOUR_GLO_BGC_L4_MY_009_104
code_mapping:
param: sensorType
index: 6
pattern: 'olci|gapfree-multi|multi-climatology|multi'
MO_OCEANCOLOUR_GLO_BGC_L4_MY_009_108:
productType: OCEANCOLOUR_GLO_BGC_L4_MY_009_108
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
collection: '{collection}'
download: !plugin
type: AwsDownload
s3_endpoint: https://s3.waw3-1.cloudferro.com
bucket_path_level: 0
auth: !plugin
type: AwsAuth
matching_url: s3://
s3_endpoint: https://s3.waw3-1.cloudferro.com
matching_conf:
s3_endpoint: https://s3.waw3-1.cloudferro.com
---
!provider # MARK: geodes
name: geodes
priority: 0
roles:
- host
description: French National Space Agency (CNES) Earth Observation portal
url: https://geodes.cnes.fr
search: !plugin
type: StacSearch
api_endpoint: https://geodes-portal.cnes.fr/api/stac/search
need_auth: false
asset_key_from_href: false
discover_queryables:
fetch_url: null
product_type_fetch_url: null
discover_product_types:
fetch_url: https://geodes-portal.cnes.fr/api/stac/collections
fetch_method: POST
fetch_body:
limit: 10000
pagination:
total_items_nb_key_path: '$.context.matched'
# 1000 is ok and 2000 fails
max_items_per_page: 1000
sort:
sort_by_tpl: '{{"sortBy": [ {{"field": "{sort_param}", "direction": "{sort_order}" }} ] }}'
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: identifier
startTimeFromAscendingNode: start_datetime
completionTimeFromAscendingNode: end_datetime
platformSerialIdentifier: platform
cloudCover: eo:cloud_cover
sort_order_mapping:
ascending: asc
descending: dsc
metadata_mapping:
uid: '$.id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- '{{"query":{{"dataset":{{"eq":"{productType}"}}}}}}'
- '$.properties.dataset'
instrument:
- '{{"query":{{"instrument":{{"eq":"{instrument}"}}}}}}'
- '$.properties.instrument'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '$.properties.identifier'
keyword: '$.properties.keywords'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- '{{"query":{{"sat:absolute_orbit":{{"eq":{orbitNumber}}}}}}}'
- '$.properties."sat:absolute_orbit"'
relativeOrbitNumber:
- '{{"query":{{"sat:relative_orbit":{{"eq":{relativeOrbitNumber}}}}}}}'
- '$.properties."sat:relative_orbit"'
cloudCover:
- '{{"query":{{"eo:cloud_cover":{{"lte":{cloudCover}}}}}}}'
- '$.properties."eo:cloud_cover"'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"query":{{"end_datetime":{{"gte":"{startTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.start_datetime'
completionTimeFromAscendingNode:
- '{{"query":{{"start_datetime":{{"lte":"{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.end_datetime'
swathIdentifier:
- '{{"query":{{"swath":{{"eq":"{swathIdentifier}"}}}}}}'
- '$.properties.swath'
# Custom parameters (not defined in the base document referenced above)
id:
- '{{"query":{{"identifier":{{"eq":"{id}"}}}}}}'
- '$.properties.identifier'
tileIdentifier:
- '{{"query":{{"grid:code":{{"contains":"{tileIdentifier}"}}}}}}'
- '{$.properties."grid:code"#replace_str(r"^T?(.*)$",r"\1")}'
providerProductType:
- '{{"query":{{"product:type":{{"eq":"{providerProductType}"}}}}}}'
- '$.null'
downloadLink: '$.assets[?(@.roles[0] == "data") & (@.type != "application/xml")].href'
quicklook: '$.assets[?(@.roles[0] == "overview")].href.`sub(/^(.*)$/, \\1?scope=gdh)`'
thumbnail: '$.assets[?(@.roles[0] == "overview")].href.`sub(/^(.*)$/, \\1?scope=gdh)`'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
products:
S1_SAR_OCN:
providerProductType: OCN
productType: PEPS_S1_L2
metadata_mapping:
cloudCover: '$.null'
S1_SAR_GRD:
providerProductType: GRD
productType: PEPS_S1_L1
metadata_mapping:
cloudCover: '$.null'
S1_SAR_SLC:
providerProductType: SLC
productType: PEPS_S1_L1
metadata_mapping:
cloudCover: '$.null'
S2_MSI_L1C:
productType: PEPS_S2_L1C
S2_MSI_L2A_MAJA:
productType: MUSCATE_SENTINEL2_SENTINEL2_L2A
S2_MSI_L2B_MAJA_SNOW:
productType: MUSCATE_Snow_SENTINEL2_L2B-SNOW
S2_MSI_L2B_MAJA_WATER:
productType: MUSCATE_WaterQual_SENTINEL2_L2B-WATER
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: HTTPDownload
ignore_assets: true
archive_depth: 2
auth_error_code:
- 401
- 403
auth: !plugin
type: HTTPHeaderAuth
matching_url: https://geodes-portal.cnes.fr
headers:
X-API-Key: "{apikey}"
---
!provider # MARK: geodes_s3
name: geodes_s3
priority: 0
roles:
- host
description: French National Space Agency (CNES) Earth Observation portal with internal s3 Datalake
url: https://geodes.cnes.fr
search: !plugin
type: StacListAssets
api_endpoint: https://geodes-portal.cnes.fr/api/stac/search
s3_endpoint: https://s3.datalake.cnes.fr
need_auth: true
asset_key_from_href: false
discover_queryables:
fetch_url: null
product_type_fetch_url: null
discover_product_types:
fetch_url: https://geodes-portal.cnes.fr/api/stac/collections
fetch_method: POST
fetch_body:
limit: 10000
pagination:
total_items_nb_key_path: '$.context.matched'
# 1000 is ok and 2000 fails
max_items_per_page: 1000
sort:
sort_by_tpl: '{{"sortBy": [ {{"field": "{sort_param}", "direction": "{sort_order}" }} ] }}'
sort_by_default:
- !!python/tuple [startTimeFromAscendingNode, ASC]
sort_param_mapping:
id: identifier
startTimeFromAscendingNode: start_datetime
completionTimeFromAscendingNode: end_datetime
platformSerialIdentifier: platform
cloudCover: eo:cloud_cover
sort_order_mapping:
ascending: asc
descending: dsc
metadata_mapping:
uid: '$.id'
# OpenSearch Parameters for Collection Search (Table 3)
productType:
- '{{"query":{{"dataset":{{"eq":"{productType}"}}}}}}'
- '$.properties.dataset'
instrument:
- '{{"query":{{"instrument":{{"eq":"{instrument}"}}}}}}'
- '$.properties.instrument'
# INSPIRE obligated OpenSearch Parameters for Collection Search (Table 4)
title: '$.properties.identifier'
keyword: '$.properties.keywords'
# OpenSearch Parameters for Product Search (Table 5)
orbitNumber:
- '{{"query":{{"sat:absolute_orbit":{{"eq":{orbitNumber}}}}}}}'
- '$.properties."sat:absolute_orbit"'
relativeOrbitNumber:
- '{{"query":{{"sat:relative_orbit":{{"eq":{relativeOrbitNumber}}}}}}}'
- '$.properties."sat:relative_orbit"'
cloudCover:
- '{{"query":{{"eo:cloud_cover":{{"lte":{cloudCover}}}}}}}'
- '$.properties."eo:cloud_cover"'
# OpenSearch Parameters for Acquistion Parameters Search (Table 6)
startTimeFromAscendingNode:
- '{{"query":{{"end_datetime":{{"gte":"{startTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.start_datetime'
completionTimeFromAscendingNode:
- '{{"query":{{"start_datetime":{{"lte":"{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
- '$.properties.end_datetime'
swathIdentifier:
- '{{"query":{{"swath":{{"eq":"{swathIdentifier}"}}}}}}'
- '$.properties.swath'
# Custom parameters (not defined in the base document referenced above)
id:
- '{{"query":{{"identifier":{{"eq":"{id}"}}}}}}'
- '$.properties.identifier'
tileIdentifier:
- '{{"query":{{"grid:code":{{"contains":"{tileIdentifier}"}}}}}}'
- '{$.properties."grid:code"#replace_str(r"^T?(.*)$",r"\1")}'
providerProductType:
- '{{"query":{{"product:type":{{"eq":"{providerProductType}"}}}}}}'
- '$.null'
downloadLink: '$.properties.endpoint_url'
quicklook: '$.assets[?(@.roles[0] == "overview")].href.`sub(/^(.*)$/, \\1?scope=gdh)`'
thumbnail: '$.assets[?(@.roles[0] == "overview")].href.`sub(/^(.*)$/, \\1?scope=gdh)`'
# storageStatus set to ONLINE for consistency between providers
storageStatus: '{$.null#replace_str("Not Available","ONLINE")}'
products:
S1_SAR_OCN:
providerProductType: OCN
productType: PEPS_S1_L2
metadata_mapping:
cloudCover: '$.null'
S1_SAR_GRD:
providerProductType: GRD
productType: PEPS_S1_L1
metadata_mapping:
cloudCover: '$.null'
S1_SAR_SLC:
providerProductType: SLC
productType: PEPS_S1_L1
metadata_mapping:
cloudCover: '$.null'
S2_MSI_L1C:
productType: PEPS_S2_L1C
S2_MSI_L2A_MAJA:
productType: MUSCATE_SENTINEL2_SENTINEL2_L2A
S2_MSI_L2B_MAJA_SNOW:
productType: MUSCATE_Snow_SENTINEL2_L2B-SNOW
S2_MSI_L2B_MAJA_WATER:
productType: MUSCATE_WaterQual_SENTINEL2_L2B-WATER
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: AwsDownload
s3_endpoint: https://s3.datalake.cnes.fr
ignore_assets: True
auth: !plugin
type: AwsAuth
auth_error_code: 403
s3_endpoint: https://s3.datalake.cnes.fr
matching_conf:
s3_endpoint: https://s3.datalake.cnes.fr
---
!provider # MARK: cop_ewds
name: cop_ewds
priority: 0
description: CEMS Early Warning Data Store
roles:
- host
url: https://ewds.climate.copernicus.eu
anchor_time_day_month_year: &time_day_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_day_month_year: &day_month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"],
"day": {_date#interval_to_datetime_dict}["day"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_month_year: &month_year
completionTimeFromAscendingNode:
- |
{{
"year": {_date#interval_to_datetime_dict}["year"],
"month": {_date#interval_to_datetime_dict}["month"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_time_hday_hmonth_hyear: &time_hday_hmonth_hyear
completionTimeFromAscendingNode:
- |
{{
"hyear": {_date#interval_to_datetime_dict}["year"],
"hmonth": {_date#interval_to_datetime_dict}["month"],
"hday": {_date#interval_to_datetime_dict}["day"],
"time": {startTimeFromAscendingNode#get_ecmwf_time}
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_hday_hmonth_hyear: &hday_hmonth_hyear
completionTimeFromAscendingNode:
- |
{{
"hyear": {_date#interval_to_datetime_dict}["year"],
"hmonth": {_date#interval_to_datetime_dict}["month"],
"hday": {_date#interval_to_datetime_dict}["day"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
anchor_hmonth_hyear: &hmonth_hyear
completionTimeFromAscendingNode:
- |
{{
"hyear": {_date#interval_to_datetime_dict}["year"],
"hmonth": {_date#interval_to_datetime_dict}["month"]
}}
- '{$.completionTimeFromAscendingNode#to_iso_date}'
auth: !plugin
type: HTTPHeaderAuth
ssl_verify: true
headers:
PRIVATE-TOKEN: "{apikey}"
download: !plugin
type: HTTPDownload
timeout: 30
ssl_verify: true
extract: false
auth_error_code: 401
order_enabled: true
order_method: POST
order_on_response:
metadata_mapping:
orderId: $.json.jobID
orderStatusLink: https://ewds.climate.copernicus.eu/api/retrieve/v1/jobs/{orderId}?request=true
searchLink: https://ewds.climate.copernicus.eu/api/retrieve/v1/jobs/{orderId}/results
order_status:
request:
method: GET
metadata_mapping:
status: $.json.status
order:date: $.json.created
creationDate: $.json.created
publicationDate: $.json.finished
modificationDate: $.json.updated
ecmwf:dataset: $.json.processID
request_params: $.json.metadata.request.ids
error:
status: failed
success:
status: successful
on_success:
need_search: true
metadata_mapping:
downloadLink: $.json.asset.value.href
search: !plugin
type: ECMWFSearch
need_auth: true
ssl_verify: true
end_date_excluded: false
dates_required: True
remove_from_query:
- dataset
- date
discover_queryables:
fetch_url: null
product_type_fetch_url: null
constraints_url: https://ewds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset}/constraints.json
form_url: https://ewds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset}/form.json
metadata_mapping:
productType: $.productType
startTimeFromAscendingNode: '{$.startTimeFromAscendingNode#to_iso_utc_datetime}'
completionTimeFromAscendingNode:
- date={startTimeFromAscendingNode#to_iso_date}/{completionTimeFromAscendingNode#to_iso_date}
- '{$.completionTimeFromAscendingNode#to_iso_utc_datetime}'
# The geographic extent of the product
geometry:
- '{{"area": {geometry#to_nwse_bounds}}}'
- $.geometry
qs: $.qs
orderLink: 'https://ewds.climate.copernicus.eu/api/retrieve/v1/processes/{dataset}/execution?{{"inputs": {qs#to_geojson}}}'
products:
EFAS_HISTORICAL:
dataset: efas-historical
metadata_mapping:
<<: *time_hday_hmonth_hyear
EFAS_FORECAST:
dataset: efas-forecast
metadata_mapping:
<<: *time_day_month_year
EFAS_SEASONAL:
dataset: efas-seasonal
metadata_mapping:
<<: *month_year
EFAS_REFORECAST:
dataset: efas-reforecast
metadata_mapping:
<<: *hday_hmonth_hyear
EFAS_SEASONAL_REFORECAST:
dataset: efas-seasonal-reforecast
metadata_mapping:
<<: *hmonth_hyear
GLOFAS_HISTORICAL:
dataset: cems-glofas-historical
metadata_mapping:
<<: *hday_hmonth_hyear
GLOFAS_FORECAST:
dataset: cems-glofas-forecast
metadata_mapping:
<<: *day_month_year
GLOFAS_SEASONAL:
dataset: cems-glofas-seasonal
metadata_mapping:
<<: *month_year
GLOFAS_SEASONAL_REFORECAST:
dataset: cems-glofas-seasonal-reforecast
metadata_mapping:
<<: *hmonth_hyear
GLOFAS_REFORECAST:
dataset: cems-glofas-reforecast
metadata_mapping:
<<: *hday_hmonth_hyear
FIRE_HISTORICAL:
dataset: cems-fire-historical-v1
metadata_mapping:
<<: *day_month_year
FIRE_SEASONAL:
dataset: cems-fire-seasonal
metadata_mapping:
<<: *day_month_year
GENERIC_PRODUCT_TYPE:
dataset: '{productType}'
---
!provider # MARK: fedeo_ceda
name: fedeo_ceda
priority: 0
description: CEDA datasets through FedEO Catalog
roles:
- host
url: https://fedeo.ceos.org/
search: !plugin
type: StacSearch
api_endpoint: 'https://fedeo.ceos.org/search'
ssl_verify: true
timeout: 20
discover_product_types:
fetch_url: 'https://fedeo.ceos.org/series/eo:organisationName/CEDA/'
result_type: json
results_entry: '$.links[?(@.rel=="child")]'
generic_product_type_id: '{$.href#replace_str(r".*/", "")}'
generic_product_type_parsable_properties:
productType: '{$.href#replace_str(r".*/", "")}'
title: '$.title'
single_collection_fetch_url: 'https://fedeo.ceos.org/series/eo:organisationName/CEDA/{productType}'
single_product_type_parsable_metadata:
ID: '{$.assets.enclosure.href#ceda_collection_name}'
abstract: '$.description'
instrument: '{$.summaries.instruments#csv_list}'
keywords: '{$.keywords#csv_list}'
license: '$.license'
platform: '{$.summaries.platform#csv_list}'
title: '$.title'
metadata_mapping:
title: '{$.properties.title#remove_extension}'
geometry: '$.geometry'
assets: '{$.assets#dict_filter($[?(@.href =~ ".*\/thredds\/fileServer\/.*")])}'
products:
GENERIC_PRODUCT_TYPE:
productType: '{productType}'
download: !plugin
type: HTTPDownload
Provider configuration#
The plugin structure is reflected in the internal providers configuration file. Here is a sample:
provider_name:
priority: 1
products:
# List of supported product types
# This is a mapping containing all the information required by the search plugin class to perform its job.
# The mapping is available in the config attribute of the search plugin as config['products']
S2_MSI_L1C:
a-config-key-needed-by-search-plugin-to-search-this-product-type: value
another-config-key: another-value
# Whether this product type is partially supported by this provider (the provider does not contain all the
# products of this type)
partial: True
...
search:
plugin: CustomSearchPluginClass
api_endpoint: https://mandatory.config.key/
a-key-conf-used-by-the-plugin-class-init-method: value
another-random-key: random-value
# A mapping between the search result of the provider and the eodag way of describing EO products (the keys are
# the same as in the OpenSearch specification)
metadata_mapping:
...
...
download:
plugin: CustomDownloadPlugin
# Same as with search for random config keys as needed by the plugin class
...
auth:
plugin: CustomAuthPlugin
# Same as with search for random config keys as needed by the plugin class
...
Note however, that for a provider which already has a Python library for accessing its products, the configuration varies a little bit. It does not have the ‘search’ and ‘download’ keys. Instead, there is a single ‘api’ key like this:
provider_name:
...
api:
plugin: ApiPluginClassName
...