Plugins

Plugins#

EODAG uses a two-level plugin system. Plugin topics are abstract interfaces for a specific functionality of EODAG like Search or Download. EODAG providers are implementations of at least one plugin topic. The more plugin topics are implemented by a provider, the more functionality of eodag are available for this provider.

Plugin Management#

The plugin machinery is managed by one instance of PluginManager. The instance manager knows how to discover plugins at runtime, using setuptools entry points mechanism.

Plugins Available#

EODAG currently advertises 5 types of plugins: Search, Download, Crunch, Authentication and Api.

The providers are implemented with a triplet of Search/Authentication/Download plugins or with an Api plugin:

Provider	Search	Authentication	Download
`aws_eos`	`PostJsonSearch`	`AwsAuth`	`AwsDownload`
`cop_ads`	`ECMWFSearch`	`HTTPHeaderAuth`	`HTTPDownload`
`cop_cds`	`ECMWFSearch`	`HTTPHeaderAuth`	`HTTPDownload`
`cop_ewds`	`ECMWFSearch`	`HTTPHeaderAuth`	`HTTPDownload`
`cop_dataspace`	`QueryStringSearch`	`KeycloakOIDCPasswordAuth`	`HTTPDownload`
`cop_marine`	`CopMarineSearch`	`AwsAuth`	`AwsDownload`
`creodias`	`QueryStringSearch`	`KeycloakOIDCPasswordAuth`	`HTTPDownload`
`dedl`	`StacSearch`	`OIDCTokenExchangeAuth`	`HTTPDownload`
`dedt_lumi`	`ECMWFSearch`	`OIDCAuthorizationCodeFlowAuth`	`HTTPDownload`
`earth_search`	`StacSearch`	`AwsAuth`	`AwsDownload`
`earth_search_gcs`	`StacSearch`	`AwsAuth`	`AwsDownload`
`ecmwf`	`EcmwfApi`	`EcmwfApi`	`EcmwfApi`
`eumetsat_ds`	`QueryStringSearch`	`TokenAuth`	`HTTPDownload`
`fedeo-ceda`	`StacSearch`	`None`	`HTTPDownload`
`geodes`	`StacSearch`	`HTTPHeaderAuth`	`HTTPDownload`
`geodes_s3`	`StacListAssets`	`AwsAuth`	`AwsDownload`
`hydroweb_next`	`StacSearch`	`HTTPHeaderAuth`	`HTTPDownload`
`meteoblue`	`MeteoblueSearch`	`HttpQueryStringAuth`	`HTTPDownload`
`peps`	`QueryStringSearch`	`GenericAuth`	`HTTPDownload`
`planetary_computer`	`StacSearch`	`SASAuth`	`HTTPDownload`
`usgs`	`UsgsApi`	`UsgsApi`	`UsgsApi`
`usgs_satapi_aws`	`StacSearch`	`AwsAuth`	`AwsDownload`
`wekeo_ecmwf`	`PostJsonSearch`	`TokenAuth`	`HTTPDownload`
`wekeo_cmems`	`PostJsonSearch`	`TokenAuth`	`HTTPDownload`
`wekeo_main`	`WekeoSearch`	`TokenAuth`	`HTTPDownload`

Creating Plugins#

EODAG plugins can be separated from the core eodag package. They live in your Python site-packages as standalone Python packages or modules, with eodag’s specific entry points that will be looked at by the plugin manager to load plugins at runtime. These entry points are: eodag.plugins.{topic}, with possible values for topic being api, search, crunch, auth and download.

All you have to do to create eodag plugins is start a new Python project, add eodag as a dependancy to your project, and then create the plugin type you want in Python module(s) by importing its base class from eodag.plugins.{topic}.base and subclassing it. Here is an example for an Api plugin class located in a module named your_package.your_module:

from eodag.plugins.apis.base import Api

class SampleApiPlugin(Api):

   def query(self):
      pass

   def download(self):
      pass

Then, in the configuration file of your Python project, add this:

[project.entry-points."eodag.plugins.api"]
SampleApiPlugin = "your_package.your_module:SampleApiPlugin"

setup(
   # ...
   entry_points={
      'eodag.plugins.api': [
         'SampleApiPlugin = your_package.your_module:SampleApiPlugin'
      ]
   }
)

See what the PyPa explains to better understand this concept. In EODAG, the name you give to your plugin in the entry point doesn’t matter, but we prefer it to be the same as the class name of the plugin. What matters is that the entry point must be a class deriving from one of the 5 plugin topics supported. Be particularly careful with consistency between the entry point name and the super class of you plugin class. Here is a list of entry point names and the plugin topic to which they map:

eodag.plugins.api : Api
eodag.plugins.authentication : Authentication
eodag.plugins.crunch : Crunch
eodag.plugins.download : Download
eodag.plugins.search : Search

Read the A sample mock search plugin section to get an example of how to build a search plugin.

As an example of external plugin, you can have a look to eodag-sentinelsat (archived).

A sample mock search plugin#

In this section, we demonstrate the creation of custom search plugin. This plugin will return a list of mocked EOProducts on search requests.

Create a folder called eodag_search_mock. It will be the name of our package.
Make an empty __init.py__ file in the package folder.

Make a mock_search.py file and paste in it the following class definition.

It is our search plugin. The query() function is the one called by EODAG when searching for products.

# mock_search.py
from typing import Any, List, Optional, Tuple
from eodag.plugins.search import PreparedSearch
from eodag.plugins.search.base import Search
from eodag.api.product import EOProduct

class MockSearch(Search):
"""Implement a Mock search plugin"""

def query(
   self,
   prep: PreparedSearch = PreparedSearch(),
   **kwargs: Any,
) -> Tuple[List[EOProduct], Optional[int]]:
   """Generate EOProduct from the request"""
   return ([EOProduct(
      provider=self.provider,
      properties={
         **{
            "id": f"mock_{kwargs.get('productType')}_{i}"
            },
         **kwargs
      }
      ) for i in range(0, prep.items_per_page)],
      prep.items_per_page if prep.count else None
   )

Create a pyproject.toml file in the package folder and paste in it the following content.

The projects.entry-points is crucial for EODAG to detect this new plugin.

[project]
name = "eodag-search-mock"
version = "0.0.1"
description = "Mock Search plugin for EODAG"
requires-python = ">=3.8"
dependencies = [ "eodag" ]

[project.entry-points."eodag.plugins.search"]
MockSearch = "mock_search:MockSearch"

Your plugin is now ready. You need to install it for EODAG to be able to use it.
```
pip install eodag_search_mock
```

Plugin configuration#

class eodag.config.PluginConfig[source]#

Representation of a plugin config.

This class variables describe available plugins configuration parameters.

class Pagination[source]#

Search pagination configuration

max_items_per_page: int#: The maximum number of items per page that the provider can handle

total_items_nb_key_path: str | JSONPath#: Key path for the number of total items in the provider result

next_page_url_key_path: str | JSONPath#: Key path for the next page URL

next_page_query_obj_key_path: str | JSONPath#: Key path for the next page POST request query-object (body)

next_page_merge_key_path: str | JSONPath#

count_tpl: str#: Template to add to next_page_url_tpl to enable count in search request

next_page_url_tpl: str#: The f-string template for pagination requests.

next_page_query_obj: str#: The query-object for POST pagination requests.

count_endpoint: str#: The endpoint for counting the number of items satisfying a request

start_page: int#: Index of the starting page

class Sort[source]#

Configuration for sort during search

sort_by_default: list[tuple[str, str]]#: Default sort settings

sort_by_tpl: str#: F-string template to add to next_page_url_tpl to sort search results

sort_param_mapping: dict[str, str]#: Mapping between eodag and provider query parameters used for sort

sort_order_mapping: dict[Literal['ascending', 'descending'], str]#: Mapping between eodag and provider sort-order parameters

max_sort_params: Annotated[int]#: Maximum number of allowed sort parameters per request

class DiscoverMetadata[source]#

Configuration for metadata discovery (search result properties)

auto_discovery: bool#: Whether metadata discovery is enabled or not

metadata_pattern: str#: Metadata regex pattern used for discovery in search result properties

search_param: str | dict[str, Any]#: Configuration/template that will be used to query for a discovered parameter

metadata_path: str#: Path to the metadata in search result

search_param_unparsed: list[str]#: list search parameters to send as is to the provider

raise_mtd_discovery_error: bool#: Whether an error must be raised when using a search parameter which is not queryable or not

class DiscoverProductTypes[source]#

Configuration for product types discovery

fetch_url: str | None#: URL from which the product types can be fetched

fetch_method: str#: HTTP method used to fetch product types

fetch_body: dict[str, Any]#: Request body to fetch product types using POST method

max_connections: int#: Maximum number of connections for concurrent HTTP requests

next_page_url_tpl: str#: The f-string template for pagination requests.

start_page: int#: Index of the starting page for pagination requests.

result_type: str#: Type of the provider result

results_entry: JSONPath | str#: JsonPath to the list of product types

generic_product_type_id: str#: Mapping for the product type id

generic_product_type_parsable_metadata: dict[str, str]#: Mapping for product type metadata (e.g. abstract, licence) which can be parsed from the provider result

generic_product_type_parsable_properties: dict[str, str]#: Mapping for product type properties which can be parsed from the result and are not product type metadata

generic_product_type_unparsable_properties: dict[str, str]#: Mapping for product type properties which cannot be parsed from the result and are not product type metadata

single_collection_fetch_url: str#: URL to fetch data for a single collection

single_collection_fetch_qs: str#: Query string to be added to the fetch_url to filter for a collection

single_product_type_parsable_metadata: dict[str, str]#: Mapping for product type metadata returned by the endpoint given in single_collection_fetch_url. If ID is redefined in this mapping, it will replace generic_product_type_id value

class DiscoverQueryables[source]#

Configuration for queryables discovery

fetch_url: str | None#: URL to fetch the queryables valid for all product types

product_type_fetch_url: str | None#: URL to fetch the queryables for a specific product type

result_type: str#: Type of the result

results_entry: str#: JsonPath to retrieve the queryables from the provider result

constraints_url: str#: Search URL of the constraint file used to build queryables

constraints_entry: str#: Search Key in the json result where the constraints can be found

class OrderOnResponse[source]#

Configuration for order on-response during download

metadata_mapping: dict[str, str | list[str]]#: Parameters metadata-mapping to apply to the order response

class OrderStatusSuccess[source]#

Configuration to identify order status success during download

Order status response matching the following parameters are considered success At least one is required

status: str#: Success value for status

message: str#: Success value for message

http_code: int#: Success value for status response HTTP code

class OrderStatusOrdered[source]#

Configuration to identify order status ordered during download

http_code: int#: HTTP code of the order status response

class OrderStatusRequest[source]#

Order status request configuration

method: str#: Request HTTP method

headers: dict[str, Any]#: Request hearders

class OrderStatusOnSuccess[source]#

Configuration for order status on-success during download

need_search: bool#: Whether a new search is needed on success or not

result_type: str#: Return type of the success result

results_entry: str#: Key in the success response that gives access to the result

metadata_mapping: dict[str, str | list[str]]#: Metadata-mapping to apply to the success status result

class OrderStatus[source]#

Configuration for order status during download

request: OrderStatusRequest#: Order status request configuration

metadata_mapping: dict[str, str | list[str]]#: Metadata-mapping used to parse order status response

success: OrderStatusSuccess#: Configuration to identify order status success during download

error: dict[str, Any]#: Part of the order status response that tells there is an error

ordered: OrderStatusOrdered#: Configuration to identify order status ordered during download

on_success: OrderStatusOnSuccess#: Configuration for order status on-success during download

class MetadataPreMapping[source]#

Configuration which can be used to simplify further metadata extraction

metadata_path: str#: JsonPath of the metadata entry

metadata_path_id: str#: Key to get the metadata id

metadata_path_value: str#: Key to get the metadata value

name: str#: PluginTopic The name of the plugin class to use to instantiate the plugin object

type: str#: PluginTopic Plugin type

ssl_verify: bool#: PluginTopic Whether the ssl certificates should be verified in the request or not

s3_bucket: str#: PluginTopic Default s3 bucket

auth_error_code: int | list[int]#: PluginTopic Authentication error codes

timeout: float#: PluginTopic Time to wait until request timeout in seconds

retry_total: int#: PluginTopic urllib3.util.Retry total parameter, total number of retries to allow

retry_backoff_factor: int#: PluginTopic urllib3.util.Retry backoff_factor parameter, backoff factor to apply between attempts after the second try

retry_status_forcelist: list[int]#: PluginTopic urllib3.util.Retry status_forcelist parameter, list of integer HTTP status codes that we should force a retry on

api_endpoint: str#: Search Plugin API endpoint

need_auth: bool#: Search Whether Search plugin needs authentification or not

result_type: str#: Search Return type of the provider result

results_entry: str#: Search Key in the provider search result that gives access to the result entries

pagination: Pagination#: Search Dict containing parameters for pagination

sort: Sort#: Search Configuration for sorting the results

discover_metadata: DiscoverMetadata#: Search Configuration for the metadata auto-discovery

discover_product_types: DiscoverProductTypes#: Search Configuration for the product types auto-discovery

discover_queryables: DiscoverQueryables#: Search Configuration for the queryables auto-discovery

metadata_mapping: dict[str, str | list[str]]#: Search The mapping between eodag metadata and the plugin specific metadata

metadata_mapping_from_product: str#: Search metadata_mapping got from the given product type

assets_mapping: dict[str, dict[str, Any]]#: Search A mapping for the metadata of individual assets

remove_from_queryables: list[str]#: Search Parameters to remove from queryables

literal_search_params: dict[str, str]#: Search Parameters to be passed as is in the search url query string

dont_quote: list[str]#: QueryStringSearch Characters that should not be quoted in the url params

asset_key_from_href: bool#: QueryStringSearch Guess assets keys using their href. Use their original key if False

free_text_search_operations: dict[str, Any]#: ODataV4Search Dict describing free text search request build

per_product_metadata_query: bool#: ODataV4Search Set to True if the metadata is not given in the search result and a two step search has to be performed

metadata_pre_mapping: MetadataPreMapping#: ODataV4Search Dict used to simplify further metadata extraction

search_definition: dict[str, Any]#: CSWSearch Search definition dictionary

merge_responses: bool#: PostJsonSearch Whether to merge responses or not (aws_eos specific)

collection: list[str]#: PostJsonSearch Collections names (aws_eos specific)

max_connections: int#: StaticStacSearch Maximum number of connections for concurrent HTTP requests

dates_required: bool#: ECMWFSearch if date parameters are mandatory in the request

end_date_excluded: bool#: ECMWFSearch Whether end date should be excluded from search request or not

remove_from_query: list[str]#: ECMWFSearch List of parameters used to parse metadata but that must not be included to the query

version: str#: CSWSearch OGC Catalogue Service version

auth_endpoint: str#: EcmwfApi url of the authentication endpoint

base_uri: str#: Download Default endpoint url

output_dir: str#: Download Where to store downloaded products, as an absolute file path

extract: bool#: Download Whether the content of the downloaded file should be extracted or not

output_extension: str#: Download Which extension should be used for the downloaded file

flatten_top_dirs: bool#: Download Whether the directory structure should be flattened or not

archive_depth: int#: Download Level in extracted path tree where to find data

ignore_assets: bool#: Download Whether ignore assets and download using downloadLink or not

products: dict[str, dict[str, Any]]#: Download Product type specific configuration

order_enabled: bool#: HTTPDownload Whether the product has to be ordered to download it or not

order_method: str#: HTTPDownload HTTP request method for the order request

order_headers: dict[str, str]#: HTTPDownload Headers to be added to the order request

order_on_response: OrderOnResponse#: HTTPDownload Dictionary containing the key metadata_mapping which can be used to add new product properties based on the data in response to the order request

order_status: OrderStatus#: HTTPDownload Order status handling

no_auth_download: bool#: HTTPDownload Do not authenticate the download request but only the order and order status ones

dl_url_params: dict[str, str]#: HTTPDownload Parameters to be added to the query params of the request

bucket_path_level: int#: AwsDownload At which level of the path part of the url the bucket can be found

s3_endpoint: str#: AwsDownload S3 endpoint

credentials: dict[str, str]#: Authentication Authentication credentials dictionary

auth_uri: str#: Authentication Authentication URL

headers: dict[str, str]#: Authentication Dictionary containing all keys/value pairs that should be added to the headers

retrieve_headers: dict[str, str]#: Authentication Dictionary containing all keys/value pairs that should be added to the headers for token retrieve only

token_key: str#: Authentication The key pointing to the token in the response from the token server

refresh_token_key: str#: Authentication Key to get the refresh token in the response from the token server

matching_url: str#: Authentication URL pattern to match with search plugin endpoint or download link

matching_conf: dict[str, Any]#: Authentication Part of the search or download plugin configuration that needs authentication

requester_pays: bool#: AwsAuth Whether download is done from a requester-pays bucket or not

token_provision: str#: OIDCRefreshTokenBase How the token should be used in the request

client_id: str#: OIDCRefreshTokenBase The OIDC provider’s client ID

client_secret: str#: OIDCRefreshTokenBase The OIDC provider’s client secret

oidc_config_url: str#: OIDCRefreshTokenBase The OIDC provider’s .well-known/openid-configuration url.

allowed_audiences: list[str]#: OIDCRefreshTokenBase The OIDC token audiences

user_consent_needed: str#: OIDCAuthorizationCodeFlowAuth Whether a user consent is needed during the authentication or not

authentication_uri_source: str#: OIDCAuthorizationCodeFlowAuth Where to look for the authorization_uri

authentication_uri: str#: OIDCAuthorizationCodeFlowAuth The callback url that will handle the code given by the OIDC provider

redirect_uri: str#: OIDCAuthorizationCodeFlowAuth The URL of the authentication backend of the OIDC provider

authorization_uri: str#: OIDCAuthorizationCodeFlowAuth The authorization url of the server (where to query for grants)

login_form_xpath: str#: OIDCAuthorizationCodeFlowAuth The xpath to the HTML form element representing the user login form

user_consent_form_xpath: str#: OIDCAuthorizationCodeFlowAuth The xpath to the user consent form

user_consent_form_data: dict[str, str]#: OIDCAuthorizationCodeFlowAuth The data that will be passed with the POST request on the form ‘action’ URL

additional_login_form_data: dict[str, str]#: OIDCAuthorizationCodeFlowAuth Additional data to be passed to the login POST request

exchange_url_error_pattern: dict[str, str]#: OIDCAuthorizationCodeFlowAuth Key/value pairs of patterns/messages used for Authentication errors

token_exchange_params: dict[str, str]#: OIDCAuthorizationCodeFlowAuth A mapping between OIDC url query string and token handler query string params

token_qs_key: str#: OIDCAuthorizationCodeFlowAuth Refers to the name of the query param to be used in the query request

token_exchange_post_data_method: str#: OIDCAuthorizationCodeFlowAuth Way to pass the data to the POST request that is made to the token server

token_uri: str#: OIDCAuthorizationCodeFlowAuth The url to query to get the authorized token

signed_url_key: str#: SASAuth Key to get the signed url

req_data: dict[str, Any]#: TokenAuth Credentials json structure if they should be sent as POST data

refresh_uri: str#: TokenAuth URL used to fetch the access token with a refresh token

token_type: str#: TokenAuth type of the token

token_expiration_key: str#: TokenAuth key to get the expiration time of the token

request_method: str#: TokenAuth HTTP method to use

subject: dict[str, Any]#: OIDCTokenExchangeAuth The full OIDCAuthorizationCodeFlowAuth plugin configuration used to retrieve subject token

subject_issuer: str#: OIDCTokenExchangeAuth Identifies the issuer of the subject_token

token_expiration_margin: int#: TokenAuth OIDCRefreshTokenBase Safety buffer to prevent token rejection from unexpected expiry between validity check and request.

audience: str#: OIDCTokenExchangeAuth Audience that the token ID is intended for. client_id of the Relying Party

method: str#: GenericAuth which authentication method should be used