Download Plugins

Download plugins must inherit the following class and implement download():

class eodag.plugins.download.base.Download(provider, config)

Base Download Plugin.

A Download plugin has two download methods that it must implement:

• download: download a single EOProduct

• download_all: download multiple products from a SearchResult

They must:

• download data in the output_dir folder defined in the plugin’s configuration or passed through kwargs

• extract products from their archive (if relevant) if extract is set to True (True by default)

• save a product in an archive/directory (in output_dir) whose name must be the product’s title property

• update the product’s location attribute once its data is downloaded (and eventually after it’s extracted) to the product’s location given as a file URI (e.g. file:///tmp/product_folder on Linux or file:///C:/Users/username/AppData/Local/Temp on Windows)

• save a record file in the directory {output_dir}/.downloaded whose name is built on the MD5 hash of the product’s product_type and properties['id'] attributes (hashlib.md5((product.product_type+"-"+product.properties['id']).encode("utf-8")).hexdigest()) and whose content is the product’s remote_location attribute itself.

• not try to download a product whose location attribute already points to an existing file/directory

• not try to download a product if its record file exists as long as the expected product’s file/directory. If the record file only is found, it must be deleted (it certainly indicates that the download didn’t complete)

Parameters:

• provider (str) – An eodag providers configuration dictionary

• config (PluginConfig) – Path to the user configuration file

download(product, auth=None, progress_callback=None, wait=0.2, timeout=10, **kwargs)

Base download method. Not available, it must be defined for each plugin.

Parameters:

• product (EOProduct) – The EO product to download

• auth (AuthBase | S3ServiceResource | None, default: None) – (optional) authenticated object

• progress_callback (ProgressCallback | None, default: None) – (optional) A progress callback

• wait (float, default: 0.2) – (optional) If download fails, wait time in minutes between two download tries

• timeout (float, default: 10) – (optional) If download fails, maximum time in minutes before stop retrying to download

• kwargs (Unpack[DownloadConf]) – output_dir (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Return type:

str | None

Returns:

The absolute path to the downloaded product in the local filesystem (e.g. /tmp/product.zip on Linux or C:\\Users\\username\\AppData\\Local\\Temp\\product.zip on Windows)

download_all(products, auth=None, downloaded_callback=None, progress_callback=None, wait=0.2, timeout=10, **kwargs)

Base download_all method.

This specific implementation uses the eodag.plugins.download.base.Download.download() method implemented by the plugin to sequentially attempt to download products.

Parameters:

• products (SearchResult) – Products to download

• auth (AuthBase | S3ServiceResource | None, default: None) – (optional) authenticated object

• downloaded_callback (DownloadedCallback | None, default: None) – (optional) A method or a callable object which takes as parameter the product. You can use the base class DownloadedCallback and override its __call__ method. Will be called each time a product finishes downloading

• progress_callback (ProgressCallback | None, default: None) – (optional) A progress callback

• wait (float, default: 0.2) – (optional) If download fails, wait time in minutes between two download tries

• timeout (float, default: 10) – (optional) If download fails, maximum time in minutes before stop retrying to download

• kwargs (Unpack[DownloadConf]) – output_dir (str), extract (bool), delete_archive (bool) and dl_url_params (dict) can be provided as additional kwargs and will override any other values defined in a configuration file or with environment variables.

Return type:

list[str]

Returns:

List of absolute paths to the downloaded products in the local filesystem (e.g. ['/tmp/product.zip'] on Linux or ['C:\Users\username\AppData\Local\Temp\product.zip'] on Windows)

generate_record_hash(product)

Generate the record hash of the given product.

The MD5 hash is built from the product’s product_type and properties['id'] attributes (hashlib.md5((product.product_type+"-"+product.properties['id']).encode("utf-8")).hexdigest())

Parameters:

product (EOProduct) – The product to calculate the record hash

Return type:

str

Returns:

The MD5 hash

This table lists all the download plugins currently available:

http.HTTPDownload(provider, config)	HTTPDownload plugin.
aws.AwsDownload(provider, config)	Download on AWS using S3 protocol.
creodias_s3.CreodiasS3Download(provider, config)	Download on creodias s3 from their VMs (extension of AwsDownload)

Download methods call graph

Here is a graph showing how download methods are called from API to plugin:

Progress bars

While downloading, progress bars are generated using ProgressCallback. They are instantiated and closed in EOProduct.download() which means that plugins do not need to handle this part. Using download_all(), one progress bar will be created and closed for each product.

The same progress bar will always be used to track both individual product download and extraction.

If a custom ProgressCallback is passed as argument of download() or download_all(), it will be used and duplicated to create bars to track download_all/download. The same bar will be used for all products, and will not be closed by eodag.

ProgressCallback handling

	plugin.download_all()	product.download()	Output
download(product)		Default ProgressCallback created for product download/extraction, then closed
download(product, progress_callback=bar)		Passed ProgressCallback used for product download/extraction, kept open
download_all(products)	Default ProgressCallback created for products count, then closed	Default ProgressCallback created per product for download/extraction, then closed
download_all(products, progress_callback=bar)	Passed ProgressCallback used for products count, copied for product.download(), then closed.	Passed ProgressCallback used for all products download/extraction, kept open