cs-si / eodag Goto Github PK
View Code? Open in Web Editor NEWEarth Observation Data Access Gateway
Home Page: https://eodag.readthedocs.io
License: Apache License 2.0
Earth Observation Data Access Gateway
Home Page: https://eodag.readthedocs.io
License: Apache License 2.0
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
At the moment, AwsSearch plugin is a subclass of RestoSearch AND AmazonWS which implements AwsSearch for search is in fact a doublon of eocloud provider when it comes to search. This means that if in the same search eocloud and AmazonWS providers are used, we will end up having partially duplicated products in our result (in terms of id for example).
Furthermore, plugins are supposed to be independent from each other, and these two are not.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Support ASF to retrieve EO data, mainly SAR one.
A python package is available for this provider https://github.com/asfadmin/Discovery-asf_search under BSD-3-Clause license (compatible with eodag).
pypi: https://pypi.org/project/asf-search/
conda-forge: https://anaconda.org/conda-forge/asf_search
Documentation: https://docs.asf.alaska.edu/asf_search/basics/
A python api plugin will be preferred (faster development time and easier maintainability), but API is also documented:
API documentation: https://docs.asf.alaska.edu/api/basics/
search example: https://api.daac.asf.alaska.edu/services/search/param?intersectsWith=POLYGON((-2.671%2042.2314,4.3776%2042.2314,4.3776%2045.9707,-2.671%2045.9707,-2.671%2042.2314))&platform=SENTINEL-1&instrument=C-SAR&start=2020-05-02T22:00:00Z&end=2020-05-20T21:59:59Z&maxResults=250&output=jsonlite2
{
"results": [
{
"b": [],
"bm": "IW",
"d": "Sentinel-1A",
"du": "https://datapool.asf.alaska.edu/RAW/SA/{gn}.zip",
"f": 146,
"fd": "ASCENDING",
"fl": null,
"fn": "{gn}.zip",
"fr": null,
"gid": "S1A_IWDV_0146_0153_032652_030",
"gn": "S1A_IW_RAW__0SDV_20200520T175617_20200520T175649_032652_03C822_715C",
"i": "C-SAR",
"in": false,
"mn": null,
"o": [
"32652"
],
"on": null,
"p": 30,
"pa": null,
"pid": "{gn}-RAW",
"po": "VV+VH",
"pt": "RAW",
"ptd": "L0 Raw Data (RAW)",
"s": 1494.7302169799805,
"ss": null,
"st": "2020-05-20T17:56:17.000000",
"stp": "2020-05-20T17:56:49.000000",
"t": null,
"w": "POLYGON((0.742400 45.103700,-2.362300 44.824200,-2.905900 46.772000,0.311000 47.055600,0.742400 45.103700))",
"wu": "POLYGON ((0.742400 45.103700, -2.362300 44.824200, -2.905900 46.772000, 0.311000 47.055600, 0.742400 45.103700))"
},
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
Currently, eodag list
command only list available product types for search. It should also list available providers.
More generally, the provider's priority configuration of an eodag install should be customizable from the command line:
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
e.g: from a shapefile, wkt, geojson, ...
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
We need search parameters to be uniform with eo products metadata throughout eodag project (cli and api). The providers config file should advertise which metadata is queryable by indicating how it is mapped to its search parameters.
Eodag should provide a way for plugin developers to retrieve the search parameters suitable to their plugin, using only the information available in the mapping.
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).
Update setup.py to deploy "user_conf_template.yml" into standard user configuration folder.
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Peps products are not downloadable : authentication depends of a bearer token not yet managed.
A direct access to the url gives a '404' error.
We can check that archives exists and is accessible with a CLI request to download url with curl, but the problem occurs with requests module.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Since EOProduct id has changed in 2a256a9, eodag send wrong id to sentinelsat download section, resulting in error when we want to download on SciHub provider.
how-to-fix-it : in eodag.plugin.apis.sentinelsat l.75 & 78, replace product.id
by product.properties['provider_id']
.
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
get_data print too much informations like if it was in debug. For example, extraction and download are printed on multiple lines:
extraction and download should be on one line.
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
We need a way to express how eodag metadata map to each provider metadata. This is how EOProduct properties are built from provider search results.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
See to replace the direct use of tqdm by a python callback that allows to choose the progress bar type (Using tqdm_notebook allows to display the download progress on a single bar).
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Sort products by wkb_hex is not optimal. There are a lot of cases where products doesn't share exactly the same extent.Below, the products of first and last index of sorted_products
have the same tile id, but are not sorted in the same SearchResult.
#!python
product_type = 'S2_MSI_L1C'
extent = {
'lonmin': -106.130161,
'lonmax': -106.102524,
'latmin': 39.627243,
'latmax': 39.647336
}
dag.set_preferred_provider(provider='airbus-ds')
sorted_products = dag.sort_by_extent([
dag.search(product_type,
startTimeFromAscendingNode='2018-05-01',
completionTimeFromAscendingNode='2018-05-18',
geometry=extent),
dag.search(product_type,
startTimeFromAscendingNode='2018-06-05',
completionTimeFromAscendingNode='2018-06-19',
geometry=extent)
])
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
For Sentinel-2:
http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/
https://aws.amazon.com/fr/public-datasets/sentinel-2/
Warning it could necessary to regenerate the SAFE format:
https://github.com/sinergise/sentinelhub/
For Landsat:
https://aws.amazon.com/fr/public-datasets/landsat/
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Related to SNAP feature
the DEM data is very useful in several applications.
Several DEM are publicly available:
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
If a get_data on a product has been done one time, re-trying the operation, by doing search and get_data on the same product, doesn't work and eodag can't find path to product bands.
Find a way to link get_data to product informations if the product is already found on local.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Support
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
S2_MSI_L2Ap product type is not yet searched on eocloud.
Add something like that (works on my side) in providers.yml in eocloud section :
#!yaml
S2_MSI_L2Ap:
product_type: L2A
collection: Sentinel2
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
See documentation of this feature here:
https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
EOPRODUCT_PROPERTIES contains a "description" field that will induce a KeyError for a Theia search, as it is no longer provided. Remove it.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
sentinelsat API must have the footprint as a area parameter, and not as a keyword argument.
In eodag.plugins.apis.sentinelsat , replace l.108 with :
#!python
query['area'] = geometry.box(*box_values).to_wkt()
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
CODEDE is another collaborative ground segment which provide Sentinel Data and other data.
https://code-de.org/
Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
On some providers, downloading some products is an asynchronous process: when the client makes a get request to download the product, the provider returns a response with status code 202 (accepted).
Right now, the http download plugin does not understand this response and treat it as it was a 200 (Success) response, resulting in a downloaded "product" not containing any data (the payload of a 202 response is an empty bytes).
The plugin must better behave to enable getting the actual product by either :
RETRY-AFTER
, not exceeding a certain amount of time (roughly half-an hour)Location
for the status of the downloadExample url leading to a 202 response: https://peps.cnes.fr/resto/collections/S2/420901c4-2f0e-59ad-a792-9043de6453fe/download?issuerId=peps (make sure you are connected to peps to reproduce). Exemple request with curl:
#!bash
curl --user id:pass "https://peps.cnes.fr/resto/collections/S2/420901c4-2f0e-59ad-a792-9043de6453fe/download?issuerId=peps"
Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).
#!python
[2018-06-07 05:34:18,767] INFO - product:download:99 : downloading product
Downloading products
[2018-06-07 05:34:19,011] DEBUG - core:__get_downloaders:207 : Looking for the appropriate Download instance to use for product: EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps)
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:96 : Creating configured *DOWNLOAD* plugin instances
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:103 : Creating 'DOWNLOAD' plugin instance with name 'peps'
[2018-06-07 05:34:19,011] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'DOWNLOAD' plugin instance with config '{'base_uri': u'https://peps.cnes.fr/resto', u'extract': None, 'dl_url_params': {'issuerId': u'peps'}, 'plugin': u'HTTPDownload'}'
[2018-06-07 05:34:19,011] DEBUG - http:__init__:34 : Images will be downloaded to directory /tmp
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:96 : Creating configured *API* plugin instances
[2018-06-07 05:34:19,012] DEBUG - instances_manager:__get_instances:114 : Skipping 'API' plugin creation for instance 'peps': not an instance of 'API'
[2018-06-07 05:34:19,012] DEBUG - core:__get_downloaders:220 : Found 1 Download instance(s) for product EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps) (ordered by highest priority): [HTTPDownload(instance_name=peps, priority=0, topic=Download)]
[2018-06-07 05:34:19,012] DEBUG - core:__download:143 : Using interface for download : HTTPDownload on instance *peps*
[2018-06-07 05:34:19,012] DEBUG - core:__get_authenticator:174 : Authentication initialisation for instance peps
[2018-06-07 05:34:19,012] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'AUTH' plugin instance with config '{'auth_uri': u'https://peps.cnes.fr/resto/api/users/connect', 'credentials': {'password': None, 'email': None}, 'plugin': u'GenericAuth'}'
[2018-06-07 05:34:19,012] DEBUG - core:__get_authenticator:183 : Initialized GenericAuth(instance_name=peps, priority=0, topic=Authentication) Authentication plugin for instance 'peps'
[2018-06-07 05:34:19,012] DEBUG - http:download:43 : Download url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download
[2018-06-07 05:34:19,016] DEBUG - connectionpool:_new_conn:824 : Starting new HTTPS connection (1): peps.cnes.fr
[2018-06-07 05:34:19,398] DEBUG - connectionpool:_make_request:396 : https://peps.cnes.fr:443 "GET /resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps HTTP/1.1" 404 44
https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps
[2018-06-07 05:34:19,404] ERROR - http:download:75 : Error while getting resource : 404 Client Error: Not Found for url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps
[2018-06-07 05:34:19,406] ERROR - product:main:215 :
Traceback (most recent call last):
File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 198, in main
process_cfg['params']['inputFile'] = download(input_dir)
File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 103, in download
product_path = next(eodag_api.download_all(product))
StopIteration
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Information about API
https://scihub.copernicus.eu/userguide/5APIsAndBatchScripting
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
https://cloud.google.com/storage/docs/public-datasets/sentinel-2
https://cloud.google.com/storage/docs/public-datasets/landsat
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Add a way to get the path to the product in one time, or at least set an attribute with the name of the product (e.g product.filename = '<product_basename>.SAFE'
)
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
In the case where a product has been already downloaded but we make a new search on it, eodag don't retrieve it and download it again. The verification should be done at least on zipfile existence. Or, for the check of the existence of de-zipped product, the .downloaded folder should contain informations like archive depth, filename, etc.
As the use of eodag goes hand in hand with non-manipulation of files, these checks on the default download directory should be sufficient.
Possibly linked to #39
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
For all providers that give the choice to select way to access to data (file or http), get_data don't look if a product is already downloaded, and the download is always done.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).
#!python
[2018-06-07 04:43:40,761] INFO - product:download:99 : downloading product
Downloading products
[2018-06-07 04:43:42,850] DEBUG - core:__get_downloaders:207 : Looking for the appropriate Download instance to use for product: EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps)
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:96 : Creating configured *DOWNLOAD* plugin instances
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:103 : Creating 'DOWNLOAD' plugin instance with name 'peps'
[2018-06-07 04:43:42,851] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'DOWNLOAD' plugin instance with config '{'base_uri': u'https://peps.cnes.fr/resto', u'extract': None, 'dl_url_params': {'issuerId': u'peps'}, 'plugin': u'HTTPDownload'}'
[2018-06-07 04:43:42,851] DEBUG - http:__init__:34 : Images will be downloaded to directory /tmp
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:96 : Creating configured *API* plugin instances
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:114 : Skipping 'API' plugin creation for instance 'peps': not an instance of 'API'
[2018-06-07 04:43:42,852] DEBUG - core:__get_downloaders:220 : Found 1 Download instance(s) for product EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps) (ordered by highest priority): [HTTPDownload(instance_name=peps, priority=0, topic=Download)]
[2018-06-07 04:43:42,852] DEBUG - core:__download:143 : Using interface for download : HTTPDownload on instance *peps*
[2018-06-07 04:43:42,852] DEBUG - core:__get_authenticator:174 : Authentication initialisation for instance peps
[2018-06-07 04:43:42,852] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'AUTH' plugin instance with config '{'auth_uri': u'https://peps.cnes.fr/resto/api/users/connect', 'credentials': {'password': None, 'email': None}, 'plugin': u'GenericAuth'}'
[2018-06-07 04:43:42,852] DEBUG - core:__get_authenticator:183 : Initialized GenericAuth(instance_name=peps, priority=0, topic=Authentication) Authentication plugin for instance 'peps'
[2018-06-07 04:43:42,852] DEBUG - http:download:43 : Download url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download
[2018-06-07 04:43:42,853] ERROR - product:main:215 : [Errno 17] File exists: '/tmp/.downloaded'
Traceback (most recent call last):
File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 198, in main
process_cfg['params']['inputFile'] = download(input_dir)
File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 103, in download
product_path = next(eodag_api.download_all(product))
File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/api/core.py", line 119, in download_all
for path in self.__download(product):
File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/api/core.py", line 152, in __download
for local_filename in maybe_generator(iface.download(product, auth=auth)):
File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/utils/__init__.py", line 87, in maybe_generator
for elt in obj:
File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/plugins/download/http.py", line 49, in download
os.makedirs(download_records)
File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/tmp/.downloaded'
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
https://www.terrascope.be/
currently not functionnal
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Actually product_location_scheme is set to 'file' for eocloud search, making eodag searching in /eodata/...
Default parameter should be 'http' for general users.
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
error :
#!python
UnsupportedDatasetAddressScheme: eo product EOProduct(id=5619902d-d6de-41ca-a295-e4d9efe970c2, provider=airbus-ds) is accessible through a location scheme that is not yet supported by eodag: {base}/5619902d-d6de-41ca-a295-e4d9efe970c2
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
This would enable the EOProduct
to trigger effective download in the situation when a previous download was done (changing the location
attribute to the file system path) and the said product was deleted from the file system before a subsequent call to EOProduct.download
, leading to a more intelligent object which knows exactly when to download itself .
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Since 2016-12-06, Sentinel-2 collection is S2ST as the products are provided per tiles, and S2 for earlier dates. Actually, only S2 is managed and search requests won't find any products on Peps.
Original report by clement cazorla (Bitbucket: ccazorla, GitHub: ccazorla).
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Some information are here: https://eocloud.cloudferro.com/knowledgebase.php?action=displayarticle&id=66
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Add a method that return geojson object from a SearchResult:
#!python
def as_geojson_object(self):
return {
'type': 'FeatureCollection',
'features': [product.as_dict() for product in (
self.__final_result if self.__crunch_calls_count > 0 else self.__original
)]
}
@property
def __geo_interface__(self):
"""Implements the geo-interface protocol.
See https://gist.github.com/sgillies/2217756
"""
return self.as_geojson_object()
Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).
eodag must wraps products data in a xarray
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
#!yaml
S2_MSI_L2Ap:
product_type: S2MSI2Ap
Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).
Add quicklooks to properties EOProduct attribute, or directly as a new attribute. A simple look on the data allows to quickly check if the product correspond to what the user want before downloading it.
In an IDE like Jupyter notebook, it can be a real plus as the images are very well embedded to the structure.
Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).
Support:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.