With commit 2ffb60f, you set set custom url to search for Sentinel-2. We must also apply this rule in the line 149

Currently, its points to Collection/Year-Month/PathRow/Scene_id. It should use the following syntax: Collection/Version/path/row/year/scene_id

The base collection BaseLandsat is not listing the nbar_* files generated by sensor-harm.

Since the Copernicus ODATA API with DHUS is deprecated, we should support the new API. See more details in https://documentation.dataspace.copernicus.eu/Home.html

We should have a provider to deal with any HTTP/FTP remote servers using temporal mask like:

catalog = Catalog('https://myurl.com/data')

scenes = catalog.search('%Y/file_%j_%H%M.hdf', start_date='2021-01-01')

We should add way to encapsulate errors related remote server closed. The pymodis library gives an AttributeError.

The command line should be able to download using tiles, scenes, list of seach result downloads

• Landsat-8 - Machine to Machine (M2M)
• Sentinel-2 - Copernicus API
• DIAS

Since the ESA has moved their API to the new endpoint News, we should support to pass option corresponding the new API url.

It seems that USGS Provider (EarthExplorer) has changed the login method. The option ncform was been removed.

The following error occurs when trying to parse Landsat-4 (LANDSAT_TM_C1):

File "/home/user/bdc-collectors/bdc_collectors/usgs/parser.py", line 29, in __init__
  raise RuntimeError(f'Invalid Landsat scene {scene_id}')
RuntimeError: Invalid Landsat scene LT04_L1TP_220068_19880927_20170205_01_T1

Right now, we have get_files which returns all raster files in dir. We should have a function which results other essential files:

Landsat - MTL, ANG, etc.
Sentinel - MTD, AOT, etc.

We should add bdc-catalog in extras

We should move to the Landsat API 1.5 since the others are deprecated and will be removed in December, 2020.

Ref #76

Since this module does not strict require Flask, we should consider to remove it from install_requires.

We should improve the documentation in how to create custom providers and attach it into collectors. It should contain the following entries:

• Add example for module initialization and class for this
• brief entrypoint management and how it is loaded by bdc-collectors

For example, consider an extension of current provider SciHub named MyProvider which uses different api location. You may have few ways to use this

• Change runtime parameter api_url:

from bdc_collectors.scihub import SciHub

catalog = SciHub(username="user", password="pass", api_url="https://my-url-provider-hub")
catalog.search(...)

# or
# from bdc_collectors.ext import CollectorExtension
# ext = CollectorExtension(app)  # flask_app
# catalog = ext.get_provider("SciHub")(username="user", password="pass", api_url="https://my-url-provider-hub")

• Custom Provider named MyProvider (in bdc-collectors bdc_collectors/my_provider/__init__.py or in other github repo):

from typing import Type

from ..base import BaseCollection
from ..scihub import SciHub

def init_provider():
    return dict(MyProvider=MyProviderAPI)


class MyProviderAPI(SciHub):
    def __init__(self, *args, **kwargs):
        kwargs.setdefault("api_url", "https://my-url-provider-hub")
        super(MyProviderAPI, self).__init__(*args, **kwargs)
    def get_collector(self, collection: str) -> Type[BaseCollection]:
        # ... logic for custom data collection
        return super().get_collector(collection) # optional

Register in setup.py the entrypoint for loading this provider:

entry_points={
        ...,
        'bdc_collectors.providers': [
            ...  # other providers
            'my_provider = bdc_collectors.my_provider'
        ]
}

and use it with:

from bdc_collectors.ext import CollectorExtension
ext = CollectorExtension(app)  # flask_app
catalog = ext.get_provider("MyProvider")(username="user", password="pass")

We have noticed that the EarthExplorer API not providing anymore the filter Day/Night indicator flag for Landsat C2 L2. I think we should consider to set as optional and set it only if its provided by default.