Scihub API returns `Too many requests` about sen2agri-system HOT 11 CLOSED

No, it's not implemented. I think you can throttle them at the OS level, but it's a bit of work.

Put different: can I manually download the files and inject them into sen2agri?

Sure!

from sen2agri-system.

Ok - I found this section in the user manual that mentions a local store.

UPDATE:

it seems this is not the goto solution according to #39

There is a command in the FAQ about importing existing products, but even calling it with the -h (help) flag errors

from sen2agri-system.

I realized the downloaders are not at all open-sourced (besides a lot of other functionalities) - is this intentional?

The data source modules are all open-source, but they are in another repository (since some of them are used by SNAP, Sen2Agri and other projects).

from sen2agri-system.

This is indeed a SciHub change. I'm not sure whether we actually do a lot of requests or they have very strict limits. Some HTTP APIs actually return rate limit headers which clients can check for and try to honor, but SciHub doesn't.

from sen2agri-system.

I can see that the 3 queries

a) page 1 query : 48 results
b) page 2 query: 0 results
c) downloading the first result

Happen very fast. Can I somehow throttle this?

Put different: can I manually download the files and inject them into sen2agri?

from sen2agri-system.

Put different: can I manually download the files and inject them into sen2agri?

Sure!

Since I do not have any working example, how does sen2agri expect the folder structure to be looking like so it will recognize the downloaded files?

from sen2agri-system.

I'll have to ask, but I think we support the original folder structure.

from sen2agri-system.

@lnicola thank you

Since my site's folder /mnt/archive/dwn_def/s2/default/hetzerath only contains the folder failed_queries I'll still need to know what the folder of the successful downloads is expected to be. Could you send over a filetree/ screenshot of a sites folder that was successfully downloaded?

FYI: SNAP seems to have had a similar issue recently https://senbox.atlassian.net/browse/SNAP-1411

from sen2agri-system.

You need to download the products to a different place, maybe unpack them, then run a command to tell the system to import them. It's not enough to just copy them to the download location.

from sen2agri-system.

I realized the downloaders are not at all open-sourced (besides a lot of other functionalities) - is this intentional?

from sen2agri-system.

I have received a reply from Cosmin with a very useful script to change to apihub instead of dhus (also mentioned in sentinelsat/sentinelsat#353)

It seems that this issue is due to some limitations introduces by SciHub in the last time. Apparently, they introduces some time constraints for the number of queries performed in a given amount of time.
For now, a quick solution would be to change the configuration of the SciHub plugin to use apihub instead dhus:

Copy the /usr/share/sen2agri/sen2agri-services/lib/tao-datasources-scihub-1.0.X.X.jar into a distinct folder
unzip/extract from the archive (the jar is actually a zip archive)
go into ro/cs/tao/datasource/remote/scihub/scihub.properties and switch the comments as below:

    scihub.search.url = https://scihub.copernicus.eu/apihub/search
    #scihub.search.url = https://scihub.copernicus.eu/dhus/search

Put back the modified file in the jar file, in the same location
copy back the modified jar file into /usr/share/sen2agri/sen2agri-services/lib/
Restart the services

Or, you can use the following script to do this automatically:change_scihub_jar_cfg.zip (891 Bytes)

The script can be invoked to simply switch from dhus to apihub:

./change_scihub_jar_cfg.sh
./change_scihub_jar_cfg.sh -s apihub

or, to switch back to dhus:

./change_scihub_jar_cfg.sh -s dhus

Best regards,
Cosmin

Find the zip file at https://forum.esa-sen2agri.org/t/too-many-requests-from-scihub-api/829

from sen2agri-system.