Director of Engineering at Radiant Earth
-
Iโm currently working on Source Cooperative
-
On the weekends I take photos and jump out of planes
-
You can reach me via email [email protected]
Plugin for QGIS to browse and download assets from STAC APIs
License: Apache License 2.0
Iโm currently working on Source Cooperative
On the weekends I take photos and jump out of planes
You can reach me via email [email protected]
Steps:
Expected:
query vector and downloaded image are coincident.
Actual:
query vector and band are very far away from one another. The distance is probably thousands of world sizes, so it seems to be an issue with CRS.
QGIS doesn't have an easy way to allow plugins to have external dependencies. Right now the only external dependency is requests which would require users to install that package manually. The plugin should move to use urllib instead to remove this requirement.
tried to add api endpoint https://dev.api.weather.gc.ca/msc-wis-dcpc/stac
fails with a generic http error,
would be good to see some more details in log why connection failed
seems i had to add it without /stac:
https://dev.api.weather.gc.ca/msc-wis-dcpc
then is added fine
in next step to query the catalog fails also with very generic error
Would be good to have in error message some indication of an expected element in server answer
in this case the browser may not have been able to detect the available stac collections in the service, the payload was:
{
"collections": [],
"bbox": [223181.425, 480387.771, 245110.0, 503192.755],
"time": "2000-01-01T00:00:00Z/2020-09-10T09:34:00Z",
"limit": 50,
"page": 1
}
so browser should have indicated it was not able to detect any stac collections in service
Version: dev/f5a35fba113c72129efc397a8a036ca5b8080af8
Repro:
AttributeError: 'NoneType' object has no attribute 'extent'
Traceback (most recent call last):
File "/Users/pvarner/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/stac_browser/ui/query_dialog.py", line 59, in on_search_clicked
self.get_time_period())
File "/Users/pvarner/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/stac_browser/stac_browser.py", line 51, in on_search
extent_rect = extent_layer.extent()
AttributeError: 'NoneType' object has no attribute 'extent'
The extent attribute makes me think that the Extent dialog shouldn't be empty?
Hello,
Is it possible to easiliy add this stac ?
https://www.geo.admin.ch/en/geo-services/geo-services/download-services/stac-api.html
Nothing happens when I try to add: data.geo.admin.ch as a URL.
And is it OK to work with a different EPSG, for exemple EPSG:2056? Because the plugin seems to work only with EPSG:4326 (WGS84).
Thank you for your help!
The current layer extent only approach is very inconvenient in the general case.
Why limit only to layer extent, when you can use the current zoom level of the map canvas to get the extent? Further, the user should be also able to draw a rectangle extent on the map to be used.
Similar behavior control already exists in QGIS, however, it's part of the processing framework and it's not a reusable component (check the second param in "Clip raster by extent" algorithm). I would like to write it first for this plugin first and then maybe reimplement it directly in C++ to make it available to all plugin developers.
I have made a catalog using PySTAC. Can I make it stream in QGIS using the STAC Browser?
Accessing the Sentinel AWS S3 bucket requires an AWS account and requests to be authenticated.
See https://forum.sentinel-hub.com/t/how-to-access-s3-data-from-aws/171
I can't find support for adding AWS credentials in the plugin?
Handle network errors, authentication errors, etc.
The loading bar window hangs if the server is down. For example, I added to Config.STAC_APIS 'http://localhost:22902/api/v2', and started the plugin, even though I'd neglected to start the server locally.
This is likely a general issue with how the API requests are being done, with no timeouts. Recommendation is to at least add connections and socket timeouts, and possibly add code like [https://pypi.org/project/timeout-decorator/] that will timeout a request even if its not causing the socket timeout, but the data return rate is intolerably slow.
I'm creating this issue for discussion about the style guide in the project, that has to be developed.
I would suggest thinking carefully before you decide to switch to one style or another. I've checked the QGIS repo and even there is no consistency, for example here. I've checked the top most downloaded plugins about camelOrPascalCase vs pep8 recommendation: Use the function naming rules: lowercase with words separated by underscores as necessary to improve readability.
. The conclusions from my investigations are: I see a lot of diversity :) . Either lack of consistency or one of the two styles is used. However, it seems there is a little preference towards QT style.
IMHO when dealing with QGIS/QT, it's better to follow their style guide, when everything is depending so much on them. Otherwise mixing is inevitable in the same file. For lib files that are completely separate and independent (therefore might be reused outside the current project), I would use the PEP8 recommendation. PEP8 is not a religious book, even python core libraries break the style guide: logging.getLogger(name).
As for PEP484, I see that in the beginning, the syntax might seem a bit funky to people who are not fully in the CS field. However, in the long term, I think it pays off as smart and easy documentation. Even for beginners, who often use IDEs, can get type checking for free, therefore making it easier to contribute with fewer bugs. PEP484 is a relatively new feature in the Python world (v3.5), QGIS supports Python 3 since Feb 2018 and it's understandable why it's not widely adopted in the community yet. So at least for separate, completely independent files, I would suggest to use it.
As a short summary, I would suggest the following convention.
Of course, all of this is my subjective opinion and it is based on writing and investigating a few public and private QGIS plugins. Feel free to disagree but I think it's a good starting point for a discussion.
# ./utils.py
def reverse_msg(msg: str):
return msg[::-1]
# ./controllers/deals_with_ui.py
from QtWidgets import QDialog
from ..utils import reverse_msg
class RandomDialog(QDialog):
def showReversedMsgDialog(self, msg: text = '!ymmuD') -> None:
importantLabel = self.widgetIdLabel
importantLabel.setText(reverse_msg(msg))
importantLabel.clicked.connect(self._onImportantLabelClicked)
def _onImportantLabelClicked(self) -> None:
importantLabel.setVisible(False)
On certain machines the gdalbuildvrt binary is not found within the path that python is searching
When I try to add https://franklin.nasa-hsi.azavea.com/ I get Connection Failed; Http error:
It is the main Franklin test instance. Looks to be STAC rc.2 implementing API 1.0.0-beta.1 at the moment.
Blocked by the API spec in STAC. Currently there is no standard way for paging through items so until that gets added (most likely v0.8.0) then this plugin will use unofficial paging APIs.
Queries made by the STAC browser are likely not compatible with the STAC API specification, in which time is parsed with the datetime argument.
For example, an HTTP request sent was:
POST https://.../api//search HTTP/1.1
Accept-Encoding: identity
Host: ...
User-Agent: Python-urllib/3.7
Content-Type: application/json; charset=utf-8
Content-Length: 183
Connection: close
{"collections": ["markham"], "bbox": [-123.6308868661519, 36.02789736179574, -28.151779495509228, 52.37970698302373], "time": "1959-01-01T00:00:00Z/1961-01-01T00:00:00Z", "limit": 50}
Which uses the time keyword for temporal filtering. This leads to ignoring the filter from the API's standpoint and loading all the items.
@kbgg congrats on this awesome project!
I'm tinkering with it to submit a PR to include the CBERS stac api server, and noticed that the browser assumes that title is always available for collections, more specifically here:
This breaks if no title is available. Since this field is not currently required it would be better to use id if title is not available. This is what I did in my fork,
Line 189 in stac_browser.py:
logging.error(f'Window {self.current_window} does not exist')
fails on Ubuntu 16.04
Tracking it down f formatting of strings was added in Python 3.6, this version of Ubuntu has 3.5.
Making a note here in case others hit this issue. Not sure it warrants a change in the code of this plugin, as most people have moved on to Python 3.6+ (osgeo4w, mac installer, and Ubuntu 18.04)
I get an http error when trying to add https://api.radiant.earth/mlhub/v1/
Would be great to make sure that works.
APIs should only be included in the query window if the STAC version the API is providing is compatible with the plugin. Currently the plugin is targeting >= 0.7.0
Just an idea, but I think you could request available STAC datasets / APIs from the STAC Index API:
https://stacindex.org/api/catalogs
It's not stable yet, but I hope it doesn't change too much...
I've just installed this extension to try and test it with my new STAC API, and I've found it seems to keep hanging whenever it tries to make a network request.
If I go to Configure APIs, and try and add my new API, it hangs when testing the API. Similarly, it hangs whenever testing any of the APIs that are originally listed there.
Do you know if there is some major network access problem with this plugin?
Trying to add https://data.geo.admin.ch/api/stac/v0.9/ I get 'Connection Failed; http error':
It looks to be a 0.9.0 STAC API/core, not sure what the backend is.
Trying to add https://tamn.snapplanet.io/ lets you add it to the list of api's, but then on search it says 'method not supported'. It is a latest beta.1 API and rc.3/rc.4 STAC implementation.
It's backed by https://github.com/jjrom/resto
It looks to me like the list of endpoints is built by retrieving all of the links to collections, and then creating one entry per collection. I'd like to be able to run my queries against the "search" ref link in the /stac Catalog without needing to choose a collection.
Blocked by #4
Format all code to follow PEP 8 style
Currently the plugin hangs when an error is encountered when building the VRT
QGIS tells me there's a new version, but the upgrade doesn't work. I'm not sure if the plugin manager handles letters in version numbering or not. This is the only plugin I've ever seen with a letter. Might be an upstream bug.
Also I can't find which branch in this repo has either version number, all the code still shows 0.1
I can add microsoft's STAC service at https://planetarycomputer.microsoft.com/api/stac/v1 to the api list, but when I hit search I get 'Network Error: [501] Not Implemented'.
It looks to be a STAC API 1.0.0-beta.1, with STAC core 1.0.0-beta.2, running https://github.com/stac-utils/stac-fastapi
I'm using the latest stable plugin and qgis version 3.8. I can view Sentinel and Landsat images intersecting with an aoi drawn on the map but when I try to download the true color image I get an HTTP ERROR. Downloading Landsat works (the Extent layer invalid error was because I accidently used an empty extent). Here is the traceback.
2019-09-21T18:18:48 INFO [1/1] Downloading https://sentinel-s2-l1c.s3.amazonaws.com/tiles/35/J/MF/2019/5/12/0/TKI.jp2
2019-09-21T18:18:51 CRITICAL Error : Failed to load S2A_35JMF_20190512_0; HTTPError
2019-09-21T18:23:11 INFO [1/1] Downloading https://sentinel-s2-l1c.s3.amazonaws.com/tiles/35/J/MF/2019/1/12/0/TKI.jp2
2019-09-21T18:23:11 CRITICAL Error : Failed to load S2A_35JMF_20190112_0; HTTPError
2019-09-21T18:26:03 CRITICAL Error : Extent layer is not valid
2019-09-21T18:26:23 INFO [1/1] Downloading https://landsat-pds.s3.amazonaws.com/c1/L8/170/082/LC08_L1TP_170082_20190319_20190325_01_T1/LC08_L1TP_170082_20190319_20190325_01_T1_thumb_large.jpg
Thanks for developing this plugin, looks really useful!
I can't get request Landsat or Planet using Staccato, here is my Messages console
2019-09-21T18:08:35 INFO Plugin installed successfully
2019-09-21T18:09:17 CRITICAL Error : Extent layer is not valid
2019-09-21T18:10:14 INFO Plugin installed successfully
2019-09-21T18:13:32 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:13:45 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:13:54 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:14:26 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:16:15 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:16:26 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:16:44 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:16:57 CRITICAL Error : Network Error: [400] Bad Request
2019-09-21T18:17:04 CRITICAL Error : Network Error: [400] Bad Request
I'm on Ubuntu 18.04 and QGIS 3.8 Zanzibar. I drew multiple different sized bounding box extents using the tool to test it out, I think only the first box was invalid because I didn;t get the Extent layer is not valid
error multiple times. I'm using the most recent version of the plugin.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.