geoscienceaustralia / dea-knowledge-hub Goto Github PK

[email protected] is listed at https://www.dea.ga.gov.au/contact , but nowhere in the user guide.

There's currently a bit of duplication between the content in this guide:

https://github.com/GeoscienceAustralia/dea-docs/blob/develop/setup/jupyter.rst

And the content in the 01_Jupyter_notebooks.ipynb notebook in the Beginners_guide:

https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/Beginners_guide/01_Jupyter_notebooks.ipynb

Do we mind? Should we combine these into one?

The examples are provided as Jupyter Notebooks, which can be run in either instance. Jupyter Notebooks contain code and explanations side-by-side so that you can understand the steps involved in an analysis. Our Jupyter Notebooks page provides a quick introduction to using Jupyter Notebooks. For a more detailed introduction, visit the Jupyter Notebook Documentation.

Background

In Knowledge Hub, we currently do a pretty good job of catering to our QGIS users, e.g.:

• https://knowledge.dea.ga.gov.au/guides/setup/gis/web_coverage_service/#how-to-connect-to-wcs-using-qgis
• https://knowledge.dea.ga.gov.au/guides/setup/gis/web_map_service/#how-to-connect-to-wms-using-qgis

However, many DEA users use ESRI software like ArcGIS Pro, which has its own different ways of accessing our data. ArcGIS Pro users currently need to work out how to load our data themselves, which can lead to a frustrating user experience (or inability to access DEA data at all).

Similarly, we currently don't provide guides on accessing DEA data via STAC and AWS in either QGIS or ArcGIS Pro, despite these being powerful and easy ways to access DEA data.

Proposed improvement

It would be extremely valuable to provide matching instructions for accessing DEA data using QGIS and ESRI software wherever appropriate. Ideally, I would love to update our docs to include subsections like this:

# SpatioTemporal Asset Catalogue (STAC)
## How to load data in Python using `odc-stac`
## How to load data using STAC in QGIS
## How to load data using STAC in ArcGIS Pro

# Amazon Web Services (AWS)
## How to load data from AWS using QGIS
## How to load data from AWS using ArcGIS Pro

# Web Map Service (WMS)
## How to connect to WMS using QGIS
## How to connect to WMS using ArcGIS Pro

# Web Coverage Service (WCS)
## How to connect to WCS using QGIS
## How to connect to WCS using ArcGIS Pro

Future work could also involve documenting how to load DEA data using other programming languages, e.g. R:

# SpatioTemporal Asset Catalogue (STAC)
## How to load data in Python using `odc-stac`
## How to load data in R using `gdalcubes`
## How to load data using STAC in QGIS
## How to load data using STAC in ArcGIS Pro

Open Data Cube Slack is now deprecated and replaced with Discord. We should replace all Slack links with Discord links:
https://discord.com/invite/4hhBQVas5U

When I try to search for something in the DEA User Guide, the search page opens, but no results are ever returned.

It looks like something has broken with the build of the site. If I look at the Javascript Console in my browser, there's an error message:

search.html?q=landsat&check_keywords=yes&area=default:146 Uncaught ReferenceError: jQuery is not defined
    at search.html?q=landsat&check_keywords=yes&area=default:146:7

JQuery is a Javascript library that should be included on the page, but for some reason is not. It could be something broken with the theme we use, or an incompatibility between something else and the theme.

I believe NCI is moving from Strudel + TurboVNC to OOD:

https://opus.nci.org.au/display/OOD/OOD+FAQ

A placeholder has been added to the doco saying we will update the connection instructions in the new year
https://docs.dea.ga.gov.au/setup/NCI/install.html

Adding a GitHub issue so that it's not forgotten.

The new DEA Changelog is fantastic, and provides a great place for us to document changes and updates across DEA. However, we do run the risk of forgetting to also update the DEA Sandbox "Updates" banner, which is the other high profile location where we share updates with our users:

I would like to propose that we add two additional checklist items to the current pull request template, to encourage (but not force) users to keep our update sections up to date:

• If required, update the DEA Tech Alerts and Changelog page
• If required, update the DEA Sandbox Updates banner

The following commit prevented updates to material on DEA Notebooks from being published to the DEA User Guide:
f6233fb

We temporarily reverted this committ so that we could publish the docs here:
ea3eba2

However, this should be revisited so that the previous changes can be kept in a way that allows the docs to publish as intended.

References to Jupyter Notebook should be replaced with JupyterLab, e.g.:

https://github.com/GeoscienceAustralia/dea-docs/blob/develop/setup/NCI/nci_basics.rst

This will also require a change to the bash install script on the NCI:

https://github.com/GeoscienceAustralia/dea-docs/blob/develop/setup/NCI/install.rst#setting-up-dea

New VDI made changes that stopped the VDI from working for some users. It is likely for some external users to cause issues, and maybe if we put a comment in the docs linking to the changes this will help them!

https://opus.nci.org.au/display/Help/4.+VDI+FAQ

The recent upgrade to the documentation on accessing DEA from ARE is very welcome. However, recently I helped a PhD student at ANU get running on the NCI and the experience led to a few suggestions to improve the current docs.

• This sentence is confusing for the user: It's also possible to use DEA on the NCI's Gadi HPC Supercomputer, however unless you're experienced with SSH and HPC systems, it's much easier to get started with ARE. Unlike the earlier iteration of Jupyterlab on the NCI which was hosted on NIRIN Cloud ("OOD"), ARE is on GADI and has full access to the supercomputer so this is something of a false dichotomy that leads to confusion. In my opinion, the docs should start with something like..."There are three ways to access DEA on NCI's GADI: VDI, ARE, or command line, here's how you do those..."

• Overall, the docs for ARE are good, but for a new user the docs are a little sparse. I think more explicit detail might be necessary to help a new user get started (my test subject found it confusing without me guiding them through it). For example, some description of why you need to add v10 to the Storage options, or what the DEA environment is (just a sentence or two helps a lot), etc.

• It would also be helpful to have a section on cloning dea-notebooks into a folder on ARE. As well as mentioning the need to change the sys.path.insert() location in dea-notebooks to make dea_tools work i.e.,

import sys
sys.path.insert(1, '/g/data/path/to/dea-notebooks/Tools')

• For a completely new user getting started on ARE, is there a need to generate a pygpass in the home directory to get access to the database? I assumed this was created when a user runs sh /g/data/v10/public/digitalearthau/install.sh on the VDI (but I'm not sure). If so, then this step would need to be added as a first step whether using ARE, VDI, or command line.

• Lastly, this is not directly related to the docs, but the default dea environment still hasn't been updated (it loads 20221025 env by default), which results in ModuleNotFoundError: No module named 'odc.geo' errors when trying to run any of the dea-notebooks. This is a huge blocker for people to get started as working around the issue is quite tricky if you're unfamiliar with managing python environments.

Happy to review any further updates to the docs

Thanks,
Chad

All references to the Raijin HPC needs to be replaced with Gadi. For example:

https://github.com/GeoscienceAustralia/dea-docs/blob/develop/setup/NCI/nci_basics.rst