see: #26 (comment)

In moving from NetCDF to more cloud optimised datasets like parquet we need to address the changes in how "meta-data" is addressed. The bottom line is that without the global attributes available in NetCDF we'll need to cary over, for each record in the dataset, some of this "meta-data" as "data" columns for each spatial and time point record.

The assumption is that while a duplication of bytes in the file and "wasteful" of storage that the real-world impact of the extra size in terms of resource costs or access time won't matter. (??)

To be calculated per individual/tag

• Time at depth
• Time at temperature

Divide summaries based on shelf and off-shelf (200 m bathy line). @fjaine to confirm how to identify on and off shelf locations from summary data.

Check AODN data object for presence in WOD archive

GFW data download has been sorted in #29

Documentation needed with instructions on how to use it.

variables, frontal index, depth, sst, chl-a, salnity, currents, ssh, climatologies?

Start simple . . . . just a dictionary.

• type of data
• list of variables & metadata
• link to to UID geo network record

Comparing the outputs from the manual selection of MLD depth using our GUI to different automated methods.

Here we will need some work done, as we need to find a way for the same profiles that were selected at random to be loaded.

• We are thinking maybe the timestamp or even altering the GUI to record an index for them.
• Other options would be to just slice the moorings data into collections and then run entire collections for testing.

Ultimate plan: send to people as if it were a game, so we can build a good size randomic dataset to use for the purpose of comparison with automated methods.

How can we leverage "lazy loading" with the GetAodn class?

• Can GetAodn provide a lazy data frame object - dask dataframe? Access parquet with dask?
• Can we "show headers" and then load based on column filters? suggestion from @leonardolaiolo

Global attributes from netcdf files don't get carried through to parquet format.
What are the implications for the user with loss of metadata with parquet formats?

Depends on issue #32

Plots to be done per species and per individual/tag

Michael has some code that does this that we can modify or use to close this issue.

To be calculated per tag

Calculations per tag/individual per day

• Distance travelled (cumulative)
• Distance travelled (daily)
• Depth summaries (min, mean, maximum, etc.)
• Temperature summaries (min, mean, maximum, etc.)
• Distance to shelf (200 meter depth)
• Distance to shore
• Maximum depth at each position (sea floor depth)

Explore use of YAML registry files - first steps

In AODN reloaded there will be YAML registry file for each parquet or zarr collection. The ideas discussed by @lbesnard point at the opportunity to make use of the YAML registry files for data discovery and cataloging.

Explore the possibility of creating a catalogue of existing datasets (from the cloud-optimised products) and accessing them via Python.

• Read about STAC catalogues
• Read about Intake catalogues
• Implement a PoC catalogue in one/both of the above
• Have a look at intake-stac as well

Create a Parquet version of the IMOS - Moorings - Hourly time-series product

Looks like the DMS have also done this, but probably only for sites on the GBR: https://github.com/aodn/rimrep-examples-private/blob/main/examples/poc/python/anmn-ltsp-hourly.ipynb

Figures TBA

Explore the "parquet-ready" datasets - what do we have today?