We currently run ruff checks in the CI using chartboost/ruff-action@v1, pointing to ./src. This triggered an issue where it was checking .ipynb files that we didn't want checked. We should configure how ruff runs its checks more explicitly, probably in the pyproject.toml file.

Currently we do not ingest any data from the PCS into the data stored in the archive. @jameshod5 wrote a script to parse the data out of the old system and write it out as a Zarr file. We need to integrate this into the ingestion scripts.

Add license:

• - To the website
• - To the s3 bucket
• - To the database.
• - To the shot files.

Currently if we get a very large result the response is paginated. This is not great if I want to get all of the results as dataframe in memory. We can maybe do better by providing a file response or streaming response to a query:

https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse

Hugging face has the datasets library which can be used to share and download datasets in a nice way for ML applications. We can write our own dataset repo and plugin for this.

Graham McArdle has provided a text file containg information about how shot settings were modified.

From the email chain with Graham:

there is a history.txt file that is continuously updated with info about edits made when PCS is running. I've assembled all the changes in the history of MAST into a file on my freia home drive ~gmcardle/mast_history.txt (it's too big for email attachment)

As I mentioned the other day, each of the shot setup files also record changes but they recursively repeat the history of shots that were restored. There are pros and cons of each here. The history file appears to be more robust and monotonic, but it also contains history of irrelevant changes made for fiddling about and running test shots (note that shot numbers in the 900000 range are test shots). If you specifically want to look at changes from one shot to the next you would get that from this history file, but if a shot is restored from an earlier shot you'd have to look back at that earlier shot to find what was done. Where this falls down at times is that we can also prepare a setup offline and restore the setup. In that case the change history of that prepared setup can't be found here but is embedded in the shot setup file.

I'm not entirely sure what you'd like to do with this data but please let me know if you think the shot setup history content is also useful. As I mentioned, there are also issues with shot setup history corruption but I might be able to repair some of that based on the contents of this contiguous log file

We need to investigate this data and consider how we can sensibly integrate this information.

This is for pushing through the IP whitelisting

Currently, the ingestion script for parsing data from pyuda is failing for the M7 campaign. This is blocking us proceeding with the ingestion of more data into the s3.

title

We should investigate a data versioning system like lakefs

At the moment, the tests are testing the API on the full database (which will change). Instead we should create a test database that will store some shots, and test the API on that instead.

The following link has an example we should follow:

https://fastapi.tiangolo.com/advanced/testing-database/

Adding this as a more generic issue (I've closed the LakeFS specific investigation).

We need a data versioning system for our data that can integrate with our current S3 storage, and not add too much complexity for the user to access our data

A lot of the units are inconsistent and could be simplified. Take a look at the result of this query

https://mastapp.site/json/signals/aggregate?data=units$distinct:,units$count:&groupby=units

We should at least try and map them to a minimal consistent set, or do the best we can.

We have already started to use podman for Mac users on the development side of things, and have changed the README for this.

The production set up still uses Docker though, need to look into if we can straight swap Docker for Podman for the commands.

When bulk downloading the data, it can be quite slow, especially when we are being selective with the data stored in s3.

This is because we have many, many small files. When we want to download everything we have to page through all the of the keys in the s3 which can take some time.

I am wondering how we can make this process more performant.

• One solution is to have a file index and query this, but that would incur introducing an access layer.
• Another solution is we push users towards local caching and they just load what they need:

endpoint_url = f"https://s3.echo.stfc.ac.uk"
url = 's3://mast/test/shots/tiny/30390.zarr'
ds = xr.open_dataset("filecache::" + url, engine='zarr', group=f'rba', 
                     storage_options={'s3': {'anon': True, 'endpoint_url': endpoint_url}, 'filecache': {'cache_storage':'/tmp/files'}})
ds

The second solution has implications for how you access the data on HPC systems. For CSD3 we should have an internally facing s3 storage. For other sites they will need an internet connection.

After working with Ale on Defuse we identified a few signals that are both useful and can be transformed into a nice representation.

We should upgrade the ingestion workflow to make these transforms

For example, we now know how to transform

• LCFS
• xsx tangential and horizontal cameras

In this issue we should also address mapping to different signal names (I.e. imas).

We should be able to map both from uda and from zarr

We should also address mapping units.

We should support re ingesting only a subset of signals/datasets

Look at the outputs of DEFUSE and how we can integrate that into the metadata/database.
IMAS still does not have this, so we should decide that ourselves.
Finding more to feed into DEFUSE.

We have missed a few of the different names for time axes. We should create a mapping file for this and try to fix the missing ones. For example, AYC_TE_CORE is not normalised.

We should make the script and mappings file similar to how we will handle units in #22

We need something that shows what sources are called and what they represent in the docs.

e.g. ayc -> Thomson Scattering

Let's take a more detailed look at SciCat and how this can facilitate future requirements for managing our data.

... just to spare us waiting for environments to build.

Nathan has a tool for parsing the provenance data out from each of the signals on Freia. We need to modify this tool to be parallelised across shots to be more efficient so that we can run it over the entire MAST back catalogue.

Such as airflow or galaxy.

See here:

https://www.w3.org/TR/vocab-dcat-3/#Class:Data_Service

Investigate how we can use podman instead of docker desktop. We need to make sure we can still use docker compose. Please make sure to write some documentation in a readme to describe how to deploy it!

uv can now install python. It likely makes sense to start from a base image like alpine, then install uv, then create venv with python version of our choosing.

For example:

http://localhost:8081/json/shots/filters=shot_id$leq:30000

Produces the following output:

Which is not very informative. Can we do better?

Need to start building a CI pipeline. Best to start with the simple things like linting and formatting (my vote goes to ruff as it covers all the things that black, flake8 and isort do) and then probably look at testing just the API. May need to spin up a temporary database using Sqlite or similar for this. See #3.

Make it so the user can directly query the database with pandas.

We need a read only user
We need to add intake definition to abstract away the boilerplate
We need to add user examples, including using duckdb

The docker compose config still sets up a minio instance. Now that we have our Ceph storage at STFC, is this still required?

It could be useful for testing, as a build-up and tear-down option.

Pros:

• We won't need to have test data in the production store.
• Tests are still valid if we move storage elsewhere.

Cons:

• Need to figure out how to handle test data. Git LFS/pull from production/etc...
• Another component to understand and manage.

When accessing the list of signals (https://mastapp.site/json/signals), which contains a very large number of records, it is noticeably slow to load. I have traced the source of this slow loading to how we are currently doing pagination.

Currently, we are paginating using offset pagination where we have to count the number of items in the table for every query. The slow line of code is line 177 in the snippet below:

I think the correct solution is to swap to using cursor based pagination. See the difference here. For most tables we should be able to use the UUID as the cursor. For shots we might use the shot_id, although for consistency it might be nicer to give shots UUIDs as well.

The unit tests are currently failing because I've changed stuff and not kept up with them. Please can you help fix these tests. It is most likely the test and not the code which is wrong at the moment.

To run the tests you can do:

python -m pytest tests

See what breaks and make them all green!

This will likely be a common use-case. Although, we should perhaps also add a section that says, "Did you know about lazy loading? Build your pipeline like this so you don't have to bulk download!"

Context: We have moved to using fastapi.paginate for the requests now, which is straight forward for most requests and provides cursor based pagination for our responses. This paginate function only needs the database and the query. The function needs to return a CursorPage[model] object.

Problem: When trying to do cursor pagination for aggregate queries however, it returns an error about columns being missing based on what the aggregate query is asking for:

mast-api | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "min_shot_id" does not exist
mast-api | LINE 1: ..._id) AS min_shot_id, max(shot_id) AS max_shot_id, min_shot_i...
mast-api | ^
mast-api |
mast-api | [SQL: SELECT min(shot_id) AS min_shot_id, max(shot_id) AS max_shot_id, min_shot_id AS _sqlakeyset_oc_3
mast-api | FROM shots ORDER BY _sqlakeyset_oc_3 DESC
mast-api | LIMIT %(param_1)s]
mast-api | [parameters: {'param_1': 51}]
mast-api | (Background on this error at: https://sqlalche.me/e/14/f405)

Current attempts: The first attempt was to create a model based on what we should expect from the response, however creating this dummy model did not fix the issue.

How to reproduce:

• Using the alternate aggregate function for shots:

@app.get("/json/shots/aggregate")
def get_shots_aggregate(
    request: Request,
    response: Response,
    db: Session = Depends(get_db),
    params: AggregateQueryParams = Depends(),
) -> CursorPage[AggModel]:
    
    query = crud.aggregate_query(
        ShotModel, params.data, params.groupby, params.filters, params.sort
    )
    return paginate(db, query)

• Use the request: http://localhost:8081/json/shots/aggregate?data=shot_id$min:,shot_id$max:&groupby=campaign&sort=-min_shot_id