alexanderkotsev / sta_client Goto Github PK

Thought this would be the best place to leave a few comments.
First of all: glad to see both STA and Python here!

Hopefully understanding what the sta_client does: ASE DataPush from SQLite DB to remote STA Server, keeping time of latest Observation sent within the STA Server, using that time to read from SQLite.

Within the Smart Emission project we have developed a STA Client with very similar functionality to read calibrated/aggregated (hour-values) measurement records from PostgreSQL DB and publish to a remote STA Server, including the time sync by tracking the latest measurement record id (no need to use time). This client was integrated with both SensorUp and GOST STA servers. We took some different approaches, this may be of help.

The STA-T Client implementation is part of a Stetl pipeline such that Input (Postgres) and Output (STA) is separated.

Some general suggestions from our experience what I will call SE STA Client.

Lazy STA Entity Creation

sta_client assumes all other other STA Entities are already present in the Server (Things, Sensors, Datastreams etc). This would require some (manual) administration upfront, plus carefully configuring Datastream id's etc.

SE STA Client on init will check if any Entities already exist within the STA Server, create them (lazily) if not.

STA Requests Using Templates

Based on our experience with SOS-T (and WFS-T) we found that building up (POST/PUT) request-bodies programmatically, requires lots of code and is error-prone. For that reason we have been using template requests, basically a full JSON (or XML/GML) request with parameters symbolically defined. See for example the STA Request templates.

At runtime, via standard Python String formatting the parameters are filled-in. In other projects even Jinja2 templates were used when having to deal with lists and optional elements.

Authentication

Think this should be implemented, i.e. not leave the STA Server open for anyone to modify.
For that we have shielded POST, PUT, PATCH, DELETE HTTP verbs.
The Python requests package has great support for e.g. Basic Auth.

Update GPS Location

As ASE may be moved it is important to dynamically update Location, initially from Lazy STA Entity Creation and then from the GPS ASE data. In SE STA Client the Location of the Thing is updated in each session using PATCH (though we could have used HistoricalLocation as well.

Using Thing.properties for Sensor Station Id and latest update time

In SE STA Client we use Things free properties field to keep for example the local ASE Id and the latest update time. This makes it easy to "find back" the Thing for a specific ASE and to keep track of latest update time. Getting the latest Observation for that is a very expensive operation once there are millions of Observations (at least in GOST now). (though for time-sync we use a local database record with the latest record id of Observation successfully sent).

Concluding

Don't see this as critic, the above suggestions were based on my experience since 2014 working with SOS-T and later STA-T. Through the years found patterns that worked, with minimal manual intervention and administration.

The Stetl framework has helped a lot. The ASE Datapush in general would be an ideal Stetl-case...