rivm-syso / atdatabase Goto Github PK

If data is requested but the API call results in a empty data set then what should we do? Know this situation is just ignored and NULL is returned, see:

However, the cache table is not updated and the sensor/range can be requested again and again (which takes time). What if the requested data is simply not available? And what if an empty data set is returned due to a hiccup of the API (and, in theory, can be retrieved later)?

The name of the sensor table is somewhat ubiquitous, rename it into location (that's what it is about)

Database tables should have indices

@jspijker , Nice and clear vignette. Here some suggestions

• Add an extra column to each table (in the explanation) to mention the format. You can read it now in the text, to be extra clear place it in the table, too.

• We discussed about the name of the station table, to rename it to location table. At some points in the text the name ' station table' is still used. (In the text in paragraph: Downloading and storing sensor info)

• For the lat, lon and the end, start time, are that separated columns in the database table, or are those lists-like?

• How to get the meta data from your download stored? The download_data function only takes the measurements. The same question for the location table.

• Minor detail there are some typos

In the explanation of the tables, there is said that for the measurements table, only 1 aggregation level per station/parameter is allowed. Why is that?

For the station table, I assume that the " timestamp" in this table is in seconds, too?

add_doc doesn't check if document already exists, resulting in duplicated documents, and hence, garbage JSON

To keep the caching size manageable, a last in, first out policy can be applied. After a certain time span data will be removed from the cache. But how to track when data was downloaded. Maybe attach a UUID and timestamp to each download?

Every time this function is used, new locations are stored. Even in these locations already exist. Create a check if a new location already exists, based on coordinates, and only add new location (with timestamp) if coordinates differ (i.e. the station has moved). If the coordinates are the same, only update the timestamp.

I see the timestamps saved as integer in seconds,

What about the information of the timezone?

Is assumed the timezone is UTC?

So you can use lubridate::as_datetime(timestamp) to transform the timestamp to a date?

Current sample set is relatively large and not added into repo. Create new, smaller set. Set is needed to build vignette.