eccc-msc / geomet-data-registry Goto Github PK

At the moment, when we want to add a file in the tileindex (geomet-data-registry data add --file /path/to/file.grib2) and this file is not part of the configuration, the script fails with a KeyError.

For example : KeyError: 'CWAT_EATM_0'

These error shoudl fail gracefully.

CC @tomkralidis @Dukestep

Radar files are being indexed in ES but each file has a null value for register_datetime.

This is because we are not setting self.filepath in the radar handler.

Hi,

I'm using the leaflet.timeDimension plugin to make WMS requests of RADAR precip data from geomet. An issue I've found is with the plugin formatting the ISO8601 datestring with milliseconds, even though the value of milliseconds are always zero. E.g. &time=2019-07-23T11:50:00.000Z. The problem is that (at least for the layer I'm testing) geomet does not accept that 2019-07-23T11:50:00.000Z is a valid time, instead only accepting values with no milliseconds, e.g. 2019-07-23T11:50:00Z. I'll find a workaround in the meantime by modifying the plugin code, but I'm wondering if your service can be configured to fully implement the ISO8601 standard, including milliseconds, and make equivalence checks that can handle varying amounts of precision provided. E.g. 2019-01-01T00Z == 2019-01-01T00:00Z == 2019-01-01T00:00:00Z ==2019-01-01T00:00:00.000Z

Thanks, and apologies if this Issue is opened on the wrong repo.

When we add a new feature to our tileindex, looks like all the features register_datetime is updated, not only the one matching the filename.

we need to fix the _update_by_query filter to only update register_datetime where the filename is exactly the same.

CC @tomkralidis @Dukestep

Giving we are thinking of downloading on-the-fly data when we get a GetMap or GetFeatureInfo, we should add a property in ES mapping for the source data.

For example, this source data could be the https path to a particular file on datamart.

This information (I think) could come from the amqp notification we get, something like parent.msg.baseurl. This could then be passed to the handler as a variable and written to ES for a feature.

The befit here is that we would only need to modify general files, like the layer base.py and the handler. And if the source changes, it will automatically be updated because it's directly from the amqp notifications.

We could call that property source_path ? Other ideas ?

cc @tomkralidis @Dukestep

in layer/base.py we should add a error logger for when a file is added in a new model run and the previous model run did not get the total required number of files.

This way we could know if we missed a layer from the amqp feed.

We should add in the code a condition that add a key in the store (Redis) with the latest timestep which will act as default wms time for the mapfile and will enable us to see if we missed any radar composite.

Example key could be like: RADAR_1KM_RRAI: 2019-10-23T12:10:00Z

Recent changes introduced in #40 is causing issues when using the directory (-d) flag with the CLI tool.

i.e geomet-data-registry data add -d /data/geomet/feeds/dev/ensemble/.../00 errors out when a variable cannot be found and all further processing stops.

Code (example from layer/reps.py):

        if self.wx_variable not in var_path:
            msg = 'Variable "{}" not in ' \
                  'configuration file'.format(self.wx_variable)
            LOGGER.exception(msg)
            raise LayerError(msg)

Instead I suggest the identify method returns False, effectively stopping the processing for that file since handler/core.py only goes on to register if identify returns True. We could still log a message saying that the variable could not be found (maybe a warning instead of an exception?).

Thoughts @tomkralidis & @RousseauLambertLP?

We are currently calculating the register_datetime property of a tile in the tileindex by using elasticsearch.update_by_query once the file is indexed.

i.e File is registered in ES --> calculate datetime.now() --> update file's document in ES with es.update_by_query().

I suggest that we look at a different approach that could use ElasticSearch's ingest pipelines to calculate register_datetime at the time the document is going to be indexed. This would avoid having to do call the update_by_query call and remove the need for the refresh='wait_for' argument when we call es.index() for the received file.

We would have to set this pipeline during the initial creation of the index.

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html and https://kb.objectrocket.com/elasticsearch/how-to-create-a-timestamp-field-for-an-elasticsearch-index-275

@tomkralidis, @RousseauLambertLP, thoughts on this?

This new repos will need an updated README.md to match GeoMet Data Registry content.

Currently, an indexed document looks like this

    "properties": {
      "elevation": "2m",
      "identifier": "GDPS.ETA_TT-20190701000000-20190701000000",
      "reference_datetime": "2019-07-01T00:00:00",
      "file_creation_datetime": "2019-07-01T03:46:46.100000",
      "filepath": "/data-san/geomet/dev/feeds/model_gem_global/25km/grib2/lat_lon/00/000/CMC_glb_TMP_TGL_2_latlon.24x.24_2019070100_P000.grib2",
      "forecast_hour_datetime": "2019-07-01T00:00:00",
      "member": null,
      "identify_datetime": "2019-09-27T18:55:51.066768",
      "receive_datetime": "2019-09-27T18:55:50.893790",
      "layer": "GDPS.ETA_TT",
      "register_datetime": "2019-09-27T18:55:51.067903"
    }

I suggest that all datetimes for each layer handler (i.e forecast_hour_datetime, reference_datetime, file_creation_datetime, identify_datetime, etc...) should contain the Zulu designator (Z).

For ElasticSearch, this will allow us to more strictly define the format expected in the ES mapping. For example:

                    'reference_datetime': {
                        'type': 'date',
                        'format': 'date_time_no_millis'
                    },
                    'file_creation_datetime': {
                        'type': 'date',
                        'format': 'date_time'
                    },

Unfortunately, datetime does not seem to support this out of the box and the only solution I've found is to isoformat() + 'Z'. I don't find this very desirable. @tomkralidis, any other options?

According to ES documentation, the process to insert new entries in ES might be faster if let ES assign the _id field by itself instead of assigning it ourself.

I don't see any impact of letting ES create the _id, any thoughts @tomkralidis and @Dukestep ?

Variables with an interval of PT0H (ex. HGT_SFC_0 in the model_gem_global store YAML config) causes the model_gem_global.py layer handler to error out.

We should modify the ES tileindex mapping to add a "model" column so that we can easily filter a particular model for development and analysis.

This means tat we have to modify the ES mapping and every model handler to add this column.

CC @Dukestep