in geocoder admin, add support for export/import configuration data about api-specs HOT 28 CLOSED

@cmhodgson @mraross I would prefer import/export as json file, it can be edited/verified with JSON editor like http://www.jsoneditoronline.org/ or https://chrome.google.com/webstore/detail/json-editor/lhkmoheomjbkfloacpgllgjcamhihfaj?hl=en

from api-specs.

I agree with Leo. JSON accommodates changes to the config file schema more gracefully than csv.

from api-specs.

I can't believe you are telling me that JSON is easier to edit than CSV but if that's what you prefer I can do that.

from api-specs.

I’m not. Leo is. I’m saying that because JSON is self-describing, if new properties are added to the schema, a json ready can read it without problem.

From: Chris Hodgson [mailto:[email protected]]
Sent: Wednesday, August 17, 2016 3:41 PM
To: bcgov/api-specs
Cc: Ross, Michael RA JTST:EX; Mention
Subject: Re: [bcgov/api-specs] Need a way to transfer config data between BCGov and Refractions (#27)

I can't believe you are telling me that JSON is easier to edit than CSV but if that's what you prefer I can do that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/27#issuecomment-240571288, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKy1LrZilF65svaw3uRW1lNW1tmzwjGlks5qg43lgaJpZM4JmpM4.

from api-specs.

Here is a sample of the export I intend to use, please let me know if this is acceptable.
(note it downloads as a .json but I cannot attach .json files here so it has been renamed as .txt)
ols_admin_config_export.txt

from api-specs.

Using http://jsonlint.com/ , I load the json file, press validate JSON, and get the following error:

SyntaxError: Duplicate key 'BGEO_ABBREVIATION_MAPPINGS' on line 1419

from api-specs.

Leo, please review json format.

from api-specs.

Revised export sample:
ols_admin_config_export.txt

from api-specs.

jsonlint says format is valid so it looks good to me.

from api-specs.

structure and format looking good, @cmhodgson, @mraross did you need to verify the content?

from api-specs.

This is an example as extracted from my local cassandra. The intent that once this is added to the admin UI, I can extract exactly what is in YOUR cassandra, and include it in my delivery as the default starting data (in the case of starting from scratch with an empty cassandra DB). The contents are not important. You specifically requested that I use JSON format; I just wanted to make sure this is acceptable.

from api-specs.

Would a date/timestamp be useful to be able to tell when an extract was done?

from api-specs.

@cmhodgson ok, that's fine. @mraross date/timestamp will be created when file is created on filesystem. is there way to ensure the integrity of the file?

from api-specs.

I like the idea of including a timestamp, as the file's create date is not always reliable in the case of copying, etc. As far as the integrity goes; it will have to parse as valid JSON, it is unlikely that a random section could be lost and still be valid JSON. I think we actually need/want to be able to make changes to the file, as that is how I would add a new config parameter. I could include a row count per table as a basic integrity check, that would be easy enough to modify when adding an additional record.

from api-specs.

I like the table row counts for integrity provided you use them on import to verify the expected number of rows were present.

from api-specs.

Yes, of course the intent is to validate the actual number of rows of data against the expected number before modifying the actual database. I've also provided an option to just validate an export file, possibly useful.

from api-specs.

Validation support would be very useful.

from api-specs.

Confirmed in delivery.

from api-specs.

Did an export then a validation which produced this report:

Validate Configuration
Validation Successful

Messages

BGEO_CONFIGURATION_PARAMETERS count: 117 (matches expected)
BGEO_ABBREVIATION_MAPPINGS count: 324 (matches expected)
BGEO_UNIT_DESIGNATORS count: 18 (matches expected)
BGEO_LOCALITY_MAPPINGS count: 4596 (matches expected)

My understanding is the check count in the file matched the actual count for each category of config variable. Could this be displayed a little more explicitly as in:

Including the geocoder actual counts helps you compare a config file you just exported to the current geocoder configuration.

from api-specs.

After our phone discussion Michael and I agreed that the "validate" step would perform a diff between the file and the current config data, and show the specific differences, ie. keys with different values in the two places, or keys that exist in one but not the other. In addition to providing the summary count data specified above.

from api-specs.

How does this look Michael?

from api-specs.

Looks great. Can you include the config file name and its creation date in this report?

from api-specs.

The File name and creation date are now included at the top of the validation/diff report.
The import function now requests a confirmation before submitting.
Confirmed in Delivery.

from api-specs.

copyright character in file not matching live config. Is there a character encoding issue here?

from api-specs.

The copyright character looks fine in the config file when I load it into pspad. Maybe its just the text comparison code?

from api-specs.

The export was in the default 8859-1 encoding instead of UTF-8 which I was expecting the input to be in. I updated it to export in UTF-8, now the comparison works. Just waiting to get it built in delivery...

from api-specs.

Export and validation now work since validation now reports no differences. JSON also reports that config file is valid json.

from api-specs.

Verified

from api-specs.