hbz / geodata Goto Github PK

@fsteeg I'm going to come by and ask you for the best how-to. Or maybe that is a good thing for our pair-programming?

Currently, the service is running on gaia. When running locally or on quaoar1 (where the other services run), it hangs. @philboeselager suggested this might be due to the fact that unlike gaia, they are not externally accessible. Any ideas about that, @dr0i?

Provide a /geodata/search route with a q parameter supporting Elasticsearch query string syntax.

See new data module specification at: http://hbz.github.io/#data-module-specification

Geonames is a promising service that might be helpful. Maybe we could add it as a third data source.

To build and test automatically in pull requests.

Though working on DEV machine, umlauts are not processed properly on gaia.

The current geodata index size is only 3.5 MB. To ease deployment and increase self-containedness, we should use an embedded Elasticsearch index as we do in lobid-organisations. Since filling up the geodata index takes time, the index should not be recreated on app start (unlike lobid-organisations).

See and copy relevant parts from https://github.com/hbz/lobid-organisations

Can be copied from https://github.com/hbz/lobid-organisations/tree/master/.settings

See http://hbz.github.io/#dev-process-coding-conventions

In order to use geodata in integration tests (like in hbz/lobid-organisations), this microservice should be accessible from the web resp. under a constant URL. Therefore, find a suitable (sub-) domain and make geodata be accessible that way.

Currently, a new empty index is created on every application's startup, possibly leading to unwanted loss of data.
Preferably, a new empty index only should be created if there is no such index existing, yet.

Also, there should be no deletion of productive data while running tests.

All data, no matter if coming from Nominatim oder Wikidata, should be provided in uniform JSON format.

See new data module specification at: http://hbz.github.io/#data-module-specification

All routes should be prefixed with 'geodata'.

See new data module specification at: http://hbz.github.io/#data-module-specification

All requests to this microservice that no result could be delivered for should be listed. By doing so, notifications to the owners of the requested organisations could be generated.

...to prevent Nominatim being requested subsequently for the same set of data. Currently, that behaviour occurs, because "lat", "long" and "postcode" are being requested in separate, subsequent commands from geodata.

An error is thrown if the postcode cannot be found in JSON coming from Nominatim.

Sample error output:

[error] play - Cannot invoke the action, eventually got an error: org.json.JSONException: JSONObject["postcode"] not found.
[error] application - 

! @711781k9f - Internal server error, for (GET) [/geodata/long/J%205/Mannheim/DE] ->

play.api.Application$$anon$1: Execution exception[[JSONException: JSONObject["postcode"] not found.]]
    at play.api.Application$class.handleError(Application.scala:296) ~[play_2.11-2.3.4.jar:2.3.4]
    at play.api.DefaultApplication.handleError(Application.scala:402) [play_2.11-2.3.4.jar:2.3.4]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.11-2.3.4.jar:2.3.4]
    at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.11-2.3.4.jar:2.3.4]
    at scala.Option.map(Option.scala:145) [scala-library-2.11.1.jar:na]
Caused by: org.json.JSONException: JSONObject["postcode"] not found.
    at org.json.JSONObject.get(JSONObject.java:476) ~[json-20141113.jar:na]
    at controllers.geo.NominatimQuery.getPostcode(NominatimQuery.java:47) ~[classes/:na]
    at controllers.geo.NominatimQuery.createGeoNode(NominatimQuery.java:59) ~[classes/:na]
    at controllers.geo.GeoInformator.getFirstGeoNode(GeoInformator.java:95) ~[classes/:na]
    at controllers.geo.GeoInformator.getLatLong(GeoInformator.java:82) ~[classes/:na]

Coming from hbz/nwbib#83 , we need an additional query technique, collecting geo data from Wikidata.

Should be done on quaoar1, thus this depends on #23

Elasticsearch cluster, index, and server in GeoElasticsearch.

Set up like other Play apps on quaoar1.

Depends on #23

Make sure that not more than one request is sent per time, in order not to violate the Nominatim user policy.

Add a documentation page at geodata/ (see organisations/ example).

See new data module specification at: http://hbz.github.io/#data-module-specification