waagsociety / citysdk-ld Goto Github PK

The CitySDK LD API stores data per object, per layer. An object is something with a physical representation in a city (train station, bus stop, building, etc.), each object can have one or more layers with data.

We plan to define the following layers, based upon GTFS:

1. gtfs.stops - layer containing all GTFS stops, plus basic data (stop_id, stop_name, etc.)
2. gtfs.routes - layer containing all GTFS routes, plus basic data (route_id, agency_id, etc.)
3. gtfs.stops.routes - extra layer for each GTFS stop, containing routes calling at stop
4. gtfs.routes.stops - extra layer for each GTFS route, containing stops for route in right order
5. gtfs.stops.schedule - work-in-progress, layers 5 & 6 contain schedule data (probably for current day or week), for routes and stops
6. gtfs.routes.schedule
7. gtfs.stops.now - current departure times, per stop

When possible, we should use OpenTransport's vocabulary.

Use RuboCop to make code guideline-compliant!

Go to project directory, and run rubocop.

To override RuboCop's default settings, edit .rubocop.yml.

Improve map viewer:

• show layer semantics (JSON-LD, RDF/Turtle);
• select layers;
• apply filters.

GTFS code is old! Refactor!

GTFS code also includes lots of large SQL queries. We should move those to postgres functions.

Categories + subcategories in write API and all serializations

Write preliminary v1.0 documentation:

• URI structure;
• concept;
• serializations;
• Semantic data - JSON-LD & RDF/Turtle;
• missing pieces.

If a layer is deleted which has dependant virtual layers, those should be deleted too.

See http://api.citysdk.waag.org/layers/parking.garage/objects?per_page=25

Add column default to table fields, of type json.

The execute_write functions in models (owner.rb, layer.rb) now check for required and optional fields in POST data on write requests. Let postgres handle checking if too many (or few) fields are send to DB, not Ruby.

Object gtfs.stops.14894 has data on three layers (http://api.citysdk.waag.org/objects/gtfs.stops.14894/layers/):

• gtfs.stops
• gtfs.stops.now
• gtfs.stops.schedule

But http://api.citysdk.waag.org/objects/gtfs.stops.14894 returns only 2!

i.e. /context maybe should return:
{ "@context": { "@vocab": "http://blabla"} }

Manually copy old documentation markdown files from old API (v0.9) to wiki.

Create endpoint status page. Also incorporate data from daemons and web services (see #16).

Generate JSON-LD context (or add to existing JSON-LD context) when a layer has fields.

v0.9 had support for routes, objects which had one or more child objects. This is used for GTFS line/stop relationships, OSM relations and is useful for API users who want to add data to a set of objects.

• Add API functions to support creating and deleting sets,
• Add DB constraints to make sure objects cannot be deleted when still in a set,
• Decide what happens to objects without data which are still in a set.

in the S/P/O the P and the O do not handle the @base URI correctly

http://api.citysdk.waag.org/layers/admr/fields/LayerData
should be: cdk:LayerData

http://api.citysdk.waag.org/layers/admr/fields/admn_level
should be: <layers/admr/fields/admn_level>

Modify map viewer to support v1.0 GeoJSON output. Use Leaflet's GeoJSON layer to do this.

• http://localhost:9292/objects.json is allowed, while
• http://localhost:9292/objects/1234.json is not.

Grape, by default, allows URLs to end with a file extension. This does not work well together with URL patterns ending with a regex pattern:

resource '/:cdk_id', requirements: { cdk_id: ::Helpers.alphanumeric_regex } do
   ...
end

Or, maybe we should not allow file extension URLs and always use the format parameter (?format=json) or the Content-Type request header.

In line with calls to /layers/ and /layers/:layer (the first returns a JSON array, the second a single JSON object), should we decide to return a GeoJSON FeatureCollection on calls to /objects and a single GeoJSON Feature on calls to /objects/:cdk_id?

Currently, all object calls return FeatureCollection.

Make sure all errors (Grape, Ruby, Rack, nginx) are correctly presented in output in production environment.

When all JSON-LD serializations are finished, an RDF ontology should be created.

Add text field in layers table (and serializations) for layer's subcategory.

It should be possible to create and edit layers which depend on other layers via the API.

http://api.citysdk.waag.org/layers/gtfs.routes.schedule/objects/gtfs.routes.iffns.18868.0

Using the TODOs in api_spec.rb, create tests for uncovered API calls.

Build API authentication. Users should only be able to do write queries when logged in, and on layers they have write access on.

Use GitHub's API authentication for inspiration.

Grape authentication documentation.

Doing a PATCH request to a URL to which only GET and POST requests are allowed should result in an error. Example:

curl --request PATCH --data "{\"a\": \"b\"}" http://localhost:9292/owners

X-Result-Count 23643
etc,

When i try to run the below command during installation

ruby importers/osm/osm_importer.rb -f ~/Downloads/amsterdam_netherlands.osm.pbf -d

I get the following error:

root@ns553739:~/citysdk-ld# ruby importers/osm/osm_importer.rb -f ~/Downloads/amsterdam_netherlands.osm.pbf -d osm
importers/osm/osm_importer.rb:99:in osm2pgsql': undefined method +@' for #String:0x0000000190c530 (NoMethodError)
from importers/osm/osm_importer.rb:180:in `

Create internal API which lets web services and daemons to report their status into memcached.

Add to wiki text describing API, layers, linking of data to objects, with images (as was available prev.).

Add layer's fields to JSON-LD and Turtle serializations.

Add RDF class/type per layer, and use this class in serializations.

Hi
I am new to this.

getting following error when i try
ruby importers/osm/osm_importer.rb -f ~/Downloads/amsterdam_netherlands.osm.pbf

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require': cannot load such file -- citysdk (LoadError) from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
from importers/osm/osm_importer.rb:6:in `

Every x minutes, run update_layer_hashes, to ensure the latest layer geometries are cached.

If user is logged in and is owner of an object itself, it is possible to edit the object (geometry + title) with PATCH /objects/:cdk_id.

Add call to edit list of objects: PATCH /objects.

Make test for admin and authoritative filters.

Setting the layer_id of an object to -1 when the object's layer or owner is deleted is now done in Ruby code (here, here and here).

Instead, see if it's possible to use BEFORE trigger on objects table to do this.

Build support for:

1. API BluePrint
2. JSON API

First version of JSON API support is already included, but should be improved.

When importing an OSM extract (like e.g. Amsterdam oder an own) using the osm_importer.rb script, I get this output:

Now note that I used a custom user for the postgres DB, which is configured correctly in the config.development.json file. The problem is that the script when calling osm2pgsql uses still the user postgres as default (-U postgres):

osm2pgsql = "export PGPASS=#{config[:db][:password]}; osm2pgsql --slim -j -d #{config[:db][:database]} -H #{config[:db][:host]} -l -C6000 -U postgres #{options[:osm_filename]}"

However when connecting to the db earlier for example, it uses the user specified in the config file:

database = Sequel.connect "postgres://#{config[:db][:user]}:#{config[:db][:password]}@#{config[:db][:host]}/#{config[:db][:database]}", encoding: 'UTF-8'

So it would probably make sense to change that fixed username with the one from the config.
Could be like this:

osm2pgsql = "osm2pgsql --slim -j -d #{config[:db][:database]} -H #{config[:db][:host]} -l -C6000 -U #{config[:db][:user]} -W #{config[:db][:password]} #{options[:osm_filename]}"

(Still in earlier tries it didn't even work with the standard user postgres (when there was one), because no password with the option -W was specified, the result was also failed authentication.)
Now when running this fixed version of the script, it can authenticate itself (although it's curious that I still have to enter the password manually), however the import fails because of other reasons:

Here's the output:

Looks like some tables are missing. . What could be the reason for that??
thank you very much for your help!

See whether it's useful to disallow real large requests (100s of MBs) to the write API by putting a limit on request's Content-Length header.

The database connection is done in api.rb. See if this is the correct place, and how the API behaves after nginx spawns multiple instances.

Grape's and Capistrano's configuration directories are both config. Untangle!