tilezen / vector-datasource Goto Github PK

I would open this at https://github.com/mapzen/TileStache, but it doesn't have tickets, and this is sort of a higher level question.

OSM data differs from other data in that it can be updated minutely. How is expiry of vector tiles handed in vector-datasource? The traditional raster stacks use a notion of a "dirty" tile that is rendered but not up to date. There are no public vector tile stacks with full diff support, but some providers are rendering the entire planet, then re-rendering any dirty tiles.

https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service makes no reference to diffs, including reading them with osm2pgsql.

Is it possible to use SRID 4326 in vector-datasource? I understand I need to load the data into Postgis with SRID 4326, but what about querying the data? Should I change queries/*.pgsql files accordingly or there is more to accomplish this? My goal is to use an unprojected SRID in the database and project on the fly either at the application or at a client level. I don't want to use 900913 in the database, I'd like to store unprojected data in PostGis. And another question: what are the drawbacks with this approach?

Thanks!

The SF zoo is missing, and the animals there are sad.
tourism:zoo

I can figure how the URL /tilejson/tilejson.json is served with actual code. Need an additional ruby server or preprocessor to serve this file with the python server part ?

Adding ramps to the cartography at zooms less than 15 is problematic now because they often z-stack order over highways, adding noise that visually breaking up the highway line. Not displaying ramps is not ideal as there are often highway segments that connect via ramps and it looks like we're missing data when the ramps aren't shown at zooms 12, 13, 14. Ignore the OSM layer information when calculating the feature sort at zooms will allow ramps to sort under highways for these zooms.

We need to pass in the zoom level to mz_calculate_road_sort_key() and only exercise
https://github.com/mapzen/vector-datasource/blob/master/data/functions.sql#L271
if the zoom >= 15
else do the simple bridge/tunnel & highway type sorting.

Example near: San José, California at 13/37.2554/-121.8593

Currently we don't show ramps, creating a visual gap:

Adding the ramps in fills the gap, but too much visual noise at Blossom Hill Road:

I've been recommended this project, but there's no instructions for use / basic walkthrough.

It's pretty easy to figure out the higher level-stuff (where you store queries, etc.) but the lack of tutorial has me hesitant to spend the time forking it and figuring out all the pieces.

When a road intersects a landuse polygon, split the road up and tag the segment inside the landuse polygon with the polygon's landuse kind property.

Light grey roads are hard to see crossing a park, and white casings that look good over generic earth background pop too much over green parks.

Previously in #126 we suppress parking POIs until zoom 17. Turns out some of these are landuse labels (from relation AOIs) and we need to do this for the other data type, too.

Ways with the tag railway=preserved are not included in the vector tiles.
If I understand how things work, including this tag in the mz_calculate_road_level function to set a minimum zoom level should solve it.

Currently they are zoom 18+

The same POI kind in different countries sometimes needs different artwork for the icon. This will also allow fast filtering by country client side.

I tried using the MapBox style PBFs and get this:
Request: http://vector.mapzen.com/osm/water/1/1/1.mapbox
Responds:
Exception!
addFeatures() takes at most 4 arguments (5 given)

Do you have a sense for how much space is used by mapbox tiles for the planet?

I'm sure you're not pre-rendering for an area that large, but it would be interesting to know how this compares to raster tiles for the planet (which I believe, in theory, would be 50+ TB).

Currently, one can query http://vector.mapzen.com/osm/buildings/15/17603/10750.json but not http://vector.mapzen.com/osm/buildings:residential/15/17603/10750.json or such. Would be a great feature.

There are functions in data/functions.sql that are no longer used, and should be removed.

Is there a Mapnik XML style file corresponding to what you're sending back for the Mapnik vector tiles (e.g. .mapbox extension)?

FYI:

$ http http://vector.mapzen.com/osm/earth,landuse,roads,buildings,places/6/9/13.mapbox
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 5139
Content-Type: text/plain
Date: Fri, 07 Nov 2014 09:29:47 GMT
Date: Fri, 07 Nov 2014 09:29:47 GMT
Server: Jetty(9.2.z-SNAPSHOT)
Via: 1.1 8e78f48eca361058a6b2b308102cca1d.cloudfront.net (CloudFront)
X-Amz-Cf-Id: DI3fYvleU8TjyrOzsnpGOpJUELUxBjKhxO9aEVWRZN9-RV-IIo6l-A==
X-Cache: Error from cloudfront

Exception!
','

May be linked to #45.

Thanks :)

At line
https://github.com/mapzen/vector-datasource/blob/master/data/functions.sql#L249

We should start including trunk and primary along with motorway right off the bat.

    CASE WHEN highway_val IN ('motorway','trunk', 'primary') THEN 7
         WHEN highway_val IN ('secondary') THEN 10

There are situations where the kind value doesn't have the most appropriate tag's value in it. We should also return back the value that is in the landuse tag on the feature to help with these situations.

Hi :)

I don't see the river lines in the vector tiles (i.e. waterway=river or waterway=canal for example).

Am I missing it, or is it on purpose, or is it a "to be done" issue? :)

This layer would be needed even in a generic style to display the river labels along the river line (now, having only polygons, we can't).

If it's to be done, I may work on a PR.

Thanks!

Yohan

Ways with the tag highway=living_street are not included in vector tiles.
Including it in the function mz_calculate_road_level (maybe at the same level as highway=service) should probably be the solution, if I understand it correctly.

We're using the default DP geometry simplification which create angular, chunky shapes instead of beautiful shapes.

Turns out newer PostGIS includes Visvalingam-Whyatt geometry simplification popularized in the MapShaper.org tool: http://postgis.net/docs/manual-dev/ST_SimplifyVW.html

Let's start using VW for our vector tiles.

We kind of have this in the ref attribute on roads now, but it's not normalized (and the network attribute on the road's route relation is, more often than not).

We need this for:

1. Road shields
2. Improve road connectivity based on highway designation, not just road surface / width classification.

For instance: US 101 goes thru San Francisco, but for 1/2 it's length thru the northern part of the city it's shown only as highway:primary and kind:major_road. The southern 1/2 is marked highway:motorway and kind:highway.

For the road shields it seems like the network attribute is very useful.

To improve road connectivity we might be able to override the kind to show as highway?

Sometimes we need to localize road colors by country.

In the docs you state that polygons "represent building outlines" and include "building footprint at lower zoom levels, and individual building:part geometries [...] at higher zoom levels."

However, it seems it seems you do not use multi-polygons for buildings having outer and inner features, e.g. for inner court yards, etc. Any way of accessing these via your vector tiles?

Example:
http://vector.mapzen.com/osm/buildings/18/140827/85996.json

We're using Natural Earth from zooms 0-8. The source data is already generalized for display at that zoom, but it looks like we're applying additional generalization on top of that.

Hey mapzen folks,
thanks for keeping your source open. The Mapzen Vector Tile Service was a great start for me to understand vector tiles.
Now i would like to build my own Vector Tile Service with custom queries for specific OSM features (which your service doesn't provide). I was already able to run the basic TileStache service with GeoJSON output of my osm database. But unfortunately I can't manage to extend TileStache with your repo to use your queries and extended output formats. Is it enough to import the .shp files in the data directory and use the tilestache-server.py with your tilestache.cfg? Or is it necessary to follow additional steps?
When I start the TileStache server using your .cfg (with adjusted DB settings) I don't get any errors, but the output of

http://localhost:8080/pois/9/267/166.json

is an empty GeoJSON

{"type":"FeatureCollection","features":[]}

However the default TileStache .cfg with the TileStache.Goodies.Providers.PostGeoJSON.Provider is producing the correct GeoJSON with the exact same query. What am I doing wrong? Since there are no instructions I thought you guys could help me out a bit.

Thanks!

We need to stroke the water coastlines directly.

Right now stroking the existing water polygons doesn't work because that adds blue lines at tile boundaries / feature chunks.

Sometimes less is more. There's too much stuff on the map now, we need the ability to filter data by country.

We got a support request which I'm moving here, per @rmarianski's suggestion

We followed your installation guide on https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service# https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service# and found that the installed API always only returns empty arrays, even though, we have downloaded the latest Switzerland file from geofabrik.

There were also no errors that occurred during the installation process.

This repository lacks any licensing terms, or even a copyright statement. Is that intentional—because Mapzen intends to reserve all rights—or is it an oversight? I'm looking to deploy an instance of this, but it's not clear to me if that's permissible.

Same deal as #136.

Hi,

AFAIK, to integrate a non MapBox vectortiles service in MapBox Studio, this service needs to expose a tilejson URL.
I'm working on implementing vector tiles support in Kosmtik, and I'm thinking about supporting tilejson too (and raw TMS too btw), so I wonder if this is something that could be added to this repo somewhere in the future.

Thanks for you lights on this :)

Yohan

Hallo kind folks at Mapzen,

I'm trying to setup your fork of TileStache (in order to be able to serve my vector-tiles), but I'm running into the following error:

Error loading Tilestache config:
Traceback (most recent call last):
  File "./scripts/tilestache-server.py", line 55, in <module>
    app = TileStache.WSGITileServer(config=options.file, autoreload=True)
  File "/Library/Python/2.7/site-packages/TileStache-1.49.11-py2.7.egg/TileStache/__init__.py", line 342, in __init__
    self.config = parseConfigfile(config)
  File "/Library/Python/2.7/site-packages/TileStache-1.49.11-py2.7.egg/TileStache/__init__.py", line 107, in parseConfigfile
    return Config.buildConfiguration(config_dict, dirpath)
  File "/Library/Python/2.7/site-packages/TileStache-1.49.11-py2.7.egg/TileStache/Config.py", line 217, in buildConfiguration
    config.layers[name] = _parseConfigfileLayer(layer_dict, config, dirpath)
  File "/Library/Python/2.7/site-packages/TileStache-1.49.11-py2.7.egg/TileStache/Config.py", line 447, in _parseConfigfileLayer
    layer.provider = _class(layer, **provider_kwargs)
TypeError: __init__() got an unexpected keyword argument 'ignore_cached_sublayers'

What seems to cause the problem?
If I simply run Tilestache without specifying any .cfg file it works just fine..
I also checked if I'm actually running your fork of Tilstache by exporting the folder, containing the clone of your github repo, to my PYTHONPATH.

Thanks in advance for your help!,
Cheers!

At zoom 9, 10, and 11 we're missing major highway (motorway) links. Let's close up those gaps by adding the major highway links at the same time we add the major highways.

Looks like this is controlled here:
https://github.com/mapzen/vector-datasource/blob/master/data/functions.sql#L100

         WHEN highway_val IN ('motorway_link', 'trunk_link', 'residential', 'unclassified', 'road') THEN 12

Seems fine for trunk and other links to remain at zoom 12 (that's when the minor roads come in already).

Example around Stockton, California (#10.925/37.8195/-121.4428):

Is it normal for it to take 8 minutes to fetch a tile? I have been trying to get a vector tile server up with tilestache for some time now & it always seems to take forever to fetch/seed tiles. My latest attempt I followed all the instructions in the wiki.

time curl http://localhost:5000/buildings/16/19293/24641.mvt
real 8m14.025s

It's an empty tile too...I mean all it's doing is making a psql query & then returning the data, no?
Are there any gotchas I'm missing?

Any help would be greatly appreciated!

Thank you, Luke

We're missing the polygon part of piers in San Francisco, but do show paths on the piers creating a weird walking on water feeling.

An example:
http://www.openstreetmap.org/edit#map=18/37.79884/-122.39577

TagInfo:
https://taginfo.openstreetmap.org/keys/man_made?filter=relations#values

These are common enough & interesting enough to include by default:

pier
wastewater_plant
works
bridge
tower
breakwater
water_works
groyne
dike
cutline

Hi All,

I'm trying to use simple webpage using vector-datasource, and there is a CORS issue even when my start page is loaded from the same host. I'm not sure if it is port number (since I use another web server to serve the static content). It would be great if documentation describes on how to configure Tilestache and/or werkzeug on how to serve both static and dynamic vector datasource content and route it properly.

This is what FF console prints:

uncaught exception: [object XMLHttpRequest]
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://myserver:8080/all/2/1/2.json. This can be fixed by moving the resource to the same domain or enabling CORS.

Thanks!

Parking comes in too early at z15 now, bus_stop too early at z16 now.

Potentially I can see including bus_stop earlier for TRANSIT layer only, but not for a general purpose reference map.

Currently, there is no information in the GeoJSON response if or how simplified the geometries are.

It seems different zoom levels may have the exact same geometries, so with such a field one could check whether to replace the geometry for the same object (having same ID) for a new zoom level, or not.

It could be a worthwhile optimization to have separate layers for labels. See thread in #84.

Hi,

I'm not sure if this is the right place to ask this question, but I haven't been able to find other resources in this space, so I figured I'd ask here.

I'm trying to adapt this and the mapzen TileStache fork to a differently formatted dataset (a subset of OSM data for now, but extending into other things in the future). What are the assumptions of what gets returned from the SQL queries for each zoom level? I've read through the various .pgsql files, and tried to replicate them for my own dataset, but it looks like there's some assumptions being made in the VecTiles code for the .mapbox extensions. Are these assumptions enumerated someplace?

If not, could they be documented? I'd prefer not to figure them out myself, but can if needed.

Thanks

Hey all,

Unfortunately I can't post issues to mapzen/TileStache, so I'll add this here. I am still having issues with seams on tile borders. I haven't looked into it too much, but I thought I'd let ya'll know. It looks like its just occurring with water.

tilezen/TileStache#34

Compare a geojson of "buildings" layer:

and a geojson of the same tile for "roads":

"Roads" seems to be clipped without any buffer, which then ends with some rendering artefacts, see for example:

Seems to me that we should keep a buffer, and let consumers of the tile clip it themselves when needed.

https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service

our work-in-progress WebGL rendering library results in a 403

Hi again :)

This is more a reflection/remark than an issue.

When using vectortiles with Mapnik to create map, geometries and labels are paint with a subtle difference:

• geometries are paint using the "painter algorithm", which means that every layer is paint over the layer above, so in a way the last layer to be paint has more priority than the ones that came before
• on the other hand, labels are paint following the rule "first paint blocks following labels", i.e. the first label paint will prevent other labels to be paint on the same place (there is some configuration that can change this on the edge), so we can say that the first layer to be paint has more priority than the ones that came after

For this reason, when doing "old school" mapping with direct connection to the data (say a PostGIS database), for a given type (say the roads) we usually have one layer for the geometries and one for the labels; just to take an example, that could be "roads" and "roads_labels" layers.

When using vector tiles as sources, as it is now, we have only one layer for both geometries and layers, so this means, for example, that one can only put roads before or after places, while in reality we would like to put the places labels before the roads labels (so a place label has priority over a road label) but we want the place label to be after the roads geometries, so that roads are not paint over the places labels.

I understand this repository is meant for generic purpose only, and may not go in great details, but I feel like this one hurts even when doing very basic cartography.

So it may that the answer would be to have separate layers again for geometries and labels. Not sure how this would impact performance, though.

Now, I may just have missed something, or I may not be considering the issue from the right point of view.

Yohan

Some tourism and man_made features are being included in the landuse layer, but the kind coalesce was not updated to include those, and these features are missing the kind property as a result.

I have done lots of changes in https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service. Update, reorder and add some small missing steps. I hope I have not done big mistakes.

Nevertheless, is still missing some data when you follow only this guide. Where are from the land_polygons and water_polygons ? There are not included nor fetched.

Hi (again ;) ),

Seems that to be able to add multiple source in a MapBox Studio project, those sources are separated by a comma: "source1,source2". (That's a very poor format, but that's another debate.)
As I'm working on adding MapBox Studio styles support to Kosmtik, I know I will face this too. And I would love to be able to use MapZen tiles, as they are open so they can be a good entry point for beginners.
I have a workaround in mind, but I'm giving you the information, for discussion.

Thanks! :)

Yohan