I am creating this issue for open discussion and a note for future releases. Currently the ClosePath command does not move the cursor. This was not described in v1 and most existing vector tiles are currently encoded in such a way that ClosePath does not move the cursor. From a logical standpoint, it might make sense to have ClosePath move the cursor to the initial point of that linear ring.

I.e. that a given key cannot be present more than once.

What are the benefits of delta encoding, it immediately rules out transmission over UDP which removes a few UAV use cases. It would be nice to have this as a request option, e.g. encoding=raw or encoding=delta.

They keys and values use non-packed repeated fields which is inefficient compared to packed repeated fields. In addition the values are each in its own message which needs extra space for the length header. If the values are small integers that could otherwise be encoded in a single byte, this can easily mean they need six times as much space as would be needed otherwise (Value message length + Value message wire type + Value message field number + Layer message wire type + Layer message field number + the value itself).

Unfortunately this can't be changed easily, because the values can be of different types. One option would be to have a repeated string string_value [packed=true], repeated float float_value [packed=true], etc. in the Layer message. But this would mean reordering the keys or having multiple sets of keys for the different types.

Is there any requirement for polygon holes and shell to follow a specific winding ? (clockwise/counterclockwise) ?

Do rings have to end with a ClosePath ?

The spec should say how MultiPoint, MultiLineString, and MultiPolygon geometries are to be encoded. As I understand it, currently:

• A MultiPoint geometry is encoded using GeomType POINT and >1 MoveTo commands in the geometry field.
• A MultiLineString geometry is encoded using GeomType LINESTRING and >1 MoveTo, LineTo* sequences in the geometry field.
• A MultiPolygon geometry is encoded using GeomType POLYGON and >1 MoveTo, LineTo*, ClosePath sequences in the geometry field.

These should be documented. But they have the following deficiencies:

• It's not possible to encode a MultiPoint geometry with a single point in a way that is distinguishable from a Point geometry. Thus VTs cannot round-trip GeoJSON or other formats in which a singleton MultiPoint is valid. The same is true for singleton MultiLineString and MultiPolygon.
• It's not possible to reconstruct MultiPolygons without complex heuristics for determining inner/outer rings. (Ref mapbox/mapnik-vector-tile#59)

For these reasons, I would rather see the spec adopt explicit MultiPoint, MultiLineString, and MultiPolygon types.

According to the README, a given tile is converted to screen coordinates from spherical mercator. Being that paths are delta encoded, in what form is the origin for a given path? Is it pixels from some origin in the corner of a tile?

Also, will vector tiles be solidified to play in the world of Google/OSM tile schema where we cut Spherical Mercator into quads? Does this make any sense when we apply this format to OpenGL?

I wonder about this, because OpenGL does not need / want to think in terms of zoom levels, and I wonder if it makes sense to project vectors to pixel space for a given zoom level. I am thinking this will overcomplicate the process of creating a single vector model and apply a scaler--the essential process we would need for an OpenGL map application.

Also, how does this fit in a coming world of adaptive map projection web map applications?

http://cartography.oregonstate.edu/demos/AdaptiveCompositeMapProjections/

https://github.com/mapbox/vector-tile-spec/wiki/Implementations could be clearer if it split apart into producers and consumers of vector tiles. Some do both, but many are just one side.

There may also be some duplication - I see more than one link which is Mapnik under the hood. Is it useful to list all libraries and applications which use mapnik-vector-tile?

Hey,

the move to Vector spec 2.0 puts more constraints to the polygon type than previously.

What is best practice that custom exported data should follow to adhere to the specification?

Reason for asking is that we're running a postGIS -> node mapnik -> .mbtiles production chain, and have noticed some issues with the mapbox-gl-js earcut branch. Entry data has a tendency to be slightly dirty, but we haven't had any issues with mapbox-gl-native or other tessellation-based renderers previously.

Currently experimenting with:

PostGIS processing: ST_MakeValid, ST_ForceRHR, ST_ForceSFS
Node Mapnik: Bumping versions,
Mbtile: Bumping versions

We're seeing both artifacts constrained within a single tile, but also covering multiple tiles. None of these seem to be affecting gl-native.

One essential use case that arises from having tiles is recreating a complete feature that spans many tiles.

Being that a given tile will clip a geometry, how do we know which paths are actually just clipping a feature as opposed to an actual path describing that features?

Also, how do we uniquely identify each feature so that we know that we can join it to geometries in other tiles?

Protocol Buffers offers a style guide for .proto files. Please consider following it.
https://developers.google.com/protocol-buffers/docs/style

Hello,

This is more of a question than an issue. In the specs https://github.com/mapbox/vector-tile-spec/tree/master/1.0.1 it says that "Geometries are clipped, reprojected into spherical mercator, converted to screen coordinates".
Is it valid for every kind of geometry? If you look at the Java encoder implementation here:
https://github.com/ElectronicChartCentre/java-vector-tile/blob/master/src/main/java/no/ecc/vectortile/VectorTileEncoder.java#L100
It clips only when not a polygon so linestring/multilinestring/points will be clipped (which is also weird because it then tests for a MultiPolygon type which is impossible if you clip a line, you will get a multilinestring in the worst case).
It just checks if polygons are within bounds of the tile but does not crop them.

In this message, @kkaefer says that Mapbox uses clipped geometry (with a small buffer around the tile, probably to account for line-style widths):
#6 (comment)

This one says that the geometries can be clipped or not, the tile format is fine with either:
#8 (comment)

I am producing my own tiles and getting rendering artefacts on some segments (lines and polygon weirdness) from mapbox-gl-js:

So I was wondering if clipping obligatory, at least reduce the tile size, and what MapBox does exactly when generating tiles when it comes to clipping

Thanks a lot for your help!
Fabien

It feels weird that unique ID is optional, not mandatory, since features across tiles are going to have to be acted on together occasionally for visual coherence. I suppose a good spec finds a balance between optionality and mandatories, but a unique ID per feature seem not unreasonable, so I can know that various areas and boundaries stretching over several tiles are the same feature.

Currently the specification does not allow compression, but some PBFs seen in the wild are compressed, and not at the transport layer (HTTP).

The specification should be explicit on if compression is allowed.

With mapnik-vector-tile implementation as I am upgrading to the stricter interpretation of V2, I have noticed an issue perhaps in our wording.

Within our decoder of geometries, I ignore lineto commands that do not move the cursor -- LineTo(0,0).

The LINESTRING geometry states only that the command count must be at least one. However, the logic in my decoder is that a lineto(0,0) doesn't contribute to the point count and therefore, I throw errors about invalid linestrings in situations where the lineto command count might be 1 or more.

The same is true in POLYGON geometries. However, this might be caught by the area requirement for rings.

The fix might be to clarify this situation in the polygon and linestring types.

Assuming specified mime type for version 2.0 is accepted, we should register the MIME type to make it official.

/cc @sgillies @pnorman

A feature MAY contain an id field. If a feature has an id field, the value of the id SHOULD be unique among the features of the parent layer.

Had a question on the example at the bottom of the 2.1 spec. It involves the conversion of some GeoJSON points into a vector tile. I noticed that the 'ID' field is the same for both point features in the vector tile. Is this valid?

Example in question:

layers {
version: 2
name: "points"
features: {
id: 1
tags: 0
tags: 0
tags: 1
tags: 0
tags: 2
tags: 1
type: Point
geometry: 9
geometry: 2410
geometry: 3080
}
features {
id: 1
tags: 0
tags: 2
tags: 2
tags: 3
type: Point
geometry: 9
geometry: 2410
geometry: 3080
}
keys: "hello"
keys: "h"
keys: "count"
values: {
string_value: "world"
}
values: {
double_value: 1.23
}
values: {
string_value: "again"
}
values: {
int_value: 2
}
extent: 4096
}

The specification should set an upper bound of 16,384 for the extent of a tile, so that decoders can safely use signed 16 bit integers to represent (potentially buffered) tile coordinates (as certain widely used decoders already do).

It should probably also specify that tile extents must be a power of 2.

cc @ansis

This is a bit of a corner case, but. When rendering linear features with distinct end-point rendering styles (a symbol at the end, or a dumbell or something) there is no way in the current spec to distinguish between a line that ends at a tile boundary, and one that is carried over the tile boundary by another line in the neighboring tile.

Hi,
I read that vector tiles would be much more space efficient than raster tiles. But comparing what your HTML or iOS demo apps are downloading it doesn't look like so.

For example:

http://a.tiles.mapbox.com/v4/mapbox.mapbox-streets-v6/14/4823/6160.vector.pbf?access_token={token}

https://api.mapbox.com/v4/mapbox.streets/14/4823/6160.png?access_token={token}

Results in downloading 16.725 bytes PNG versus 35.206 bytes PBF file.

How is it possible?

The spec contains this sentence: "The version field MUST be the first field within the layer." I am not sure what this means in the context of a Protobuf encoded layer. Generally the Protobuf-libraries do not allow you to define the order of fields in a message. Some low-level libraries (such as Protozero) allow this, but generally this is not the case.
See the last section "Field Order" in https://developers.google.com/protocol-buffers/docs/encoding for details. This looks to me like the version field will be written last in the Layer message because it has the highest field number.

Is there some program that checks vector tiles for their adherence to the spec? So something that checks all the MUSTs and SHOULDs etc. and writes out a report with warnings or errors? Could be useful to check implementations against different versions of the spec.

I don't like the fact that the projection is fixed to spherical Mercator. Because it might limit usage of the format by several official agencies due to the inaccuracies.

Maybe it would be better to follow a WMS or WMTS system. Where the possible projections are defined on a server manifest, and are accessed by different URIs. Then the definition of the projection can be taken out of the tile spec.

Currently, we have only one example of the structure of an entire tile encoded. Perhaps more examples would be helpful to people?

Hi, I've got a question and I hope this is an alright place to put it.

There are a number of vector tile sources out there that use geojson. Is it possible to convert the tiles defined in this repo into geojson, or vice versa? If so, are there any known implementations?

After I have created an object, each layer has a field called _features. The value is an integer. Is this integer some sort of index for the buffer itself?

How do I go further to decode the actual paths from the buffer?

Encoding a huge number of points in VT is currently pretty inefficient in terms of output size. This is especially a big problem on the highest zoom levels where POIs and address points are encoded. Let's discuss what we could do about that in a future major spec revision to make encoding optimal.

This is how points are currently encoded in a layer:

{
  "id": 3359,
  "tags": [0, 561, 1, 123, 2, 456],
  "type": 1,
  "geometry": [9, 1050, 8692]
},
{
  "id": 3360,
  "tags": [0, 548, 1, 234, 2, 457],
  "type": 1,
  "geometry": [9, 1071, 8694]
},
{
  "id": 3361,
  "tags": [0, 549, 2, 458],
  "type": 1,
  "geometry": [9, 1072, 8693]
},
...

Problems with the current approach:

• each point is a separate message, so each needs at least two bytes for a tag + message length
• tag for each field (id, tags, type, geometry) is repeated over and over
• additionally, tags and geometry are packed fields so need 2 bytes for lengths
• the same type is repeated everywhere
• each geometry has a superfluous moveTo command
• the same key indexes are typically repeated over and over
• geometries are not delta-encoded

Some of the repetition gets compressed away by gzip, but there's still a lot of data we could eliminate with a smarter encoding approach. Here a rough idea how the sample above could be encoded optimally:

{
  "id": [3359, 3360, 3361, ...],
  "tags": [561, 123, 456, 548, 234, 457, 549, 0, 458, ...],
  "geometry": [3336, 8692, 21, 2, 1, 1, ...]
}

• a set of points is one big message with a few packed fields
• ids are packed together
• tags are packed together and only have value indexes; each layer key has a corresponding value encoded, so if a particular key is not present in a feature, we encode 0 (this won't occur too often so the gain from dropping keys should far outweigh the need to include such zeroes)
• point geometries are packed together and delta-encoded (which should be a big size win); for efficiency of delta-encoding, we sort points before encoding (e.g. in a z-curve order, by a morton hash)

To understand how much of a win this will be in terms of compression, I'll make a quick proof of concept encoding on sample z15 point data and post back here.

cc @yhahn @springmeyer @flippmoke @kkaefer @tmcw @sgillies @ericfischer

Describing the implications of polygon handling in a tile setup would be useful, I think. Presumably users will want to transport polygons twice, once as a closed path, for filling, and once as an unclosed path for rendering the boundary.

Do I understand correctly that the "tags" for each feature are references to field names (even indexes) and values (odd indexes) expanded at the tile.layer level ?

Does this mean that different features (in a layer) having the same value (for any field) could point to the same element in the tile.layer.values array ? Is that done by any implementation ?

Also, doesn't this mean that a feature covering multiple tiles forces writing the same key/value pair in each and every tile it appears in ?

I know that @pnorman @pramsey and @ShibaBandit all commented in #39 about the experimental state of rasters. Here is the best clarification I can offer:

A decoder using v2 can simply ignore the raster field, we realize that a more detail specification for rasters is needed. We currently aren't sure how we think rasters should best be put into this field and some research/experimenting is required. We did want to include the field however, as we will be working in the mapnik-vector-tile repository to expand the capabilities for rasters in vector tiles. Additionally, @springmeyer has told me that some groups have already started to experiment with this as a feature. The goal is that we will have this as an experimental field for version 2 and it will lay the groundwork for the community to work on how specifically how the raster data will be stored.

While a great amount of text has changed in the version 2 text specification README, the protob file has not been heavily changed. #39

The comments in the file should be updated to match the wording of the specification and more comments could be added to perhaps better explain some fields (or link to their requirements in the the README?)

Currently the spec states:

For example, if a tile has an extent of 4096, values between 0 and 4095 are considered within the tile's extent. A point at (0,10) or (4095,10) is within the extent of the tile. A point at (-1,10) or (4096,10) is outside the extent of the tile.

This is incorrect because the edges of the tile should be 0 and 4096, therefore, values between 0 and 4096 are considered within the tile's extent!

This was a failure of my part during writing this section and does not reflect how "v1 tiles were being created". I think it is important to change the wording of this section to properly reflect that the edges of the tile are 0 and 4096 for an extent of 4096.

The comments in the .proto doc for Feature.tags say "[...] refer to the nth value in the [...] list". Does that mean they are 1-based rather than 0-based ?

Any reason we need this namespace? Afaik, this is just used to ensure the namespace when using protoc to generate bindings.

The documentation for a layer containing only points is lacking. Is it a moveTo with repetition, followed by a series of coords? Or a moveTo and then a lineTo the same spot? The spec appears incomplete for this scenario. A worthwhile addition would be an example for a layer containing points. (A brief answer here would be very gratefully received too ;)

I'm confused by the relation of this specification with the PBF one:
http://wiki.openstreetmap.org/wiki/Pbf

The confusion arises by the fact that mapnik-vector-tile refers to these specs as "vector tile" (https://github.com/mapbox/mapnik-vector-tile#implementation-details) but then have a "vector tile reader" that only supports PBF:
https://github.com/mapbox/mapnik-vector-tile/tree/master/examples/c%2B%2B

Are them the same format ? Different formats sharing the same extension ?

Consider this WKT geometry: LINESTRING (1 1, 0 0). The vector_tile encoding would be [9 1 1 17 -1 -1] (moveto 1,1; lineto 0,0), but negative deltas can't be stored in the geometry field as it's of uint32 type (repeated uint32 geometry = 4 [ packed = true ];).

@springmeyer @kkaefer by "zigzag" do we mean replace all deltas n by (n << 1) ^ (n >> 31) as described in https://developers.google.com/protocol-buffers/docs/encoding? That would give a vector_tile encoding of [9 2 2 17 1 1].

The documentation for "extent" says "The bounding box in this tile spans from 0..4095 units", but the "extent" is a parameter of a layer rather than a tile. Can different layers in a tile have different "extent" ?

Where can we find some good vector tiles to play with and decode? I'm talking about maybe a big zip full of tiles, not an obfuscated SQLite MBTiles container.

The google protobuf v3 spec doesn't allow you to define default values any more. In v3 defaults are what you would expect: 0 for integers, empty strings for strings, whatever has value 0 in enums.
(For details see https://developers.google.com/protocol-buffers/docs/proto3 and https://github.com/google/protobuf/blob/master/CHANGES.txt#L309).

We are using protobuf v2 currently, but to keep our options open for later switching to protobuf v3 I suggest we get rid of default values that don't correspond to the default values in the protobuf spec in the .proto and move any defaults we need into the vector tile spec.

Specifically this is the version field which has a default of 1 and the extent field which has a default of 4096.

Hi,

We are considering to switch to vector tiles but our main concern is about the data confidentiality.

Unlike raster tiles, with the vector tiles it seems quite easy to decode the vector data from each tile to rebuild the original sources. We invested huge amount of time and money to build some very specific GIS databases, we cannot take this risk. Is there any way to encrypt or protect the data contained in a vector tile ?

The spec says about the UNKNOWN geometry type: "The specification purposefully leaves an unknown geometry type as an option. This geometry type encodes experimental geometry types that an encoder MAY choose to implement. Decoders MAY ignore any features of this geometry type."

What is the purpose for this wording? Do we have a use case for this? Are encoders really going to use that value for their experiments? Why only one value? If we really need values reserved for experimentation I suggest we define, say, all enum values above 256 as experimental or something like it. It is often useful to keep the zero value as some kind of unspecified value that should never appear in the data, but implementations can use the value internally as a default after initialization of some datastructure or to signify a deleted feature or so.

Currently within message Layer, the extent field is optional. In order to not change the format of the .proto in version 2 this has been only noted with a comment that is it required. In a later version of the release this should be changed to required.

Per discussion with @springmeyer, it seems important to be explicit about what .mvt stands for or, rather, what we'd like it to stand for from now on. There has been some confusion in regards to the recent blog post calling them Mapbox Vector Tiles. Since .mvt is not a new concept, it can be conflated with Mapnik Vector Tiles.

Proposal: officially call these Mapbox Vector Tiles. by adding to 2.1 file extension ... something like " ... .mvt (Mapbox Vector Tile) ... "

Would love some guidance on PR workflow @flippmoke!

Currently in our description of ParameterIntegers we specify the range to be:

Parameter values greater than pow(2,31) - 1 or less than -1 * (pow(2,31) - 1) are not supported.

This is incorrect it should likely be:

Parameter values greater than pow(2,30) - 1 or less than -1 * pow(2,30) - 1 are not supported.

This is an oversight be me as I was writing the specification. The minimum value is assuming non two's complement int32 we are storing.

I am playing with the Mapsplit Demo. This demo uses straight OSM PBF files that have been split into tiles. This is then rendered into GeoJSON, features are joined, and it puts it on the map.

How does OSM PBF relate to vector.pbf tiles that mapnik is generating? Could someone provide a high level explanation of how the two relate and some insight with this?

In particular, I'm looking at OSM's PBF Spec:

http://wiki.openstreetmap.org/wiki/PBF_Format

It would be nice if the spec were more in the spec, maybe with some diagrams to explain what it means to clip and transform coordinates into image space. It would be nice if the valid draw commands were not hidden in the protobuffer file, and had a bit more explanation (only three, right? 1, 2, and 7?) I guess, I'd either expect the spec description to be more complete, or for it to not exist and for the protobuffer file to be the whole thing (which it almost is now).

Being that a layer set can be constructed by concatenating vector tiles for z/x/y from multiple sources, one could go a step further and concatenate layers gradually through a socket or a websocket connection.

This could be very useful if we are constructing the contents of a tile that is non-determinate and not pre-composed.

Thinking one step further, the z/x/y of a vector tile could be thought of more as a subscription end point, and real-time data could be routed to an appropriate set of vector tiles so that the renderers will automatically show what needs to be shown.

Have there been any experiments in this direction?