mapbox / osm-boundaries Goto Github PK

MapBox OSM boundary relation processing

License: BSD 3-Clause "New" or "Revised" License

JavaScript 46.73% Python 53.27%

osm-boundaries's Introduction

MapBox OSM Boundaries

This program will process and import boundary relations from a OSM PBF file. The process is optimized for rendering - instead of building (multi)polygons, boundary relation way members are imported as unconnected linestrings allowing different segments of a single boundary to be styled independently. This also avoids overlapping lines where different boundaries and admin levels meet, and allows renderers like Mapnik to draw dashed lines correctly.

Data processing

The data is manipulated and simplified for easier styling:

The lowest admin_level value of any of a way's parent relations will be the admin_level value of the resulting linestring.
Various tag combinations are checked to see if each way is a maritime boundary. This information is simplified to a single maritime field with a value of either 0 (false) or 1 (true).
Various tag combinations are checked to see if each way is a disputed boundary. This information is simplified to a single disputed field with a value of either 0 (false) or 1 (true).
Boundaries that are also coastlines (natural=coastline) are not imported.
Boundaries that are closure segments (closure_segment=yes) are not imported. (Closure segments are ways added at the limits of the projection to close boundaries for valid multipolygon building. They are not actual borders.)
Geometries are imported to a Spherical Mercator projection (900913).

Known issues

boundaries that are not part of any boundary=administrative relation are ignored.

Dependencies

Python & Psycopg2 in a Unixy environment
Osmosis (requires version >= 0.42 for planet files newer than Feb 9 2013)
PostgreSQL (tested with 9.2)
PostGIS (tested with 2.0)
Osmium - make sure osmjs is compiled and in your PATH

Running

Make sure you have a PostgreSQL database set up with PostGIS enabled.
Run run.py -f 2 -t 4 data.osm.pbf with appropriate options set for your database and desired admin levels. See run.py --help for available options.

The process will take quite some time and require lots of free disk space for temporary storage. Processing a full planet file might take over six hours and require at least 60 GB of free disk space.

osm-boundaries's People

Contributors

Stargazers

Watchers

Forkers

mjaros teneighty xkwangy pnorman ospreyx johndpope testbigorg isabella232

osm-boundaries's Issues

tiny code-modification

in the last line it has to be:

osmium.apply(reader,location_handler, handler);

otherwise i get the following error:

reader.apply(location_handler, handler);
TypeError: reader.apply is not a function
at Object. (/root/process-boundaries.js:110:8)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Function.Module.runMain (module.js:441:10)
at startup (node.js:139:18)
at node.js:974:3

Don't rely on Overpass API

It's a bit much for most admin levels. Processing boundaries out of a planet file could be an alternative. Test this:

osmosis \
  --rb planet.pbf \
  --tf accept-ways boundary=administrative \
  --tf accept-relations boundary=administrative \
  --un \
  --uw \
  --wb dc_bound.pbf

Add simplified geometries

maritime ended up as char(varying)

Not sure how this happened.

Thread for task 1-read-pbf failed

I've tried ./run.py -d osm_boundaries -f 2 -t 6 planet-latest.osm.pbf but it failed with this:

INFO: Pipeline executing, waiting for completion.
Apr 29, 2015 3:47:27 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent 2312448328 as an integer.
        at org.openstreetmap.osmosis.core.util.LongAsInt.longToInt(LongAsInt.java:33)
        at org.openstreetmap.osmosis.core.filter.common.ListIdTracker.set(ListIdTracker.java:118)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:96)
        at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(WayContainer.java:60)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:65)
        at org.openstreetmap.osmosis.tagfilter.v0_6.UsedWayFilter.complete(UsedWayFilter.java:131)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
        at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
        at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
        at crosby.binary.file.BlockInputStream.process(Unknown Source)
        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:37)
        at java.lang.Thread.run(Thread.java:745)

Apr 29, 2015 3:47:27 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:328)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:408)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:351)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:31)

Table carto_boundary is empty.

Ubuntu 14.04.2 LTS
python-psycopg2
osmosis 0.40.1
PostgreSQL 9.3
PostGIS 2.1
osmium 0.0~20111213-g7f3500a-3build5

password authentification missing

I think my server has the appropriate settings in the pg_hba.conf to let user postgres access the DB automatically. But unfortunately I get a
psycopg2.OperationalError: fe_sendauth: no password supplied when running. So it would be nice to be able to authenticate with password.

run.py fails at osmosis due to unknown error

When running run.py, I get the following error in the osmosis part:

SCHWERWIEGEND: Thread for task 1-read-pbf failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent 2191925799 as an integer.
    at org.openstreetmap.osmosis.core.util.LongAsInt.longToInt(LongAsInt.java:33)
    at org.openstreetmap.osmosis.core.filter.common.ListIdTracker.set(ListIdTracker.java:118)
    at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:96)
    at org.openstreetmap.osmosis.core.container.v0_6.WayContainer.process(WayContainer.java:60)
    at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:65)
    at org.openstreetmap.osmosis.tagfilter.v0_6.UsedWayFilter.complete(UsedWayFilter.java:131)
    at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
    at org.openstreetmap.osmosis.tagfilter.v0_6.TagFilter.complete(TagFilter.java:142)
    at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)
    at crosby.binary.file.BlockInputStream.process(Unknown Source)
    at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:37)
    at java.lang.Thread.run(Thread.java:679)
14.03.2013 12:03:29 org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
    at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
    at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
    at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:328)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:408)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:351)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:31)

Include additional OSM tags

This program summarizes various OSM feature tags into simple fields like disputed or maritime, which can later be easily styled with CartoCSS. I would like to preserve some of the underlying OSM tags, so I could use them to style features in Mapbox Studio (drawing from Mapbox.com's hosted data, which seems to use osm-boundaries output). For example, I would like to style just US state boundaries, perhaps using a CartoCSS filter like [iso3166 =~ "US-.*"].

Is this possible? Or would this unacceptably increase the storage space or processing required? I notice the Mapbox features have an osm_id field, but it is always 0. If that were actually set to a relation ID, perhaps I could address the features by ID.

Write build script

Just a few commands to handle downloading the data & running osmjs.

Consider osmfilter, or --tee with osmosis

Osmosis is difficult to package, and generally considered slower than osmfilter.

On the other hand, Osmosis allows you to use --tee to read the planet file once and split it to multiple tasks, each of which writes out its own PBF.

Failing to create table

Error Message:

ProgrammingError                          Traceback (most recent call last)
/root/<ipython-input-7-facf155e379a> in <module>()
      8 geom_gen1 geometry(Geometry,900913),
      9 geom_gen0 geometry(Geometry,900913)
---> 10 );'''.format(boundary_table))

ProgrammingError: type modifier is not allowed for type "geometry"
LINE 7: geom geometry(Geometry,900913),
             ^

admin_level=6 and maritime boundaries

Hi! I've tried to display boundaries with admin_level=6 and it seems that there is no boundary=maritime for this boundaries.

Here is an example — maritime is red and boundaries with admin_level=6 is green. So is it OSM issue or something else?