gagan-bansal / osm-for-my-country Goto Github PK

View Code? Open in Web Editor NEW

50.0 5.0 11.0 355 KB

It's small effort to make OSM maps usage simple. This is a boilerplate combining many packages related to OSM. (beta)

License: MIT License

JavaScript 88.97% Shell 11.03%

openstreetmap osm osm2pgsql geofabrik kosmtik map-tiles region stack cartocss map

osm-for-my-country's Introduction

osm-for-my-country

Note: This is a boilerplate combining many packages related to OSM.

You are working on a project that require map of your country and don't have much time to understand the OpenStreetMap data and map creation with it. Don't worry here is a completely automated solution to create OpenStreetMap map tiles of your region.

This project is stack of many libraries and framework stitched together to create map tiles of your region. You just specify the country name and required data is downloaded through scripts from different sources. Even all the dependent software modules and libraries are also installed in a single go. You only need a plain Linux machine.

Main libraries, packages, data sources and framework those are part of this project are: OpenStreetMap data from Overpass API and geofabrik, osm2pgsql, PostgreSQL with PostGIS, OpenStreetMap tools osmconvert and osmfilter, node.js using nvm, pm2, http-server, main mapping framework Kosmtik that is developed on Mapnik node.js bindings, CartoCSS based map style framework openstreetmap-carto, Google Noto Fonts and many OpenStreetMap data related node.js modules geosjon2poly, osm-tile-list, osmtogeojson, simplify-geojson, sphericalmercator.

Consider this project as a boilerplate. You can learn all the components of a OSM project: map rendering, caching the tiles, updaating the OSM data and serving map tiles.

Dependency

python command is required on shell prompt. If using python3 create alias python for python3.

Installation

All the above mentioned packages would be installed with single scripts.

git clone https://github.com/gagan-bansal/osm-for-my-country.git
cd osm-for-my-country
bash install.sh

Please go through install.sh to see what all the packages are being installed.

Usage

How does this project work? Basic flow is

after installation first data is downloaded and processed for the country name you specify (command: init)
a map rendering engine is started (command: start-kosmtik)
all the map tiles are exported. Why all, can't we render dynamically and serve? No, idea of this project is to create a complete map cache then you can server map tiles with simple http server or even s3. With this stack you can achieve similar to Mapbox export (command: export)
exported map tiles are served with light weight http-server (command: serve) Now the map can be previewed (command: demo). You can also easily update OpenStreetMap data and map tiles with single command with daily updates from geofabrik with single command(command: update). It's nice to keep update command as cron job.

Here are the required commands:

node index.js --help

Usage: node index.js <command> [options]

For command help:
node index-dev.js <command> --help

Commands:
  init           Initiate the data by downloading and inserting into postgres
                 and many more things
  start-kosmtik  Start kosmtik server with mapnik as map rendering engine
                 At this stage you can preview map with ksomtik at
                 http://127.0.0.1:6789/
                 You can make changes in CartoCSS and preview immediately.
  export         Export map tiles
                 By default all the options are read from
                 './config/default.json'
  serve          Serve map tiles at http
                 This will enable map tiles to be served at
                 http://127.0.0.1:4040/$z/$x/$y.png
  demo           Privew map tiles with help of Leaflet
                 Check your map at http://127.0.0.1:4141/
  update         Update the OSM data and map tiles based on daily update from
                 GEOFABRIK.
                 You can set this command in your cron job to update on daily
                 basis.

Options:
  -s, --save  save parameters to config file           [boolean] [default: true]
  -h, --help  Show help                                                [boolean]

init

node index.js init --help

Usage:node index.js init [options]

Examples:
  node index.js init --region 'Asia, Nepal'

Options:
  -r, --region  Region to download with complete path as per GEOFABRIK
                like: --region 'Asia, Nepal'                            [string]
  -h, --help    Show help                                              [boolean]

start-kosmtik

node index.js start-kosmtik --help

Usage: node index.js start-kosmtik
You can preview your map at http://127.0.0.1:6789/

Options:
  -h, --help  Show help                                                [boolean]

export

node index.js export --help

Usage:node index.js export -u [str] -t [str] -o [str]
by default all these options are read from './config/default.json'

Options:
  -u, --tileServerURL  base url serving map tile                        [string]
  -t, --tileList       a file for tiles list                            [string]
  -o, --dir            output tile directory                            [string]
  -h, --help           Show help                                       [boolean]

serve

node index.js serve --help

Usage: node index.js serve
  Serve map tiles at http
  This will enable map tiles to be served at http://127.0.0.1:4040/$z/$x/$y.png

Options:
  -h, --help  Show help                                                [boolean]

demo

node index.js demo --help

Usage: node index.js demo
  Privew map tiles with help of Leaflet
  Check your map at http://127.0.0.1:4141/

Options:
  -h, --help  Show help                                                [boolean]

update

node index.js update --help

Usage: node index.js update
  Update the OSM data and map tiles based on daily update from GEOFABRIK
  You can set this command in your cron job to update on daily basis

Options:
  -h, --help  Show help                                                [boolean]

Please go through config/defualt.json to change different project options. Recommended minimum system required is 4 core with 8GB memory.

TODO

skip installation if package is already installed
improve log

License

This project is licensed under the terms of the MIT license.

osm-for-my-country's People

Contributors

Stargazers

Watchers

Forkers

rt2yrru ujjwalnepal ikyandhi torbenskov vdedyukhin anastasiaclark methekill kmpoppe klinsc

osm-for-my-country's Issues

Slow generation of tiles using "node index.js export"

I have a server with dual xeon cpu (8 cores with HT, 32 cores all in total), and 32 gig of ram, running Ubuntu 18.04 server. After downloading and running osm-for-my-country for Serbia using node index.js init "Europe, Serbia" and starting kosmtik and export, I only get output of about 1 tile per second.

Attached are the logs of top -b

Export has been running for few days now (done levels up to 15)

free -mh shows:

          total        used        free      shared  buff/cache   available
Mem: 31G 3.0G 15G 659M 12G 26G
Swap: 8.0G 0B 8.0G

Attached is the 5 minute output of "top -b":
top.log

Should this not be much faster on machine like this?

Extend to any sub region of a country

It should allow to create map for sub regions also like for single state of a country.
Data can be downloaded with overpass API wherever it does not match with Geofabrik sub regions.

node index.js init not completing conversion to "my-area-final-data.osm"

I get the following error when trying to run `node index.js init --region 'Europe, Norway'. Seems to be a problem with the last step creating the "my-area-final-data.osm". All the other steps run fine. Any idea what could be wrong?

init ...
downloadCountryData ...
downloading region data: http://download.geofabrik.de/europe/norway-latest.osm.pbf
curl: echo  `curl -s --connect-timeout 10 --retry 10 --retry-delay 10 --retry-max-time 500 -w '%{http_code}' http://download.geofabrik.de/europe/norway-latest.osm.pbf -o my-area.osm.pbf `

Using config file: './config/default.json'

200

Writing current state of data ...
Current state: http://download.geofabrik.de/europe/norway-updates/state.txt
The number of request attempts: 1
pbf2osm ...
dropCountriesBorder ...


done
downloading countries border data with overpass api..
curl overpass: echo  `curl -s --connect-timeout 10 --retry 10 --retry-delay 10 --retry-max-time 500 -H "Host: overpass-api.de" -H "Content-Type: text/xml" -w '%{http_code}' -d 'relation["admin_level"="2"];(._;>;); out body;' http://overpass-api.de/api/interpreter -o countries-border.osm `
000

downloaded coutries borders
downloading my country border data with overpass api..
curl my country overpass: echo  `curl -s --connect-timeout 10 --retry 10 --retry-delay 10 --retry-max-time 500 -H "Host: overpass-api.de" -H "Content-Type: text/xml" -w '%{http_code}' -d 'relation["admin_level"="2"] ["name:en"="Norway"];(._;>;); out body;' http://overpass-api.de/api/interpreter -o my-area-border.osm `
200

downloaded my country border
osm2geojson ...

done
simplifyGeoJSON ...
/Users/aleksandereilertsen/osm-for-my-country/node_modules/.bin/simplify-geojson /Users/aleksandereilertsen/osm-for-my-country/data/my-area-border.geojson --tolerance 0.001 > /Users/aleksandereilertsen/osm-for-my-country/data/my-area-border-simplified.geojson

done
view file saved
geojson2poly ...
Done!

done

some error in exec.
/Users/.../osm-for-my-country/cmd-init.js:491
    throw err //callback(err)
    ^

Error: Command failed: ./osmconvert --hash-memory=12025 /Users/.../osm-for-my-country/data/my-area-dropped-countries.osm /Users/.../osm-for-my-country/data/countries-border.osm --out-osm -o=/Users/.../osm-for-my-country/data/my-area-final-data.osm
osmconvert Error: could not open input file: /Users/.../osm-for-my-country/data/countries-border.osm

    at ChildProcess.exithandler (child_process.js:270:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:897:16)
    at Socket.stream.socket.on (internal/child_process.js:340:11)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at Pipe._handle.close [as _onclose] (net.js:548:12)

Output image as png8

Output image should be png8 that reduce the size of the image. As most of the cased transparent images are not required.

Error running init command

Hi, while trying to run your project i get this error

init ... downloadCountryData ... downloading region data: http://download.geofabrik.de/europe/portugal-latest.osm.pbf curl: echo curl -s --connect-timeout 10 --retry 10 --retry-delay 10 --retry-max-time 500 -w '%{http_code}' http://download.geofabrik.de/europe/portugal-latest.osm.pbf -o my-area.osm.pbf `

Using config file: './config/default.json'

200

Writing current state of data ...
Current state: http://download.geofabrik.de/europe/portugal-updates/state.txt
The number of request attempts: 1
pbf2osm ...
/home/query/osm-for-my-country/cmd-init.js:39
if (err) throw err
^

Error: spawn /home/query/osm-for-my-country/osmconvert ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)`