Comments (5)
Don't worry about the "wrong orientation" messages, they're fairly standard and don't mean much other than for debugging geometry processing - that's why they're hidden behind --verbose
:)
--fast
is simply a shorthand preset that chooses options for you ("faster runtimes, at the expense of a little more memory"). If you're specifying individual options it's best not to confuse things by adding --fast
.
--store
(of course), --shard-stores
and possibly (if you renumber the pbf first) --compact
will do the best job of minimising RAM usage.
from tilemaker.
Tried again overnight, with little luck - unfortunately the terminal seems to have completely crashed out so I can't inspect any outputs . Inspecting the system log and cross comparing against the time that the VM disk IO dropped off a cliff, systemd-oomd seems to kill the process due to memory pressure being 76.55% > 50.00% for > 20s with reclaim activity.
, which seems insanely aggressive.
Systemd-oomd seems to be enabled by default in Ubuntu 22.04 - so it's likely that other users running on this particular OS will face similar problems during planet generation.
Going to try to disable systemd-oomd and up the disk space - the process seems to have resulted in 359 mmap files (each 1GB) that aren't cleaned up, which coupled with the size of the osm.pbf (~80GB), output mbtiles (100+GB) and ubuntu itself is rapidly hitting the limit of the disk size. I'll report back on progress.
from tilemaker.
At the risk of stating the bleeding obvious, is it perhaps worth trying a smaller extract (or several smaller extracts sequentially) and measuring resource usage when processing those, to try and see where the issues might be?
from tilemaker.
A few hours later - it works!
I bumped the storage up to 650GB, and disabled systemd-oomd
via the following commands:
sudo systemctl disable --now systemd-oomd
sudo systemctl mask systemd-oomd
I can't say how wise it is to leave this disabled permanently, as opposed to tweaking the internal settings - but if you're spinning up a box just for the sake of generating a planet, it's definitely better than having your hard work and processing time killed off silently and without warning.
It may be worth having a troubleshooting FAQ or something - as this is a fairly insidious little issue that will catch people out for smaller machines and Ubuntu 22.04.
from tilemaker.
Nice debugging!
--shard-stores
and --fast
together should probably be rejected by tilemaker before it does any real work, since they try to achieve mutually exclusive goals. I'll try to send a patch for that.
Perhaps a note in a subsection of https://github.com/systemed/tilemaker/blob/master/docs/RUNNING.md#performance-and-memory-tuning would be the right spot to mention OOM killing debugging techniques for systemd-oomd?
from tilemaker.
Related Issues (20)
- lua error: table is not a function HOT 1
- lua missing, ubuntu 23.04 HOT 1
- STL Vector assertion failed when running maptiles HOT 9
- Assertion failed: (has_wire_type(pbf_wire_type::varint) && "not a varint"), function get_sint64, file pbf_reader.hpp, line 570 HOT 2
- Invalid polygons after clipping at tile boundaries HOT 15
- terminate called after throwing an instance of 'std::out_of_range' HOT 1
- tilemaker-server terminated by signal SIGKILL (Forced quit) HOT 9
- build problems HOT 1
- Tilemaker does not support maritime boundaries HOT 2
- Huge file for simple map. HOT 1
- Roundabouts missing if the first node reference is the same as the last one HOT 2
- Making Tiles With Zoom 15 Removes Place Names HOT 4
- terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
- CI is broken
- Support writing MultiPoint into vector tiles HOT 1
- Ocean and land area is missing HOT 13
- Possible issue with multipoint processing HOT 4
- How to avoid duplicate processing of OSM object
- Cross-Platform Inconsistency in Attribute Handling with Identical Configs on Ubuntu and macOS HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tilemaker.