Planet generation issues on hardware-constrained machine about tilemaker HOT 5 OPEN

Don't worry about the "wrong orientation" messages, they're fairly standard and don't mean much other than for debugging geometry processing - that's why they're hidden behind --verbose :)

--fast is simply a shorthand preset that chooses options for you ("faster runtimes, at the expense of a little more memory"). If you're specifying individual options it's best not to confuse things by adding --fast.

--store (of course), --shard-stores and possibly (if you renumber the pbf first) --compact will do the best job of minimising RAM usage.

from tilemaker.

Tried again overnight, with little luck - unfortunately the terminal seems to have completely crashed out so I can't inspect any outputs . Inspecting the system log and cross comparing against the time that the VM disk IO dropped off a cliff, systemd-oomd seems to kill the process due to memory pressure being 76.55% > 50.00% for > 20s with reclaim activity., which seems insanely aggressive.

Systemd-oomd seems to be enabled by default in Ubuntu 22.04 - so it's likely that other users running on this particular OS will face similar problems during planet generation.

Going to try to disable systemd-oomd and up the disk space - the process seems to have resulted in 359 mmap files (each 1GB) that aren't cleaned up, which coupled with the size of the osm.pbf (~80GB), output mbtiles (100+GB) and ubuntu itself is rapidly hitting the limit of the disk size. I'll report back on progress.

from tilemaker.

At the risk of stating the bleeding obvious, is it perhaps worth trying a smaller extract (or several smaller extracts sequentially) and measuring resource usage when processing those, to try and see where the issues might be?

from tilemaker.

A few hours later - it works!

I bumped the storage up to 650GB, and disabled systemd-oomd via the following commands:

sudo systemctl disable --now systemd-oomd
sudo systemctl mask systemd-oomd

I can't say how wise it is to leave this disabled permanently, as opposed to tweaking the internal settings - but if you're spinning up a box just for the sake of generating a planet, it's definitely better than having your hard work and processing time killed off silently and without warning.

It may be worth having a troubleshooting FAQ or something - as this is a fairly insidious little issue that will catch people out for smaller machines and Ubuntu 22.04.

from tilemaker.

Nice debugging!

--shard-stores and --fast together should probably be rejected by tilemaker before it does any real work, since they try to achieve mutually exclusive goals. I'll try to send a patch for that.

Perhaps a note in a subsection of https://github.com/systemed/tilemaker/blob/master/docs/RUNNING.md#performance-and-memory-tuning would be the right spot to mention OOM killing debugging techniques for systemd-oomd?

from tilemaker.