Threading not working well for ctb-tile? about cesium-terrain-builder HOT 4 CLOSED

Since writing this I have read some things that make me suspect that the bottleneck is the file system - not disk I/O, but creating files. Interesting...

from cesium-terrain-builder.

Hello - CTB's use of threads is simple and in an ideal world I'd expect it to scale linearly with the number of cores. As such any failure to meet this expectation in practice will almost certainly be due to a bottle neck in a subsystem. The problem will be narrowing it down to which one!

GDAL itself may be the culprit as it is a pretty complex beast with a plethora of config options and drivers which can interact in subtle ways (especially cache related options such as GDAL_MAXCACHE). There is also room for improvement: for example I notice you are using a VRT and if it is referencing many source datasets then implementing this issue may help.

However, your hunch that it could be the filesystem may well turn out to be true; I have had problems with this in the past. In this case it may be worth looking at a filesystem optimised for many small files, such as ReiserFS running on solid state disks.

Assuming funds and time I would like to implement different cache backends to CTB in addition to the filesystem (e.g. memcache, redis, postgis, leveldb, sqlite) which could be used to help scale and overcome filesystem bottlenecks.

I would be interested in any findings you have, as it would be useful to set up a tuning section in the README to help with these issues.

from cesium-terrain-builder.

Thanks @homme! The VRT in this case was only referencing a couple of hundred images, which I imagine wouldn't have been too taxing for it.

We're done with our big processing task now, so may not get to put a lot more time into testing different file systems but will keep ReiserFS in mind. We will let you know if we have the opportunity to experiment further and find out anything useful :)

from cesium-terrain-builder.

In htop on 32 core machine I can see that with -c 32 utilization of cores is not full.

Also any recommendations about -m parameter?

from cesium-terrain-builder.