Investigate string pools about osmexpress HOT 5 CLOSED

Yes, I attempted it at the beginning. The storage space savings were 5-10%, which is quite a bit, but OSMExpress eats lots of disk anyways, so I decided it wasn't worth making the code more complicated :)

from osmexpress.

I followed the same scheme imposm3 uses which is to encode common tags in the unicode Private Use Area: https://github.com/omniscale/imposm3/blob/master/cache/binary/tags.go#L140

from osmexpress.

Have you tried :List(UInt16) as a sibling to :List(Text)? The Unicode PUA character hack needs 4 bytes per tag, whereas an explicit integer only 2. However, one would lose the original tag ordering. Interestingly, imposm3 uses only 166 common tags, so a :List(UInt8) would do. Intuitively, I’d have used a larger set of common tags (64K), but of course one would have to measure the size impact.

from osmexpress.

I didn't investigate it further, maybe I will when I get a chance to look at this again (and likely migrate to flatbuffers instead of capnproto). Compact storage isn't really a design goal of this system in favor of keeping a small and maintainable codebase.

from osmexpress.

Moving to #1 since that would be the next opportunity to make a breaking change in the data format.

from osmexpress.