augmented diff example program about osmexpress HOT 7 CLOSED

🎉 great summary!

An open question is if the 1:1 strategy described above is good enough.

As far as we've gotten, the answer to this depends on your use case and the tradeoffs you can make with regards to when specific changes show up based on time or transaction boundaries. We're not sure there's one correct answer for everyone, attempted to summarize a bit here: https://github.com/azavea/onramp/blob/master/docs/index.md#use-cases.

osmx do not include all metadata for nodes

Just double checking and clarifying -- all metadata is included for tagged nodes. It's the untagged nodes that only have (or will soon) version info?

from osmexpress.

Just double checking and clarifying -- all metadata is included for tagged nodes. It's the untagged nodes that only have (or will soon) version info?

Yes, correct. The precise way to put it is "Nodes with 0 tags do not have timestamp, changeset, username or UID." All other information in the current state of OSM should be present.

from osmexpress.

https://github.com/protomaps/OSMExpress/blob/master/python/examples/augmented_diff.py is now usable to generate a partial augmented diff with create, modify, delete elements. I have tested the output with achavi: you can drag the file onto the map to visualize the diff: https://overpass-api.de/achavi/

Two things I have not implemented yet:

• bounding box for ways and relations. this should be easy but requires one more pass over the elements at the end to compute the bounding box.
• propagation from changed node location > ways, and from changed node location > way > relation or from way member list change > relation. This is quite tricky, because we want don't want to include nodes or ways who only have changed tags. but we can't distinguish these two cases until we have already made a pass over all actions filling in refs.

In addition I made some notes while implementing:

• In the Geofabrik regional OSCs I'm using, the delete element has the metadata and version of the object before It was deleted, but osm.org has the metadata and version for the changes of the delete itself.
• I haven't tested with cases where an object has multiple modifications in a single OSC
• This method in general is very sensitive to the timestamp of the osmx db and the OSC file. They must align exactly. In the case where a modify in the OSC happens to an element the osmx does not have, this modify is swizzled into a create action instead.
• Untagged nodes have ? for their old username, uid, changeset and timestamp of course by design.

from osmexpress.

I've implemented propagating changes as well as bounds.

I haven't tested this in many situations but it should be a complete proof of concept. I did need to add some functions to the Python bindings in order to retrieve node->way, node->relation and way->relation references.

Once propagation is added in, the # of elements in the augmented diff increases a lot; this code is slow in general.

For regional extracts that are not reference-complete for all relations (e.g. admin boundaries) I skip propagation when a referenced member is missing from the osmx.

@CloudNiner the output of python augmented_diff.py may be useful to compare to Onramp.

from osmexpress.

🎉!

this code is slow in general

How slow is that, taking a ballpark guess at an average running time per OSM OSC file?

from osmexpress.

🎉!

this code is slow in general

How slow is that, taking a ballpark guess at an average running time per OSM OSC file?

I just tested it on planet.osmx with a minutely diff, it seemed to only take a few seconds, but OSCs vary widely in how complex they are.

from osmexpress.

https://github.com/azavea/onramp seems to be using this successfully, so marking this as closed.

from osmexpress.