Merging diff files about osmdbt HOT 2 OPEN

Yep, it looks like I wrote most of this code back in 2009 so my memory is a bit hazy on the details.

Osmosis does use timestamps heavily. While timestamps in the state files may be a bit fuzzy, there are some guarantees available. The timestamp in a replication state file is greater than or equal to all data inside that changeset and those preceding it (I hope I have this right, it's important). Similarly, data in subsequent replication files may also be less than this timestamp due to late arriving data. It's eventually consistent. You can use the timestamps to determine when to start replicating and know that you'll eventually see all data created after that timestamp.

Osmosis doesn't count files, or even really care what interval size you're using, it's generic in that sense. You can specify interval lengths in seconds if you wish, and that's what's I was striving for when I first wrote it. Osmosis is setup to create minute files initially because that's the minimal interval in cron, but it is possible to create a file every second if you wish. --receive-replication and the --send-replication-xxxx tasks were written to support always connected streaming but I never got around to getting them up and running or iron out the remaining kinks.

If at least one interval of data is available then Osmosis starts processing. It calculates a maximum interval prior to processing that is a multiple of the interval length based on what's available on the server. If a partial interval is available it won't process. If 1.5 intervals are available, it will process one interval and stop.

Osmosis using the local state file at the root of the directory tree to figure out what the current local interval number (and associated timestamp) is. Therefore resetting is a simple matter of copying an earlier state file over the top and it will start from there again.

If you want to understand the Osmosis date logic, the two main classes to look at are:

• BaseReplicationDownloader - It has a method called calculateMaximumTimestamp which determines when to stop downloading replication files. It limits to data available on the server and also has a maximum bound that avoids downloading half of the Internet in a single run if processing hasn't been run for a long time. Under normal operation this limit won't get hit. The basic ReplicationDownloader class (--read-replication-interval) used for streaming changes inherits this basic functionality.
• ReplicationFileMerger - This is the implementation of --merge-replication-files and a sub-class of BaseReplicationDownloader. It imposes additional timestamp constraints for file merging. It augments the calculateMaximumTimestamp method by further limiting the maximum timestamp to align to file boundaries. It has all the logic to produce multiple output files if multiple intervals of data are available.

Hope that helps.

from osmdbt.

The Osmosis code is really hard to read so I haven't figured out yet what it is doing exactly.

Maybe @brettch still remembers some of those details, or has some good pointers to the relevant bits in the code to get started.

from osmdbt.