Comments (2)
Yep, it looks like I wrote most of this code back in 2009 so my memory is a bit hazy on the details.
Osmosis does use timestamps heavily. While timestamps in the state files may be a bit fuzzy, there are some guarantees available. The timestamp in a replication state file is greater than or equal to all data inside that changeset and those preceding it (I hope I have this right, it's important). Similarly, data in subsequent replication files may also be less than this timestamp due to late arriving data. It's eventually consistent. You can use the timestamps to determine when to start replicating and know that you'll eventually see all data created after that timestamp.
Osmosis doesn't count files, or even really care what interval size you're using, it's generic in that sense. You can specify interval lengths in seconds if you wish, and that's what's I was striving for when I first wrote it. Osmosis is setup to create minute files initially because that's the minimal interval in cron, but it is possible to create a file every second if you wish. --receive-replication
and the --send-replication-xxxx
tasks were written to support always connected streaming but I never got around to getting them up and running or iron out the remaining kinks.
If at least one interval of data is available then Osmosis starts processing. It calculates a maximum interval prior to processing that is a multiple of the interval length based on what's available on the server. If a partial interval is available it won't process. If 1.5 intervals are available, it will process one interval and stop.
Osmosis using the local state file at the root of the directory tree to figure out what the current local interval number (and associated timestamp) is. Therefore resetting is a simple matter of copying an earlier state file over the top and it will start from there again.
If you want to understand the Osmosis date logic, the two main classes to look at are:
- BaseReplicationDownloader - It has a method called
calculateMaximumTimestamp
which determines when to stop downloading replication files. It limits to data available on the server and also has a maximum bound that avoids downloading half of the Internet in a single run if processing hasn't been run for a long time. Under normal operation this limit won't get hit. The basic ReplicationDownloader class (--read-replication-interval
) used for streaming changes inherits this basic functionality. - ReplicationFileMerger - This is the implementation of
--merge-replication-files
and a sub-class of BaseReplicationDownloader. It imposes additional timestamp constraints for file merging. It augments thecalculateMaximumTimestamp
method by further limiting the maximum timestamp to align to file boundaries. It has all the logic to produce multiple output files if multiple intervals of data are available.
Hope that helps.
from osmdbt.
The Osmosis code is really hard to read so I haven't figured out yet what it is doing exactly.
Maybe @brettch still remembers some of those details, or has some good pointers to the relevant bits in the code to get started.
from osmdbt.
Related Issues (20)
- Limit total number of changes per diff generation (osmosis compatibility) HOT 10
- Q: fake-log process and duplicates in diff files HOT 6
- launchpad.net Ubuntu PPA build HOT 23
- Unit test for relations table HOT 2
- Osm-logical: sanity checks HOT 2
- Unit tests for create-diff HOT 5
- Exploring wal2json HOT 2
- Should get-log check for existing log files with same LSN? HOT 1
- Fetching objects in create-diff (osmosis performance regression) HOT 2
- fake-log, isolation level and concurrent txns HOT 4
- state.txt timestamp - and osmosis compatibility HOT 2
- Order of tags in object HOT 4
- Docker compose cluster HOT 1
- Documenting state.txt file format HOT 2
- CI is broken HOT 4
- Build for Ubuntu 22.04 HOT 3
- Ubuntu 20.04 changes HOT 14
- replication_slot in osmdbt-config.yaml example
- disable-replication should check for remaining entries to be processed HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from osmdbt.