Comments (4)
The planet-dump-ng
software only sets the current time in the PBF header, not the sequence number. This is because the planet dump is an independent process from the replication diffs and neither depends on the other. Also, there are minutely, hourly and daily replication streams and each has a different (independent) sequence number.
There are tools to synchronise a planet dump with a chosen replication stream, for example pyosmium's up-to-date tool. This works by looking at the timestamp of the planet file, rewinding a bit and replaying the diffs covering that period.
The general reason why these streams are all independent is that it previously wasn't easy to identify a linear point in time in Postgres, hence all the stuff in Osmosis' state file about txnActiveList
and the xid
column index in the database. More recently, Postgres made it easier to get access to the internals of the replication log, which made more robust tools like osmdbt
possible and allows talking about a specific linear point in the log.
In summary; planet-dump-ng
won't write the sequence number header in PBF files, you'll have to use something else (e.g: pyosmium-up-to-date
) to merge replication stream info into the planet file.
Hope that helps!
from planet-dump-ng.
Hi,
Sorry for hijacking your thread, but it's kind of related.
For the life of me if I can figure this out.
As a background, to understand my approach, we're building a private OSM server and a tiles server, part of a bigger app.
With the intention of having a small DB and new clients as up-to-date as possible with the main OSM server, we decided to import only the client's bits, not the whole country.
When a new client joins, we re-download the country.pbf, slice his bit and import it without impacting our other clients (I think).
Clients will edit the map using iD so I've setup a tiles server in sync.
Replication using osmdbt is done and I'm currently working on importing the chages using imposm or osm2pgsql and here is the tricky bit.
I can't manage to generate the correct state.txt. Using osmium fileinfo on the generated PBF shows the latest change's timestamp, but no sequenceNumber.
To summarise, when a new client joins:
- download "country.pbf", slice his turf using a poly and import it using osmosis
- dump the updated database to a PBF using your tool (kudos, nice work)
- drop postgis and mapnik tiles and re-import with imposm or osm2pgsql
I'd appreciate some help, thanks :)
from planet-dump-ng.
it does, thanks for the explanation.
Edit: technically, I'd rather reset the sequence numbers every time
from planet-dump-ng.
Hi,
I wanted to contribute to your project so I'm pasting our dockerfiles here. Maybe you guys need it.
Postgres version can be bumped to 12 without any hiccups, we're just not there and haven't tested it.
I removed line, might hiccup at permissions on the volume but I don't think so.
Dockerfile:
FROM debian:buster-slim
ARG PLANET_DUMP_URL=https://github.com/zerebubuth/planet-dump-ng/archive/v1.2.0.tar.gz
RUN set -eu; \
apt-get update; \
apt-get install -y --no-install-recommends \
build-essential \
autoconf \
automake \
ca-certificates \
curl \
libboost-date-time-dev \
libboost-dev \
libboost-filesystem-dev \
libboost-iostreams-dev \
libboost-program-options-dev \
libboost-thread-dev \
libosmpbf-dev \
libprotobuf-dev \
libxml2-dev \
osmpbf-bin \
pkg-config \
postgresql-client-11; \
useradd -u 999 -r planetdump; \
mkdir /opt/build; \
curl -sL $PLANET_DUMP_URL | tar xz -C /opt/build --strip-components=1; \
cd /opt/build; \
./autogen.sh; \
./configure; \
make -j $(nproc); \
make install; \
cd /; \
rm -rf /opt/build; \
mkdir /dumps; \
chown planetdump:planetdump /dumps
COPY entrypoint /usr/local/bin/entrypoint
VOLUME /dumps
USER planetdump
WORKDIR /dumps
ENTRYPOINT ["/usr/local/bin/entrypoint"]
CMD ["bash"]
entrypoint (chmod +x)
#!/bin/sh
set -eu
PBF_FILE=${PBF_FILE:-latest.pbf}
case "$1" in
dump)
cd /dumps
rm -rf users changeset* node* way* relation*
echo "dumping OSM db"
DUMP_FILE=$(mktemp)
pg_dump -F custom > $DUMP_FILE
echo "creating PBF"
planet-dump-ng -f $DUMP_FILE -p "$PBF_FILE"
rm -rf users changeset* node* way* relation*
;;
*) exec "$@";;
esac
Edit: added missing file name, removed osmium.
from planet-dump-ng.
Related Issues (20)
- PBF doesn't use DenseNodes HOT 3
- PBF History file contains too large blobs HOT 1
- No separate changesets file. HOT 3
- All member types of relation objects are from type="relation" HOT 1
- Changeset discussion dumps HOT 1
- Add comments_count to changesets dump HOT 1
- Changeset dump file contains no comments after February 23rd HOT 3
- Reference failures in planet file HOT 1
- Check references in current output
- Update github description HOT 1
- Set osmosis_replication_base_url HOT 1
- Build instructions fail on `make` HOT 5
- handle directory-based input for parallel pgdump HOT 2
- Invalid xml data in planet osm HOT 15
- Occasional lockups dumping planet HOT 12
- Exception dumping recent planets HOT 10
- .
- OSM AWS Athena Tables Not Updated HOT 1
- v1.2.4 fails to compile on Debian 11 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from planet-dump-ng.