Comments (23)
We want 12 for testing but we probably want 9.5 initially for prod as I assume we will switch to this first before trying the database upgrade.
from osmdbt.
As you seem to already know how to do all this, can you provide a PR? Or somebody else? Setting up the Debian config is as far as my knowledge goes.
from osmdbt.
There's one topic I'd need your opinion, namely how we should go about with the postgresql-plugin. All the command line tools in src/ only depend on the postgresql client library, while the plugin depends on the postgresql server + additional dev libraries.
I'm wondering if should create two separate PPAs for the client tools on the one hand, and the plugin on the other hand. It would certainly add a bit more effort, but we could update each of those two parts independent of each other. In particluar I'm thinking about keeping the plugin part as small as possible, so we can leave it alone, even if we find bugs later on in the command line tools. Also it would facilitate a use case, where the command line tools run on another server and connect via network to the db server.
from osmdbt.
Unless @tomhughes want something different I would say: Keep it simple. One package with everything is fine for the moment. We can always change things later if we want to, after all we are the only users of this.
from osmdbt.
Here's a first test build on Ubuntu Launchpad:
- Changes: master...mmd-osm:ubuntu/focal
- Test PPA build on Ubuntu Launchpad: https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test1/+packages
I can't get those tests to run in the Ubuntu PPA build, as they depend on the osm-logical.so to be deployed to the Postgresql library directory. pbuilder builds packages in some chroot environment, and simply copying osm-logical to /usr/lib/postgresql/*/lib as part of the dh_auto_test override is rejected with a Permission denied error.
By the way, debuild -I.
seems to have a similar issue. Unit tests fail, unless you install the shared library via sudo make install
beforehand. That's easily doable on your local machine, but it doesn't work if you build packages on some external build server.
The way the build process works at the moment, it would take the Postgresql version that happens to be the current default - that's Postgresql 12 in case of Focal. Is this the version we want to use later in prod?
from osmdbt.
Chef repo seems to pull all Postgresql packages from https://wiki.postgresql.org/wiki/Apt - which appears to be the only way to deploy 9.5 packages on Focal. I think this dependency on an external repository is still missing in the current debian/* files.
from osmdbt.
I don't know if we can get this to run on launchpad. The biggest challenge is that Postgres 9.5 is no longer part of Ubuntu 20.04, and we have to resort to an external repository, as I mentioned in my previous post. While we can add external package dependencies on Launchpad, those packages need to be in another PPA on Launchpad itself (not somewhere on the Postgresql APT repo). As a consequence, we would have to download source packages from the Postgresql repo, and somehow upload them to Launchpad. This seems all massively complicated.
Meanwhile, the following Dockerfile installs all Postgresql 9.5 files on Ubuntu 20.04, builds everything, installs it, then builds the debian package in a defined, clean environment. It can be found afterwards as /root/osmdbt_0.1_amd64.deb
. This deb package can then be installed on an Ubuntu 20.04 installation via dpkg -i osmdbt_0.1_amd64.deb
.
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -qq && \
apt-get -y -q install software-properties-common curl ca-certificates gnupg \
libosmium2-dev libboost-filesystem-dev libboost-program-options-dev \
libbz2-dev zlib1g-dev libexpat1-dev cmake libyaml-cpp-dev \
libpqxx-dev gettext-base git pandoc dpkg-dev devscripts debhelper
RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main" > /etc/apt/sources.list.d/pgdg.list
RUN apt-get update && apt-get install -y apt-utils apt-transport-https ca-certificates
RUN apt-get -y -q install postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 \
postgresql-common postgresql-server-dev-9.5 \
gcc g++ \
--no-install-recommends
WORKDIR /root/osmdbt
COPY ./ ./
RUN mkdir build && \
cd build && \
cmake .. -DCMAKE_INSTALL_PREFIX=/usr && \
cmake --build . && \
make install && \
ctest
RUN debuild -I
Resulting debian package reads as:
osmdbt_0.1_amd64.deb:
new Debian package, version 2.0.
size 325740 bytes: control archive=1128 bytes.
487 bytes, 11 lines control
1306 bytes, 19 lines md5sums
Package: osmdbt
Version: 0.1
Architecture: amd64
Maintainer: Jochen Topf <[email protected]>
Installed-Size: 1805
Depends: libboost-filesystem1.71.0, libboost-program-options1.71.0, libc6 (>= 2.14), libgcc-s1 (>= 3.0), libpqxx-6.4 (>= 6.4), libstdc++6 (>= 9), libyaml-cpp0.6 (>= 0.6.2), zlib1g (>= 1:1.2.6)
Section: utils
Priority: optional
Homepage: https://github.com/openstreetmap/osmdbt
Description: OSM Database Tools
Tools for creating replication feeds from the main OSM database.
*** Contents:
drwxr-xr-x root/root 0 2020-06-04 13:21 ./
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/bin/
-rwxr-xr-x root/root 294264 2020-06-04 13:21 ./usr/bin/osmdbt-catchup
-rwxr-xr-x root/root 440704 2020-06-04 13:21 ./usr/bin/osmdbt-create-diff
-rwxr-xr-x root/root 178536 2020-06-04 13:21 ./usr/bin/osmdbt-disable-replication
-rwxr-xr-x root/root 178536 2020-06-04 13:21 ./usr/bin/osmdbt-enable-replication
-rwxr-xr-x root/root 289136 2020-06-04 13:21 ./usr/bin/osmdbt-fake-log
-rwxr-xr-x root/root 235888 2020-06-04 13:21 ./usr/bin/osmdbt-get-log
-rwxr-xr-x root/root 182632 2020-06-04 13:21 ./usr/bin/osmdbt-testdb
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/lib/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/lib/postgresql/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/lib/postgresql/9.5/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/lib/postgresql/9.5/lib/
-rw-r--r-- root/root 14328 2020-06-04 13:21 ./usr/lib/postgresql/9.5/lib/osm-logical.so
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/share/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/share/doc/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/share/doc/osmdbt/
-rw-r--r-- root/root 3459 2020-06-04 13:21 ./usr/share/doc/osmdbt/README.md.gz
-rw-r--r-- root/root 140 2020-06-04 13:21 ./usr/share/doc/osmdbt/changelog.gz
-rw-r--r-- root/root 865 2020-06-04 13:21 ./usr/share/doc/osmdbt/copyright
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/share/man/
drwxr-xr-x root/root 0 2020-06-04 13:21 ./usr/share/man/man1/
-rw-r--r-- root/root 835 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-catchup.1.gz
-rw-r--r-- root/root 1534 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-create-diff.1.gz
-rw-r--r-- root/root 744 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-disable-replication.1.gz
-rw-r--r-- root/root 710 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-enable-replication.1.gz
-rw-r--r-- root/root 937 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-fake-log.1.gz
-rw-r--r-- root/root 883 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-get-log.1.gz
-rw-r--r-- root/root 712 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-testdb.1.gz
-rw-r--r-- root/root 1777 2020-06-04 13:21 ./usr/share/man/man1/osmdbt.1.gz
from osmdbt.
@tomhughes : what will be our target architecture in the future? osmosis runs on ironbelly connecting to karm as db master. Do you plan to run the same set up with the new tool as well? Is it ok that we "pollute" ironbelly's filesystem with those /usr/lib/postgresql/9.5/lib/ files?
from osmdbt.
Why would they be on ironbelly? They won't be any use there.
from osmdbt.
We would have to deploy the single debian package to both karm and ironbelly. The shared library would be relevant for Postgresql on karm (and otherwise pretty much useless on ironbelly). The command line tools would run on ironbelly, if this will be set up in the same way osmosis is running today.
That was the trade off of having a single debian package only...
from osmdbt.
Well if it's packaged then I really don't care. I haven't really though about how things will work in production in the new world anyway.
I'm more concerned that it's not really practical to include the postgres bits for all the reasons you have discovered, that we're using the postgres packages not the ubuntu ones, and that we may need multiple versions of them available and to map them to the right version of postgres in different environments.
It may well be better to build the postgres bits from source I suspect.
from osmdbt.
Building osm-logical.so from source would of course reduce a lot of the complexity with multiple versions.
Assuming postgresql-server-dev-9.5 (or some other version) has been installed already, that compile process would be straightforward:
cd postgresql-plugin
mkdir build
cd build
cmake ..
make install
All command line tools query the server version and run different statements depending on the Postgresql server version, so there's no need to keep any version specific packages around.
CMakeLists.txt may need a bit of tweaking, as the parent project is missing. Those seem to be warnings only.
from osmdbt.
Since c39dcbe the postgresql plugin is no longer build in the Debian build and not included in the package. This should remove that complexity.
from osmdbt.
Can we get a new build then @mmd-osm? The last one I see is from July?
from osmdbt.
Assuming testing will be initially done on the dev instance, can we build the binaries directly on the box like we did before with cgimap? I think that would speed up testing a bit, as any commit on the master branch would be almost immediately available for testing.
I think for production, we should at least revisit the PPA topic, as the plugin would have to be built directly in the box anyway, i.e. we need to have all libraries in place.
A good fit for a PPA would be e.g. if we want to deploy the dbt tools on another box and connect to the database via network.
Regardless of that decision it would probably make sense for @joto to sign up on
https://launchpad.net/ in the meantime, so that Iβm not the bottleneck.
from osmdbt.
Yes of course I can build them from source but that is a last resort and I don't really want to be live tracking master in any case, certainly not for production, and I don't plan to chef anything for testing on dev,
from osmdbt.
Right, tracking the master branch was only meant for tests on the dev instance, certainly not for production.
Iβm not exactly sure if there were some previous discussions outside of this GitHub repo on how those tests should be run, so itβs a bit difficult for me to comment on the chef topic. The binaries could be compiled manually on the box, deploying the plugin probably needs sudo powers, a minutely job would need to be scheduled and ideally osc files being made available via Apache.
from osmdbt.
We discussed it with Jochen on the OWG call this week - as testing on dev is only of marginal use my plan is to do a quick bit of manual testing.
I've already built and installed the plugin this morning, and was going to install the tools from your build but found it was up to date which is why I asked. Obviously I can build manually instead, it's just more work.
from osmdbt.
I've triggered a new build as PPA https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test2, based on commit
b867f22. PPA publishing step is currently pending and should complete shortly.
debian/changelog
still reports as version 0.1 back from June, which may be a bit confusing. Maybe @joto can bump the version number to 0.2 and add a new entry to the debian changelog.
from osmdbt.
I updated the version.
from osmdbt.
Nothing left to do, closing here...
from osmdbt.
As an FYI:
Since compiling and deploying a custom plugin seemed a bit cumbersome, I've been looking into pgoutput, a fast binary format-based plugin that's built into Postgresql.
This might also be beneficial in case of cloud environments with limited options for deploying custom plugins.
Link: https://github.com/mmd-osm/osmdbt-pgoutput
from osmdbt.
@mmd-osm Can you please open a new issue for this?
from osmdbt.
Related Issues (20)
- Limit total number of changes per diff generation (osmosis compatibility) HOT 10
- Q: fake-log process and duplicates in diff files HOT 6
- Unit test for relations table HOT 2
- Osm-logical: sanity checks HOT 2
- Unit tests for create-diff HOT 5
- Exploring wal2json HOT 2
- Should get-log check for existing log files with same LSN? HOT 1
- Fetching objects in create-diff (osmosis performance regression) HOT 2
- fake-log, isolation level and concurrent txns HOT 4
- state.txt timestamp - and osmosis compatibility HOT 2
- Order of tags in object HOT 4
- Docker compose cluster HOT 1
- Merging diff files HOT 2
- Documenting state.txt file format HOT 2
- CI is broken HOT 4
- Build for Ubuntu 22.04 HOT 3
- osmdbt and pgoutput HOT 2
- replication_slot in osmdbt-config.yaml example
- disable-replication should check for remaining entries to be processed HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from osmdbt.