Ubuntu 20.04 changes about osmdbt HOT 14 CLOSED

What about changes in https://github.com/joto/osm-logical/commits/master? Will you create a pull request for those?

from osmdbt.

osmdbt is only ever needed for the main OSM database. So we should support whatever version of the OS, libraries etc are running there and don't care about anything else. Of course we might have to temporarily support two versions to help migrating, but can remove support for the old version soon after.

from osmdbt.

My assumption was that this code would be deployed to the dev instance for testing first. Versions on dev usual differ a bit from prod. As an example, dev already runs on Postgres 12 iirc.

I don’t know if prod will have been migrated to Ubuntu 20.04 and a newer Postgres version by the time this code is in use.

By the way, speaking of prod, some ready made Ubuntu ppa‘s would also be needed for deployment. I believe those are not yet available.

from osmdbt.

Now that the debian/ subdirectory contains all necessary files to build a debian package, I tried a local build process via debuild on Ubuntu Bionic. Unfortunately, this failed due to the unsatisfied dependency on libosmium2-dev 2.15, which is only present on Ubuntu Focal. On which environment did you run your tests?

Next on was a test of the build process on Ubuntu Launchpad. To use the right Ubuntu distribution, I had to change the first debian/changelog entry to osmdbt (0.1-focal1) focal; urgency=medium
After packaging, signing and uploading the file to Launchpad, I got the following result:
https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test1/+packages

Unfortunately, the build failed as most of the database unit tests cannot be executed in this environment, and would be typically skipped to avoid a build failure.

The following tests FAILED:
3 - 01-get-log (Not Run)
4 - 02-create-diff (Not Run)
5 - 03-fake-log (Not Run)
6 - 04-log-pid-fail (Not Run)
7 - 05-create-diff-state (Not Run)
8 - 06-create-diff-missing-state (Not Run)
9 - 07-cmdline (Not Run)
10 - 08-redaction (Not Run)
Errors while running CTest

https://launchpadlibrarian.net/483795483/buildlog_ubuntu-focal-amd64.osmdbt_0.1-focal1_BUILDING.txt.gz

Question to @tomhughes : do you see an option to use those locally built debian packages for deployment to production? I don't think you can directly upload those to Launchpad, but maybe there's some other way to use them. If that doesn't work, we need to find a way to get this Launchpad process up and running for osmdbt.

from osmdbt.

Sorry I don't understand the problem. Surely we just backport libosmium? That's never been a problem before.

I'm not sure why we're worrying about how to deploy this to production though - is it actually ready to be used or something?

from osmdbt.

In fact we already have that backported to Bionic in the admins PPA.

from osmdbt.

I was testing the new debian build process for osmdbt, which will be needed when we switch to the new diff replication. So far, the README file describes an option to build osmdbt as a local package: https://github.com/openstreetmap/osmdbt#debian-package

What I was wondering about is, if we can use such a locally built debian package for deployment to production later on, or if we need to go via Launchpad in any case.

So this isn't so much about libosmium, and rather about osmdbt.

from osmdbt.

So @joto has added the missing dependency on postgresql-common that caused the test failure.

If you can get that working on focal then I can rebuild it in our PPA for bionic.

Obviously we could install a deb some other way but I don't understand why we would want to.

from osmdbt.

OK, great, I'll give it another try 👍

New build on osmdbt-test2 PPA: https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test2/+build/19441690

Build log: https://launchpadlibrarian.net/483800054/buildlog_ubuntu-focal-amd64.osmdbt_0.1-focal1_BUILDING.txt.gz

3/10 Test #3: 01-get-log .......................***Failed 0.01 sec
Could not determine PostgreSQL version, are any PostgreSQL server packages installed?

from osmdbt.

postgresql-common alone didn't do the trick. I took a look at how Postgis runs their unit tests and adjusted debian/control accordingly. My set of local changes now looks as follows:

diff --git a/debian/changelog b/debian/changelog
index 90cca66..909b1a3 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,4 +1,4 @@
-osmdbt (0.1) UNRELEASED; urgency=medium
+osmdbt (0.1-focal1) focal; urgency=medium
 
   * Initial release
 
diff --git a/debian/control b/debian/control
index 8032b27..51eac42 100644
--- a/debian/control
+++ b/debian/control
@@ -13,7 +13,9 @@ Build-Depends: debhelper (>= 10~),
                libyaml-cpp-dev,
                libpqxx-dev,
                pandoc,
-               postgresql-common,
+               postgresql-all,
+               postgresql-common (>= 148~),
+               postgresql-server-dev-all,
                zlib1g-dev
 Standards-Version: 4.3.0
 Homepage: https://github.com/openstreetmap/osmdbt

New test PPA: https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test3

Build log: https://launchpadlibrarian.net/483818516/buildlog_ubuntu-focal-amd64.osmdbt_0.1-focal1_BUILDING.txt.gz

This looks much better now. Postgresql is starting up, pg_virtualenv seems to be working.

Test cases are still failing because we don't have the osm-logical plugin installed. I'm not sure what the best way to integrate https://github.com/zerebubuth/osm-logical would be.

travis.yml builds and installs the osm-logical.so on the fly via:

  - git clone --quiet --depth 1 https://github.com/zerebubuth/osm-logical ../osm-logical
  - cd ../osm-logical
  - make PG_CONFIG=/usr/lib/postgresql/$PG_VERSION/bin/pg_config
  - sudo make install PG_CONFIG=/usr/lib/postgresql/$PG_VERSION/bin/pg_config

from osmdbt.

I believe this all works now.

from osmdbt.

I think we would still need to get the Ubuntu PPA build process working on Focal.

Even with the two changes in debian/changelog and debian/control I mentioned earlier on, 9 out of 11 unit tests are currently failing with "ERROR: could not access file "osm-logical": No such file or directory". It seems that the osm-logical shared library isn't part of the installation process yet.

from osmdbt.

Not sure what you mean, all tests including on Focal a green? This issue is a bit of a mess mixing up several things, if you still see problems, please open a new issue with just one thing.

from osmdbt.

My comment wasn't really about the unit tesst on Travis, which are working fine. I was trying the Ubuntu PPA build process on launchpad.net, as that would be the place where @tomhughes fetches binaries for deployment to production. I'll open another issue for it...

from osmdbt.