See https://bugzilla.gnome.org/show_bug.cgi?id=738954

We can do this better by splitting up the "create installroot" and "commit" parts of treecompose. The "commit" portion only needs privileges to write to the target repo.

I think the easiest path here would be (if --commit-as-repo-owner or so is specified)

• Detect owner/group of target repo
• Call a separate binary /usr/libexec/rpm-ostree/commit after set{g,u}id()

I looked briefly at doing the setuid() inside the main compose binary just before commit but there's a wrinkle - we need to be able to clean up our temporary data. In general when doing privilege separation stuff we really want to have clearly defined roles and input/output for each process.

OSTree has them, we should support people writing them for a few reasons.

One use case I see is it's very common to maintain metadata in a git repository, and it'd be nice to have that git repo URL + commit in the message.

After a tree is composed an ostree summary file should be generated so that the tree can be probed from non-local filesystem. This can be done manually with ostree -u --repo=PATH and the resulting summary file will land in the repo. There is an API in ostree for this, so it just needs to be implemented.

The version output when upgrading/rolling back uses the same version A->A, which is wrong. Should be A->B. Here is an example:

-bash-4.3# rpm-ostree rollback
Moving '06a63ecfcf053d1625e9ddf406429eef3c7fe3ecccbe636a54b90175a5899e7d.0' to be first deployment
Transaction complete; bootconfig swap: yes deployment count change: 0
Changed:
  acl 2.2.52-8.fc23 -> 2.2.52-8.fc23

where it should be:

-bash-4.3# rpm-ostree rollback
Moving '06a63ecfcf053d1625e9ddf406429eef3c7fe3ecccbe636a54b90175a5899e7d.0' to be first deployment
Transaction complete; bootconfig swap: yes deployment count change: 0
Changed:
  acl 2.2.52-8.fc23 -> 2.2.52-7.fc22

At least some components of rpm-ostree should be an introspectable shared library.

• RPM database inspection and diffs
• Server side composes

See projectatomic/rpm-ostree-toolbox#70

OSTree supports arbitrary metdata per commit, and we added a "version" member which is displayed (but no semantics are ascribed to it). rpm-ostree-toolbox supports one scheme to generate it, going from e.g. 7.0.0 to 7.0.1 by referencing the previous commit.

Version numbers seem important enough for the UX that we need to make having them the default. There are two paths for that: building a flexible and generic mechanism into rpm-ostree, or having more consumers use -toolbox (and adding a flexible policy there).

Support for flexible schemes

Red Hat Enterprise Linux Atomic Host follows a scheme implemented in -toolbox now, which looks like 7.0.2 -> 7.0.3, or possibly 7.0.2.1.

There's been no discussion of a version numbering scheme for Fedora. But let's strawman here that it would be a basic number.digit(+) scheme. Something like 21.0 -> 21.1, where the digit denotes how many times the compose has changed since gold.

A "version" member in treefile

Suppose the treefile gained something like:

"version": "7.0.{N}"

Then rpm-ostree would have logic like:

• No previous commit? {N} = 0
• Does everything before {N} match? Then substitute {N} with {N} + 1.
• Otherwise, {N} = 0

This would work out well for the RHELAH case, as the build administrator could change in the treefile:

"version": "7.1.{N}"

when switching to 7.1, and things would correctly reset to 7.1.0.

Supporting more complex policies

External build systems could simply not use version in the treefile and instead pass it on the command line as is supported today.

Some OS generators would like to be able to strip down the content beyond what the RPM set does.

We should investigate supporting something like lorax's language for this:
https://git.fedorahosted.org/cgit/lorax.git/tree/share/runtime-cleanup.tmpl

Show me the rpm-level difference between two trees. Take a command line option to do filesystem-level diff?

If the update stream for a particular branch ends, it would be nice if the OS vendor could make a new commit that has metadata like endoflife with a string value for an arbitrary message.

# rpm-ostree upgrade
....
warning: endoflife: This version of Fedora is no longer being updated.  For more information about how to upgrade, see https://fedoraproject.org/wiki/End_of_life

I'm thinking this would be a bit in the detached metadata. Or if we require a new commit, could be in the in-band metadata.

Regression from #22

$ rpm-ostree rpm diff --repo=repo fedora-atomic/rawhide/x86_64/docker-host
error: "rpm-ostree db diff" takes exactly 2 arguments

Whenever I try to add the mesosphere repository and the mesos Packaged to the tree, I receive this error:
Moving /etc to /usr/etc error: rename: Directory not empty

The problem is that the mesos rpm will create the directory usr/etc/mesos.

Currently, all %posts are run on the server side. This means we need nss-altfiles to help OSTree's 3 way merge of /etc.

The systemd people added systemd-sysusers, with the intention that the updates happen on boot.

A new hybrid option is running systemd-sysusers in the new root just before reboot. This would avoid system mutation on boot with all the problems that entails (e.g. booting read-only should work, etc.)

Ideally, we would have something for "make check"/InstalledTests that runs though a basic tree compose. I was a bit surprised to see that at least yum doesn't appear to have any test repo metadata or packages.

librepo does: https://github.com/Tojaj/librepo/tree/master/tests/test_data
and it also has tests that use it. A technical reason this is nice for librepo is that it can run as non-root.

But unfortunately 'compose tree' would bomb out with yum if we're running "make check" as non-root (as we're constrained to do by RPM %check in a Koji/Brew environment).

Ideas:

• LD_PRELOAD or something to fake yum (and rpm) into thinking it's running as root. (This might be easier as a branch?)
• Have a sudo make sudo-check that uses containers (Docker?) Basically the Docker container would yum install rpm-ostree, then sudo make install DESTDIR=/tmp/tmpdir.XXXX/ && tar cf /tmp/tmpdir.XXXX, then unpack that tarball into a Docker container. (Or we just make a local RPM, and install that)

This happened at the end of an "atomic rebase foo:".

       PID: 836 (atomic)
       UID: 0 (root)
       GID: 0 (root)
    Signal: 11 (SEGV)
 Timestamp: Thu 2014-09-25 18:03:21 UTC (47min ago)

Command Line: atomic rebase local:
Executable: /usr/bin/rpm-ostree
Control Group: /user.slice/user-1000.slice/session-1.scope
Unit: session-1.scope
Slice: user-1000.slice
Session: 1
Owner UID: 1000 (walters)
Boot ID: 6e1f2de7f9114eba9275cbf00de7f87a
Machine ID: 45bb3b96146aa94f299b9eb43646eb35
Hostname: localhost.localdomain
Coredump: /var/lib/systemd/coredump/core.atomic.0.6e1f2de7f9114eba9275cbf00de7f87a.836.1411668201000000.xz
Message: Process 836 (atomic) of user 0 dumped core.

            Stack trace of thread 836:
            #0  0x00007fe8f0782aba g_list_remove_link (libglib-2.0.so.0)
            #1  0x00007fe8f077d128 g_key_file_remove_group_node (libglib-2.0.so.0)
            #2  0x00007fe8f077d284 g_key_file_clear (libglib-2.0.so.0)
            #3  0x00007fe8f077f149 g_key_file_unref (libglib-2.0.so.0)
            #4  0x00007fe8f127d7b3 ostree_deployment_finalize (libostree-1.so.1)
            #5  0x00007fe8f0a8ad86 g_object_unref (libgobject-2.0.so.0)
            #6  0x00007fe8f127c651 ostree_sysroot_upgrader_finalize (libostree-1.so.1)
            #7  0x00007fe8f0a8ad86 g_object_unref (libgobject-2.0.so.0)
            #8  0x0000000000407d5b rpmostree_builtin_rebase (rpm-ostree)
            #9  0x000000000040697a main (rpm-ostree)
            #10 0x00007fe8ef3f90e0 __libc_start_main (libc.so.6)
            #11 0x0000000000406b74 _start (rpm-ostree)

This is related to https://bugzilla.gnome.org/show_bug.cgi?id=731725

Basically, we want to encourage tree creators to GPG sign commits. And we want to have a way to show the verification status on the client side.

Likely this issue would involve adding new APIs ostree side, then consuming them for atomic status.

On my system I have:

dnf-yum-0.6.4-1.fc21.noarch
dnf-0.6.4-1.fc21.noarch

Running my compose as:
rpm-ostree compose tree --repo=/srv/rpm-ostree/centos-atomic-host/7/ --cachedir=/srv/rpm-ostree/cache /tmp/atomic-treecomposeKtoBYS.tmp/centos-atomic-base.json
Generated from rpm-ostree-toolbox treecompose.

This in turn calls:
Starting 'yum' '-y' '--disablerepo=*' '--setopt=reposdir=/tmp/atomic-treecomposeKtoBYS.tmp' '--enablerepo=CentOS-Base' '--enablerepo=CentOS-updates' '--enablerepo=CentOS-extras' '--enablerepo=atomic7-testing' '--enablerepo=virt7-testing' '--setopt=keepcache=0' '--setopt=cachedir=/var/tmp/rpm-ostree.QNWFWX/yum-cache' '--installroot=/var/tmp/rpm-ostree.QNWFWX/rootfs.tmp' 'shell'

This throws the error:

Config error: releasever not given and can not be detected from the installroot.

It appears that dnf is expecting releasever to be explicitly set when given to it.

Running this by hand proves this with:
yum -y --disablerepo='*' --setopt=reposdir=/tmp/atomic-treecomposeKtoBYS.tmp --enablerepo=CentOS-Base --enablerepo=CentOS-updates --enablerepo=CentOS-extras --enablerepo=atomic7-testing --enablerepo=virt7-testing --setopt=keepcache=0 --setopt=cachedir=/var/tmp/rpm-ostree.QNWFWX/yum-cache --installroot=/var/tmp/rpm-ostree.QNWFWX/rootfs.tmp shell
Config error: releasever not given and can not be detected from the installroot.

yum -y --disablerepo='*' --setopt=reposdir=/tmp/atomic-treecomposeKtoBYS.tmp --enablerepo=CentOS-Base --enablerepo=CentOS-updates --enablerepo=CentOS-extras --enablerepo=atomic7-testing --enablerepo=virt7-testing --setopt=keepcache=0 --setopt=cachedir=/var/tmp/rpm-ostree.QNWFWX/yum-cache --installroot=/var/tmp/rpm-ostree.QNWFWX/rootfs.tmp --releasever=7 shell
WORKS

hif has a function hif_context_set_release_ver which doesn't seem to be set by rpm-ostree.

A start to fix this is:

diff --git a/src/rpmostree-compose-builtin-tree.c b/src/rpmostree-compose-builtin-tree.c
index efb566c..f895e1a 100644
--- a/src/rpmostree-compose-builtin-tree.c
+++ b/src/rpmostree-compose-builtin-tree.c
@@ -172,6 +172,7 @@ install_packages_in_root (RpmOstreeTreeComposeContext _self,
char *_packages,
gboolean _out_unmodified,
char *_out_new_inputhash,

•                      gchar         *release_ver,
                     GCancellable    *cancellable,
                     GError         **error)
  

  {
  @@ -213,6 +214,8 @@ install_packages_in_root (RpmOstreeTreeComposeContext *self,

  hif_context_set_repo_dir (hifctx, gs_file_get_path_cached (contextdir));

• hif_context_set_release_ver(hifctx, release_ver);

{ JsonNode *install_langs_n =
json_object_get_member (treedata, "install-langs");

@@ -872,6 +875,7 @@ rpmostree_compose_builtin_tree (int argc,
(char**)packages->pdata,
opt_force_nocache ? NULL : &unmodified,
&new_inputhash,

•                                //////RELEASEVER HERE
                              cancellable, error))
  

  goto out;

However, I cannot see a location in src/rpmostree-compose-builtin-tree.c where the target version is accessible.

Tips for me to build a fix to resolve this issue would be great.

Rather than doing the [=== ] bit, let's do something like emit an output line every 5 seconds or so, so that people doing log files + live-tailing (as e.g. Jenkins does) see something.

Wrap ostree admin switch, but preserve layered packages.

#79 caused a regression in combination with #69 because the data is in both files, we end up accumulating copies of it in the temporary /etc/passwd, and then when re-split, that duplicate data ends up in the new /usr/lib/passwd.

The fix here will be to ignore duplicates when creating the temporary /etc/passwd.

Should be equivalent of "yum check-updates" - tell me if I have a new commit.

We should have something like atomic operation-history that is similar to yum history in binding changes to the audit loginuid that initiated them. One possible approach is to keep this in the systemd journal as structured data.

As of recently, rpm-ostree supports live updates ( #639 ) particularly for "pure package additions".

kpatch is logically a pure package addition.

RHEL spends a lot of effort on generating kpatches, and we should support their use on rpm-ostree systems too.

• https://github.com/dynup/kpatch
• https://www.redhat.com/en/blog/overcoming-vulnerabilities-live-kernel-patching-red-hat-enterprise-linux-85

Currently admins install RPMs (but yum plugin has been prototyped). These RPMs include a kernel module that is loaded; the data is dropped in /var/lib/kpatch (SCAP knows to scan there).

The RPM also regenerates the initramfs to ensure that the kpatch is also applied on the next boot (but that's optional).

I think the most important thing to happen here is to figure out the desired administrator UX, as well as what the update provider supports. There are a lot of fundamental questions.

The RPM-delivery model has kpatches as distinct stream of RPMs (yum repository). Perhaps the simplest start is to just match the kpatch-rpm experience via something like:

$ rpm-ostree install -A "kpatch-patch = $(uname -r)"

(This also heavily intersects with #2883 in that we can make it literally the same command)

There's a bunch of other tweaks to do; ideally rpm-ostree status shows you the kpatches that are applied)

Following #101 I pulled the /usr/lib/passwd and /usr/lib/group out of a tree, then cleaned up and tried feeding them back in via check-passwd with file.

However, the generated /etc/passwd was missing the root user, with obvious bad consequences when I actually tried to run it.

Matches Fedora direction

This is the direction Fedora is taking, with these libraries underlying dnf (and PackageKit-on-Fedora). The more we share code, the better.

Package layering

We can only do package layering via more precise control over package installation. See https://github.com/cgwalters/atomic-pkglayer

Treecompose improvements

We can make significant improvements in the treecompose side - for example, download metadata only first, check the repository timestamp and skip doing a compose if it's unchanged.

New Docker container packer

I would like to develop a tool like rpm-ostree that knows how to build Docker (and other types of) containers from the outside. (Perhaps even rename the tool to 'atomic' and have it know how to build containers too). Basically, rather than having yum/dnf inside the container, we compute the content on the host and inject the RPMs inside. There are a lot of powerful benefits to this model:

• Containers become much smaller, potentially even don't have bash
• We can easily have a unified package cache on the host system. I find it really irritating how I have to keep downloading the same packages from Dockerfile
• A declarative format would allow more robust build systems

Is rpm-ostree still considered "beta" given that it's shipping in several projects? I know it's still evolving, but I wonder if "beta" is still apt? (Perhaps if we're using the Gmail sense of "beta"...)

Suppose that an OS vendor wants to provide a custom tree just to a specific user that contains a hotfix.

Of course one mechanism for hot fixes could be #107
but let's presume for this bug that we are focusing on the "new tree" approach. (Although possibly provided as a delta, knowing the customer is running the latest tree).

Regardless, let's assume that this tree data is provided out of band - in a tarball, Docker container, or whatever that is only accessible to that user.

We want a user experience here where the user can:

• Switch to the tree
• The tree is clearly labeled as a hotfix
• There is an easy command to swtich back to the mainline tree (e.g. rpm-ostree upgrade should clearly show a message that you're on a dead-end hotfix stream)

Things are obviously simpler here if the OS vendor provides online-accessible hotfixes, possibly in the same repository; then one can use rebase to switch to it.

PR: ostreedev/ostree#1503

--

If an admin runs "atomic upgrade", then makes changes to /etc, then reboots, those changes are "lost" (they're still in the old tree).

Fixing this is a bit nontrivial. One simple approach would be to encourage "atomic upgrade -r". Or alternatively, split "atomic prepare-upgrade" -> "atomic apply-upgrade". (The latter gets into making atomic a daemon, so we can use a systemctl isolate target for doing the /etc merging just before reboot)

See http://marc.info/?l=selinux&m=139576819325212&w=2

The "one week" default is pretty broken for many use cases, e.g. where I have a GA repo I'm using as file:/// that I want to honor forever.

(Maybe another alternative is to consider file:/// always up to date...I think I wrote a patch?)

Anyways let's call hif_context_set_cache_age (G_MAXUINT)

One trick that https://github.com/libguestfs/supermin does to make VMs as non-root is simply ignore RPM %post. I think %post scripts come in two forms:

• Optional optimizations ldconfig, gtk-update-icon-cache
• Required: useradd, systemctl restart foo, systemctl enable

However for useradd, if we're making trees that only run as one uid, we can ignore useradd. We want to ignore things like systemctl restart always. systemctl enable is best done via presets or the treefile.

Particularly if we're making small containers that run as non-root, I think we can entirely ignore %post.

Related:
rpm-software-management/rpm#8

On "rpm-ostree upgrade", we should support detecting:

1. An unconfigured remote
2. An HTTP 403

And allow calling into an external tool to perform configuration (and potentially a retry).

In the scenario where an OS vendor has multiple repositories with the same branch (e.g. a continuous and released one), one can use the following to switch repositories:

ostree remote add other http://example.com/other/repo
atomic rebase other:

However, 90% of the people that have tried this have omitted the trailing :. We might as well have a convenient:

atomic set-origin other http://example.com/other/repo <optional branch>

With the new summary file, if the branch is not provided, we could download it and offer one.

We want the composes to be fast for many obvious reasons.

See related: https://bugzilla.gnome.org/show_bug.cgi?id=707827

James has an idea here for rpm-ostree.

I was probably too agressive with OSTree in relying on only on https://wiki.gnome.org/GnomeGoals/InstalledTests . Let's take a look at a project which is doing both make check and installed, and handle both. glib/json-glib? Cockpit also has invested a lot in tests, should look there too.

If we're going to do package layering, we need metadata caching, particularly for Fedora.

See https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-addressable-web-proxy/#content

Would it make sense to reuse the local OSTree repo for this? I think probably not...for a cache you want a robust access time, and I'm not sure about reusing the file atimes.

So a /var/cache/objects? For things like package metadata, tools could take a hardlink to them to ensure they're kept around. For downloaded packages, just leave them in the cache, and rely on a regular caching algorithm for eviction. How about API? Filesystem based? DBus? Trust clients or not?

9a20073

Regressed upgrades from existing systems as e.g. the "docker" group will drop out of /usr/lib/group and end up in /usr/etc/group, which will likely be modified downstream. Then docker will fail to start.

I think we should write to both files.

Small nit but I think we can fix it to properly get it's own line. See example output below:

-bash-4.2# rpm -q rpm-ostree-client
rpm-ostree-client-2015.3-3.atomic.el7.x86_64
-bash-4.2# rpm-ostree rebase lab:badtree

26 metadata, 37 content objects fetched; 101802 KiB transferred in 7 secondsCopying /etc changes: 26 modified, 8 removed, 70 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 180.1 MB
Deleting ref 'lab:labtree'
Changed:
  etcd-2.0.11-2.el7.x86_64
  kubernetes-0.17.1-1.el7.x86_64
Removed:
  setools-console-3.3.7-46.el7.x86_64

We need something like yum updateinfo/yum update-security that can show an administrator when a tree update contains CVE fixes and the like. To implement this, we need to parse updateinfo.xml on the server side, and then I suggest converting it to a saner GVariant format, and embedding that in the commit metadata.

See https://bugzilla.gnome.org/show_bug.cgi?id=729585

The checking code from #56 landed, and started triggering for me on the dockerroot user. It's nice to know it works. Then the issue is... "what now"?

It turns out in the case of dockerroot it's actually unused, so we could fix this by deleting it. But in general we need to support dynamic uids/gids/. And we can't yet take a hard dep on #49.

So here's an idea: why don't we just (optionally, but probably by default) take a copy of the passwd/group data from the previous commit? We would merge the data from the altfiles split into /etc before doing any package operations at all.

Red Hat uses a tool called "subscription-manager" to allow access to content. Right now it has support for configuring the ostree remote with the new tls-ca-cert bits.

However, we need a bit more than that:

1. Unconfigured state: if you type "atomic upgrade", but the system is unconfigured, and you need to run subman.

Rather than plugins, we could just have build-time hooks, something like: --with-unconfigured-tool=subscription-manager

2. Handling of 403 due to expired certificates. We could add a config option to ostree like: tls-cert-tool=subscription-manager

This would just be a string that we would use when encountering errors, we could say:

error: Certificate rejected, use the external command "subscription-manager" to adjust configuration

systemd started shipping tmpfiles.d snippets for a variety of directories; this causes conflict warnings at runtime.

rpm-ostree would need to scan the tmpfiles.d snippets at install time and skip synthesizing entries if they are already specified.

Now that rpm-ostree compose doesn't use yum, it seems to have become more fragile to network failures. In particular, it doesn't seem to try mirrors or retry when a package can't be downloaded:

Downloading packages [=                                                   ]   3%
error: cannot download Packages/e/e2fsprogs-1.42.12-3.fc22.x86_64.rpm to /var/tmp/rpm-ostree.0BB5WX/cache/fedora-22/packages/: Curl error (56): Failure when receiving data from the peer for ftp://ftp.uni-bayreuth.de/pub/linux/fedora/linux/development/22/x86_64/os/Packages/e/e2fsprogs-1.42.12-3.fc22.x86_64.rpm [response reading failed]

Because ostree ships numeric uid/gid, we need some mechanism to ensure that packages get assigned the same uids across tree composes.

See https://bugzilla.gnome.org/show_bug.cgi?id=729118

A possible fix would be to force the mapping to be listed in the .json file.

Right now it's hardcoded to rpm-sign which is unnecessary. Actually, we should probably move this code to ostree, something like "ostree commit-sign"?

Since 827e711 we now migrate all password entries except root. This regressed running "usermod" to add users to existing groups - namely wheel.

I haven't debugged shadow-utils yet, but from strace it's not getting EROFS or EPERM, just silently failing (exit code 0).

Right now we drop the input treefile in /usr/share/rpm-ostree/treefile.json - which is important and useful for downstreams to know how to rebuild the system.

Except we don't include external files such as passwd/group, nor postprocess-script. We should define a "flattened" format similar to what ksflatten does so that all of the inputs can be known downstream.

(Note: not called atomic-upgraded since the "d" there parses as English past tense, not Unix daemon)

Some deployments would like to enable automatic updates. We need a mode to make that work well.

See: #40
See: https://github.com/coreos/locksmith

As a followup to the "rpm" subcommand, due to concerns about duplication, let's rename it to 'db'.

It is confusing if you pull from an unversioned tree and suddenly status loses versions.

We should figure out how to fit both it and the timestamp I think.