Test data for composes about rpm-ostree HOT 16 CLOSED

Also, the Docker approach would give us uid 0 in a container, which would allow us to write much more interesting tests that more fully exercise the "running as uid 0" parts of ostree/yum while still providing system saftey. Unfortunately, these tests couldn't be run in Brew, but that'd be OK, there are other systems that could support running them.

from rpm-ostree.

Yeh, on the yum side we have unit tests for the internal stuff which runs at %check time. But we just rely on developer testing, mock/etc. and shipping it in rawhide for external testing. QA has some infrastructure that allows them to easily create test packages and repos., and they use that to create small test cases with real data (I think the infrastructure code is public, but it'd still be annoying to create 100+ packages I think).

Not sure if you know about: http://fakeroot.alioth.debian.org/ ... but that is in Fedora.

from rpm-ostree.

neither fakeroot or yocto's pseudo project are able to emulate all that is needed by "rpm-ostree compose tree".

I've started working on an LD_PRELOAD library that should shadow all the calls that require root from rpm-ostree, it doesn't seem like a trivial task, there are many cases that must be handled. I am not yet convinced that this is the correct way though, it seems very fragile.

from rpm-ostree.

pesudo is fairly actively maintained at least. What system calls failed? If it's some of the container bits, we could just turn that off (though obviously, the more we do this, the less we're testing the real system).

from rpm-ostree.

Options I see:

Run as non-root

• LD_PRELOAD
• Use linux-user-chroot

Installed root tests: sudo make rootcheck

This is like https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests but as root. We have to do a sudo make install over the running system, but there's a test suite that operates using the installed binaries, but doesn't break the host. This is similar to what https://github.com/projectatomic/atomic/blob/master/test.sh is doing.

Inject code into a local Docker container

A variant of the above where we do make rpm using a local spec file, then inject it into a Docker (or other) container. Or just make install DESTDIR=, then copy that over into a container. There are a lot of advantages to this - e.g. we could support testing a rawhide build from a RHEL7 host, etc.

Inject code into a VM

Like the above, but with VMs. This is what Cockpit (and other projects I'm sure) do: https://github.com/cockpit-project/cockpit/blob/332c0e4813a541304014603c87cb1e394a3fe307/test/testvm.py#L127

Where the VM approach would be useful for us is that ostree obviously wants to change the kernel configuration. However, this type of testing I think just doesn't fit very well into the make check model. It's best to have it live externally IMO, and make check remains unprivileged unit tests.

from rpm-ostree.

For example pseudo misses emulation unshare.

I've continued adding wrappers of libc functions, until I realized I was ending up with a clone of pseudo. So I reverted to using pseudo and add a new library for the missing pieces, it is much less code to maintain now:

https://github.com/giuseppe/rpm-ostree/tree/giuseppe/test-compose

TODO:

1. Move the emulation of unshare to pseudo.
2. Properly support execle in pseudo, so we can test the post-process phase as well. It seems that it doesn't honor a chroot.
3. RPM-ostree calls directly the clone syscall unless mount fails with EINVAL. This probably should be still done, or we need to wrap it in a way to allow us to emulate it through LD_PRELOAD.

from rpm-ostree.

I think we could skip emulation of some of the container syscalls safely enough. Emulating clone is a bit trickier...maybe we could just degrade it to fork without any container features?

Can we depend on pseudo as a package at some point instead of a submodule? A submodule is ok for now though.

from rpm-ostree.

IMHO, it is safer if the version used in the tests is more loose than the version used by rpm-ostree and we keep using clone and emulate it as best as we can as the "make check" version of compose is not much more than a smoke test. I plan to remove the hack in the fork wrapper once the exec* wrappers in pseudo honors the chroot settings (I have a patch here) and thus we will be able to run depmod and dracut within the rootfs and do the no-op there instead of the fork wrapper itself.

from rpm-ostree.

after thinking more about it, I think such a complex test as running a compose should rather be done using an installed tests and run as root. We should introduce "make check tests" only for unit testing as we do with ostree. What do you think?

from rpm-ostree.

I pushed some other commits to:

https://github.com/giuseppe/rpm-ostree/tree/giuseppe/test-compose

I reverted the usage of pseudo, as it looks too fragile environment for any serious testing.

I've moved the compose test under installed tests, and expect they are executed as root user. There is one warning coming from glibc inotify implementation that makes tests fail on Fedora 21/22, this could be fixed by changing the G_DEBUG setting in libtest.sh (or overloading it in the compose tests) but since the tests run without issues on Rawhide, I left it unchanged.

from rpm-ostree.

Root installed tests sound OK to me...it hasn't been defined in the spec, but that doesn't block us from doing it =) Maybe something like Type=headless-root or something?

from rpm-ostree.

Nothing blocks us though from doing the sudo make rootcheck option, right? That might be easier than root-installed tests.

from rpm-ostree.

Yes, that can be done as well. Would you drop installed tests completely or provide both functionalities?

from rpm-ostree.

it doesn't look like the test repository will work smoothly in the same way from installed tests and when we run tests locally with make check.

Do you think it makes sense to take rid of the installed tests completely and rewrite the tests to be run only locally and maintain only one version?

I will add a check to check if the tests requiring root permissions run with UID=0 or skip them otherwise.

from rpm-ostree.

I've sent a pull request for the compose test. I think we can make the cleanup later, in case we decide to drop installed tests and use only make check (and we should probably make this uniformly with ostree)

from rpm-ostree.

This was done.

from rpm-ostree.