In openshift/redhat-release-coreos#6 there was mention of either disabling docker by default or removing it completely from the compose.

Rather than litter that PR with design discussion, I've opened this issue. 😄

While I'm in favor of both of these (maybe disable first, then incrementally move towards removal) the use case that immediately comes to mind is oc cluster up or minishift. The oc cluster up case still requires docker (not sure about minishift). I believe one or both of those were being considered as a development path for OpenShift devs?

So outright removal doesn't seem wise right now?

In all of our discussions of "Ignition runs once", in fact we will almost certainly need it to run twice when doing a pivot. Alternatively, it can be the installer's job. We will need to decide this soon.

I can imagine the installer doing this by basically laying down ssh creds, then adding a systemd unit like this:

[Unit]
ConditionPathExists=!/usr/lib/os.release.d/os-release-rhcos
[Service]
ExecStart=/opt/installer/bin/pivot ...

Then the other systemd units all do the opposite:

[Unit]
ConditionPathExists=/usr/lib/os.release.d/os-release-rhcos
[Service]
ExecStart=/opt/installer/bin/install ...

So we don't do Ignition twice, but all of the systemd units are conditionalized to only run in one case or the other.

There's a window in the current pipeline where the image bucket will 404 while the new images are being sync'd.

In an early meetings we had talked about gating ssh access via a menu of some kind. Over the weekend I threw together this as a possible starting point. The questions I have are:

1. Do we still want to gate ssh access via a menu in RHCOS?
2. If so, what menu options should be present?
3. If so, does this look like a good starting point?

/cc @cgwalters @bgilbert @dustymabe @lucab

Note: This isn't something we need to figure out now, but since I put together the above code I figured we might as well restart the conversation.

Shouldn't be. We should being using NM. Offhand...I suspect this is rd.neednet=1 which we're using for Ignition - dracut may have some rules somewhere which is telling the target to enable network.service? If so I'm not finding it.

/etc/os-release currently has:

ID="coreos"

That ID has long been used by Container Linux, so this seems likely to cause confusion. Define a new ID instead.

currently ignition disks stage is missing from RHCOS because we pulled it out in the dracut modules. We need to re-evaluate this and put it back it and also make sure our "boot once detection" still works.

Bug

Operating System Version

• RHCOS ~4392

Ignition Version

0.27.0-2.gitcc7ebe0.fc30

Environment

• libvirt
• openstack

What hardware/cloud provider/hypervisor is being used to run Ignition?

• libvirt
• openstack

Expected Behavior

If ignition is unable to honor the config the boot will fail with log output

Actual Behavior

If ignition is unable to honor the config the booting hangs with no log output.

Reproduction Steps

1. Provide an ignition config to RHCOS (example to be provided)
2. Booting hangs with no logs

Other Information

The following PRs addressed similar issues earlier on:

• #158
• https://github.com/dustymabe/ignition-dracut/pull/13

This was reported by @eparis.

I'm seeing the following over and over in the cloud build:

+ ostree --repo=repo rev-parse 'fef5001fa01e0a4fc6537568ee0649b7420fb67a97e4a849cda33fe304fd6972^'
error: Commit fef5001fa01e0a4fc6537568ee0649b7420fb67a97e4a849cda33fe304fd6972 has no parent

+ ostree --repo=repo rev-parse '9b6ede677640701890b4ac647bfe2ff6d5694c8312f0b4f70c53fecb2c66c748^'
error: Commit 9b6ede677640701890b4ac647bfe2ff6d5694c8312f0b4f70c53fecb2c66c748 has no parent

Line is https://github.com/openshift/os/blob/master/Jenkinsfile.cloud#L168

Right now, we have some magic in bootengine for detecting AWS based on Xen and OpenStack based on dmidecode output, though this is clearly not ideal. IIUC, CL had different OEM packages for each cloud that drops in a small grub cfg script which is used at runtime to build the kernel cmdline, e.g. https://github.com/coreos/coreos-overlay/blob/master/coreos-base/oem-gce/files/grub.cfg.

One idea is to have a small libguestfs script that does something similar for each cloud-specific image we provide before uploading them?

New group entry: systemd-network

(rpm-ostree compose tree:444): GLib-GObject-CRITICAL **: 14:48:48.544: g_object_get_qdata: assertion 'G_IS_OBJECT (object)' failed

(rpm-ostree compose tree:444): GLib-GIO-CRITICAL **: 14:48:48.544: g_file_get_path: assertion 'G_IS_FILE (file)' failed

Ugh. Not sure if it's related to the recent merge or not. Can't repro locally yet. Debugging.

So I was setting up my local dev, following the docs for openshift on libvirt. I downloaded the latest rhcos image and went through the install, but bootkube failed because it couldn't read a config file. I know this seems like an installer issue, but I think it's an rhcos issue, and you will see why in a second.

Essentially, it was mounting some volumes onto a container to render some assets (see https://github.com/openshift/installer/blob/master/modules/bootkube/resources/bootkube.sh#L7-L12 and https://github.com/openshift/installer/blob/master/modules/bootkube/resources/bootkube.service#L11) and the renderer needed /opt/tectonic/kco-config.yaml (mounted into the container at /assets/kco-config.yaml). However only the directory structure of /opt/tectonic was making it into the container (I used an alpine container to figure that out). In the course of debugging, I tried adding the :z suffix to the volume mount with the alpine container, and the problem went away. When I tried to repro the original issue after that, it was gone, even without the :z suffix.

Reading up on the :z volume mount suffix, it seems like it can have consequences on the host machine (https://stackoverflow.com/a/35222815), particularly, it seems to manipulate selinux labels in some way. My guess is that something changed in the base image kind of recently having to do with the labels on /opt/tectonic (or /var/opt/tectonic which afaict is hardlinked one way or the other), and now the docker command fails to mount the files in properly, and the :z suffix inadvertently fixes the issue.

I'm not sure what could've changed to cause this, but I figured I would throw it out there and see if anyone had some insight. It was definitely a thorn at any rate.

We should provide a tool that handles all the bits to do a container pivot ➰, something like:

$ coreos-container-pivot registry.example.com:exampleos:latest

Also in order to make things work better with ignition/cloud-init, maybe have it exit 77 if nothing changed, and optionally add a --reboot arg, so that one can do (in shell script):

do_pivot() {
  set +e
  coreos-container-pivot registry.example.com:exampleos:latest
  rc=$?
  set -e
  case "${rc}" in
    0) systemctl reboot;;;
    77) ;;
    *) exit "${rc}"
  esac
}

Or just:

coreos-container-pivot --reboot registry.example.com:exampleos:latest

and that's a no-op if we're already targeting that container.

Supporting some of this nicely is probably going to require driving down into rpm-ostree's origin data what the container sha256 is or so.

As a result of running kola run against RHCOS I've found that Ignition is failing when attempting to create directories inside of /sysroot.

[    8.262414] Starting Setup Virtual Console...ignition[734]: CRITICAL : files: createFilesystemsFiles: createFiles: op(4): [failed]   writing file "/resource/data": mkdir /sysroot/resource: permission  denied

I'm unsure if this is something that's intended.

Located here for lack of a better location to put this issue up.

Ignition should only run once. On CL today this is handled by detecting a file on the boot partition in the grub config. If this file is noticed a kernel command line argument is set, which a systemd generator in the initrd notices and then enables the Ignition units. One of these units then deletes the file once Ignition succeeds.

The way I see it there are two main options for achieving this in red hat land:

Detect a file on the boot partition via the grub config

Similar (identical, really) logic as CL could be used in the grub config of RHCOS to trigger Ignition. Additionally it's really straightforward to enable networking in the initrd by adding some kernel command line arguments, so adding those only when Ignition is going to run would be easy.

The biggest downside I see is that this solution becomes much trickier to implement in an RPM for a system that doesn't include Ignition by default (e.g. Fedora). Modifying the grub config for the sake of Ignition on those platforms seems like a bad idea, so if RHCOS goes with this solution a different one will need to be devised for Fedora/etc.

Detect a file on the boot partition via Ignition's systemd generator

A systemd generator is used in the initrd to determine if Ignition should run. This generator could look for the file on the boot partition itself, and enable the network and Ignition when its present.

The downsides of this are:

• I already tried this, but the generator was failing for reasons I don't understand. I may have simply had a programming error, but the generator would succeed when manually run from an emergency shell, so I think the generators run early enough to be racing with udev.
• The generator needs to be able to find the boot partition, so we'll need to mandate that the boot partition has a well known label (e.g. boot). Such a mandate may not be possible on other platforms such as Fedora.
• I have yet to find a reasonable way to start the network once the initrd is running. Finding someone more experienced with dracut may make this a non-issue.

I'd love people's thoughts on this, especially on if I'm missing any other big positives/negatives for this approach or if we should strive to keep the RHCOS run-Ignition-once solution the same as we use on other platforms.

We should make use of tags; today we just push :latest, and I manually pushed a :alpha tag.

It'd be great to have someone look at automatically tagging :alpha after it passes tests - and also ensuring that once this is done we have corresponding images generated (and hmm....we should probably put the kickstart in the oscontainer too!)

Also, to make things more complex: I think we should defer uploading AMIs until after basic qemu sanity checking. Right now the AMI upload is 20 minutes; we could have run tons of basic qemu tests in that time.

So maybe the flow is e.g.:

treecompose -> :latest
basic sanity tests -> :alpha-sanity -> upload to EC2
Once it's in EC2 we do a lot more extensive testing, maybe of a cluster -> :alpha-cluster-sanity

Promotion from :alpha-cluster-sanity to e.g. :beta would be manual?

Yet another tricky thing in all of this is that we will almost certainly need to (in practice) decouple OS-basics testing from kube-and-above testing.

I think the idea of https://github.com/coreos/toolbox is really critical - even if the implementation is very basic.

My strawman: Write a new tool that uses podman via the varlink API. Have some opinionated defaults, like mounting /srv:/srv for passthrough, and also e.g. -v /etc:/host/etc:ro -v /usr:/host/usr:ro say. But also just pass through some basic options like --pid=host.

In my dev containers I ended up doing -v /home:/srv/container-home so that I get persistent history. There's a whole big decision point about using --rm by default here (which forces storing state elsewhere, IMO a good idea) or not.

@mrunalp @lsm5 have noted that cri-o will need cri-tools. This should be added to the RHCOS composes.

I have identified two compose runs that obviously have different content in them that is reporting "no changes":

The jenkins pipeline view shows 1168 building with ignition-0.23.0-0.7.el7.x86_64 (dustymabe-ignition) and 1169 buildilng with ignition-0.23.0-0.8.el7.x86_64 but 1169 reports at the bottom: No apparent changes since previous commit

Right now all build logic for the OS is in Jenkinsfiles. This is great for a pipeline, terrible for people who want to build locally and test things. I believe we need to find a way to separate building from the pipeline so that developers can build local dev versions of RHCOS.

See #107

Not entirely sure what's going wrong...looking through the logs quickly I see some etcd cert issues at least.

# rpm-ostree status
State: idle; auto updates disabled
Deployments:
● ostree://rhcos:openshift/3.10/x86_64/os
                   Version: 3.10-7.5.23 (2018-05-29 16:09:02)
                    Commit: d953d2db12ae9f778814d879ef5ad7a7636c5a9de86b0fc8c72ac46f9aa255db
# rpm -q origin
origin-3.10.0-0.alpha.0.1323.836f8e3.x86_64

Currently we use the Fedora guildelines for ExclusiveArch but our buildroot today isn't pulling in the necessary golang macros, so we end up with ExclusiveArch: (nothing).

This is a painful type of issue when mixing fedora/el spec builds. We can likely just drop the ExclusiveArch for now in our spec - but it might also not be too hard to add the golang macros to our buildroot definition.

Ignition expects that systemd-preset will run and take care of creating relevant symlinks.

As a result currently any systemd unit created by Ignition will not run despite being enabled.

Per @cgwalters systemd-preset won't run on RHCOS:
that's another consequence of the ConditionFirstBoot= not being set - it only works for systems with empty /etc (neither classic fedora/rhel nor ostree-based systems)

The problem now is https://github.com/cgwalters/qcow2-to-vagrant doesn't handle all of the formatting goop for e.g. VirtualBox too. That may not be hard to add...but at this point it's probably saner to have the Vagrant box be a separate ImageFactory run like it is with FAH today.

That said...conceptually a nice approach for Vagrant would be to inject a default Ignition config that sets it all up rather than the systemd unit we do now - but only if we can extract the "make an OVA" bits from ImageFactory without duplicating too much code.

When a non-default user is specified in an Ignition config it fails during the useradd trying to create the home directory.

[   19.778871] ignition[723]: CRITICAL : files: failed to create users/groups: failed to create users: failed to create user "core": exit status 12: Cmd: "/usr/sbin/useradd" "--root" "/sysroot" "--       create-home" "--password" "*" "core" Stdout: "" Stderr: "useradd: cannot create directory /home/core\n"] Started Emergency Shell.

Some additional investigation into the ordering might be required if the /home symlink is non-functional currently.

Full serial console run @ https://slack-files.com/T027F3GAJ-FB33Q74Q1-bef7bf3c2c

etcdctl is present on Container Linux.

Is this something that we'd want to ship in the base OS or something that we expect users to run inside of a container?

The RHCOS ostree repo only has a single commit in it at the end of the pipeline. This makes collecting pkg diffs between commits via rpm-ostree db diff not quite as easy as before.

I think it would be useful to generate these kinds of changes and publish them as part of the metadata we make available. But I think we should decide on which versions we are diff'ing.

In #77, @cgwalters suggested starting to tag buildmaster, latest, beta, and stable versions of the oscontainer, where buildmaster has no testing and latest has minimal smoke testing.

Using the above tags, I suggest to start with a latest -> beta diff and beta -> stable diff as the set of pkg diff metadata we would publish to users.

We could keep diffs between each buildmaster or each latest internal for our own debugging purposes?

Any thoughts or suggestions?

The patches all got merged, the problem seems to be for some reason we don't have /boot/coreos-firstboot in /boot in the images. Debugging.

We dropped cloud-init.

Today we don't include coreos-metadata.

Hi,
As title suggests, I was unable to use pivot/(docker + rebase) locally. Below is the method I have tried.

-------docker login is done earlier ------
[root@localhost ~]# ./pivot registry.svc.ci.openshift.org/rhcos/os:latest
pivot version 0.0.0 (56ac45679f34297b39fdc74b874900efa33eb8e5)
Running: skopeo inspect docker://registry.svc.ci.openshift.org/rhcos/os:latest
Trying to pull registry.svc.ci.openshift.org/rhcos/os:latest@sha256:05815042ae65eabac9c1406b7a8cc8fae45345407fe95cc939367f2d2a031ea2...Getting image source signatures
Copying blob sha256:9fd0835072117c2226c8ae1334aaf880e02edbba573329c3e3bcb7f6136a0dac
 71.23 MB / 71.23 MB [======================================================] 8s
Copying blob sha256:b3ccca4efa9f6752e8706e6de91a5db79717ce248517a4e9f374f06e7dca7097
 709.60 MB / 709.60 MB [=================================================] 1m27s
Copying config sha256:03b98184dc2939e423efefaa044d31ed06d717ee09351affab5b68ed0b7562c6
 2.21 KB / 2.21 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
03b98184dc2939e423efefaa044d31ed06d717ee09351affab5b68ed0b7562c6
Running: podman run -d --name ostree-container-pivot --entrypoint sleep registry.svc.ci.openshift.org/rhcos/os:latest@sha256:05815042ae65eabac9c1406b7a8cc8fae45345407fe95cc939367f2d2a031ea2 infinity
Running: podman mount 83ff4446d6fd4bec3d90f74c340b0e853d1a0a9546b76fcd22ff6bd662416152
Running: ostree --repo=srv/tree/repo refs
error: /var/lib/containers/storage/overlay/8ffccd95b35cabd32a7ff839004623a483a1b1d85b5bc5bc49cd651fb705d73b/merged/srv/tree/repo: opendir(/var/lib/containers/storage/overlay/8ffccd95b35cabd32a7ff839004623a483a1b1d85b5bc5bc49cd651fb705d73b/merged/srv/tree/repo): No such file or directory
F0706 14:33:40.715089    2900 run.go:34] ostree: exit status 1
goroutine 1 [running]:
github.com/ashcrow/pivot/vendor/github.com/golang/glog.stacks(0xc420016e00, 0xc420066a20, 0x3f, 0x10b)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/golang/glog/glog.go:769 +0xa7
github.com/ashcrow/pivot/vendor/github.com/golang/glog.(*loggingT).output(0x6e7d40, 0xc400000003, 0xc420094160, 0x6c99d8, 0x6, 0x22, 0x0)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/golang/glog/glog.go:720 +0x36b
github.com/ashcrow/pivot/vendor/github.com/golang/glog.(*loggingT).printf(0x6e7d40, 0xc400000003, 0x5ee520, 0x6, 0xc4200cb7e8, 0x2, 0x2)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/golang/glog/glog.go:655 +0x14f
github.com/ashcrow/pivot/vendor/github.com/golang/glog.Fatalf(0x5ee520, 0x6, 0xc4200cb7e8, 0x2, 0x2)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/golang/glog/glog.go:1148 +0x67
github.com/ashcrow/pivot/utils.RunGetOut(0x5ee742, 0x6, 0xc4200cbab0, 0x2, 0x2, 0xc4200438f0, 0x6b)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/utils/run.go:34 +0x310
github.com/ashcrow/pivot/cmd.Execute(0x6e58c0, 0xc420083a70, 0x1, 0x1)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/cmd/root.go:89 +0x5e2
github.com/ashcrow/pivot/vendor/github.com/spf13/cobra.(*Command).execute(0x6e58c0, 0xc42000a1d0, 0x1, 0x1, 0x6e58c0, 0xc42000a1d0)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/spf13/cobra/command.go:766 +0x2b9
github.com/ashcrow/pivot/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x6e58c0, 0x4, 0x6ce500, 0xc420083a20)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/spf13/cobra/command.go:852 +0x37e
github.com/ashcrow/pivot/vendor/github.com/spf13/cobra.(*Command).Execute(0x6e58c0, 0xb, 0x5ee21a)
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/home/steve/Tech/Envs/go/src/github.com/ashcrow/pivot/main.go:25 +0x154
[root@localhost ~]# rpm-ostree status
State: idle; auto updates disabled
Deployments:
* ostree://fedora-atomic:fedora/28/x86_64/atomic-host
                   Version: 28.20180527.0 (2018-05-27 19:05:29)
                    Commit: 291ea90da29bc5abe757b5a50813b3de1396b08412939a89b3b671aba9856093
              GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1

As above suggests, the image seems to be pulling down correctly, but neither pivot or docker way are able to rebase/use the /repo content from the container. I have seen this behaviour for two hosts.. Am I missing something? Thanks :).

By default it doesn't seem like any groups exist in /etc/sudoers. On Container Linux both the %sudo group & the core user have NOPASSWD.

The default sudoers for CL (from /usr/share/baselayout/sudoers):

## Default sudoers for CoreOS

## Pass LESSCHARSET through for systemd commands run through sudo that call less.
## See https://github.com/coreos/bugs/issues/365.
Defaults env_keep += "LESSCHARSET"

## enable root and wheel
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL

## enable passwordless access for sudo
%sudo ALL=(ALL) NOPASSWD: ALL

## the core user has no password by default
core ALL=(ALL) NOPASSWD: ALL

## Read drop-in files from /etc/sudoers.d
## (the '#' here does not indicate a comment)
#includedir /etc/sudoers.d

We dropped cloud-init which has disk resizing code, and it turns out Ignition doesn't do this, see https://github.com/coreos/init/blob/master/scripts/extend-filesystems

Now the next problem we have is that script doesn't handle LVM...in practice what we did with Fedora Atomic was that it was actually container-storage-setup that does this, I need to figure out why that's not working.

There are a ton of files in the top level and they are for various purposes. It would be nice if we could group things together in subfolders and have a structure that was more user friendly.

Thoughts?

I find this very convenient myself. We could use https://github.com/cgwalters/qcow2-to-vagrant or add a separate imagefactory run.

AFAIK we have enforcing=0 as a hack to bringup Ignition. We need to ship in enforcing mode.

Ignition fails to pull user_data due to what appears to be a networking issue. I'm using rhcos-openstack.qcow2.gz from http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/cloud/latest/ dated some time last week. I can replicate this on both OpenStack v9 and v10. The console logs are as follows:

Right now we don't tag releases, and our CI doesn't gate anything.

Our first step here should be having kola or whatever actually be involved in promoting a tag.

We also need an explicit tagging/promotion model of e.g. "alpha/beta/stable".

Using the tag strategy in #77, are there any opinions on how many versions of buildmaster (no testing completed on that tag) we should retain?

@cgwalters suggested 3-4 in an internal Jira issue, which seems like a good amount. Any other opinions on the matter?

I think the removal of subscription-manager and its associated dependencies caused RHCOS to lose the Red Hat CA. This is impacting the ability to pull container images from the Red Hat registry.

This looks similar to CentOS/sig-atomic-buildscripts#329

# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://rhcos:openshift/3.10/x86_64/os
                   Version: 3.10-7.5 (2018-07-17 19:50:05)
                    Commit: 195f9e0cf36d04682fc8d380093b40eef1dd96009e85ce86a7e5419029e4aeea

# podman --log-level debug pull registry.access.redhat.com/rhel
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: override_kernelcheck=true           
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true 
INFO[0000] CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf 
INFO[0000] Initial CNI setting succeeded                
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]registry.access.redhat.com/rhel:latest" 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]registry.access.redhat.com/rhel:latest" does not resolve to an image ID 
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]registry.access.redhat.com/rhel:latest" 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]registry.access.redhat.com/rhel:latest" does not resolve to an image ID 
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]registry.access.redhat.com/rhel:latest" 
Trying to pull registry.access.redhat.com/rhel...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.access.redhat.com/rhel:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com 
DEBU[0000]  crt: /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt 
Failed
ERRO[0000] error pulling image "registry.access.redhat.com/rhel": unable to pull registry.access.redhat.com/rhel 

# docker pull registry.access.redhat.com/rhel
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel ... 
open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

# ls -l /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt 
lrwxrwxrwx. 1 root root 27 Jul 17 20:25 /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem

# ls -l /etc/rhsm/ca/redhat-uep.pem
ls: cannot access /etc/rhsm/ca/redhat-uep.pem: No such file or directory

# rpm -qf /etc/rhsm/ca/redhat-uep/.pem
error: file /etc/rhsm/ca/redhat-uep/.pem: No such file or directory

@crawford has found in tests that /sys/fs/cgroup/cpuacct,cpu is being expected during his testing but RHCOS provides /sys/fs/cgroup/cpu,cpuacct.

kubernetes/kubernetes#32728 (comment) denotes a similar issue. The workaround is to setup a link from one to the other.

We're still not getting the crio change; looks like treecompose isn't picking up the updated rdgo repo:

rpm-md repo 'rhcos-continuous'; generated: 2018-07-13 14:50:15

Something going wrong with the s3 sync?

Not sure what's going on. It seems like we're missing the BR on golang.

I was getting images from http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/ but that just 404s now...

We need to decide whether to include a docker group or not. Today CL does, Fedora does not. Context: https://www.projectatomic.io/blog/2015/08/why-we-dont-let-non-root-users-run-docker-in-centos-fedora-or-rhel/

1. Move to a jenkins instance in api.ci; see openshift/release#1000
   1a) Use GCE nested virt for vmimage builds
2. Split out the pipeline to a separate repo
3. Move to pinning all of the base (non-rdgo) RPMs in the manifest; updates to them are driven a bot that submits PRs. Think like what Cockpit does to update their VMs.
4. Change rdgo to work the same way but with git commits
   4a) Related, rewrite or replace/rework rpmdistro-gitoverlay
5. Containerize the imagefactory bits so we can consistently use containers and not classic systems

While debugging coreos.selinux.enforcing on RHCOS, the test was reporting command not found for getenforce. Yet when I logged in directly or via Manhole(), I was able to run getenforce just fine.

When I finally inspected the value of $PATH in both cases, I found the issue. When logged in directly (as core user), I get the following $PATH

PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/core/.local/bin:/home/core/bin

...but when running a test via kola:

PATH=/usr/local/bin:/usr/bin

And since getenforce lives in /usr/sbin, the kola tests are unable to find it during execution.

I think I was able to make sense of how $PATH was getting set for the login shell case (thanks man bash!), but I haven't been able to figure out where the default path is configured for the non-login case.

Related to this, the value for $PATH in the non-login case on Container Linux looks like this:

PATH=/usr/bin:/bin:/usr/sbin:/sbin

Today openshift-ansible sets a bunch like e.g. container_manage_cgroup. Let's just turn those on in our compose-post.sh.

Today we're building the ostree twice (once in the Dockerfile triggered by prow, and once in Jenkins), and pushing two different containers to two different registries.

The Dockerfile path entirely tosses the ostree commit history concept and loses our "did something change" handling. We are also not using this container content to build virt images.

We really need to get to one ostree commit stream.

Further, we need to handle versioning for this in a saner way - just pushing :latest isn't going to cut it. I'd strawman we have buildmaster, latest, beta, stable or so - the buildmaster concept here means that it hasn't undergone any tests yet. latest is what I've used smoketested for in the past. Then beta and stable are manual promotions?

Perhaps I am missing something, but as part of our pipeline, we should be pushing the newest oscontainer image to registry.svc.ci.openshift.org, like shown here:

https://github.com/openshift/os/blob/master/Jenkinsfile.treecompose#L121

However, in my own testing (and in the Pull and run oscontainer stage), the :latest image on the registry is showing up as 3 weeks old; last built on 2018-06-27.

The pipeline keeps on chugging along as if everything is fine, though.

Any ideas?

Like https://alpha.release.core-os.net/amd64-usr/current/coreos_production_ami_all.json

Software such as a cluster installer should be able to easily and programmatically find our AMIs (and extend later for similar clouds).