Giter Club home page Giter Club logo

fedora-coreos-config's Introduction

Fedora CoreOS Config

next-devel status

Base manifest configuration for Fedora CoreOS.

Use https://github.com/coreos/coreos-assembler to build it.

Discussions in https://discussion.fedoraproject.org/c/server/coreos. Bug tracking and feature requests at https://github.com/coreos/fedora-coreos-tracker.

About this repo

There is one branch for each stream. The default branch is testing-devel, on which all development happens. See the design and tooling docs for more information about streams.

All file changes in testing-devel are propagated to other branches (to next-devel, branched, and rawhide through config-bot, and to testing and eventually stable through usual promotion), with the following exceptions:

  • manifest.yaml: contains the stream's name, yum repos used during composes, and the releasever.
  • lockfiles (manifest-lock.* files): on testing-devel and next-devel, lockfiles are pushed by the bump-lockfile job. Production streams receive them as part of usual promotion. Overrides (manifest-lock.overrides.*) are managed independently with the help of some GitHub Actions (see sections below).

Layout

We intend for Fedora CoreOS to be used directly for a wide variety of use cases. However, we also want to support "custom" derivatives such as Fedora Silverblue, etc. Hence the configuration in this repository is split up into reusable "layers" and components on the rpm-ostree side.

To derive from this repository, the recommendation is to add it as a git submodule. Then create your own manifest.yaml which does include: fedora-coreos-config/ignition-and-ostree.yaml for example. You will also want to create an overlay.d and symlink in components in this repository's overlay.d.

Overriding packages

By default, all packages for FCOS come from the stable Fedora repos. However, it is sometimes necessary to either hold back some packages, or pull in fixes ahead of Bodhi. To add such overrides, one needs to add the packages to manifest-lock.overrides.yaml (there are also arch-specific variants of these files for the rare occasions the override should only apply to a specific arch). There is a tool to help with this, and for simple cases, an automated workflow that runs the tool and submits a PR.

Note that comments are not preserved in these files. The lockfile supports arbitrary keys under the metadata key to carry information. Some keys are semantically meaningful to humans or other tools.

Fast-tracking

Example:

packages:
  selinux-policy:
    evra: 34.10-1.fc34.noarch
    metadata:
      type: fast-track
      bodhi: https://bodhi.fedoraproject.org/updates/FEDORA-2021-f014ca8326
      reason: https://github.com/coreos/fedora-coreos-tracker/issues/850
  selinux-policy-targeted:
    evra: 34.10-1.fc34.noarch
    metadata:
      type: fast-track
      # you don't have to repeat the other keys for related packages

Whenever possible, it is important that the package be submitted as an update to Bodhi so that we don't have to carry the override for a long time.

Fast-tracked packages will automatically be removed by the remove-graduated-overrides GitHub Action in this repo once they reach the stable Fedora repos (or newer versions). They are detected by the type: fast-track key.

Pinning

Example:

packages:
  dracut:
      evr: 053-5.fc34
      metadata:
        type: pin
        reason: https://github.com/coreos/fedora-coreos-tracker/issues/842
  dracut-network:
      evr: 053-5.fc34
      metadata:
        type: pin
        reason: https://github.com/coreos/fedora-coreos-tracker/issues/842

All pinned packages must have a reason key containing more information about why the pin is necessary.

Once an override PR is merged, coreos-koji-tagger will automatically tag overridden packages into the pool.

Adding packages to the OS

Since testing-devel is directly promoted to testing, it must always be in a known state. The way we enforce this is by requiring all packages to have a corresponding entry in the lockfile.

Therefore, to add new packages to the OS, one must also add the corresponding entries in the lockfiles:

  • for packages which should follow Bodhi updates, place them in manifest-lock.$basearch.json
  • for packages which should remain pinned, place them in manifest-lock.overrides.$basearch.yaml

There will be better tooling to come to enable this, though one easy way to do this is for now:

  • add packages to the correct YAML manifest
  • run cosa fetch --update-lockfile (this will only update the lockfile for the current architecture, most likely x86_64)
  • copy the new lines to the lockfiles for other architectures (i.e. aarch64)
  • commit only the new package entries (skip the timestamped changes to avoid merge conflicts with the lockfile updates made by the bot)

Moving to a new major version (N) of Fedora

Create a rebase checklist in fedora-coreos-tracker.

CoreOS CI

Pull requests submitted to this repo are tested by CoreOS CI. You can see the pipeline executed in .cci.jenkinsfile. For more information, including interacting with CI, see the CoreOS CI documentation.

Tests layout

Tests should follow the following format:

#!/bin/bash
## kola:
##   exclusive: false
##   platforms: aws gcp
##   # See all options in https://coreos.github.io/coreos-assembler/kola/external-tests/#kolajson
#
# Short summary of what the test does, why we need it, etc.
#
# Recommended: Link to corresponding issue or PR
#
# Explain the reasons behind all the kola options:
# - distros: fcos
#   - This test only runs on FCOS due to ...
# - platforms: qemu
#   - This test should ...
# - etc.

set -euxo pipefail

. $KOLA_EXT_DATA/commonlib.sh

foo_bar() <-- Other function definitions

if ...    <-- Actual test code
          <-- Errors must be raised with `fatal()`
          <-- Does not need to end with a call to `ok()`

fedora-coreos-config's People

Contributors

aaradhak avatar arithx avatar bgilbert avatar bh7cw avatar c4rt0 avatar cgwalters avatar coreosbot avatar dustymabe avatar gursewak1997 avatar huijinghei avatar jbtrystram avatar jlebon avatar jmarrero avatar jschintag avatar kelvinfan001 avatar kshithijiyer avatar lakshmiravichandran1 avatar lorbuschris avatar lucab avatar marmijo avatar miabbott avatar mike-nguyen avatar nikita-dubrovskii avatar prashanth684 avatar prestist avatar ravanelli avatar saqibali-2k avatar sinnykumari avatar sohankunkerkar avatar travier avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fedora-coreos-config's Issues

mount generators fail when rebased from Atomic Host/Silverblue

This is probably an unsupported migration path, but I have a system that was running fedora silverblue, planning to rebase it to fedora coreos afterwards. I rebased it today and I have issues with /boot not mounting properly.

Actually, the file /run/systemd/generator/boot.mount is empty.

I debugged coreos-boot-mount-generator (by adding some shell debugging at the top of the file) and I get the following log and error:

+ cat /proc/mounts
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=1899000k,nr_inodes=474750,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/sda5 /sysroot btrfs ro,seclabel,relatime,space_cache,subvolid=256,subvol=/system 0 0
/dev/sda5 / btrfs ro,seclabel,relatime,space_cache,subvolid=256,subvol=/system/ostree/deploy/fedora/deploy/f73012a35a12a34c6f05b78c46df8709f33e4fdbccfad8bdd5fafd0914298266.0 0 0
/dev/sda5 /usr btrfs ro,seclabel,relatime,space_cache,subvolid=256,subvol=/system/ostree/deploy/fedora/deploy/f73012a35a12a34c6f05b78c46df8709f33e4fdbccfad8bdd5fafd0914298266.0/usr 0 0
selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
+ '[' '!' -f /run/ostree-live ']'
+ add_wants boot.mount
+ local name=boot.mount
+ shift
+ local wants_dir=/run/systemd/generator/local-fs.target.wants
+ mkdir -p /run/systemd/generator/local-fs.target.wants
+ ln -sf ../boot.mount /run/systemd/generator/local-fs.target.wants/boot.mount
+ cat
/usr/lib/systemd/system-generators/coreos-boot-mount-generator: line 20: cannot create temp file for here-document: Read-only file system

I didn't know shell here documents required temporary files, but apparently they do and it makes the generator fail. I found some reference about that here: https://www.oilshell.org/blog/2016/10/18.html and https://github.com/bminor/bash/blob/d233b485e83c3a784b803fb894280773f16f2deb/redir.c#L452

As you can see /tmp is not yet mounted, and the shell script is probably failing because of that.

A workaround is to set at the top of the generator:

TMPDIR=$UNIT_DIR

With this fix, now the boot continues (but fails later on because /boot/efi fails because the partition does not have the right label). may I suggest that the generator should also read /etc/fstab in case /boot and /boot/efi are specified, and if they are specified, it should not generate the units...

Note: generator documentation suggest writing generators in C (however this would have made this issue harder to understand): https://www.freedesktop.org/software/systemd/man/systemd.generator.html

Add support for NTP from DHCP

Chrony ignores NTP server from DHCP. I see that some script exists for AWS and other cloud providers, but it definetly doesn't work with old-school DHCP OPTION 42.

Here is my dnsmasq template:

dhcp-option=42,{{ ntpserver }}

Current workaround is to override *.fedora.pool.ntp.org zone in DNS:

{% for ntpserver in ntpservers %}
address=/.fedora.pool.ntp.org/{{ ntpserver }}
{% endfor %}

BTW, may be it is worth to consider to replace chrony with systemd-timesyncd...

`kdump` not working without `dracut-squash`

dracut-squash and squashfs-tools were made weak dependencies by kexec-tools in version 2.0.21-4.fc33.x86_64. However, without these two packages, kdump's generated initramfs will be unable to boot.
xref: https://bugzilla.redhat.com/show_bug.cgi?id=1925585

Testing on Fedora Server, kdump works properly without dracut-squash as expected, so this seems to be a CoreOS-only issue.

This workaround adds back dracut-squash and squashfs-tools to allow kdump to be functional for now, but ideally we shouldn't need these two additional packages.

Assumption of `/dev/disk/by-label/{boot,root}` for non-live targets

The coreos-boot-mount-generator makes a broad assumption that there are devices for boot and EFI-SYSTEM.

We use the best-practice for using systemd mount units with:

[Unit]
Requires=dev-disk-by\2xdlabel-boot.device
After=dev-disk-by\2xdlabel-boot.device

In [1]. The problem, that I traced down is that the device link is created for the simple device. Device Mapper and Raid 0 devices will need to use dev-mapper-<name>.device. I'm playing with [2] to ensure that we select the correct device link.

This assumption in [3] for Ignition-Dracut is that the ignition-diskful.target is around finding a device with boot. Then [4] comes along and blindly expects /dev/disk/by-label/root to be ready [5]. On second boot [6] has no dependencies on the actual device being there. @cgwalters suggested that for second boot we use explicit kargs of root=... which could address this issue.

In the Cloud world, these assumptions are perfectly fine. But once we break the assumption that the disk is simple (i.e. partitions on a physical disk) as is the case with bare-metal, LUKS, iSCSI, complicated disk setup via @arithx's work we need to re-jigger the Ignition-Diskful.

[1] https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator#L28-L42

[2] https://github.com/darkmuggle/fedora-coreos-config/blob/1b59759cba1468d10882ebfefc0e0e531d0bc0df/overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator#L25-L56.

[3] https://github.com/coreos/ignition-dracut/blob/master/dracut/30ignition/ignition-generator#L55-L57

[4] https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-mount-firstboot-sysroot.service

[5] https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-mount-sysroot.sh

[6] https://github.com/coreos/ignition-dracut/blob/master/dracut/30ignition/ignition-subsequent.target

selinux issue during compose

Jonathan reported this to me the other day and I'm now seeing it too in the FCOS pipeline running in centos CI.

lvm2.post: rpm-ostree-systemctl: Ignored non-preset command: enable lvm2-monitor.service
lvm2.post: rpm-ostree-systemctl: Ignored non-preset command: enable lvm2-lvmetad.socket
lvm2.post: rpm-ostree-systemctl: Ignored non-preset command: enable lvm2-lvmpolld.socket
chrony.post: rpm-ostree-systemctl: Ignored non-preset command: daemon-reload

66 done
Writing rpmdb... error: Plugin selinux: hook tsm_pre failed
�[31m�[1merror: �[22m�[0mFailed to update rpmdb (rpmtsRun code -1)

Not really sure what's going on yet and haven't had a chance to try to repro locally not in a supermin VM, but figured I would open the issue to track. I don't know if the issue is in Fedora or in CoreOS Assembler (my branch with supermin code). We'll move it if we need to.

Fedore CoreOS enables pwscore dict check, but does not install dictionary

On current Fedora CoreOS (30.20191014.0), pwscore fails:

# echo foobarfoo | pwscore 
/usr/share/cracklib/pw_dict.pwd.gz: No such file or directory
Password quality check failed:
 The password fails the dictionary check - error loading dictionary

This breaks e. g. Cockpit's Accounts page, it's not possible to add accounts there. This calls pwscore, but passwords are always rejected due to this.

This is essentially the same as https://bugzilla.redhat.com/show_bug.cgi?id=1410450 (in the libpwquality package itself). It got closed as WONTFIX, as from the package's perspective everything is alright: The package recommends dictionaries:

# rpm -q --recommends libpwquality
cracklib-dicts >= 2.8

So for Fedora CoreOS, either cracklib-dicts should be installed (i. e. Recommends honored), or it should modify /etc/security/pwquality.conf to set dictcheck=0.

Version-Release number of selected component (if applicable): libpwquality-1.4.0-12.fc30.x86_64

systemd-networkd cannot be used in CoreOS

I'd prefer to use systemd-networkd instead of Networkmanager in a server setup.

This is made impossible by:

- [systemd, /etc/systemd/networkd.conf,
/usr/lib/systemd/systemd-networkd,
/usr/lib/systemd/systemd-networkd-wait-online,
/usr/lib/systemd/network/.*,
/usr/lib/systemd/system/systemd-networkd.service,
/usr/lib/systemd/system/systemd-networkd.socket,
/usr/lib/systemd/system/systemd-networkd-wait-online.service]

Instead of removing files from installed packages, wouldn't it be possible to split the systemd package in two, and just not install systemd-networkd, so it would be possible to layer systemd-networkd manually after.

FCOS asking for two DHCP leases

Right now, booting FCOS will result in two DHCP leases being handed out: once during the initrd, and once more in the real root. I initially thought this was because unlike in FAH, we weren't shipping initscripts, which has a legacy script to copy networking data over from the initrd (though in addition to that, FAH also doesn't set up networking since we don't have rd.neednet=1).

Anyway, I was playing with variations of:

diff --git a/fedora-coreos-base.yaml b/fedora-coreos-base.yaml
index 95c2c6f..4b81036 100644
--- a/fedora-coreos-base.yaml
+++ b/fedora-coreos-base.yaml
@@ -122,12 +122,29 @@ postprocess:
     WantedBy=multi-user.target
     EOF

+    # Make sure we carry any DHCP lease obtained during initrd
+    cat > /usr/lib/systemd/system/coreos-carry-initrd-lease.service <<'EOF'
+    [Unit]
+    Description=Carry over DHCP lease from initramfs
+    DefaultDependencies=no
+    ConditionDirectoryNotEmpty=/run/initramfs/state/var/lib/dhclient
+    Before=sysinit.target
+    After=local-fs.target systemd-tmpfiles-setup.service
+    [Service]
+    Type=oneshot
+    ExecStart=/usr/bin/sh -c "cp -v /run/initramfs/state/var/lib/dhclient/*.lease /var/lib/NetworkManager"
+    ExecStart=/usr/bin/sh -c "grep UUID= /run/initramfs/state/etc/sysconfig/network-scripts/ifcfg-eth0 >> /etc/sysconfig/network-scripts/ifcfg-eth0"
+    RemainAfterExit=yes
+    [Install]
+    WantedBy=sysinit.target
+    EOF
+
     cat >/usr/lib/systemd/system-preset/42-coreos.preset << EOF
     # Presets here that eventually should live in the generic fedora presets
     # This one is from https://github.com/coreos/ignition-dracut
     enable ignition-firstboot-complete.service
     enable coreos-growpart.service
     enable coreos-useradd-core.service
+    enable coreos-carry-initrd-lease.service
     enable console-login-helper-messages-*.service
     enable console-login-helper-messages-*.path
     EOF

Though for some reason, NetworkManager still calls dhclient again. The lease file then ends up with two also identical entries.

Interestingly, on RHCOS maipo, this does not happen, even though rhel-import-state is disabled, and NetworkManager does end up calling dhclient again. Though it looks like dhclient is happy reusing the same lease.

missing `ignition-diskful` and `ignition-complete` targets when building initrd

Saw the following when doing coreos-assembler build; they don't appear to be fatal or even affect operation, but thought I would log the messages anyways

dracut: *** Including module: i18n ***                                                                                                                                                                                                                                                                                        
dracut: *** Including module: coreos-network ***                                                                                                               
Failed to add dependency on unit, unit ignition-diskful.target does not exist.                                                                                                                                                                                                                                                
dracut: *** Including module: live ***                                                                                                                                                                                                                                                                                        
Failed to add dependency on unit, unit ignition-complete.target does not exist.                                                                                
dracut: *** Including module: coreos-udev ***                                                                                                                                                                                                                                                                                 
dracut: *** Including module: afterburn ***      

failed adding package

I followed https://github.com/coreos/coreos-assembler/blob/master/docs/building-fcos.md and build success
I would like to add rkt package, I found rkt on https://fedora.pkgs.org/32/fedora-x86_64/rkt-1.30.0-3.20190512git0c87656.fc32.x86_64.rpm.html, and added the following lines to manifest-lock.x86_64.json

"rtk": {
  "evra": "1.30.0-3.20190512git0c87656.fc32.x86_64"
},

it result in failure in step "cosa fetch", error message:

error: Couldn't find locked package 'rtk-1.30.0-3.20190512git0c87656.fc32.x86_64' (pkgs matching NEVRA: 0; mismatched checksums: 0)

My question is: where can I found packages for FCOS?

Thanks

Add a preset file for Fedora CoreOS in the release rpm

This issue tracks moving the systemd presets FCOS needs to a proper place. Currently, FCOS writes a preset file with the needed additional presets during post-processing of the build.

The preset file would fit cleanly into fedora-release-coreos, named 80-coreos.preset, analogous to how fedora-release-workstation contains 80-workstation.preset.

Once the list of needed presets in FCOS stabilizes, we could open a PR to https://src.fedoraproject.org/rpms/fedora-release with the new preset file to go in fedora-release-coreos in Rawhide and backport to F29 (to use in testing).

cc @dustymabe @sgallagher

add test for podman network

The reproducer for coreos/fedora-coreos-tracker#923 is simple enough we should consider adding a test for it.

# as non-root user
podman network create testnetwork
podman run --rm -it --network=testnetwork registry.fedoraproject.org/fedora:34 getent hosts google.com
echo $?

[s390x] s390utils-base requires initscripts

Bug Report

Not sure this is a bug or the bug is not from assembler rather than fedora, the cosa build failed with Could not depsolve transaction; 1 problem detected: - package initscripts-10.02-2.fc31.s390x is filtered out by exclude filtering.

Environment

What operating system is being used to run coreos-assembler?
RHEL 8.1 s390x + local build localhost/coreos-assembler based on clean Fedora 31 s390x

What operating system is being assembled?
coreos

Is coreos-assembler running in Podman or Docker?
Podman

If Podman, is coreos-assembler running privileged or unprivileged?
Privileged

Actual Behavior

[xiaoyuz@zcx-gus fcos]$ cosa init https://github.com/coreos/fedora-coreos-config
COREOS_ASSEMBLER_CONTAINER=localhost/coreos-assembler
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /home/xiaoyuz/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa localhost/coreos-assembler init https://github.com/coreos/fedora-coreos-config
+ mkdir -p src
+ cd src
+ test -e config
+ case "${source}" in
+ git clone --recurse-submodules https://github.com/coreos/fedora-coreos-config config
Cloning into 'config'...
remote: Enumerating objects: 2955, done.
remote: Total 2955 (delta 0), reused 0 (delta 0), pack-reused 2955
Receiving objects: 100% (2955/2955), 640.82 KiB | 16.43 MiB/s, done.
Resolving deltas: 100% (1532/1532), done.
+ '[' -n '' ']'
+ set +x
Config commit: d18bcd20d359ba03e5b47faba25810cb26a0fa32
+ manifest=config/manifest.yaml
+ '[' -f config/manifest.yaml ']'
+ mkdir -p cache
+ mkdir -p builds
+ mkdir -p tmp
+ mkdir -p overrides/rpm
+ mkdir -p overrides/rootfs
+ rc=0
+ set +x

[xiaoyuz@zcx-gus fcos]$ cosa fetch
COREOS_ASSEMBLER_CONTAINER=localhost/coreos-assembler
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /home/xiaoyuz/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa localhost/coreos-assembler fetch
Config commit: d18bcd20d359ba03e5b47faba25810cb26a0fa32
Using manifest: /srv/src/config/manifest.yaml
Running: rpm-ostree compose tree --repo=/srv/tmp/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/build/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --download-only
RPM-OSTree Version: 2020.1
No previous commit for fedora/s390x/coreos/testing-devel
Enabled rpm-md repositories: fedora-coreos-pool fedora fedora-updates fedora-modular fedora-updates-modular
Updating metadata for 'fedora-coreos-pool'... done
rpm-md repo 'fedora-coreos-pool'; generated: 2020-04-09T02:11:45Z
Updating metadata for 'fedora'... done
rpm-md repo 'fedora'; generated: 2019-10-23T22:52:27Z
Updating metadata for 'fedora-updates'... done
rpm-md repo 'fedora-updates'; generated: 2020-04-11T21:34:34Z
Updating metadata for 'fedora-modular'... done
rpm-md repo 'fedora-modular'; generated: 2019-10-23T22:53:11Z
Updating metadata for 'fedora-updates-modular'... done
rpm-md repo 'fedora-updates-modular'; generated: 2020-04-09T20:25:33Z
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
 Problem: package s390utils-base-2:2.10.0-1.fc31.s390x requires network-scripts, but none of the providers can be installed
  - package network-scripts-10.02-2.fc31.s390x requires initscripts(s390-64) = 10.02-2.fc31, but none of the providers can be installed
  - conflicting requests
  - package initscripts-10.02-2.fc31.s390x is filtered out by exclude filtering
+ rc=1
+ set +x

Reproduction Steps

cosa() ✓
cosa init ✓
cosa fetch X

Other Information

[xiaoyuz@zcx-gus fcos]$ podman run -it 0027939a6dd7 shell
[coreos-assembler]$ sudo dnf install initscripts
Repository 'f31-coreos-continuous' is missing name in configuration, using id.
Repository 'f32-coreos-continuous' is missing name in configuration, using id.
f31-coreos-continuous                                                   70 kB/s |  46 kB     00:00
f32-coreos-continuous                                                   42 kB/s |  24 kB     00:00
Fedora Modular 31 - s390x                                              4.7 MB/s | 5.1 MB     00:01
Fedora Modular 31 - s390x - Updates                                    1.6 MB/s | 4.0 MB     00:02
Fedora 31 - s390x - Updates                                             20 MB/s |  20 MB     00:00
Fedora 31 - s390x                                                       11 MB/s |  65 MB     00:06
Package initscripts-10.02-2.fc31.s390x is already installed.

Looking for advice that why is the build excluding the package with specific version and which side issue it belongs to?

ISO mount still racing?

Even after #411 I just saw this in a CI run for RHCOS:

"Found device QEMU_DVD-ROM rhcos-46.82.202005292045-0."
"Mounting /run/media/iso..."
"mount: /run/media/iso: /dev/sr0 already mounted or mount point busy."
"run-media-iso.mount: Mount process exited, code=exited status=32"
"run-media-iso.mount: Failed with result 'exit-code'."
"Failed to mount /run/media/iso."

I'm 87.2% sure we have that PR, though I need to triple check.

tests: identify and mark non-exclusive tests

Now that we support combining non-exclusive tests into a single VM let's identify other tests that we have written in our external tests within this repo (other than misc-ro and misc-ign-ro) that aren't exclusive and can be marked as exclusive: false).

Consider to add systemd-netword as an option

NetworkManager is good software for the desktop, but it is extremely inconvenient to use via configuration files (ingnition, machineconfigoperator). For example, try to create, say, 802.3ad bonding via NetworkManager configuration files without using nm-tui. For me, it is easy to play with /sys/proc/net than to get working NetworkManager via /etc/NetworkManager.

Please consider to add systemd-networkd as an option. I understand that RHEL8/RHCOS uses (and it is one of the most annoying things in RHEL8), but I hope that RHEL9 will move forward and replace NetworkManager via systemd-networkd for server. NetworkManager for desktop (e.g. Fedora Workstation), systemd-networkd for server (CoreOS!).

I propose to add some option to ignition to disable NetworkManager and enable systemd-networkd.

tests: covert tests/manual/ to auto

tests/manual/coreos-docs-net-testing.sh and tests/manual/coreos-network-testing.sh are networking test scripts that not included in CI, we can covert them to auto under kola. @dustymabe do you have any suggestions about this?

Add test for password authentication

Add a test that provisions a user password via Ignition (replicating docs) and checks that the password works for local login. (We should be able to do that from the command line, without adding new functionality to kola.)

Perhaps we should also test that the passwd command creates a yescrypt password hash (i.e., one starting with $y$).

systemd presets not getting evaluated

seems like the same issue as: openshift/os#243 - for example:

[root@localhost grub.d]# systemctl status coreos-firstboot-complete.service 
● coreos-firstboot-complete.service - Mark boot complete
   Loaded: loaded (/usr/lib/systemd/system/coreos-firstboot-complete.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://github.com/coreos/ignition

This was built with the rpms from FAHC:

sh-4.4$ rpm -q rpm-ostree ostree
rpm-ostree-2018.7.32-961482f088e5088230935ffd7b1a094e764835db.1f289f2410d7b49abdbf6eeaf2589c3df9b642e0.fc28.x86_64
ostree-2018.8.6-a70d2f67310bd869c6a40c4013dc51a3e2f2d2a8.5bb0359caf4facc535ee3ee8b5ec7c552637c28d.fc28.x86_64

warnings in initramfs from 90-coreos-device-mapper.rules

I'm seeing this in a RHCOS testiso run:

"invalid key/value pair in file /usr/lib/udev/rules.d/90-coreos-device-mapper.rules on line 15, starting at character 53 (',')"
"invalid key/value pair in file /usr/lib/udev/rules.d/90-coreos-device-mapper.rules on line 16, starting at character 56 (',')"

Serial console is primary

image.ks configures the kernel command line with tty0 as the primary console:

bootloader --timeout=1 --append="no_timer_check console=ttyS0,115200n8 console=tty0 net.ifnames=0 biosdevname=0 ip=dhcp rd.neednet=1 rw $ignition_firstboot"

This is consistent with Container Linux, which generally configures VGA console primary except on platforms which only provide serial console.

However, the booted system has serial console primary. /etc/default/grub has a different order:

GRUB_CMDLINE_LINUX="no_timer_check console=tty0 net.ifnames=0 biosdevname=0 ip=dhcp rd.neednet=1 rw $ignition_firstboot console=ttyS0,115200n8"

And /proc/cmdline is different still:

BOOT_IMAGE=/ostree/fedora-coreos-cc078ea1a021c12c793a99d4c53ff42ccf0ae16f1a971371fcf566a96c433775/vmlinuz-4.20.4-200.fc29.x86_64 no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 ip=dhcp rd.neednet=1 rw root=UUID=a36c52b9-bdb7-41c8-abaa-2fa5ca4d07d8 ostree=/ostree/boot.0/fedora-coreos/cc078ea1a021c12c793a99d4c53ff42ccf0ae16f1a971371fcf566a96c433775/0 coreos.oem.id=qemu

/etc/sudoers is owned by uid 1000, should be 0

[core@localhost ~]$ sudo su -                       
sudo: /etc/sudoers is owned by uid 1000, should be 0
sudo: no valid sudoers sources found, quitting      
sudo: unable to initialize policy plugin            

is this because we are running as builduser in coreos-assembler?

"enable boot-efi.mount" make ppc64le platform fail

cosa run failed on ppc64le platform.
ppc64le qcow2 generated image does not end its boot and get stuck with no access to login.

The problem is due to the "enable boot-efi.mount" added in
overlay/usr/lib/systemd/system-preset/42-coreos.preset for all platforms.
see commit cb3f240
and also #105 and #106 pull requests

I didn't find a fair way to avoid it, please @ajeddeloh could you have a look ?

ignition: not getting correct context on authorized keys file

Not sure why but ignition doesn't seem to be working with selinux in this repo right now. If I boot the resulting image and set enforcing=0 on the kernel command line in grub I get the system up.

I see that ignition-relabel.service ran but it doesn't touch authorized_keys for some reason

[root@localhost ~]# systemctl status ignition-relabel.service
● ignition-relabel.service - Relabel files created by Ignition
   Loaded: loaded (/run/systemd/system/ignition-relabel.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2018-09-07 01:45:34 UTC; 41s ago
  Process: 952 ExecStart=/usr/bin/rm /etc/selinux/ignition.relabel (code=exited, status=0/SUCCESS)
  Process: 942 ExecStart=/usr/sbin/restorecon -0vRf /etc/selinux/ignition.relabel (code=exited, status=0/SUCCESS)
 Main PID: 952 (code=exited, status=0/SUCCESS)

Sep 07 01:45:33 localhost.localdomain systemd[1]: Starting Relabel files created by Ignition...
Sep 07 01:45:34 localhost.localdomain restorecon[942]: Relabeled /etc/systemd/system/example.service from system_u:object_r:unlabeled_t:s0 to system_u:object_r:systemd_unit_file_t:s0                                                     
Sep 07 01:45:34 localhost.localdomain restorecon[942]: Relabeled /etc/systemd/system-preset/20-ignition.preset from system_u:object_r:unlabeled_t:s0 to system_u:object_r:etc_t:s0                                                         
Sep 07 01:45:34 localhost.localdomain systemd[1]: Started Relabel files created by Ignition.

and I see the AVC denials

[   32.785013] audit: type=1400 audit(1536284755.035:147): avc:  denied  { getattr } for  pid=1115 comm="sshd" path="/var/roothome/.ssh/authorized_keys" dev="dm-0" ino=14729 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
# rpm-ostree db list fedora/28/x86_64/coreos | grep ignition
 ignition-0.28.0-3.gitf707912.fc28.x86_64
 ignition-dracut-0.28.0-3.gitf707912.fc28.noarch

the startup logs from ignition are shown below:

ignition[819]: INFO     : Ignition 0.28.0                                                                                                                                                                                    
ignition[819]: INFO     : reading system config file "/usr/lib/ignition/base.ign"                                                                                                                                            
ignition[819]: INFO     : no config at "/usr/lib/ignition/base.ign"                                                                                                                                                          
ignition[819]: DEBUG    : parsed url from cmdline: ""                                                                                                                                                                        
ignition[819]: INFO     : no config URL provided                                                                                                                                                                             
ignition[819]: INFO     : reading system config file "/usr/lib/ignition/user.ign"                                                                                                                                            
ignition[819]: INFO     : no config at "/usr/lib/ignition/user.ign"                                                                                                                                                          
ignition[819]: INFO     : op(1): [started]  loading QEMU firmware config module
ignition[819]: DEBUG    : op(1): executing: "modprobe" "qemu_fw_cfg"
ignition[819]: INFO     : op(1): [finished] loading QEMU firmware config module
ignition[819]: DEBUG    : parsing config: {               
ignition[819]:   "ignition": { "version": "2.2.0" },
ignition[819]:   "systemd": {                       
ignition[819]:     "units": [{                                   
ignition[819]:       "name": "example.service",                                        
ignition[819]:       "enabled": true,                                                
ignition[819]:       "contents": "[Service]\nType=oneshot\nExecStart=/usr/bin/false\n\n[Install]\nWantedBy=multi-user.target"
ignition[819]:     }]                                                               
ignition[819]:   },                                       
ignition[819]:   "passwd": {                       
ignition[819]:     "users": [                                             
ignition[819]:       {                   
ignition[819]:         "name": "root",      
ignition[819]:         "sshAuthorizedKeys": [                             
ignition[819]:           "ssh-rsa AAAAB"
ignition[819]:         ]          
ignition[819]:       }                                                                                                                                                                                                      
ignition[819]:     ]                                                                                                                                                                                                        
ignition[819]:   }                                                                                                                                                                                                          
ignition[819]: }                                                                                                                                                                                                            
ignition[819]: INFO     : files: createUsers: op(2): [started]  creating or modifying user "root"                                                                                                                           
ignition[819]: DEBUG    : files: createUsers: op(2): executing: "/usr/sbin/usermod" "--root" "/sysroot" "root"                                                                                                              
ignition[819]: INFO     : files: createUsers: op(2): [finished] creating or modifying user "root"                                                                                                                           
ignition[819]: INFO     : files: createUsers: op(3): [started]  adding ssh keys to user "root"                                                                                                                              
ignition[819]: INFO     : files: createUsers: op(3): [finished] adding ssh keys to user "root"                                                                                                                              
ignition[819]: INFO     : files: op(4): [started]  processing unit "example.service"                                                                                                                                        
ignition[819]: INFO     : files: op(4): op(5): [started]  writing unit "example.service" at "etc/systemd/system/example.service"                                                                                            
ignition[819]: INFO     : files: op(4): op(5): [finished] writing unit "example.service" at "etc/systemd/system/example.service"
ignition[819]: INFO     : files: op(4): [finished] processing unit "example.service"                                                                                                                                        
ignition[819]: INFO     : files: op(6): [started]  enabling unit "example.service"      
ignition[819]: INFO     : files: op(6): [finished] enabling unit "example.service"                                                                                                                                          
ignition[819]: INFO     : files: op(7): [started]  processing unit "ignition-relabel.service"
ignition[819]: INFO     : files: op(7): op(8): [started]  writing unit "ignition-relabel.service" at "run/systemd/system/ignition-relabel.service"                                                                          
ignition[819]: INFO     : files: op(7): op(8): [finished] writing unit "ignition-relabel.service" at "run/systemd/system/ignition-relabel.service"
ignition[819]: INFO     : files: op(7): [finished] processing unit "ignition-relabel.service"                                                                                                                               
ignition[819]: INFO     : files: files passed    
ignition[819]: INFO     : Ignition finished successfully

[tests/kola/root-reprovision] luks reprovision test require tpm, that is not available on all platforms

Re-provision luks test requires tpm to be available in the qemu, but it is not really available in qemu yet beyond the x86_64.
I'm not sure if with the ext tests there is a way how to handle it as with the regular internal cosa kola tests(not use tpm on platforms that don't have it (yet), skip them). It seems the omitting the tpm from the ignition is also not a way, as the luks will not get unlocked. I would much appreciate any ideas, pointers.

Contents of console.txt from ppc64le without tpm in the config.ign are in the attachment.
ppc64le-luks.log

Work towards creating streams

So, we're now in a position where we can start implementing the stream work as described in https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.md.

I'd like to suggest the following concrete steps to start:

  • rename master to testing-devel
  • add two new branches: testing and bodhi-updates, which are copies of testing-devel modulo the ref (we can manually keep bodhi-updates in sync to start, but need to work towards automating this)
  • set up pipeline for periodic bodhi-updates runs (basically what it does today, but targeting the new stream)
  • push output lockfile to the bodhi-updates branch (see coreos/coreos-assembler#553) (again, can start manually to get things going, but need to automate this very soon)
  • for now, just push lockfile to testing-devel as well; we can work towards making this a PR-based + CI workflow afterwards
  • hook up coreos-koji-tagger to watch testing-devel
  • set up pipelines for testing-devel and testing runs triggered on ref changes
  • try out testing releases by manually merging from the testing-devel branch using the mythical "theirs" strategy, and work on tooling to make this less manual & error-prone.

All open for discussions of course. The goal is to get a taste of the workflow across mechanical, devel, and prod streams even if in a semi-manual capacity and unblock tooling and automation work. Once we have this down, we can start filling in more of the blanks, such as working towards automation for doing a testing release, adding the remaining prod, devel, and mechanical streams, etc...

Consider moving CI to jenkins-fedora-coreos

PR CI runs in jenkins-projectatomic-ci.apps.ci.centos.org, which is an additional namespace developers need access to in order to be able to e.g. rerun jobs. Consider moving to jenkins-fedora-coreos.apps.ci.centos.org.

replace glibc-all-langpacks with glibc-minimal-langpack

Currently Fedora coreos installs all glibc locales (glibc-all-langpacks) which takes up a lot of space (~200MB on disk).

Since the default system locale is C.UTF-8 and default shell uses POSIX (it would be better to use C.UTF-8), how about just installing glibc-minimal-langpack instead of glibc-all-langpacks to save space and bandwidth: this is also what we do now in fedora base containers.

bootupd v 1.3-2 broke builds

Latest version of bootupd is erroring on builds:

+ test -d /tmp/rootfs/ostree/deploy/fedora-coreos/deploy/d57c6a71d779f5ae28d46e61bcfe9d8654fecab77ae9f913395d80ebbc4ce8bf.0/usr/lib/bootupd/updates
+ /usr/bin/bootupctl backend install --src-root=/tmp/rootfs/ostree/deploy/fedora-coreos/deploy/d57c6a71d779f5ae28d46e61bcfe9d8654fecab77ae9f913395d80ebbc4ce8bf.0 /tmp/rootfs
error: boot data installation failed: trailing input at line 1 column 35
+ rm -rf /srv/tmp/build.qemu/supermin.out /srv/tmp/build.qemu/supermin.prepare /srv/tmp/build.qemu/supermin.build

This has been observed in CI and locally.

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=1624886

kola: Add toolbox tests

Feature Request

Add toolbox tests to make sure we do not release an FCOS image with a broken combination of toolbox & podman.

core user isn't added to groups when created via Ignition

The core user is only added to groups if the user doesn't exist on first boot:

ExecStart=/usr/bin/sh -c 'if !getent passwd core &>/dev/null; then /usr/sbin/useradd -G wheel,sudo,adm,systemd-journal core; fi'

Thus, if Ignition is used to add an SSH key to the core user, but the Ignition config doesn't explicitly add the user to groups, it will not have sudo access:

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - "ssh-rsa ..."

The above config works on Container Linux.

cc @ajeddeloh

update graduated overrides job to handle rawhide

Sometimes the rawhide pungi runs won't succeed for many days on. Rather than wait on one of these to succeed (when we want new packages that have already been built with fixes to make it into the repos) we have started to fast-track builds. We should update our graduated overrides job to handle the rawhide branch as well to clean up the overrides.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.