rawhide kernel 6.10.0 >=20240514 - podman update device-read-bps = 0 about podman HOT 20 OPEN

@edsantiago there is no testing repo for Rawhide, so if an update fails gating there isn't really a proper repo to get it from, unfortunately. You have to get it from Koji. You can use koji download-build --arch=x86_64 --arch=noarch <NVR> to download all the packages from the build, but for kernels that's a lot of packages, so I usually just cherry-pick the few packages I need to install from the web UI.

openQA does record logs, but we don't happen to pipe the output of this specific test command to a file at present. It would be easy to do that if it's useful, though.

@Luap99 it's the kernel that is causing this. The same test is passing just fine on every other Rawhide update; it fails only on kernel updates, which means the kernel is the cause.

from podman.

well, @jmflinuxtx - the Fedora kernel maintainer - is aware of the issue, so I was kinda leaving it to him to report it to the appropriate upstream venues. I find it pretty impossible to know where to send kernel issues.

from podman.

Failed again, kernel-6.10.0-0.rc0.20240516git3c999d1ae3c7.5.fc41

from podman.

Can you create simple reproducer? AFAIK cgroup setup depends podman -> crun -> systemd-> kernel so maybe check if the other components changed too.

from podman.

Can you create simple reproducer?

That has been my goal, as you might have predicted. However, dnf --enablerepo=updates-testing upgrade kernel does not bring in any affected (6.10) kernel, only 6.9, and I'm much too lazy to hunt down all the 6.10 packages. But okay, I'll find some time to do so.

from podman.

Thanks @AdamWill, I guess then we have to get a simple reproducer and file a kernel bug.

from podman.

I'm being lazy again: the failure is a 0514 kernel build. I see a 0517 koji build and have not seen any OpenQA error emails about it. Until I have reason to suspect otherwise, I'll assume the problem is fixed. (And will save myself the time of pulling the kernel and looking for a reproducer)

from podman.

sigh... never mind. 0517 did fail in OpenQA.

Reproducer:

# uname -r
6.9.0-0.rc7.20240510git448b3fe5a0ea.62.fc41.x86_64
# dnf -y install podman-tests

# podman run -d --name foo quay.io/libpod/testimage:20240123 sleep inf
<cid>
# podman exec foo cat /sys/fs/cgroup/io.max
# podman update --device-read-bps=/dev/zero:10mb foo
<cid>
# podman exec foo cat /sys/fs/cgroup/io.max
1:5 rbps=10485760 wbps=max riops=max wiops=max    <<<<< THIS IS GOOD

Then:

# wget https://kojipkgs.fedoraproject.org//packages/kernel/6.10.0/0.rc0.20240517gitea5f6ad9ad96.6.fc41/x86_64/kernel{,-core,-modules,-modules-core}-6.10.0-0.rc0.20240517gitea5f6ad9ad96.6.fc41.x86_64.rpm
# dnf install kern*rpm; reboot

Then

# uname -r
6.10.0-0.rc0.20240517gitea5f6ad9ad96.6.fc41.x86_64
# podman rm -f -a
[repeat the podman run/update/exec from above]
1:5 rbps=0 wbps=0 riops=0 wiops=0       <<<<<< THIS IS NOT GOOD

from podman.

Filed rhbz2281805

from podman.

Does this still happen with 6.10 rc1?

from podman.

If by rc1 you mean 6.10.0-0.rc1.17, then yes

from podman.

Also 6.10.0-0.rc1.20240528git2bfcfd584ff5.18

from podman.

a cli reproducer should be something like this

mkdir /sys/fs/cgroup/test-cgroup
echo "1:5 rbps=10485760" > /sys/fs/cgroup/test-cgroup/io.max
cat /sys/fs/cgroup/test-cgroup/io.max
rmdir /sys/fs/cgroup/test-cgroup

from podman.

I tried to get a rawhide VM going to test myself install but seems like something with dnf is terribly broken there as I cannot install anything due checksum errors. I tried several VM's all fail in the same way...

from podman.

huh, that seems odd? I'm running Rawhide here and not seeing anything like that, and our automated tests aren't either.

from podman.

On 1mt, a minute or two ago, I saw a ton of red checksum errors but dnf install podman ended up successful.

from podman.

I do see this mail, which might be relevant. I hadn't updated to that yet. But openQA did pass tests today...which includes doing quite a lot of package installs...

from podman.

Yeah seems to be working now again, not sure what happened.

from podman.

Tried 6.10.0-0.rc1.20240528git2bfcfd584ff5.18 and can reproduce with the shell commands above, you may need to add the io controller first on a fresh boot.

echo +io > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/test-cgroup
echo "1:5 rbps=10485760" > /sys/fs/cgroup/test-cgroup/io.max
cat /sys/fs/cgroup/test-cgroup/io.max
rmdir /sys/fs/cgroup/test-cgroup

I think this must be reported to the kernel upstream, I don't see this getting solved just sitting in the fedora bugzilla.

from podman.

Yess, I am aware, I passed this on to Waiman Long. He thought there was a patch for it and that turned out not to cover this case, so he was looking again. In the meantime, we just hit RC1 so bug fixes are coming in fast, and it is possible that someone else has a fix. Worst case, I can bisect later this week.

from podman.