podman run failed with pasta error with podman 5 about podman HOT 12 CLOSED

@escape0707, thanks for reporting this. It looks like pasta is failing to configure IPv6 connectivity for the container here. Can you please share what interfaces (ip link show), addresses (ip address show) and IPv6 routes (ip -6 route show) you have configured on the host?

By the way, as a temporary workaround, you can probably start your containers with IPv4-only connectivity, using podman run --net=pasta:-4 --rm fedora:latest.

from podman.

6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

This might explain it, the VLAN interface used for the default route is probably confusing pasta's netlink implementation. I should be able to reproduce it now, thanks, I'll keep you posted.

from podman.

Running into the same issue without VLANs:

$ podman run quay.io/podman/hello:latest
Error: pasta failed with exit code 1:
Couldn't set IPv6 route(s) in guest: No such device

Having more than one route:

$ ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default proto ra metric 600 pref medium
        nexthop via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 weight 1
        nexthop via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3 weight 1

After deleting one of them:

# ip -6 ro de default via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3
# ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 proto ra metric 600 pref medium

It works:

$ podman run quay.io/podman/hello:latest
!... Hello Podman World ...!

Maybe it is confused by multiple default routes with same weight? Hope it helps.

from podman.

pasta patch now pending review (testing also appreciated) at https://archives.passt.top/passt-dev/[email protected]/

from podman.

Thanks. I applied the patch and it seems to work.

$ podman run --rm quay.io/podman/hello:latest

!... Hello Podman World ...!

$ ip -6 ro sh
fdc0:ffee:a10::3:9216:bfae dev wlp0s20f3 proto kernel metric 600 pref medium
fdc0:ffee:a10::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium
default proto ra metric 600 pref medium
        nexthop via fe80::60df:52ff:fe8d:81cd dev wlp0s20f3 weight 1
        nexthop via fe80::3ca8:6bff:feb8:c610 dev wlp0s20f3 weight 1

IPv6 connectivity from inside a container also works.

from podman.

This is now fixed in pasta's version 2024_04_05.954589b -- I guess we can close it, unless @escape0707 prefers to wait for an updated Arch Linux package to check the fix.

from podman.

Thanks for the response. The --net=pasta:-4 trick did workaround this issue.

As for the network related info:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: enp67s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ae brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
    link/ether 50:e0:85:f3:30:95 brd ff:ff:ff:ff:ff:ff
5: enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
3: enp67s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b4:2e:99:a6:cf:ae brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 50:e0:85:f3:30:95 brd ff:ff:ff:ff:ff:ff
5: enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b62e:99ff:fea6:cfad/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
6: enp67s0f1.222@enp67s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b4:2e:99:a6:cf:ad brd ff:ff:ff:ff:ff:ff
    inet 203.178.135.37/25 brd 203.178.135.127 scope global enp67s0f1.222
       valid_lft forever preferred_lft forever
    inet6 2001:200:0:1cd1:b62e:99ff:fea6:cfad/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591922sec preferred_lft 604722sec
    inet6 2001:200:0:1cd1::37/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b62e:99ff:fea6:cfad/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

$ ip -6 route
2001:200:0:1cd1::/64 dev enp67s0f1.222 proto kernel metric 256 pref medium
2001:200:0:1cd1::/64 dev enp67s0f1.222 proto ra metric 1024 expires 2591819sec hoplimit 64 pref medium
fe80::/64 dev enp67s0f1 proto kernel metric 256 pref medium
fe80::/64 dev enp67s0f1.222 proto kernel metric 256 pref medium
default proto static metric 1024 pref medium
        nexthop via 2001:200:0:1cd1::1 dev enp67s0f1.222 weight 1
        nexthop via fe80::7afe:3dff:fed4:7181 dev enp67s0f1.222 weight 1

from podman.

Maybe it is confused by multiple default routes with same weight? Hope it helps.

Thanks, yes, it helps in the sense that we get ENODEV from the kernel also without a VLAN device, when we have RTA_NEXTHOP attributes (multipath routes) without RTA_OIF, so I'm focusing on nexthop attributes now.

I can't just reproduce it with any multipath route with two nexthops with the same weight, I think we need to have an actual mismatch of the interface numbering between host and container (expected, as we don't copy all the interfaces).

When I reproduce it with something similar to #22192 (comment), it looks like this:

[pid   159] sendto(7, [{nlmsg_len=148, nlmsg_type=0x18 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|NLM_F_MULTI|NLM_F_ACK|0x400, nlmsg_seq=14, nlmsg_pid=0}, "\x0a\x00\x00\x00\xfe\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x08\x00\x06\x00\x00\x04\x00\x00\x3c\x00\x09\x00"...], 148, 0, NULL, 0) = 148
[pid   159] recvfrom(7, [{nlmsg_len=168, nlmsg_type=NLMSG_ERROR, nlmsg_flags=0, nlmsg_seq=14, nlmsg_pid=162}, {error=-ENODEV, msg=[{nlmsg_len=148, nlmsg_type=0x18 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|NLM_F_MULTI|NLM_F_ACK|0x400, nlmsg_seq=14, nlmsg_pid=0}, "\x0a\x00\x00\x00\xfe\x03\x00\x01\x00\x00\x00\x00\x08\x00\x0f\x00\xfe\x00\x00\x00\x08\x00\x06\x00\x00\x04\x00\x00\x3c\x00\x09\x00"...]}], 65536, 0, NULL, NULL) = 168

Unfortunately strace(1) can't decode this because we have some padding in the message (I think that's the reason), but I'm now fairly sure we need to go "into" RTA_NEXTHOP attributes and also adjust interface numbers there as we copy host routes.

from podman.

@sbrivio-rh Sorry, I was kinda busy yesterday. Thank you for your hard work. I'll test the patch today.

from podman.

I can confirm the patch fixed my issue. Thank you for maintaining this amazing tool! Closing this issue.

from podman.

I think I have the same issue on OpenSUSE Tumbleweed. Will we on OpenSUSE get the update soon as well?

from podman.

I think I have the same issue on OpenSUSE Tumbleweed. Will we on OpenSUSE get the update soon as well?

I can't say as I don't maintain that package, but I suppose you'll get it reasonably soon as https://bugzilla.suse.com/show_bug.cgi?id=1221840 needs to be closed anyway, and the current upstream version includes the changes needed for that one as well.

from podman.