Network issues when using pasta instead of slirp4netns in podman 5.0.2 about podman HOT 8 CLOSED

Here's the tshark output of a pcap ran for a few minutes on the container using tcpdump -i lo -w lo.pcap while the migration was running. Can't share the full pcap because of the amount of unencrypted sql / data going through, I'd rather not have to go through the list and check every packet for sensitive data and edit it out.

You can see around 2.7s in (near the end of the file) all tcp traffic stops until around 121s in. The migration never finishes, and I stop things manually, although I haven't let it go for more than about 15 min. But usually the hundreds of migrations (when using slirp4netns) take less than 2 min total, whereas this one migration gets stuck for over 10 min using pasta with seemingly nothing actually happening.

tshark.txt

from podman.

@sbrivio-rh your patch appears to have fixed the issue. After compiling and installing it, I ran the migration three times in a row and it worked every time. I then reverted to the commit before yours, installed again, and the migration got stuck at the same place it was happening before installing your patched version.

from podman.

A packet capture would be nice to have, you can ask pasta to save one with e.g. --network=pasta:--pcap,/tmp/postgres.pcap.

You can drop packets you don't want to share using pcapedit or even Wireshark, or have a look and share what seems to be relevant with a tshark output.

from podman.

Running a packet capture seems to only pick up a few multicast listener report message v2 and router solicitation packets, without picking up any of the postgres traffic. Are there options I'm missing? After letting the container run for several minutes and running the java app that runs the db migrations, I can see the app contacting the db and doing various things, but I only get about 12 total packets in the pcap.

podman run --rm -it --name postgres --network pasta:--pcap,/tmp/postgres.pcap -v pg:/var/lib/postgresql/data -p 5432:5432 -e POSTGRES_PASSWORD=postgres docker.io/postgres:14

pcap.zip

from podman.

Running a packet capture seems to only pick up a few multicast listener report message v2 and router solicitation packets, without picking up any of the postgres traffic. Are there options I'm missing?

Ah, no, my bad: pasta implements a tap bypass path for local connections that splices TCP sockets between container and host directly, to improve throughput by avoiding Layer-2 / Layer-4 translations where not needed, see the appendage in this diagram. Packets on that path are not captured (because they're not really packets -- we just move the Layer-4 payload around).

See also #22575 (comment) (that issue might be related, but we're not sure there's an issue in pasta yet) and the following comment.

To capture traffic in this case, tcpdump -i lo -w lo.pcap in the container should do the trick.

from podman.

You can see around 2.7s in (near the end of the file) all tcp traffic stops until around 121s in.

Thanks a lot for the capture, this seems to be compatible with the current findings from #22575 -- I'm trying to reproduce that at the moment.

from podman.

@jadonclegg, assuming that #22575 is in fact the same issue, I just posted a patch to fix this at https://archives.passt.top/passt-dev/[email protected]/ (see also #22575 (comment)).

It would be great if you could test it by building from source (something on the lines of git clone git://passt.top/passt; cd passt; curl https://archives.passt.top/passt-dev/[email protected]/raw | git am; prefix=/usr sudo make install).

from podman.

This is now fixed in passt version 2024_05_10.7288448 and the corresponding Fedora 40 update.

from podman.