Comments (4)
When looking at the flow using tcpdump, we did see that the application pod did receive the SYN call from the ingress. However when the application pop was attempting to send the SYN-ACK back, the packet appears to be unable to route back, to the ingress. Followed by the ingress, attempting to restart the TCP flow.
19:23:00.745947 IP ip-100-66-155-240.us-west-2.compute.internal.46283 > ip-100-66-156-47.us-west-2.compute.internal.7654: Flags [S], seq 2885681828, win 62727, options [mss 8961,sackOK,TS val 340353580 ecr 0,nop,wscale 7], length 0
19:23:00.745967 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149138160 ecr 340353580,nop,wscale 7], length 0
19:23:00.745984 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149138160 ecr 340353580,nop,wscale 7], length 0
19:23:01.755860 IP ip-100-66-155-240.us-west-2.compute.internal.46283 > ip-100-66-156-47.us-west-2.compute.internal.7654: Flags [S], seq 2885681828, win 62727, options [mss 8961,sackOK,TS val 340354590 ecr 0,nop,wscale 7], length 0
19:23:01.755887 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755904 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755906 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755909 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
from cilium.
Thanks for your issue.
As mentioned in your comment, the workaround is to set bpf.hostLegacyRouting=true. You can probably set endpointRoutes.enabled=true as well.
Based on my previous investigation, most likely it's due to below block, especially the variable ENABLE_SKIP_FIB.
cilium/pkg/datapath/linux/config/config.go
Lines 1048 to 1059 in 451c3b2
from cilium.
Really appreciate the follow-up and suggestion.
I attempted to comment out this code to observe the behavior:
// if len(option.Config.GetDevices()) == 1 {
// if e.IsHost() || !option.Config.EnforceLXCFibLookup() {
// fmt.Fprintf(fw, "#define ENABLE_SKIP_FIB 1\n")
// }
// }
Still hitting the same issue.
* Connection #0 to host 127.0.0.1 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection timeout
Also tried running this on both arm64
and amd64
to rule out any architecture-specific behavior.
from cilium.
When looking at the flow using tcpdump, we did see that the application pod did receive the SYN call from the ingress. However when the application pop was attempting to send the SYN-ACK back, the packet appears to be unable to route back, to the ingress. Followed by the ingress, attempting to restart the TCP flow.
I think it's worth revisiting this issue once #33014 has been merged.
This would allow replies destined for the Ingress endpoint to be processed as expected here.
from cilium.
Related Issues (20)
- [DATA RACE]: k8s.(*ServiceCache).DebugStatus() HOT 1
- custom BPF code loading failes
- SourceIPVerification while chaining CNI HOT 2
- Hubble-Relay timesout running on Kubernetes 1.30
- Test [to-fqdns] fails yet manual verification works HOT 1
- [DATA RACE]: github.com/cilium/dns.(*Conn).ReadMsgHeader()
- [DATA RACE]: github.com/cilium/cilium/pkg/bgpv1/agent.(*Controller).bgppSelection()
- [DATA RACE]: github.com/cilium/cilium/operator/auth/spire.(*Client).Upsert()
- [DATA RACE]: github.com/cilium/cilium/pkg/ciliumenvoyconfig.(*envoyServiceBackendSyncer).RegisterServiceUsageInCEC()
- Collect cilium_skip_lb{4,6} maps in sysdump
- Cilium identity not correct when trying connect from outside cluster with nodeport
- Cannot forward proxied DNS lookup; bind: address already in use
- LoadBalancerClass cannot be set on Cilium Gateway API Servicess
- CI: Integration Test: TestClusterMeshMultipleAddRemove HOT 1
- Document GKE cloud DNS deployment using LRP
- Run LRP e2e test for per-packet LB HOT 1
- Policy Correlation: realized policy misses for L4-only rules HOT 1
- CFP: HOT 1
- CFP: cilium
- Documentation issues
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cilium.