Comments (3)
hzxuzhonghu
commented on September 14, 2024
please paste the bpf trace log.
You can use kmesh-daemon log --set bpf:debug within kmesh pod to turn on debug
from kmesh.
YaoZengzeng
commented on September 14, 2024
The deployed application status is as follow:
root@kurator-linux-0002:~# kgp -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
details-v1-cdd874bc9-25gjt 1/1 Running 0 28m 10.244.1.22 kmesh-testing-worker <none> <none>
httpbin-648f469544-5jkgs 1/1 Running 0 54m 10.244.1.14 kmesh-testing-worker <none> <none>
httpbin-svc-waypoint-5b777b8859-b9hpp 1/1 Running 0 37m 10.244.1.19 kmesh-testing-worker <none> <none>
productpage-v1-5bb9985d4d-dqpbc 1/1 Running 0 28m 10.244.1.27 kmesh-testing-worker <none> <none>
ratings-v1-6484d64bbc-7fw8t 1/1 Running 0 28m 10.244.1.23 kmesh-testing-worker <none> <none>
reviews-svc-waypoint-6884756fc5-mc8bp 1/1 Running 0 28m 10.244.1.29 kmesh-testing-worker <none> <none>
reviews-v1-598f9b58fc-k56kx 1/1 Running 0 28m 10.244.1.24 kmesh-testing-worker <none> <none>
reviews-v2-5979c6fc9c-59pmv 1/1 Running 0 28m 10.244.1.25 kmesh-testing-worker <none> <none>
reviews-v3-7bbb5b9cf7-dbd6k 1/1 Running 0 28m 10.244.1.26 kmesh-testing-worker <none> <none>
sleep-5577c64d7c-mzctk 1/1 Running 0 54m 10.244.1.15 kmesh-testing-worker <none> <none>
waypoint-b7bc55b9f-c2hsg 1/1 Running 0 15m 10.244.1.31 kmesh-testing-worker <none> <none>root@kurator-linux-0002:~# kgs
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
details ClusterIP 10.96.32.204 <none> 9080/TCP 28m
httpbin ClusterIP 10.96.195.202 <none> 8000/TCP 55m
httpbin-svc-waypoint ClusterIP 10.96.202.255 <none> 15021/TCP,15008/TCP 37m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 67m
productpage ClusterIP 10.96.227.210 <none> 9080/TCP 28m
ratings ClusterIP 10.96.76.254 <none> 9080/TCP 28m
reviews ClusterIP 10.96.61.192 <none> 9080/TCP 28m
reviews-svc-waypoint ClusterIP 10.96.108.93 <none> 15021/TCP,15008/TCP 28m
sleep ClusterIP 10.96.11.177 <none> 80/TCP 55m
waypoint ClusterIP 10.96.95.140 <none> 15021/TCP,15008/TCP 15mActuall the curl command was executed successfully but because reviews were not accessed, the subsequent grep failed:
kubectl exec deploy/sleep -n "$NAMESPACE" -- sh -c "curl -s http://productpage:9080/productpage | grep reviews-v.-"The Kmesh daemon log after executing the above command is as follows:
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.227.210:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.227.210:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] DEBUG: access the backend by service:2936233694\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] DEBUG: get the backend addr=[10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SOCKOPS] ERR: enable encoding metadata failed!, err is -16" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SENDMSG] DEBUG: get valid dst, do encoding...\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.32.204:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.32.204:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] DEBUG: access the backend by service:2936233694\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] DEBUG: get the backend addr=[10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.244.1.31:15019]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SOCKOPS] ERR: enable encoding metadata failed!, err is -16" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SENDMSG] DEBUG: get valid dst, do encoding...\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.61.192:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.61.192:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: record metadata origin address and port failed, ret is 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: waypoint_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[FRONTEND] ERR: service_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: frontend_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: sock_traffic_control failed: 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: record metadata origin address and port failed, ret is 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: waypoint_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[FRONTEND] ERR: service_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: frontend_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: sock_traffic_control failed: 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: connect bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: connect bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SOCKOPS] ERR: enable encoding metadata failed!, err is -16" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SENDMSG] DEBUG: get valid dst, do encoding...\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: close bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: close bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.61.192:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.61.192:9080]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: record metadata origin address and port failed, ret is 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: waypoint_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[FRONTEND] ERR: service_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: frontend_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: sock_traffic_control failed: 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: origin addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] DEBUG: bpf find frontend addr=[2:10.96.108.93:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SERVICE] DEBUG: find waypoint addr=[10.96.95.140:15008]\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: record metadata origin address and port failed, ret is 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[BACKEND] ERR: waypoint_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[FRONTEND] ERR: service_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: frontend_manager failed, ret:0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[KMESH] ERR: sock_traffic_control failed: 0\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: connect bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: connect bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SOCKOPS] ERR: enable encoding metadata failed!, err is -16" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[SENDMSG] DEBUG: get valid dst, do encoding...\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: close bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="[PROBE] ERR: close bpf_sk_storage_get failed\n" subsys=ebpf
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="can't find service correspond workload: waypoint-b7bc55b9f-c2hsg" subsys=pkg/telemetry
time="2024-08-14T07:41:51Z" level=info msg="get destination service host failed" subsys=pkg/telemetryfrom kmesh.
YaoZengzeng
commented on September 14, 2024
The config dump is as follows, only show waypoint related service:
... {
"name": "httpbin-svc-waypoint",
"namespace": "default",
"hostname": "httpbin-svc-waypoint.default.svc.cluster.local",
"vips": [
"/10.96.202.255"
],
"ports": [
{
"service_port": 15021,
"target_port": 15021
},
{
"service_port": 15008,
"target_port": 15008
}
],
"loadBalancer": {
"mode": "FAILOVER",
"routingPreferences": [
"NETWORK",
"REGION",
"ZONE",
"SUBZONE"
]
},
"waypoint": {
"destination": "/10.96.95.140"
}
},
...
{
"name": "reviews-svc-waypoint",
"namespace": "default",
"hostname": "reviews-svc-waypoint.default.svc.cluster.local",
"vips": [
"/10.96.108.93"
],
"ports": [
{
"service_port": 15021,
"target_port": 15021
},
{
"service_port": 15008,
"target_port": 15008
}
],
"loadBalancer": {
"mode": "FAILOVER",
"routingPreferences": [
"NETWORK",
"REGION",
"ZONE",
"SUBZONE"
]
},
"waypoint": {
"destination": "/10.96.95.140"
}
},
...
{
"name": "waypoint",
"namespace": "default",
"hostname": "waypoint.default.svc.cluster.local",
"vips": [
"/10.96.95.140"
],
"ports": [
{
"service_port": 15021,
"target_port": 15021
},
{
"service_port": 15008,
"target_port": 15008
}
],
"loadBalancer": {
"mode": "FAILOVER",
"routingPreferences": [
"NETWORK",
"REGION",
"ZONE",
"SUBZONE"
]
},
"waypoint": {
"destination": ""
}
},from kmesh.
Related Issues (20)
- e2e test hangs HOT 2
- Unexpected outbound accesslog HOT 3
- Panic with istio 1.24-alpha.96593644e05baa5c1b6aaab3647b2d9335a23942 HOT 3
- Headless service support
- "/debug/bpf/*" should output human-readable, real bpf information HOT 4
- Kmesh cli tool
- Add a configuration option to start accesslog HOT 1
- Can't use traffic mirroring capabilities of Kmesh HOT 1
- Kmesh-build image should be public HOT 2
- Redundant delete in update? HOT 1
- Memory Leak in /bpf/deserialization_to_bpf_map/deserialization_to_bpf_map.c HOT 1
- flaky of TestBookinfo HOT 1
- Stop duplicate logs into `/var/run/kmesh/daemon.log`
- `make gen` failed on arm64. HOT 2
- Add helm push github action HOT 2
- Unexpected error from test `unmount /mnt/kmesh_cgroup2 error: no such file or directory` HOT 1
- [E2E Enhancement] ServiceEntry related cases HOT 1
- ads deserial bug HOT 9
- [E2E Enhancement] Bypass related test cases
- Add copyright checker
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kmesh.