Comments (6)
jpegleg
commented on July 16, 2024
2
Thank you for the report!
Could you please add the logs from the tetragon container?
You're welcome!
Here is a tetragon log sample from a k3s cluster that encountered this behavior:
time="2022-07-06T02:01:55Z" level=info msg="Loaded config from directory" config-dir=/etc/tetragon
time="2022-07-06T02:01:55Z" level=info msg="Starting tetragon" version=v0.8.0
time="2022-07-06T02:01:55Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: cilium-bpf: config-dir:/etc/tetragon config-file: debug:false enable-cilium-api:false enable-export-aggregation:false enable-k8s-api:true enable-process-ancestors:true enable-process-cred:false enable-process-ns:false export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{\"event_set\":[\"PROCESS_EXEC\", \"PROCESS_EXIT\", \"PROCESS_KPROBE\"]} export-denylist:{\"health_check\":true}\n{\"namespace\":[\"\", \"cilium\", \"kube-system\"]} export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-rotation-interval:0s export-filename:/var/run/cilium/tetragon/tetragon.log export-rate-limit:-1 force-small-progs:false ignore-missing-progs:false kernel: log-format:text log-level:info metrics-server::2112 netns-dir:/var/run/docker/netns/ process-cache-size:65536 procfs:/procRoot run-standalone:false server-address:localhost:54321 verbose:0]"
time="2022-07-06T02:01:55Z" level=info msg="Available sensors" sensors=
time="2022-07-06T02:01:55Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor"
time="2022-07-06T02:01:55Z" level=info msg="Registered probe types" types="kprobe sensor, tracepoint sensor"
time="2022-07-06T02:01:56Z" level=info msg="Enabling Kubernetes API"
time="2022-07-06T02:01:56Z" level=info msg="Starting metrics server" addr=":2112"
time="2022-07-06T02:01:56Z" level=info msg="Initialized pod cache" num_pods=4
time="2022-07-06T02:01:56Z" level=info msg="Disabling Cilium API"
time="2022-07-06T02:01:56Z" level=info msg="Starting process manager" enableCilium=false enableEventCache=true enableProcessCred=false enableProcessNs=false
time="2022-07-06T02:01:56Z" level=info msg="Starting gRPC server" address="localhost:54321"
time="2022-07-06T02:01:56Z" level=info msg="Starting JSON exporter" logger="&{/var/run/cilium/tetragon/tetragon.log 10 0 5 false false 0 <nil> {0 0} <nil> {0 {0 0}}}" request="allow_list:{event_set:PROCESS_EXEC event_set:PROCESS_EXIT event_set:PROCESS_KPROBE} deny_list:{health_check:{value:true}} deny_list:{namespace:\"\" namespace:\"cilium\" namespace:\"kube-system\"}"
time="2022-07-06T02:01:56Z" level=info msg="Exporter configuration" enabled=true fileName=/var/run/cilium/tetragon/tetragon.log
time="2022-07-06T02:01:56Z" level=info msg="Using metadata file" metadata=
time="2022-07-06T02:01:56Z" level=info msg="Loading sensor" name=__main__
time="2022-07-06T02:01:56Z" level=info msg="Loading kernel version 5.14.21"
time="2022-07-06T02:01:56Z" level=info msg="tetragon, map loaded." map=execve_map path=/sys/fs/bpf/tcpmon/execve_map sensor=__main__
time="2022-07-06T02:01:56Z" level=info msg="tetragon, map loaded." map=execve_map_stats path=/sys/fs/bpf/tcpmon/execve_map_stats sensor=__main__
time="2022-07-06T02:01:56Z" level=info msg="Started watching tracing policies"
time="2022-07-06T02:01:56Z" level=info msg="tetragon, map loaded." map=names_map path=/sys/fs/bpf/tcpmon/names_map sensor=__main__
time="2022-07-06T02:01:56Z" level=info msg="tetragon, map loaded." map=tcpmon_map path=/sys/fs/bpf/tcpmon/tcpmon_map sensor=__main__
time="2022-07-06T02:01:56Z" level=info msg="BPF prog was loaded" label=tracepoint/sys_exit prog=/var/lib/tetragon/bpf_exit.o
time="2022-07-06T02:01:56Z" level=info msg="BPF prog was loaded" label=kprobe/wake_up_new_task prog=/var/lib/tetragon/bpf_fork.o
time="2022-07-06T02:01:56Z" level=info msg="Load probe" Program=/var/lib/tetragon/bpf_execve_event_v53.o Type=execve
time="2022-07-06T02:01:57Z" level=info msg="Read ProcFS /procRoot appended 70/261 entries"
time="2022-07-06T02:01:57Z" level=warning msg="Procfs execve event pods/ identifier error" error="open /procRoot/0/cgroup: no such file or directory"
time="2022-07-06T02:01:57Z" level=info msg="BPF prog was loaded" label=tracepoint/sys_execve prog=/var/lib/tetragon/bpf_execve_event_v53.o
time="2022-07-06T02:01:57Z" level=info msg="Loaded BPF maps and events for sensor successfully" sensor=__main__
time="2022-07-06T02:01:57Z" level=info msg="Using metadata file" metadata=
time="2022-07-06T02:01:57Z" level=info msg="Loading sensor" name=__main__
time="2022-07-06T02:01:57Z" level=info msg="Loading kernel version 5.14.21"
time="2022-07-06T02:01:57Z" level=info msg="Loaded BPF maps and events for sensor successfully" sensor=__main__
time="2022-07-06T02:01:57Z" level=info msg="Listening for events..."
time="2022-07-06T02:33:01Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: an error on the server (\"unknown\") has prevented the request from succeeding (get tracingpolicies.cilium.io)"
time="2022-07-06T02:33:02Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:33:05Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:33:09Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:33:17Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:33:34Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:34:12Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:34:50Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:35:29Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:36:20Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:36:56Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:37:37Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:38:24Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:39:20Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:39:58Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
time="2022-07-06T02:40:37Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: failed to list *v1alpha1.TracingPolicy: resourceVersion: Invalid value: \"\\x00\\x00\\x00\\x00\": strconv.ParseUint: parsing \"\\x00\\x00\\x00\\x00\": invalid syntax"
from tetragon.
Forsworns
commented on July 16, 2024
1
BTW, the log is too hard to read, lol. Here is the formatted version. The pod.namespace, pod.name and node_name are missing as
{
"process_exec": {
"process": {
"exec_id": "AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==",
"pid": 1732488,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/tetra",
"arguments": "status",
"flags": "execve rootcwd clone",
"start_time": "2022-06-24T01:27:13.470Z",
"auid": 4294967295,
"pod": {
"namespace": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"name": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"container": {
"id": "containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5",
"name": "tetragon",
"image": {
"id": "quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703",
"name": "quay.io/cilium/tetragon:v0.8.0"
},
"start_time": "2022-06-23T18:52:01Z",
"pid": 31562
}
},
"docker": "9001291f84d1fa67aeaafd788062186",
"parent_exec_id": "AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==",
"refcnt": 1
},
"parent": {
"exec_id": "AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==",
"pid": 1732478,
"uid": 0,
"cwd": "/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/",
"binary": "/snap/microk8s/3272/bin/runc",
"arguments": "--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5",
"flags": "execve clone",
"start_time": "2022-06-24T01:27:13.425Z",
"auid": 4294967295,
"parent_exec_id": "AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==",
"refcnt": 1
}
},
"node_name": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"time": "2022-06-24T01:27:13.470Z"
}
{
"process_exit": {
"process": {
"exec_id": "AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==",
"pid": 1732488,
"uid": 0,
"cwd": "/",
"binary": "/usr/bin/tetra",
"arguments": "status",
"flags": "execve rootcwd clone",
"start_time": "2022-06-24T01:27:13.470Z",
"auid": 4294967295,
"pod": {
"namespace": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"name": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"container": {
"id": "containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5",
"name": "tetragon",
"image": {
"id": "quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703",
"name": "quay.io/cilium/tetragon:v0.8.0"
},
"start_time": "2022-06-23T18:52:01Z",
"pid": 31562
}
},
"docker": "9001291f84d1fa67aeaafd788062186",
"parent_exec_id": "AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA=="
},
"parent": {
"exec_id": "AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==",
"pid": 1732478,
"uid": 0,
"cwd": "/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/",
"binary": "/snap/microk8s/3272/bin/runc",
"arguments": "--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5",
"flags": "execve clone",
"start_time": "2022-06-24T01:27:13.425Z",
"auid": 4294967295,
"parent_exec_id": "AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==",
"refcnt": 4294967295
}
},
"node_name": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
"time": "2022-06-24T01:27:13.476Z"
}
from tetragon.
Forsworns
commented on July 16, 2024
I use KinD and haven't encounter this problem. But I guess the problems origin from following codes
The podInfo is queried in
tetragon/pkg/process/process.go
Line 166 in 77e32bf
And the node name is queried in
tetragon/pkg/reader/node/node.go
Line 13 in 77e32bf
They are all done in golang side
from tetragon.
jpegleg
commented on July 16, 2024
One of the more severe occurrences of this behavior (observed on k3s) has only timestamp data intact, looking like this:
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:28.228Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:28.235Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:38.234Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:38.240Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:48.239Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:48.249Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:58.230Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:29:58.236Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:30:08.228Z"}
{"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-07-02T11:30:08.234Z"}
from tetragon.
kkourt
commented on July 16, 2024
Thank you for the report!
Could you please add the logs from the tetragon container?
from tetragon.
kkourt
commented on July 16, 2024
Here is a tetragon log sample from a k3s cluster that encountered this behavior:
Thanks!
time="2022-07-06T02:33:01Z" level=error msg="Kubernetes API error" error="github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:104: Failed to watch *v1alpha1.TracingPolicy: an error on the server ("unknown") has prevented the request from succeeding (get tracingpolicies.cilium.io)"
So it seems that we need to:
- Have a more sane event when this error happens (e.g., an error flag)
- Figure out whether the error is something to be expected or not
from tetragon.
Related Issues (20)
- Tetragon Go Test failing on actuated-arm64-4cpu-8gb: github.com/cilium/tetragon/pkg/sensors/tracing: TestTraceKernelModule HOT 1
- I think that we currently cannot have a TracingPolicy and a TracingPolicyNamespaced with the same name.
- LSM `security_path_*` deletion kprobes can't resolve target dentry name HOT 2
- field-filters: filter message and tags fields
- metrics: add tags support HOT 2
- policyfilter test failure on bpf-next
- gRPC server can't be disabled
- Tetragon does not allow getting healthcheck and non healthcheck events at the same time
- Using rateLimit on tcp_connect calls doesn't have any effect HOT 2
- Documentation: broken links automatic report HOT 4
- Documentation: broken links automatic report
- update docs to use NotifyEnforcer HOT 1
- v1.1.1 release HOT 1
- v1.1.2 release HOT 2
- TestDemoApp failure
- Dynamic log level change doesn't work the same for all environments
- Cannot track users attempting to follow symlinks outside of a directory root
- New `TestPolicyListingWhileLoadUnload` introduces a flake
- `tetra getevents` filtering with --pods returns no pod HOT 1
- Implement parent_binary export filter
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tetragon.