cilium / tetragon Goto Github PK

View Code? Open in Web Editor NEW

3.4K 53.0 324.0 69.67 MB

eBPF-based Security Observability and Runtime Enforcement

Home Page: https://tetragon.io

License: Apache License 2.0

Dockerfile 0.12% Makefile 0.69% C 45.39% Go 52.86% Smarty 0.16% Shell 0.72% Python 0.06%

bpf ebpf kernel kubernetes security

tetragon's People

Contributors

Stargazers

Watchers

Forkers

leewalter willfindlay xmulligan davaddi fwxiong umesh3034 rajatguptarg aws-coder sayboras dmitris git0ffs3c bryanasdev000 tixxdz dumpanalysis sjanulonoks silverfoxs muse117 jack-r-lantern percivalll chriskalix brightverge doytsujin nam-jaehyun blue-infosec walking-appa gdemarcsek paravatha cen-ngc5139 lllwan halleyhan derekhjray ethercflow arvinzhao88 tailm marwannour timb-machine-mirrors subconsciouscompute icodein eternalerrors btsgrp tklauser chancez rchittaro kimstacy andreagit97 arcilli tnorlin olsajiri dimanne new1phalley raphink molter73 echohead gmh5225 laplacekorea dreamice2012 losenineai gdzy1987 laozhudetui svaaz lic17 devops-mirror furisto rafaribe du2016 drizzlerisk uranusblockstack kjm0001 taction johnhowlett jeevanjoseph cardyok mrtc0 saxton-michael tzstoyanov shepf leifhenriksen michaelhyatt mtardy feiyunwill dnsmichi duanjiong forsworns ofriouzan godlike910716 pxp928 dsseng webvul br1ght3n anjmao acumenix chaosinthecrd infrastrukt liyanvip vincentmli mjtian13 clix-dev-llc xjas sarahfujimori mfrw

tetragon's Issues

Cannot merge two enter events error

I use minikube and install tetragon with helm. I run the File Access example in the readme, after applied ./crds/examples/sys_write_follow_fd_prefix.yaml, the output of command kubectl logs -n kube-system ds/tetragon -c export-stdout -f | tetragon observe --namespace default --pod xwing works as expected. But I see the following log in the tetragon container.

time="2022-06-14T06:58:09Z" level=warning msg="cannot merge two enter events: prev:{ev:0xc001124b40 returnEvent:false} curr:{ev:0xc001124be0 returnEvent:false}"
time="2022-06-14T06:58:09Z" level=warning msg="cannot merge two enter events: prev:{ev:0xc001124d20 returnEvent:false} curr:{ev:0xc001124dc0 returnEvent:false}"

$ kubectl get tracingpolicies.cilium.io -A
NAME                     AGE
sys-read-follow-prefix   21h

NULLs in tetragon data

When tetragon is installed via helm as instructed, some fields are populated by null strings (example in data below) instead of the expected data:

{"process_exec":{"process":{"exec_id":"AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==","pid":1732488,"uid":0,"cwd":"/","binary":"/usr/bin/tetra","arguments":"status","flags":"execve rootcwd clone","start_time":"2022-06-24T01:27:13.470Z","auid":4294967295,"pod":{"namespace":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","container":{"id":"containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","name":"tetragon","image":{"id":"quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703","name":"quay.io/cilium/tetragon:v0.8.0"},"start_time":"2022-06-23T18:52:01Z","pid":31562}},"docker":"9001291f84d1fa67aeaafd788062186","parent_exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","refcnt":1},"parent":{"exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","pid":1732478,"uid":0,"cwd":"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/","binary":"/snap/microk8s/3272/bin/runc","arguments":"--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","flags":"execve clone","start_time":"2022-06-24T01:27:13.425Z","auid":4294967295,"parent_exec_id":"AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==","refcnt":1}},"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-06-24T01:27:13.470Z"}
{"process_exit":{"process":{"exec_id":"AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==","pid":1732488,"uid":0,"cwd":"/","binary":"/usr/bin/tetra","arguments":"status","flags":"execve rootcwd clone","start_time":"2022-06-24T01:27:13.470Z","auid":4294967295,"pod":{"namespace":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","container":{"id":"containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","name":"tetragon","image":{"id":"quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703","name":"quay.io/cilium/tetragon:v0.8.0"},"start_time":"2022-06-23T18:52:01Z","pid":31562}},"docker":"9001291f84d1fa67aeaafd788062186","parent_exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA=="},"parent":{"exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","pid":1732478,"uid":0,"cwd":"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/","binary":"/snap/microk8s/3272/bin/runc","arguments":"--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","flags":"execve clone","start_time":"2022-06-24T01:27:13.425Z","auid":4294967295,"parent_exec_id":"AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==","refcnt":4294967295}},"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-06-24T01:27:13.476Z"}

It isn't just tetragon container events that have these nulls like the example, other container's tetragon trace data has also been observed to be impacted.

This behavior hasn't been appearing immediately when tetragon is installed, it seems to take some time before it starts happening, perhaps a few hours or a day, then it continues to occur.

I have observed this behavior across multiple builds of k3s and microk8s clusters but only when tetragon is deployed. We can also see related data in the microk8s kubelite daemon log:

microk8s.daemon-kubelite.log:Jun 23 13:17:55 moon2 microk8s.daemon-kubelite[16056]: E0623 13:17:55.084224   16056 status.go:71] apiserver received an error that is not an metav1.Status: storage.InvalidError{Errs:field.ErrorList{(*field.Error)(0xc014330000)}}: resourceVersion: Invalid value: "\x00\x00\x00\x00\x00\x00": strconv.ParseUint: parsing "\x00\x00\x00\x00\x00\x00": invalid syntax

This behavior does not occur when tetragon is not running/installed into the clusters.

Tests for this I have done so far have been with both k3s and microk8s clusters.
Cluster nodes in the tests have been running OpenSUSE Leap 15.4, Ubuntu 22 Server, Ubuntu 22 Desktop, (kernels 5.15.0-39-generic and 5.14.21-150400.22-default at the moment).

Currently I have a microk8s cluster running with this happening.

Let me know if there is any other data that would be helpful!

replace verifier.sh with a go verifier test that uses cilium/ebpf

https://github.com/cilium/tetragon/blob/main/contrib/verify/verify.sh

build/bpf: fix bpf_execve_event_v53.o under clang-14

Currently, upstream clang-13 works fine for compiling our BPF programs. However when I attempted to upgrade to clang-14 I found that the event_execve program in bpf_execve_event_v53.o spikes its instruction count to over 1M and this fails the verifier.

Logs:

; probe_read(&dentry, sizeof(dentry),
2391: (b7) r2 = 8                     ; R2_w=8
2392: (bf) r3 = r6                    ; R3_w=scalar(id=69319) R6_w=scalar(id=69319)
2393: (85) call bpf_probe_read#4      ; R0_w=scalar() fp-8=mmmmmmmm
; mnt = parent;
2394: (79) r6 = *(u64 *)(r10 -112)    ; R6_w=scalar() R10=fp0
2395: (bf) r3 = r6                    ; R3_w=scalar(id=69320) R6_w=scalar(id=69320)
2396: (b7) r1 = 32                    ; R1_w=32
2397: (0f) r3 += r1                   ; R1_w=32 R3_w=scalar()
2398: (bf) r1 = r10                   ; R1_w=fp0 R10=fp0
;
2399: (07) r1 += -88                  ; R1_w=fp-88
; probe_read(&vfsmnt, sizeof(vfsmnt),
2400: (b7) r2 = 8
BPF program is too large. Processed 1000001 insn
processed 1000001 insns (limit 1000000) max_states_per_insn 22 total_states 38642 peak_states 2336 mark_read 177

Full verifier logs available here.

Monitor connections to IPs

Just wondering if it's possible to monitor incoming/outgoing connections to certain IPs. I've seen the tcp_connect example and I'm trying to figure out how I can look for IP addresses and trigger events based on that.
A scenario would be having a process trying to connect to a C2 server (or several) and the goal is to trigger a Sigkill action after I detect that connection.

bpf_execve_event_v53 compiled with clang-13 fails verifier on Linux 5.4

Verifying bpf/objs/bpf_execve_event_v53.o... Failed!
 R0=map_value(id=0,off=0,ks=4,vs=32768,imm=0) R1_w=inv4294967295 R6_w=map_value(id=0,off=0,ks=4,vs=32768,imm=0) R7=map_value(id=0,off=0,ks=4,vs=1848,imm=0) R8=map_value(id=0,off=248,ks=4,vs=1848,imm=0) R9=map_value(id=0,off=0,ks=4,vs=1848,imm=0) R10=fp0 fp-8=????mmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=????mmmm fp-176=mmmm???? fp-184=ctx fp-192=mmmmmmmm fp-200=ctx fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=map_value fp-232=map_value fp-248=mmmmmmmm fp-256=mmmmmmmm
1587: (7b) *(u64 *)(r10 -272) = r8
1588: (79) r7 = *(u64 *)(r10 -248)
;
1589: (0f) r7 += r9
last_idx 1589 first_idx 1583
regs=80 stack=0 before 1588: (79) r7 = *(u64 *)(r10 -248)
regs=0 stack=40000000 before 1587: (7b) *(u64 *)(r10 -272) = r8
regs=0 stack=40000000 before 1586: (15) if r6 == 0x0 goto pc+177
regs=0 stack=40000000 before 1584: (18) r1 = 0xffffffff
regs=0 stack=40000000 before 1583: (bf) r6 = r0
 R0_rw=map_value_or_null(id=14,off=0,ks=4,vs=32768,imm=0) R6=inv(id=0) R7=map_value(id=0,off=0,ks=4,vs=1848,imm=0) R8_r=map_value(id=0,off=248,ks=4,vs=1848,imm=0) R9_r=map_value(id=0,off=0,ks=4,vs=1848,imm=0) R10=fp0 fp-8=????mmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=????mmmm fp-176=mmmm???? fp-184=ctx fp-192_w=mmmmmmmm fp-200=ctx fp-208=mmmmmmmm fp-216_w=mmmmmmmm fp-224=map_value fp-232=map_value fp-248_rw=mmmmmmmm fp-256=mmmmmmmm
parent already had regs=0 stack=0 marks
math between map_value pointer and register with unbounded min value is not allowed
processed 121492 insns (limit 1000000) max_states_per_insn 17 total_states 5967 peak_states 626 mark_read 262

tests/flake: TestMultipleMountsFiltered fails with missing process info

The TestMultipleMountsFiltered tests has been failing occasionally in CI.

Example of a failed run: https://github.com/cilium/tetragon/runs/6492564815?check_suite_focus=true

The root cause seems to be due to missing process fields in the kprobe events, like so:

{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7ImJ5dGVzX2FyZyI6ImV5SndjbTlqWlhOelgydHdjbTlpWlNJNmV5SndjbTlqWlhOeklqcDdJbkJwWkNJNk1URXlNalVzSW5OMFlYSjBYM1JwYldVaU9pSXlNREl5TFRBMUxURTRWREUxT2pRNU9qSTJMalUxTjFvaWZTd2ljR0Z5Wlc1MElqcDdmU3dpWm5WdVkzUnBiMjVmYm1GdFpTSTZJbDlmZURZMFgzTjVjMTkzY21sMFpTSXNJbUZ5WjNNaU9sdDdJbVpwYkdWZllYSm5JanA3SW5CaGRHZ2lPaUl2ZEcxd01pOTBiWEF6TDNSdGNEUXZkRzF3TlM5MFpYTjBabWxzWlNKOWZTeDdJblJ5ZFc1allYUmxaRjlpZVhSbGMxOWhjbWNpT25zaVlubDBaWE5mWVhKbklqb2laWGxLZDJOdE9XcGFXRTU2V0RKMGQyTnRPV2xhVTBrMlpYbEtkMk50T1dwYVdFNTZTV3B3TjBsdVFuQmFRMGsyVFZSRmVVMXFWWE5KYms0d1dWaEtNRmd6VW5CaVYxVnBUMmM5UFNJc0ltOXlhV2RmYzJsNlpTSTZJalF4TlRFaWZYMHNleUp6YVhwbFgyRnlaeUk2SWpReE5URWlmVjBzSW1GamRHbHZiaUk2SWt0UVVrOUNSVjlCUTFSSlQwNWZVRTlUVkNKOUxDSjBhVzFsSWpvaU1qQXlNaTB3TlMweE9GUXhOVG8xTURveU55NHhOVGhhSW4wSyJ9LHsic2l6ZV9hcmciOiI0MTQifV0sImFjdGlvbiI6IktQUk9CRV9BQ1RJT05fUE9TVCJ9LCJ0aW1lIjoiMjAyMi0wNS0xOFQxNTo1MDoyNy4xNThaIn0K"},{"size_arg":"846"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}
{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7InRydW5jYXRlZF9ieXRlc19hcmciOnsiYnl0ZXNfYXJnIjoiZXlKd2NtOWpaWE56WDJ0d2NtOWlaU0k2ZXlKd2NtOWpaWE56SWpwN0luQnBaQ0k2TVRFeU1qVXNJbk4wWVhKMFgzUnBiV1VpT2lJeU1ESXlMVEExTFRFNFZERTFPalE1T2pJMkxqVTFOMW9pZlN3aWNHRnlaVzUwSWpwN2ZTd2lablZ1WTNScGIyNWZibUZ0WlNJNklsOWZlRFkwWDNONWMxOTNjbWwwWlNJc0ltRnlaM01pT2x0N0ltWnBiR1ZmWVhKbklqcDdJbkJoZEdnaU9pSXZkRzF3TWk5MGJYQXpMM1J0Y0RRdmRHMXdOUzkwWlhOMFptbHNaU0o5ZlN4N0ltSjVkR1Z6WDJGeVp5STZJbVY1U25kamJUbHFXbGhPZWxneWRIZGpiVGxwV2xOSk5tVjVTbmRqYlRscVdsaE9la2xxY0RkSmJrSndXa05KTmsxVVJYbE5hbFZ6U1c1T01GbFlTakJZTTFKd1lsZFZhVTlwU1hsTlJFbDVURlJCTVV4VVJUUldSRVV4VDJwUk5VOXFTVEpNYWxVeFRqRnZhV1pUZDJsalIwWjVXbGMxTUVscWNEZG1VM2RwV201V2RWa3pVbkJpTWpWbVltMUdkRnBUU1RaSmJEbG1aVVJaTUZnelRqVmpNVGt6WTIxc01GcFRTWE5KYlVaNVdqTk5hVTlzZERkSmJWcHdZa2RXWmxsWVNtNUphbkEzU1c1Q2FHUkhaMmxQYVVsMlpFY3hkMDFwT1RCaVdFRjZURE5TZEdORVVYWmtSekYzVGxNNU1GcFlUakJhYld4eldsTktPV1pUZURkSmJVbzFaRWRXZWxneVJubGFlVWsyU1cxV05WTnVaR3BpVkd4eFYyeG9UMlZzWjNsa1NHUnFZbFJzY0Zkc1RrcE9iVlkxVTI1a2FtSlViSEZYYkdoUFpXdHNjV05FWkVwaWEwcDNWMnRPU2s1ck1WVlNXR3hPWVd4V2VsTlhOVTlOUm14WlUycENXVTB4U25kWmJHUldZVlU1Y0ZOWWJFNVNSV3cxVkVaU1FrMVZlRlZTVkZKWFVrVlZlRlF5Y0ZKT1ZUbHhVMVJLVFdGc1ZYaFVha1oyWVZkYVZHUXliR3BTTUZvMVYyeGpNVTFGYkhGalJHUnRWVE5rY0ZkdE5WZGtWbXQ2Vlc1Q2FVMXFWbTFaYlRGSFpFWndWRk5VV2twaVJHeHRXbFZTV2sxR1ozcFVhbFpxVFZScmVsa3lNWE5OUm5CVVUxaE9TbUpWV2pWWGFrNU9ZVlU1YzJSRVpFcGlWbkIzV1d0a1YxcHNiRmxUYlRWS1lXNUJNMU5YTlVOaFIxSklXakpzVUdGVmJESmFSV040WkRBeGNFOVVRbWxYUlVZMlZFUk9VMlJIVGtWVldGcHJVbnBHTTFSc1RUVk5SbkJaVkdwQ1lXSlhlSHBYYkU1TFQxZGFWR1ZFWkVwaWJFbzFXa1pqTVdGc2JGbFZiWGhoVW1wc2NGcFciLCJvcmlnX3NpemUiOiI0OTM5In19LHsic2l6ZV9hcmciOiI0OTM5In1dLCJhY3Rpb24iOiJLUFJPQkVfQUNUSU9OX1BPU1QifSwidGltZSI6IjIwMjItMDUtMThUMTU6NTA6MjcuMTU4WiJ9Cg=="},{"size_arg":"1462"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}
{"process_kprobe":{"process":{"pid":11225,"start_time":"2022-05-18T15:49:26.557Z"},"parent":{},"function_name":"__x64_sys_write","args":[{"file_arg":{"path":"/tmp2/tmp3/tmp4/tmp5/testfile"}},{"bytes_arg":"eyJwcm9jZXNzX2twcm9iZSI6eyJwcm9jZXNzIjp7InBpZCI6MTEyMjUsInN0YXJ0X3RpbWUiOiIyMDIyLTA1LTE4VDE1OjQ5OjI2LjU1N1oifSwicGFyZW50Ijp7fSwiZnVuY3Rpb25fbmFtZSI6Il9feDY0X3N5c193cml0ZSIsImFyZ3MiOlt7ImZpbGVfYXJnIjp7InBhdGgiOiIvdG1wMi90bXAzL3RtcDQvdG1wNS90ZXN0ZmlsZSJ9fSx7ImJ5dGVzX2FyZyI6ImV5SndjbTlqWlhOelgydHdjbTlpWlNJNmV5SndjbTlqWlhOeklqcDdJbkJwWkNJNk1URXlNalVzSW5OMFlYSjBYM1JwYldVaU9pSXlNREl5TFRBMUxURTRWREUxT2pRNU9qSTJMalUxTjFvaWZTd2ljR0Z5Wlc1MElqcDdmU3dpWm5WdVkzUnBiMjVmYm1GdFpTSTZJbDlmZURZMFgzTjVjMTkzY21sMFpTSXNJbUZ5WjNNaU9sdDdJbVpwYkdWZllYSm5JanA3SW5CaGRHZ2lPaUl2ZEcxd01pOTBiWEF6TDNSdGNEUXZkRzF3TlM5MFpYTjBabWxzWlNKOWZTeDdJbUo1ZEdWelgyRnlaeUk2SW1WNVNuZGpiVGxxV2xoT2VsZ3lkSGRqYlRscFdsTkpObVY1U25kamJUbHFXbGhPZWtscWNEZEpia0p3V2tOSk5rMVVSWGxOYWxWelNXNU9NRmxZU2pCWU0xSndZbGRWYVU5cFNYbE5SRWw1VEZSQk1VeFVSVFJXUkVVeFQycFJOVTlxU1RKTWFsVXhUakZ2YVdaVGQybGpSMFo1V2xjMU1FbHFjRGRtVTNkcFdtNVdkVmt6VW5CaU1qVm1ZbTFHZEZwVFNUWkpiRGxtWlVSWk1GZ3pUalZqTVRrelkyMXNNRnBUU1hOSmJVWjVXak5OYVU5c2REZEpiVnB3WWtkV1psbFlTbTVKYW5BM1NXNUNhR1JIWjJsUGFVbDJaRWN4ZDAxcE9UQmlXRUY2VEROU2RHTkVVWFprUnpGM1RsTTVNRnBZVGpCYWJXeHpXbE5LT1daVGVEZEpibEo1WkZjMWFsbFlVbXhhUmpscFpWaFNiR014T1doamJXTnBUMjV6YVZsdWJEQmFXRTVtV1ZoS2JrbHFiMmxhV0d4TFpESk9kRTlYY0dGWFJUVTJWMFJLTUdReVRuUlBWMnhoVlRCck1scFliRXRrTWs1MFQxZHdZVmRGTlRaVFYzQjNUakJzZFZGdVFtRlJNR3N5VkZaU1JtVlZNWEZXV0U1S1ltczBkMWRXYUV0TlJtZDZWVzVDYVZZeFZuQlVNbU01VUZOSmMwbHRPWGxoVjJSbVl6SnNObHBUU1RaSmFsRjRUbFJGYVdaWU1ITmxlVXA2WVZod2JGZ3lSbmxhZVVrMlNXcFJlRTVVUldsbVZqQnpTVzFHYW1SSGJIWmlhVWsyU1d0MFVWVnJPVU5TVmpsQ1VURlNTbFF3TldaVlJUbFVWa05LT1V4RFNqQmhWekZzU1dwdmFVMXFRWGxOYVRCM1RsTXdlRTlHVVhoT1ZHOHhUVVJ2ZVU1NU5IaE9WR2hoU1c0d1N5SjlMSHNpYzJsNlpWOWhjbWNpT2lJME1UUWlmVjBzSW1GamRHbHZiaUk2SWt0UVVrOUNSVjlCUTFSSlQwNWZVRTlUVkNKOUxDSjBhVzFsSWpvaU1qQXlNaTB3TlMweE9GUXhOVG8xTURveU55NHhOVGhhSW4wSyJ9LHsic2l6ZV9hcmciOiI4NDYifV0sImFjdGlvbiI6IktQUk9CRV9BQ1RJT05fUE9TVCJ9LCJ0aW1lIjoiMjAyMi0wNS0xOFQxNTo1MDoyNy4xNThaIn0K"},{"size_arg":"1422"}],"action":"KPROBE_ACTION_POST"},"time":"2022-05-18T15:50:27.158Z"}

Chore: Update Cilium dependency

Our Cilium dependency is outdated. It would be nice to update it.

Not getting all outputs when using `kubectl logs` with `tetragon-cli`

As mentioned in the README, I was using kubectl logs -n kube-system ds/tetragon -c export-stdout -f | tetragon observe in order to observe the tetragon events. Unfortunately it seems that kubectl logs is failing to follow the logs (I think in the event that too many logs are being produced at once.

To combat this, I have used stern instead and this seems to be far more reliable:

stern -l app.kubernetes.io/name=tetragon -E tetragon -n kube-system -o raw | tetragon observe

Maybe it would be better to enable the tetragon-cli to not rely on being piped the output of logs from another binary.

Unsupported kernel issue

The container won't run, what should I do?

time="2022-06-11T01:37:21Z" level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: Kernel version \"3.10.0-1160.62.1.el7.x86_64\" BTF search failed kernel is not included in supported list. Use --btf option to specify BTF path and/or '--kernel' to specify kernel version"

Build fails on Arch Linux with unused-but-set-variable

Getting build failures on Arch Linux (up to date as of yesterday) when trying to build tetragon-bpf:

$ make tetragon-bpf
make -C ./bpf
make[1]: Entering directory '/var/abs/local/tetragon-git/src/tetragon/bpf'
...
clang -I. -I/usr/include/x86_64-linux-gnu/ -Wall -Werror -Wno-address-of-packed-member -Wno-compare-distinct-pointer-types -Wno-unknown-warning-option -O2 -I ./libbpf/ -I ./include/ -I ./lib -target bpf -emit-llvm -g -c process/bpf_generic_kprobe.c -o objs/bpf_generic_kprobe.ll
In file included from process/bpf_generic_kprobe.c:14:
process/generic_calls.h:10:24: error: variable 'a2' set but not used [-Werror,-Wunused-but-set-variable]
        unsigned long a0, a1, a2, a3, a4;

and so on about all lines 10, 138, 185, 232, & 280.
Guessing that this is a "newer compilers enable warnings as errors for absurd stuff by default" problem.

add homebrew formula for tetragon-cli

Just like installing cilium cli is a breeze on MacOS thanks to the formula, it would be super cool if there was also a formula for tetragon CLI.

tetragon: GetPodInfo() is called even a non k8s deployments

GetPodInfo() where we get the pod info for the events is being called even if Tetragon is run with enable-k8s-api:false , this is not an urgent or real issue unless it starts pulling some other bugs...

The ideal case would be to disable K8s code completely for such deployments, but before the question: what would be the side effects on k8s deployments then ? needs to be answered.

docs: describe generic kprobe/tracepoint operations

e.g., FollowFD, UnfollowFD

Add an ID field to the detection rule to help trace it back to the TracingPolicy

At the moment, all tracing events received from tetragon don't have any context to the "TracingPolicy" defined that rule in the first place.
For kprobe events for example, the best chance to correlate it is by using the function_name field, which may have several policy hooks for that specific function, so it isn't sufficient.

A sample usage could be to trace several kinds of write operations to files, that each belongs to a different kind of policy, and analyze the output properly depending on the source policy.
Many more use cases could be thought of.

That ID field could be the metadata.name for tracing policy which is usually defined:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "fd_install"
...

tests: possible flake in TestCopyFd

Just filing this for my own sanity so I remember to investigate later. Nothing to see here.

--- FAIL: TestCopyFd (4.31s)
    kprobe_copyfd_test.go:79: pid is 11464 and spec file is /tmp/copyfd-3340320829.yaml
    kprobe_sigkill_test.go:33: error reading stderr> : read |0: file already closed
    kprobe_sigkill_test.go:33: error reading stdout> : read |0: file already closed
    kprobe_copyfd_test.go:93: command failed with exit status 1. Context error: %!s(<nil>)
    filenames.go:60: deleting export file for TestCopyFd (/tmp/tetragon.gotest.TestCopyFd.3364309601.json)
FAIL

tetragon: update libbpf usage to cilium/ebpf or latest version of libbpf

We currently run a fairly old libbpf with an extra patch or two to get some functionality we needed. This issue is to update to either cilium/bpf (ideally) or at least run latest libbpf. This will clean up the build infra.

tetragon: TestCopyFd test failing on 4.19 kernel

TestCopyFd test is failing on 4.19 kernel. It looks like it does not get read event from 'dup' fd.
test.log
tetragon.gotest.TestCopyFd.720014031.json.log

repository for tetragon helm chart

Currently rather painful to be forced to download the github repo and run a local helm install.
I assume that you will start working on this when everything get's a bit more stable but always good to have an issue open.

In short please provide tetragon in a public repository, just like https://helm.cilium.io/.

Proposal: replace perf map to a ring buffer

Currently Tetragon uses tcpmon_map to send perf events, which is BPF_MAP_TYPE_PERF_EVENT_ARRAY.

Is it better to replace it by BPF_MAP_TYPE_RINGBUF?

See the blog on ringbuf vs perfbuf.

Missing fd_install events in dup system call

With the following configuration:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "copyfd"
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: int
    - index: 1
      type: "file"
    selectors:
    - matchPIDs:
      - operator: In
        followForks: true
        values:
        - <ADD_PID_HERE>

and the following program:

#define _GNU_SOURCE 
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>

#define BUFSIZE 9

#define errExit(msg) \
   do { \
      perror(msg); \
      exit(EXIT_FAILURE); \
   } while (0)

void read_with_fd(int fd)
{
    ssize_t retval;
    char buf[BUFSIZE];

    if (lseek(fd, 0, SEEK_SET) != 0)
        errExit("lseek");

    retval = read(fd, buf, BUFSIZE);
    if (retval != BUFSIZE)
        errExit("read");

    buf[BUFSIZE - 1] = '\0';
    printf("%s\n", buf);
}

int main(int argc, char *argv[])
{
    int fd, fd_dup;
    const char *pathname = "/etc/passwd";

    fd = open(pathname, O_RDONLY);
    if (fd == -1)
        errExit("open");

    read_with_fd(fd);

    fd_dup = dup(fd);
    if (fd_dup == -1)
         errExit("dup");

    read_with_fd(fd_dup);

    close(fd_dup);    
    close(fd);

    return 0;
}

In a 5.13 kernel we receive 2 fd_install events as expected (one for the open and one for the dup system calls). These events are:

{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEyMTU1Njg3NTM2MDoxOTc1",
      "pid": 1975,
      "uid": 1010,
      "cwd": "/home/apapag/tetragon",
      "binary": "/home/apapag/tetragon/contrib/tester-progs/follow-tester",
      "flags": "execve clone",
      "start_time": "2022-06-15T08:02:08.275287599Z",
      "auid": 1010,
      "parent_exec_id": "OjE2MzgwMDAwMDAwOjE1NzY=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjE2MzgwMDAwMDAwOjE1NzY=",
      "pid": 1576,
      "uid": 0,
      "cwd": "/home/apapag/tetragon",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-15T08:00:23.098396913Z",
      "auid": 0,
      "parent_exec_id": "OjE2MjYwMDAwMDAwOjE1NzU=",
      "refcnt": 1
    },
    "function_name": "fd_install",
    "args": [
      {
        "int_arg": 3
      },
      {
        "file_arg": {
          "path": "/etc/passwd"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-06-15T08:02:08.275512060Z"
}
{
  "process_kprobe": {
    "process": {
      "exec_id": "OjEyMTU1Njg3NTM2MDoxOTc1",
      "pid": 1975,
      "uid": 1010,
      "cwd": "/home/apapag/tetragon",
      "binary": "/home/apapag/tetragon/contrib/tester-progs/follow-tester",
      "flags": "execve clone",
      "start_time": "2022-06-15T08:02:08.275287599Z",
      "auid": 1010,
      "parent_exec_id": "OjE2MzgwMDAwMDAwOjE1NzY=",
      "refcnt": 1
    },
    "parent": {
      "exec_id": "OjE2MzgwMDAwMDAwOjE1NzY=",
      "pid": 1576,
      "uid": 0,
      "cwd": "/home/apapag/tetragon",
      "binary": "/usr/bin/bash",
      "flags": "procFS auid",
      "start_time": "2022-06-15T08:00:23.098396913Z",
      "auid": 0,
      "parent_exec_id": "OjE2MjYwMDAwMDAwOjE1NzU=",
      "refcnt": 1
    },
    "function_name": "fd_install",
    "args": [
      {
        "int_arg": 4
      },
      {
        "file_arg": {
          "path": "/etc/passwd"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST"
  },
  "time": "2022-06-15T08:02:08.275547122Z"
}

In 5.10 and 5.4 (at least) we receive only the first event (i.e. dup system call does not generate an fd_install event).

ci flake: kprobe_copyfd_test missing all expected events

Just got a CI flake in https://github.com/cilium/tetragon/runs/7410818480?check_suite_focus=true#step:7:10985 where we are not seeing any of the expected events in kprobe_copyfd_test. // cc @tpapagian

Here is the Tetragon json log from the test: tetragon-json.zip

avoid having to run the 'go' compiler with sudo in 'make test'

Currently you have to run 'make test' with sudo (understandably since you need to load BPF programs etc.) which also runs the go compiler with elevated priviledges - which makes some hard-core Paranoids cringe with fear and lose their sleep 😄. The compiler - even as wonderful and well-cared-for as Go - is a big and complex piece of software and can at some times have some bugs, running a compiler as sudo is against the good security practices and also is not necessary. We should build the test binaries with go test -c in a make step that doesn't require 'sudo', and then only run those test binaries with sudo.

In the output of make I already see some test binaries being made:

go build -gcflags="" -ldflags="-X 'github.com/cilium/tetragon/pkg/version.Version=v0.8.0-24-g61731ca'" -mod=vendor -o tetragon-alignchecker ./tools/alignchecker/
mkdir -p go-tests
go test -gcflags="" -c ./pkg/bugtool         -o go-tests/bugtool.test
go test -gcflags="" -c ./pkg/filters         -o go-tests/filters.test
go test -gcflags="" -c ./pkg/grpc            -o go-tests/grpc.test
go test -gcflags="" -c ./pkg/metrics         -o go-tests/metrics.test
go test -gcflags="" -c ./pkg/stacktracetree  -o go-tests/stacktracetree.test
go test -gcflags="" -c ./pkg/vtuplefilter    -o go-tests/vtuplefilter.test
go test -gcflags="" -c ./pkg/tracepoint      -o go-tests/tracepoint.test
go test -gcflags="" -c ./pkg/config          -o go-tests/config.test
go test -gcflags="" -c ./pkg/idtable         -o go-tests/idtable.test
go test -gcflags="" -c ./pkg/bpf             -o go-tests/bpf.test
go test -gcflags="" -c ./pkg/btf             -o go-tests/btf.test
make -C contrib/sigkill-tester

Are these all or only some? We should build all test binaries like this in a step other than make test (that one would run with sudo), maybe just in make itself or one of its dependencies.

Kernel version 5.18.2-1 not supported

os： CentOS Linux release 7.9.2009
problem：After upgrading the latest kernel with elrepo, the installation of tetragon prompts that the kernel version is not supported

log：
time="2022-06-07T08:10:36Z" level=info msg="Loaded config from directory" config-dir=/etc/tetragon
time="2022-06-07T08:10:36Z" level=info msg="Starting tetragon" version=v0.8.0
time="2022-06-07T08:10:36Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: cilium-bpf: config-dir:/etc/tetragon config-file: debug:false enable-cilium-api:false enable-export-aggregation:false enable-k8s-api:true enable-process-ancestors:true enable-process-cred:false enable-process-ns:false export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{"event_set":["PROCESS_EXEC", "PROCESS_EXIT", "PROCESS_KPROBE"]} export-denylist:{"health_check":true}\n{"namespace":["", "cilium", "kube-system"]} export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-rotation-interval:0s export-filename:/var/run/cilium/tetragon/tetragon.log export-rate-limit:-1 force-small-progs:false ignore-missing-progs:false kernel: log-format:text log-level:info metrics-server::2112 netns-dir:/var/run/docker/netns/ process-cache-size:65536 procfs:/procRoot run-standalone:false server-address:localhost:54321 verbose:0]"
time="2022-06-07T08:10:36Z" level=info msg="Available sensors" sensors=
time="2022-06-07T08:10:36Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor"
time="2022-06-07T08:10:36Z" level=info msg="Registered probe types" types="kprobe sensor, tracepoint sensor"
time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/sys/kernel/btf/vmlinux
time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/var/lib/tetragon/metadata/vmlinux-5.18.2-1.el7.elrepo.x86_64
time="2022-06-07T08:10:36Z" level=info msg="candidate btf file does not exist" file=/var/lib/tetragon/btf
time="2022-06-07T08:10:36Z" level=fatal msg="Failed to start tetragon" error="tetragon, aborting kernel autodiscovery failed: Kernel version "5.18.2-1.el7.elrepo.x86_64" BTF search failed kernel is

Panic with unexpected Execve events

I run Tetragon on a server out of the k8s cluster via

sudo LD_LIBRARY_PATH=$(realpath ./lib) ./tetragon --bpf-lib bpf/objs

And then I login into the server via ssh, the Tetragon on the server run into the panic

panic: runtime error: slice bounds out of range [:-1]

goroutine 72 [running]:
github.com/cilium/tetragon/pkg/sensors/exec.execParse(0x1)
        /root/tetragon/pkg/sensors/exec/exec.go:115 +0xae5

The corresponding code is

tetragon/pkg/sensors/exec/exec.go

Line 114 in a797a06

n := bytes.Index(args, []byte{0x00})

I tried to add a check before slicing to avoid the panic (see the PR #207), but I don't know if this perf event is expected during login. Maybe we have to read the ebpf part to explore the reason.

Here is the perf event causing the panic, I add a breakpoint before it:

JSON log format to increase reported timestamps resolution

Currently, the time is reported only in milliseconds, which causes the loss of chronological order of events:

Increasing the reported time resolution to microseconds or even nanoseconds or making it configurable can be useful.

[Question] How to collect `process_connect `、`process_close` and `process_listen` events

I've seen the four golden signlas here: https://github.com/cilium/tetragon/tree/main/docs/security-observability-with-ebpf/03_chapter/00_four_golden_signals . But there are only tutorials about how to collect "PROCESS_EXEC", "PROCESS_EXIT", "PROCESS_KPROBE" events.
I want to know how to collect process_connect 、process_close and process_listen events, any help would be appreciated!

Here is my tetragon configmap:

kubectl get cm -n kube-system tetragon-config -o yaml
apiVersion: v1
data:
  enable-k8s-api: "true"
  enable-process-cred: "false"
  enable-process-ns: "false"
  export-allowlist: '{"event_set":["PROCESS_EXEC", "PROCESS_EXIT", "PROCESS_KPROBE",
    "PROCESS_TRACEPOINT"]}'
  export-denylist: |-
    {"health_check":true}
    {"namespace":["", "cilium", "kube-system"]}
  export-file-compress: "false"
  export-file-max-backups: "5"
  export-file-max-size-mb: "10"
  export-filename: /var/run/cilium/tetragon/tetragon.log
  export-rate-limit: "-1"
  metrics-server: :2112
  process-cache-size: "65536"
  procfs: /procRoot
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: tetragon
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2022-06-15T01:45:41Z"
  labels:
    app.kubernetes.io/instance: tetragon
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: tetragon
    helm.sh/chart: tetragon-0.8.0
  name: tetragon-config
  namespace: kube-system
  resourceVersion: "20294151"
  uid: 1749b71a-e54f-4c4f-9eb8-881c73bf4841

tests/e2e: write some end-to-end tests with the new e2e-framework

Filing this issue to gauge who would be interested in taking a stab at writing some end-to-end tests using the new framework introduced in #226. Happy to help mentor new contributors who want to take a stab at this.

Here is a wish list:

skeleton test that can be used as a basis for writing others
pod labels test
kprobe/kretprobe test (read/write a file)
kprobe/kretprobe test (tracing tcp connections)
pod lifecycle tracing test (exec/kprobe(s)/exit with correct namespaces, pod labels, etc)

Document path resolution code

Document better the path resolution code introduced in #90.

eventchecker: support additional map[KEY]VAL types

#260 adds support in the eventchecker for map[string]string but it would be nice to future-proof this by adding support for other map[KEY]VAL types as well. Doing so would require some non-trivial refactoring of the eventchecker codegen but I think it's probably worth doing as a stretch goal.

One thing that would make this a lot easier is moving to Go 1.18 to enable us to use generics. You could imagine, for example generic MapCheck and ListChecker types that could replace what we have now.

Add LICENSE file

Standalone daemon installation package without vagrant / kubernetes?

Hey team,

After reading the announcement of the project, I've been eager and excited to play around and compare how the tool compares to others like Falco.

I was able to follow the instructions and get an environment running within a Vagrant image and monitoring the sample kuburnetes container images.

At a glance (might be wrong here), it seems that the Makefile makes some assumptions about being installed in a container and not the host itself.

Are there any plans to offer an installation package (deb, rpm) that allows for a stand-alone installation and configuration of the daemon?

codegen: get go 1.18 generics working in codegen

#112 is removing references to go 1.18 generics in our codegen because it's spitting out code that doesn't compile. It would be nice to get this fixed at some point in the future so we can add them back.

tracing tests: Mkdir failed: mkdir /tmp2: file exists

Using the new test option (-disable-tetragon-logs, #237) I'm seeing:

--- PASS: TestKprobeObjectOpen (2.78s)  
=== RUN   TestKprobeObjectOpenMount     
    kprobe_test.go:583: Mkdir failed: mkdir /tmp2: file exists
    kprobe_test.go:584:                 
--- SKIP: TestKprobeObjectOpenMount (0.00s)
=== RUN   TestKprobeObjectMultiValueOpen
    jsonchecker.go:132: jsonTestCheck: opening: /tmp/tetragon.gotest.TestKprobeObjectMultiValueOpen.359677843.json
    filenames.go:60: deleting export file for TestKprobeObjectMultiValueOpen (/tmp/tetragon.gotest.TestKprobeObjectMultiValueOpen.359677843.json)
--- PASS: TestKprobeObjectMultiValueOpen (2.75s)
=== RUN   TestKprobeObjectMultiValueOpenMount
    kprobe_test.go:583: Mkdir failed: mkdir /tmp2: file exists
    kprobe_test.go:584:

Which points to:

tetragon/pkg/sensors/tracing/kprobe_test.go

Lines 578 to 584 in 0c08c89

 mntPath := "/tmp" 

 if useMount == true { 

 mntPath = mountPath 

 if err := os.Mkdir(mntPath, 0755); err != nil { 

 t.Logf("Mkdir failed: %s\n", err) 

 t.Skip()

We should probably fix this since the test is skipped. I think the best option would be to create a temp directory (https://pkg.go.dev/os#MkdirTemp) for each test and remove it afterwards.

Helm charts: Command is hardcoded in `_container_export_stdout.tpl`

It is currently set to be hubble-export-stdout and it is a good idea to make it configurable, similar to the rest of the export template, so it is possible to plug in another exporter container for tetragon pod.

bump cilium/ebpf from 0.8.1 to 0.9.0

There were some conflicts (#91), which we need to resolve first.

loader: refactor/tighten up loader codepaths

Right now our program loading/unloading logic is a bit messy and could probably be cleaned up quite significantly. In particular, it would be nice to have one shared codepath for program loading and one for map loading that takes care of any shared logic common to all program/map types. This would benefit things like #172 where we want to start keeping better track of what programs/maps sensors and loading and unloading over time.

Such changes could either be part of the cilium/ebpf refactor or happen afterwards. In the meantime, I'm filing this as a tracking issue so we can discuss it.

Fix execve_map_value in alignchecker

C and Go structs alignment check failed: execvemap.ExecveValue(48) size does not match execve_map_value(112)

More context in #200

Is there any way to send container logs to kafka

or we must need another tool like filebeat or fluentd?

Add first documentation for contribution and developement

Add documentation guides for contribution and development.

This issue and the commit log that fixes it will be referenced in the doc as examples.

docs: add better instructions for module vendoring

Folks who are new to Golang might have trouble doing the go mod tidy/vendor/verify dance. It would be nice to provide some better instructions in developer docs to avoid headaches.

DEVELOP.md: install libelf-dev and libcap-dev

got these errors while following DEVELOP.md:

/usr/bin/ld: cannot find -lelf

test_caps.c:11:10: fatal error: sys/capability.h: No such file or directory
   11 | #include <sys/capability.h>
      |          ^~~~~~~~~~~~~~~~~~

Make tetragon bind port user-configurable

Tetragon container reports port already in use on Flatcar OS and the tetragon container fails to start.

uname -a 
Linux ip-10-1-3-24.us-west-2.compute.internal 5.15.37-flatcar #1 SMP Wed May 4 13:53:25 -00 2022 x86_64 x86_64 x86_64 GNU/Linux

kubectl logs -n kube-system    tetragon-6x6fs -c tetragon
time="2022-05-18T16:31:45Z" level=info msg="Loaded config from directory" config-dir=/etc/tetragon
time="2022-05-18T16:31:45Z" level=info msg="Starting tetragon" version=v0.8.0
time="2022-05-18T16:31:45Z" level=info msg="config settings" config="map[bpf-lib:/var/lib/tetragon/ btf: cilium-bpf: config-dir:/etc/tetragon config-file: debug:false enable-cilium-api:false enable-export-aggregation:false enable-k8s-api:true enable-process-ancestors:true enable-process-cred:false enable-process-ns:false export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist:{\"event_set\":[\"PROCESS_EXEC\", \"PROCESS_EXIT\", \"PROCESS_KPROBE\"]} export-denylist:{\"health_check\":true}\n{\"namespace\":[\"\", \"cilium\", \"kube-system\"]} export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-rotation-interval:0s export-filename:/var/run/cilium/tetragon/tetragon.log export-rate-limit:-1 force-small-progs:false ignore-missing-progs:false kernel: log-format:text log-level:info metrics-server::2112 netns-dir:/var/run/docker/netns/ process-cache-size:65536 procfs:/procRoot run-standalone:false server-address:localhost:54321 verbose:0]"
time="2022-05-18T16:31:45Z" level=info msg="Available sensors" sensors=
time="2022-05-18T16:31:45Z" level=info msg="Registered tracing sensors" sensors="tracepoint sensor, kprobe sensor"
time="2022-05-18T16:31:45Z" level=info msg="Registered probe types" types="kprobe sensor, tracepoint sensor"
time="2022-05-18T16:31:45Z" level=info msg="Enabling Kubernetes API"
time="2022-05-18T16:31:45Z" level=info msg="Starting metrics server" addr=":2112"
time="2022-05-18T16:31:45Z" level=info msg="Initialized pod cache" num_pods=6
time="2022-05-18T16:31:45Z" level=info msg="Disabling Cilium API"
time="2022-05-18T16:31:45Z" level=info msg="Starting process manager" enableCilium=false enableEventCache=true enableProcessCred=false enableProcessNs=false
time="2022-05-18T16:31:45Z" level=fatal msg="Failed to start gRPC server" address="localhost:54321" error="listen tcp 127.0.0.1:54321: bind: address already in use"

  tetragon:
    Container ID:  docker://73da3ea27f5d42c1d926594d4fe4cfe0503701c07df684c889217df5b2bb742c
    Image:         quay.io/cilium/tetragon:v0.8.0
    Image ID:      docker-pullable://quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/tetragon
    Args:
      --config-dir=/etc/tetragon
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 18 May 2022 09:30:16 -0700
      Finished:     Wed, 18 May 2022 09:30:16 -0700
    Ready:          False
    Restart Count:  4
    Liveness:       exec [tetra status] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      NODE_NAME:   (v1:spec.nodeName)

tracing/exec: fix missing parents in exec events

After installing via the helm chart, we are seeing lots of missing parents on exec events. From metrics:

isovalent_errors_total{type="exec_missing_parent"} 333

A cursory exploration of the pod indicates that procRoot is indeed mounted and that Tetragon configs are pointed to the right mount point, so something else is going on here.

Filing this as a tracking issue while I investigate.

btf: kprobe btf id warnings on recent kernels

Been seeing a lot of warnings like this on 5.17:

time="2022-05-13T15:54:56Z" level=warning msg="invalid or old kprobe spec: failed to dump paramemter type by id: failed to dump btf id 1276"

I've attached the BTF file for my kernel here.
vmlinux.tar.gz

tetragon: use released clang

We currently use an older clang release with BPF backports. This costs us time we have to maintain the clang branch and is now blocking building ARM releases.

[ ] upgrade tooling to use latest released clang
[ ] fixup errors on BPF code paths generated by clang
[ ] tests to ensure doesn't break us again (kernel, nightly, ?)

The reason for the branch was originally because we carried some out of tree fixes/patches. These have been merged upstream for a long time now (1+year) but I never managed to get the code to build on recent release so did a simpler task and backported the BPF backend patches/fixes I needed.

Lets fix this all up as original reason for the backport (extra alu32 patches) is no longer needed.

config: move all cli flags to options.Config

Similar to what was done here, it would be nice to have all cli flags be backed by values in options.Config.

Slack private token in the tetragon repo

A slack private token seemingly used for build notifications to a #bots channel has been found in the tetragon repo. It's base64 encoded but should probably be encrypted or set to a variable and the credential should be rotated
https://github.com/cilium/tetragon/blob/main/vendor/github.com/go-openapi/analysis/appveyor.yml#L27-L29

bpf: replace arch-specific pt_regs assumptions with bpf co-re macros

Our eBPF code makes some arch-specific assumptions about the layout of struct pt_regs. We need to refactor this to use BPF CO-RE macros to make multi-arch support work properly for our kprobes.

alignchecker for event_config

Once #53 is merged, add alignchecker (https://github.com/cilium/tetragon/tree/main/tools/alignchecker, https://github.com/cilium/tetragon/tree/main/pkg/alignchecker) support for event_config.

	mntPath := "/tmp"
	if useMount == true {
	mntPath = mountPath

	if err := os.Mkdir(mntPath, 0755); err != nil {
	t.Logf("Mkdir failed: %s\n", err)
	t.Skip()