bpftools / linux-observability-with-bpf Goto Github PK

Code snippets from the O'Reilly book

License: Apache License 2.0

C 46.31% Shell 2.93% Python 41.22% Go 0.91% Makefile 8.64%

linux-observability-with-bpf's Introduction

Important note for readers (Jan 30th 2022)

This repository is now archived, this book was published in 2019 and written in 2018. We have been trying to keep the repository up-to-date until now but eBPF had a tremendous evolution in the past 3 years. This does not mean that reading the book is a complete waste of your time now, many concepts are always the same: like how the bpf syscall works, the instruction set and things like how tracepoints, kprobes, uprobes, xdp and traffic control works. However, at this point, just updating the examples here is not enough anymore and many areas of the book would need to be rewritten to fit the new concepts, tools, libraries and the ecosystem around eBPF. A second edition is not yet planned (will update here if it ever happens).

We want to say thank you to all the readers and the amazing people who helped updating the examples with their findings.

What should I do then?

If you are new to eBPF and don't know where to start good news is that today there is an amazing community that didn't exist in 2018!

Here you go! There are just two links you will really need.

eBPF.io website: A website containing useful documentation, blog post, conference talks and links to many resources
The eBPF and Cilium Slack: an amazing community of people around eBPF, you'll be welcome there.

What are you waiting for? Go meet your new friends!

From now on, the original text of this readme.

Linux Observability with BPF code examples

This is the companion code repo for the book Linux Observability with BPF.

We believe that even if the examples included in the book were all tested and working when we wrote them, human error is possible and technology changes. For that reason, the purpose of this repo is to keep them as updated as possible and correct mistakes we made while writing the book.

Nota Bene: All the examples in this repository are adapted from the book to assume that you use the Vagrant environment we provide. Examples can be slightly different in this repository because of that. The reason is that we didn't want to couple the book itself to Vagrant as a tool. If you don't want a Vagrant based environment, make sure you have: bcc and clang

Environment setup

Please expand the details of the environment you want to work on.
Remember that the examples have been tested on the Vagrant based environment primarily.
Feel free to open an issue or a PR if you want to help in making this better for everyone!

Fedora 30

First, we need to install some build dependencies and all the tools needed for the examples:

sudo dnf install make glibc-devel.i686 elfutils-libelf-devel wget tar clang bcc strace kernel-devel -y

Then we need grab a copy of the source code of the current kernel.

In our case the kernel runing can be verified with uname.

$ uname -r
5.0.9-301.fc30.x86_64

Given that version, please notice the URL we fetch the sources from in the following command. Change it according to your version.

cd /tmp
wget -c https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.9.tar.gz -O - | tar -xz

Now that we have the kernel source, we can move it to the /kernel-src folder.

NOTE THAT: All the examples using kernel sources in this repo assume that the kernel sources are available there. In case you don't like it, make sure you do a search and replace!

At this point we move the kernel sources and compile libbpf. Again please notice the 5.0.9 here and change accordingly.

sudo mv linux-5.0.9 /kernel-src
cd /kernel-src/tools/lib/bpf
sudo make && sudo make install prefix=/

Ubuntu 18.04

First, we need to install some build dependencies and all the tools needed for the examples:

sudo apt update
sudo apt install build-essential git make libelf-dev clang strace tar bpfcc-tools linux-headers-$(uname -r) gcc-multilib

Note on Kernel version: make sure to have a recent kernel to run the examples, a version >=5.0.0 will do the job. Most Ubuntu 18.04 providers are shipping with the kernel 4.15 that doesn't work for most of the examples. Upgrading options are left to the reader, we've been successful on aws by installing the linux-image-5.0.0-1019-aws package.

After dependencies, we need grab a copy of the kernel source code for the current release. Since this assumes that you are running an updated Ubuntu 18.04 we can get it directly from the repo they provide.

cd /tmp
git clone --depth 1 git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git

Now that we have the kernel source, we can move it to the /kernel-src folder.

NOTE THAT: All the examples using kernel sources in this repo assume that the kernel sources are available at /kernel-src. In case you don't like it, make sure you do a search and replace!

At this point we move the kernel sources and compile libbpf.

sudo mv ubuntu-bionic /kernel-src
cd /kernel-src/tools/lib/bpf
sudo make && sudo make install prefix=/usr/local

Ubuntu doesn't have the library path that the makefile expects so we need to move our libraries to its library path now.

sudo mv /usr/local/lib64/libbpf.* /lib/x86_64-linux-gnu/

Vagrant (recommended)

We provide reproducible environment in the form of a Vagrantfile that installs all the needed to make the examples work.

The environment is based on Fedora 30.

Install Vagrant

To install Vagrant, follow the official guide here.

Once you have Vagrant installed, you will need to clone this repository and issue a vagrant up.

git clone https://github.com/bpftools/linux-observability-with-bpf.git
cd linux-observability-with-bpf
vagrant up

This Vagrant command, will start a Fedora 30 VM in Virtualbox, you can SSH into the machine using:

vagrant ssh

Before going on, make sure you download the kernel source tree in this repository. It is needed as a dependency for some examples. We will be downloading the code for Kernel 5.0.9 - We are avoiding a git clone here because the Git history of the kernel is very big.

In the machine:

cd /tmp
wget -c https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.9.tar.gz -O - | tar -xz
sudo mv linux-5.0.9 /kernel-src

At this point, we need to compile the libbpf library:

cd /kernel-src/tools/lib/bpf
make && sudo make install prefix=/

Before going to the examples, it will be useful to have a copy of this repo in your environment.

git clone https://github.com/bpftools/linux-observability-with-bpf.git ~/linux-observability-with-bpf

IMPORTANT NOTE: The examples assume that you clone the repo in your home folder ~/linux-observability-with-bpf, if you didn't do please remember to change your commands!

Yay, at this point you have everything and can follow the following code examples.

Code examples

Click on each example to follow the setup instructions.

Chapter 2

Hello World

Chapter 3

The BPF Virtual Filesystem

Chapter 4

Probes

Kernel Probes

User-Space Probes

Tracepoints

Tracepoints

User Statically Defined Tracepoints (USDT)

User Statically Defined Tracepoints

Visualizing Tracing Data

Chapter 6 - Linux Networking and BPF

Chapter 7 - eXpress Data Path (XDP)

Chapter 8 - Linux Kernel security, Capabilities and Seccomp

Seccomp BPF filter example

linux-observability-with-bpf's People

Contributors

Stargazers

Watchers

Forkers

krisnova yeze muscowite ardeshir lbqds wraithking101 wawava edsiper triplekill falcondb torque59 madorn 0xmilkmix brdly-smith palnabarun abhisheknsit dctape westonsteimel davaddi sysnap2012 zhiyuan0x ryugwang lambdai sirloon rtkaratekid banban9999 xiongzx li--paul zengkefu maxgio92 eliseutorres xueweiz devzero2000 mdnix sayboras zhxqgithub hien boat0 gosundy borgerli baiwfg2 alexb138 5l1v3r1 koh1 changyuwang make-bin lic17 startupit69 rachelsqh gongshijun anusorn sahilchug vmware-ysung yzx326 chenshaochun qcalgos afrouzmashayekhi nuc630 fanfanbj failfish wubo0067 ljrcore kanjie128 gaecom gridl yundag krybkin jac0bwang sthagen desmax74 chaturvedisulabh lethe-cc junho-falinux wanggancheng juneburning ovn-open-virtual-networks ztaoing chendotjs tb-bingel alicek106 su225 zombydogs annaic krunerge talamoig linux-kern descent meilihao ottq taskset qyzhaoxun irozzo-1a devillove084 fedepaol magnologan yueweijin wujunsoftsys lifupan udayskprojects aqqvxdx

linux-observability-with-bpf's Issues

chapter2 hello world

I'm new to BPF and kernel thing. I met this:

/home/linrl3/linux-observability-with-bpf/code/chapter-2/hello_world# make bpfload
clang-8 -O2 -target bpf -c bpf_program.c -I/home/linrl3/linux-image-bm/tools/testing/selftests/bpf -o bpf_program.o
clang-8 -o monitor-exec -lelf -I/home/linrl3/linux-image-bm/samples/bpf -I/home/linrl3/linux-image-bm/tools/lib -I/home/linrl3/linux-image-bm/tools/perf -I/home/linrl3/linux-image-bm/tools/include -L/usr/local/lib64 -lbpf \
        /home/linrl3/linux-image-bm/samples/bpf/bpf_load.c loader.c
/usr/bin/ld: cannot find -lbpf
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:28: recipe for target 'bpfload' failed
make: *** [bpfload] Error 1

And my linux is debian:

uname -r
4.14.81.bm.7-amd64

Does anyone know how to solve that?

bpf_load.h no longer exists in Linux as of 5.11

Hello,

Kernel 5.11 marks the removal of bpf_load.h which your code examples depend on:

https://elixir.bootlin.com/linux/v5.11/source/samples/bpf/bpf_load.h

【git fail】git clone --depth 1 git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git

git clone --depth 1 git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git

connection timed out

chapter4 kprobe, "error: incomplete definition of type 'struct pt_regs'"

Hello,

I'm playing with the examples on Ubuntu 18.04 on their 5.0.0-37-generic kernel. Yet, running chapter4/kprobes/example.py, I get the following errors:

/virtual/main.c:5:62: error: incomplete definition of type 'struct pt_regs'
int do_sys_execve(struct pt_regs *ctx) { char* filename = ctx->di; char** argv = ctx->si; char** envp = ctx->dx;
                                                          ~~~^
/virtual/include/bcc/helpers.h:537:8: note: forward declaration of 'struct pt_regs'
struct pt_regs;
       ^
/virtual/main.c:5:85: error: incomplete definition of type 'struct pt_regs'
int do_sys_execve(struct pt_regs *ctx) { char* filename = ctx->di; char** argv = ctx->si; char** envp = ctx->dx;
                                                                                 ~~~^
/virtual/include/bcc/helpers.h:537:8: note: forward declaration of 'struct pt_regs'
struct pt_regs;
       ^
/virtual/main.c:5:108: error: incomplete definition of type 'struct pt_regs'
int do_sys_execve(struct pt_regs *ctx) { char* filename = ctx->di; char** argv = ctx->si; char** envp = ctx->dx;
                                                                                                        ~~~^
/virtual/include/bcc/helpers.h:537:8: note: forward declaration of 'struct pt_regs'
struct pt_regs;
       ^
3 errors generated.
Traceback (most recent call last):
  File "example.py", line 14, in <module>
    bpf = BPF(text = bpf_source)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 301, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

Any idea what might be going wrong? Also, running uprobes/example.py, nothing shows up on the output so maybe this is due to my host.

Best regards

note: I had to replace void by char* char**, otherwise, this error came:

/virtual/main.c:4:18: error: 'void' must be the first and only parameter if specified
int do_sys_execve(struct pt_regs *ctx, void filename, void argv, void envp) {

Error when i ran Hello World Program!!

as per instructions i executed

make  bpfload

and i got error

clang -O2 -target bpf -c bpf_program.c -I/kernel-src/tools/testing/selftests/bpf -o bpf_program.o
clang -o monitor-exec -lelf -I/kernel-src/samples/bpf -I/kernel-src/tools/lib -I/kernel-src/tools/perf -I/kernel-src/tools/include -L/usr/local/lib64 -lbpf \
        /kernel-src/samples/bpf/bpf_load.c loader.c
/usr/bin/ld: /tmp/bpf_load-c3bfb7.o: in function `sys_perf_event_open':
bpf_load.c:(.text+0x1fde): undefined reference to `test_attr__enabled'
/usr/bin/ld: bpf_load.c:(.text+0x2015): undefined reference to `test_attr__open'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:28: bpfload] Error 1

and uname -r output is 5.4.0-39-generic
on my ubuntu 20.04

any help? why the error?

Got an error when running chapter 2 hello world program

Hi,

I got an error when running the hello world program in chapter 2. Can anyone help it?

bizconf@bizconf:~/work/bpf/hello$ make bpfload
clang -O2 -target bpf -c hello.c -I/usr/src/linux-source-5.3.0/tools/testing/selftests/bpf -o hello.o
In file included from hello.c:1:
In file included from /usr/include/linux/bpf.h:11:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
#include <asm/types.h>
         ^~~~~~~~~~~~~
1 error generated.
Makefile:25: recipe for target 'build' failed
make: *** [build] Error 1

My system is ubuntu 18.04 virtual machine.

bizconf@bizconf:~/work/bpf/hello$ uname -a
Linux bizconf 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Fedora 30, 31 Compile Errors

Tested against F30 and F31. F31 was a brand new install.

uname -r
5.4.17-200.fc31.x86_64

rpm -qa bcc
bcc-0.12.0-1.fc31.x86_64

rpm -qa clang
clang-9.0.0-1.fc31.x86_64

All steps followed for downloading, making, and moving /kernel-src, and making BPF library
Git cloned into home directory,
cd /home/$USER/linux-observability-with-bpf/code/chapter-2/hello_world
make bpfload
Result:

An initial web search shows some activity in 2014 around the perf.h header, though it doesn't seem to be resolved in the Fedora tree.

Chapter-4/usdt: TypeError: init() got an unexpected keyword argument 'usdt'

When I runing the example code example.py in code/chapter-4/usdt, there is a error as "TypeError: init() got an unexpected keyword argument 'usdt'", I can't solve it after searching google. Can anyone help me?

The least kernel version under which the examples are able to run ?

Now I'm experimenting on company's machine, whose kernel version is relatively low, 3.10.

I couldn't find samples/bpf directory under 3.10 source tree. So I want to know the least kernel version, thanks.

bpf_load.h missing?

Hi, bpf_load.h isn't in the repository, please clarify, thanks

perf_events example compiling is failed as its syntax is wrong

I try to run perf_events in the vagrant and find the error below.

[vagrant@bpfbook perf_events]$ sudo python3 ./bcc_example.py
/virtual/main.c:6:18: error: 'void' must be the first and only parameter if
      specified
int do_sys_execve(struct pt_regs *ctx, void filename, void argv, void envp) {
                 ^
/virtual/main.c:6:18: error: 'void' must be the first and only parameter if
      specified
/virtual/main.c:6:18: error: 'void' must be the first and only parameter if
      specified
3 errors generated.
Traceback (most recent call last):
  File "./bcc_example.py", line 17, in <module>
    bpf = BPF(text = bpf_source)
  File "/usr/lib/python3.7/site-packages/bcc/__init__.py", line 347, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

Is this perhaps I use the different version of bcc?

[vagrant@bpfbook kprobes]$ pip3.7 show bcc
Name: bcc
Version: 0.12.0
Summary: BPF Loader Library
Home-page: https://github.com/iovisor/bcc
Author: Brenden Blanco
Author-email: [email protected]
License: UNKNOWN
Location: /usr/lib/python3.7/site-packages
Requires:
Required-by:

Chapter 7 bcc example on Ubuntu: warning 'KBUILD_MODNAME' macro redefined

After modifying the loader.py file for my device and invoking python3, this warning is displayed. Is this a problem with my setup? Is it something I can "fix"?

root@xxxxxx:~/eBPF/linux-observability-with-bpf/code/chapter-7/bcc# python3 loader.py
/virtual/main.c:1:9: warning: 'KBUILD_MODNAME' macro redefined [-Wmacro-redefined]
#define KBUILD_MODNAME "program"
        ^
<command line>:6:9: note: previous definition is here
#define KBUILD_MODNAME "bcc"
        ^
1 warning generated.
Printing packet counts per IP protocol-number, hit CTRL+C to stop
1: 1 pkt/s
1: 1 pkt/s
1: 1 pkt/s
1: 1 pkt/s
1: 1 pkt/s
1: 1 pkt/s
166: 2 pkt/s
1: 1 pkt/s
166: 0 pkt/s
1: 1 pkt/s
166: 0 pkt/s
1: 1 pkt/s
166: 0 pkt/s
1: 0 pkt/s
166: 0 pkt/s
1: 0 pkt/s
166: 0 pkt/s
^CRemoving filter from device

System is:

Ubuntu 18.04 LTS
5.3.0-26-generic kernel
Python 3.6.9
Following bcc-related pkgs installed:
- bcc-tools/stable,now 0.10.0-1 all [installed]
- libbcc/stable,now 0.10.0-1 all [installed,automatic]
- libbcc-examples/stable,now 0.10.0-1 amd64 [installed]
- libcc1-0/bionic-updates,now 8.4.0-1ubuntu1~18.04 amd64 [installed,automatic]
- python-bcc/stable,now 0.10.0-1 all [installed,automatic]
- python3-bcc/stable,now 0.10.0-1 all [installed]

However, I also built bcc from sources per bcc's INSTALL.md

And, just FYI, there are two references to 'trough' instead of 'through' in the README.md file for this example.

undefined reference to `bpf'

#include <linux/bpf.h>
union bpf_attr my_map = {
.map_type = BPF_MAP_TYPE_HASH,
.key_size = sizeof(int),
.value_size = sizeof(int),
.max_entries = 100,
.map_flags = BPF_F_NO_PREALLOC
};
int main(){
int fd = bpf(BPF_MAP_CREATE,&my_map,sizeof(my_map));
}

clang test.c -lbpf
undefined reference to bpf
ch3 example , and i'am using ubuntu18.04 5.4.0-55-generic
thanks

Make sure all the examples work in the Vagrant environment and have a description

Work continuation from #6

Items with a tick have already been tested in the Vagrant environment and have a detailed description in this repo and in the book.

Chapter 3 how to compile

Is there instruction on how to compile map_create.c ?
I tried copying Makefile from hello_world , initially it failed on "bpf_load.h not found" error

I then modified it to include additional lib it compiled but failed on running

below is the make output

make
clang -O2 -target bpf -c map_create.c -I/kernel-src/tools/testing/selftests/bpf -I/kernel-src/tools/lib/bpf -I/kernel-src/samples/bpf -I/kernel-src/tools/lib -o map_create.o
clang -o map_manage -lelf -I/kernel-src/samples/bpf -I/kernel-src/tools/lib -I/kernel-src/tools/perf -I/kernel-src/tools/include -L/usr/local/lib64 -lbpf
/kernel-src/samples/bpf/bpf_load.c loader.c

[root@Milindrh8 maps]# ls
loader.c Makefile map_create.c map_create.o map_manage

[root@Milindrh8 maps]# ./map_manage
invalid relo for insn[6].code 0x85 <<< error
The kernel didn't load the BPF program

My OS details
uname -a
Linux Milindrh8 4.18.0-305.3.1.el8_4.x86_64 #1 SMP Mon May 17 10:08:25 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

hello-world compiles and run ok on same system

vagrant failing at downloading VirtualBox image for fedora/30-cloud-base

Running vagrant up fails since the image for the specific box version (30.20190425.0) cannot be downoaded:

==> bpfbook: Box 'fedora/30-cloud-base' could not be found. Attempting to find and install...
    bpfbook: Box Provider: virtualbox
    bpfbook: Box Version: 30.20190425.0
==> bpfbook: Loading metadata for box 'fedora/30-cloud-base'
    bpfbook: URL: https://vagrantcloud.com/fedora/30-cloud-base
==> bpfbook: Adding box 'fedora/30-cloud-base' (v30.20190425.0) for provider: virtualbox
    bpfbook: Downloading: https://vagrantcloud.com/fedora/boxes/30-cloud-base/versions/30.20190425.0/providers/virtualbox.box
    bpfbook: Download redirected to host: download.fedoraproject.org
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

The requested URL returned error: 404 Not Found

Chapter 2 "Hello World" Not compiling

Spun up a Vagrant instance today with the provided Vagrantfile. Immediately cloned the repo and ran make and had an error. I'm not a Makefile whiz so maybe someone here could find the issue faster than myself.

[vagrant@bpfbook ~]$ git clone https://github.com/bpftools/linux-observability-with-bpf.git ~/linux-observability-with-bpf
Cloning into '/home/vagrant/linux-observability-with-bpf'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 284 (delta 18), reused 27 (delta 9), pack-reused 226
Receiving objects: 100% (284/284), 183.59 KiB | 657.00 KiB/s, done.
Resolving deltas: 100% (97/97), done.

[vagrant@bpfbook ~]$ cd ~/linux-observability-with-bpf/code/chapter-2/hello_world

[vagrant@bpfbook hello_world]$ make bpfload
make: *** No rule to make target '/kernel-src/samples/bpf/bpf_load.c', needed by 'build'.  Stop.

I tried building manually, but I have to figure out where the bpf_load.h file is first and link it to the compilation.

The book is fantastic though and I'm really excited to learn more about BPF programming. I'll be working through all the examples so I can verify whether or not they're all working if that's helpful at all?

File "bpf_load.h" is not present in Ch2. listings

The source code from the book mention the presence of bpf_load.h which contains the helper boiler plate to compile and load the bpf program. This file is not currently present in the source repository for Ch 2.

Please check-in the file or provide correct updated instructions on how it can be found or elaborate on how the functionality present in that file can be implemented within another re-usable file.

My dev workstation is running the following:
DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

Kernel version : 4.15.0-62-generic
One more point for clarification - would the libbpf explicitly need to be synced/forked from this location? https://github.com/libbpf/libbpf

Chapter2 undefined reference to `read_trace_pipe'

load_bpf_file.c:11:5: warning: implicit declaration of function 'read_trace_pipe' is invalid in C99 [-Wimplicit-function-declaration] read_trace_pipe(); ^ 1 warning generated. /tmp/bpf_load-e24bf7.o: In functionsys_perf_event_open':
bpf_load.c:(.text+0x1f5e): undefined reference to test_attr__enabled' bpf_load.c:(.text+0x1f95): undefined reference to test_attr__open'
/tmp/load_bpf_file-e4f8ca.o: In function main': load_bpf_file.c:(.text+0x4e): undefined reference to read_trace_pipe'
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:28: bpfload] Error 1
`

CentOS 8
kernel: 5.9.3-1.el8.elrepo.x86_64

Need GPL License

As follow code/chapter-7/iproute2/ I got errors as below.

[root@fedora y]# vim xdp_drop_udp.c
[root@fedora y]# clang -g -c -O2 -target bpf -c xdp_drop_udp.c -o xdp_drop_udp.o
[root@fedora y]# ip link set dev enp0s7 xdp obj xdp_drop_udp.o sec mysection

Prog section 'mysection' rejected: Permission denied (13)!
 - Type:         6
 - Instructions: 6 (0 over limit)
 - License:

Verifier analysis:

0: (61) r1 = *(u32 *)(r1 +0)
1: (71) r1 = *(u8 *)(r1 +23)
invalid access to packet, off=23 size=1, R1(id=0,off=23,r=0)
R1 offset is outside of the packet
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

Error fetching program/map!
[root@fedora y]# uname -a
Linux fedora 5.15.12-200.fc35.x86_64 #1 SMP Wed Dec 29 15:03:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Where can I find helper code for the book?

While reading the book I am stuck at couple of examples and believe that I will need helper scripts mentioned in the book to continue. Is this the right place?

No such directory "/sys/kernel/debug/tracing/events/bpf" in vagrant fedora instance.

Couldn't find /sys/kernel/debug/tracing/events/bpf directory mentioned in p.12 and p.52:

[root@bpfbook vagrant]# ls /sys/kernel/debug/tracing/events/bpf
ls: cannot access '/sys/kernel/debug/tracing/events/bpf': No such file or directory

and maybe because of this, the tracepoint example(with void ctx thing fixed) in p.52 failed.

[vagrant@bpfbook tracepoints]$ sudo python3 example.py
open(/sys/kernel/debug/tracing/events/bpf/bpf_prog_load/id): No such file or directory
Traceback (most recent call last):
  File "example.py", line 14, in <module>
    bpf.attach_tracepoint(tp = "bpf:bpf_prog_load", fn_name = "trace_bpf_prog_load")
  File "/usr/lib/python3.7/site-packages/bcc/__init__.py", line 815, in attach_tracepoint
    (fn_name, tp))
Exception: Failed to attach BPF program b'trace_bpf_prog_load' to tracepoint b'bpf:bpf_prog_load'

FYI:

[root@bpfbook vagrant]# uname -a
Linux bpfbook 5.0.9-301.fc30.x86_64 #1 SMP Tue Apr 23 23:57:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@bpfbook vagrant]# find /sys/kernel/debug/tracing/events -type d | grep bpf
/sys/kernel/debug/tracing/events/syscalls/sys_enter_bpf
/sys/kernel/debug/tracing/events/syscalls/sys_exit_bpf

[root@bpfbook vagrant]# mount | grep kernel
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime,seclabel)

bpf_lpm_trie_key: excess elements in array initializer

The following used in Chapter 3 initialization produces an "excess elements in array initializer" warning:

struct bpf_lpm_trie_key route_3 = {.data = {192, 168, 1, 0}, .prefixlen = 24};

This is because data has zero elements:

/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry */
struct bpf_lpm_trie_key {
        __u32   prefixlen;      /* up to 32 for AF_INET, 128 for AF_INET6 */
        __u8    data[0];        /* Arbitrary size */
};

The warning is correct: trying to insert data into BPF map in such manner results in garbage key.

vagrant vm cannot compile hello world program because of missing bfd.h

Hello! I followed the guide to set up a vagrant fedora VM. And then I downloaded the kernel source into /kernel-src and tried to compile it, but it failed. It said bfd.h: No such file or directory, how to fix this? What other dependencies should I install?

[vagrant@bpfbook bpf]$ sudo make && sudo make install prefix=/

Auto-detecting system features:
...                        libbfd: [ OFF ]
...        disassembler-four-args: [ OFF ]

  CC       bpf_jit_disasm.o
/kernel-src/tools/bpf/bpf_jit_disasm.c:23:10: fatal error: bfd.h: No such file or directory
   23 | #include <bfd.h>
      |          ^~~~~~~
compilation terminated.
make: *** [Makefile:57: bpf_jit_disasm.o] Error 1

Cannot compile chapter2-helloworld on Ubuntu

I hope run this sample code on Ubuntu.

Environment

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic
# uname -a
Linux ubuntu18 5.0.0-1027-azure #29~18.04.1-Ubuntu SMP Mon Nov 25 21:18:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Symptons

Problem 1: Couldn't find asm/types.h.

root@ubuntu18:~/linux-observability-with-bpf/code/chapter-2/hello_world#
root@ubuntu18:~/linux-observability-with-bpf/code/chapter-2/hello_world# make
clang -O2 -target bpf -c bpf_program.c -I/kernel-src/tools/testing/selftests/bpf -o bpf_program.o
In file included from bpf_program.c:1:
In file included from /usr/include/linux/bpf.h:11:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
#include <asm/types.h>
         ^~~~~~~~~~~~~
1 error generated.
Makefile:29: recipe for target 'build' failed
make: *** [build] Error 1

I could fix problem the below way:

 ln -s /usr/include/asm-generic /usr/include/asm

Problem 2: unknown type name 'size_t'

root@ubuntu18:~/linux-observability-with-bpf/code/chapter-2/hello_world# make
clang -O2 -target bpf -c bpf_program.c -I/kernel-src/tools/testing/selftests/bpf -o bpf_program.o
clang -o monitor-exec -lelf -I/kernel-src/samples/bpf -I/kernel-src/tools/lib -I/kernel-src/tools/perf -I/kernel-src/tools/include -I/kernel-src/usr/include/linux -I/usr/lib/clang/6.0/include -L/usr/local/lib64 -lbpf \
        /kernel-src/samples/bpf/bpf_load.c loader.c
In file included from /kernel-src/samples/bpf/bpf_load.c:2:
In file included from /usr/include/stdio.h:41:
/usr/include/x86_64-linux-gnu/bits/libio.h:306:3: error: unknown type name 'size_t'
  size_t __pad5;
  ^
/usr/include/x86_64-linux-gnu/bits/libio.h:309:67: error: use of undeclared identifier 'size_t'; did you mean 'sizeof'?
  char _unused2[15 * sizeof (int) - 4 * sizeof (void *) - sizeof (size_t)];
                                                                  ^
/usr/include/x86_64-linux-gnu/bits/libio.h:309:66: error: reference to overloaded function could not be resolved; did you mean to call it?
  char _unused2[15 * sizeof (int) - 4 * sizeof (void *) - sizeof (size_t)];
                                                                 ^~~~~~~~
/usr/include/x86_64-linux-gnu/bits/libio.h:337:62: error: unknown type name 'size_t'
typedef __ssize_t __io_read_fn (void *__cookie, char *__buf, size_t __nbytes);

Finaly

I gave up make on Ubuntu and sucess at Fedora 31..

Couldn't pass the parameters of system call to kprobe in ebpf program

Hi,

Thanks for writing such a nice book to help us in understanding the ebpf fundamentals.

I am new to eBPF world. I am running ubuntu18.04 with kernel version 5.3.0-42-generic.

I want to observe the parameters passed to system call (sendto/recvmsg). Can you please let me know the way to get the values passed to these params?

I took the program explained in chapter-4 under kprobe section. But it is throwing following error.

/virtual/main.c:2:18: error: 'void' must be the first and only parameter if specified
int do_sys_execve(struct pt_regs *ctx, void filename, void argv, void envp) {
^
/virtual/main.c:2:18: error: 'void' must be the first and only parameter if specified
/virtual/main.c:2:18: error: 'void' must be the first and only parameter if specified
3 errors generated.
Traceback (most recent call last):
File "test.py", line 9, in
bpf = BPF(text = bpf_source)
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 343, in init
raise Exception("Failed to compile BPF module %s" % (src_file or ""))
Exception: Failed to compile BPF module

chapter 6: packet filter returns 0 does not drop the packet

I am running the example code in vagrant, after loading the code and ping localhost, I can see the output from bpf code but the packet is not dropped. Based on doc, when the filter returns 0 the packet should be dropped for SO_ATTACH_FILTER

Attach a classic BPF (SO_ATTACH_FILTER) or an extended BPF
(SO_ATTACH_BPF) program to the socket for use as a filter of
incoming packets. A packet will be dropped if the filter pro‐
gram returns zero. If the filter program returns a nonzero
value which is less than the packet's data length, the packet
will be truncated to the length returned.

[vagrant@bpfbook packet-filtering-raw-sockets]$ sudo ./loader-bin bpf_program.o
TCP 0 UDP 0 ICMP 0 packets
TCP 0 UDP 0 ICMP 0 packets
TCP 0 UDP 0 ICMP 0 packets
TCP 0 UDP 0 ICMP 4 packets
TCP 0 UDP 0 ICMP 8 packets

ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.046 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.071 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.073 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.070 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.071 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.072 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.064 ms
^C
--- 127.0.0.1 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 53ms
rtt min/avg/max/mdev = 0.046/0.066/0.073/0.013 ms

I am seeing the same behavior that packets are not dropped when I am running examples from bcc (on my local machine w/ 5.0 kernel) e.g. https://github.com/iovisor/bcc/tree/master/examples/networking/http_filter I am not sure if missed some obvious things due to my lack of knowledge on network and bpf ....

Is this all the code?

Hello.

Per book text there should be examples starting from Chapter 1, including code for bpf_load.c and bpf_load.h. All of this seems missing.

need to build BPF programs for real-world target

Its great you have a vagrantfile that spins up a VM that runs all your example code just perfectly. I'm guessing many people who buy your book (like me!) and run your examples want to use what they learn on a real linux host with a recent linux kernel. Do you provide any assist with using your sample code directly on an installed linux distro with a recent 5.x kernel?

i am running debian 10.5, stock kernel replaced with v5.8.5, LLVM/clang 7.0.1-8, bcc built from iovisor github download. your simplest example doesnt build:

clang -O2 -target bpf -c bpf_program.c -I/kernel-src/tools/testing/selftests/bpf -o bpf_program.o
clang -o monitor-exec -lelf -I/kernel-src/samples/bpf -I/kernel-src/tools/lib -I/kernel-src/tools/perf -I/kernel-src/tools/include -L/usr/local/lib64 -lbpf
/kernel-src/samples/bpf/bpf_load.c loader.c
loader.c:10:3: warning: implicit declaration of function 'read_trace_pipe' is invalid in C99 [-Wimplicit-function-declaration]
read_trace_pipe();
^
1 warning generated.
/usr/bin/ld: /tmp/bpf_load-27b31c.o: in function sys_perf_event_open': bpf_load.c:(.text+0x204e): undefined reference to test_attr__enabled'
/usr/bin/ld: bpf_load.c:(.text+0x2085): undefined reference to test_attr__open' /usr/bin/ld: /tmp/loader-e37255.o: in function main':
loader.c:(.text+0x51): undefined reference to `read_trace_pipe'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:28: bpfload] Error 1

thanks for any help!!

bpf_load.h doesn't existing in Linux above 5.11?

Is there any alternative of this? Or is it recommended to copy the old one back?

Ch2 Hello World fails on compiling BPF loader, with either redefinition errors or missing stdbool.h

OS: CentOS Linux release 8.0.1905 (Core) (from /etc/redhat-release)
Kernel version: 5.4.0-1.el8.elrepo.x86_64 (output from uname -r; upgraded kernel is from the https://elrepo.org/tiki/kernel-ml repo)

Steps to reproduce:

#I have kernel-ml-devel because that's the package name for the kernel development package in the ML repo that corresponds to my running kernel version
dnf -y install clang bpftool bison flex bcc strace kernel-ml-devel

cd /tmp
wget -c https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.tar.gz -O - | tar -xz
mv linux-5.4 /kernel-src

TOOLS=/kernel-src/samples/bpf
INCLUDE=/kernel-src/tools/lib
PERF_INCLUDE=/kernel-src/tools/perf
KERNEL_TOOLS_INCLUDE=/kernel-src/tools/include
clang -o loader -lelf\
  -I${INCLUDE} \
  -I${PERF_INCLUDE} \
  -I${KERNEL_TOOLS_INCLUDE} \
  -I${TOOLS} \
  ${TOOLS}/bpf_load.c \
  loader.c

Error output:

In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:48:2: error: redefinition of enumerator 'BPF_REG_0'
        BPF_REG_0 = 0,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:48:2: note: previous definition is here
        BPF_REG_0 = 0,
        ^
In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:49:2: error: redefinition of enumerator 'BPF_REG_1'
        BPF_REG_1,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:49:2: note: previous definition is here
        BPF_REG_1,
        ^
In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:50:2: error: redefinition of enumerator 'BPF_REG_2'
        BPF_REG_2,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:50:2: note: previous definition is here
        BPF_REG_2,
        ^
In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:51:2: error: redefinition of enumerator 'BPF_REG_3'
        BPF_REG_3,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:51:2: note: previous definition is here
        BPF_REG_3,
        ^
In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:52:2: error: redefinition of enumerator 'BPF_REG_4'
        BPF_REG_4,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:52:2: note: previous definition is here
        BPF_REG_4,
        ^
In file included from loader.c:5:
In file included from /kernel-src/samples/bpf/bpf_load.h:5:
In file included from /kernel-src/tools/lib/bpf/bpf.h:26:
/usr/include/linux/bpf.h:53:2: error: redefinition of enumerator 'BPF_REG_5'
        BPF_REG_5,
        ^
/kernel-src/tools/include/uapi/linux/bpf.h:53:2: note: previous definition is here
        BPF_REG_5,

... It repeats for quite a bit, for every BPF register and many built-in functions.

Running clang with -nostdinc (and excluding the files in /usr/include/linux/) causes the redefinition errors to go away but I then get a different error about stdbool.h being missing:

clang -o loader -lelf\                                                                                                                                                                                                                              ⏎
  -nostdinc \
  -I${INCLUDE} \
  -I${PERF_INCLUDE} \
  -I${KERNEL_TOOLS_INCLUDE} \
  -I${BPF_PERF_INCLUDE} \
  -I${TOOLS} \
  ${TOOLS}/bpf_load.c \
  loader.c
In file included from /kernel-src/samples/bpf/bpf_load.c:2:
In file included from /kernel-src/tools/perf/include/bpf/stdio.h:3:
In file included from /kernel-src/tools/perf/include/bpf/bpf.h:5:
In file included from /kernel-src/tools/include/uapi/linux/bpf.h:11:
/kernel-src/tools/include/linux/types.h:5:10: fatal error: 'stdbool.h' file not found
#include <stdbool.h>
         ^~~~~~~~~~~
1 error generated.
In file included from loader.c:3:
In file included from /kernel-src/tools/perf/include/bpf/stdio.h:3:
In file included from /kernel-src/tools/perf/include/bpf/bpf.h:5:
In file included from /kernel-src/tools/include/uapi/linux/bpf.h:11:
/kernel-src/tools/include/linux/types.h:5:10: fatal error: 'stdbool.h' file not found
#include <stdbool.h>
         ^~~~~~~~~~~
1 error generated.

The only instance of stdbool.h I have in /kernel-src/ is in /kernel-src/arch/powerpc/boot/stdbool.h

archlinux run ./monitor-exec error

I compiled chapter-2/hello_world example on archlinux successfully,but run is error:
[mod@mod hello_world]$ ./monitor-exec
bpf_load_program() err=1
The kernel didn't load the BPF program

my compile information is:
[mod@mod hello_world]$ make
clang -O2 -target bpf -c bpf_program.c -I/home/mod/code/linux/kernel/linux-5.4.77/tools/testing/selftests/bpf -o bpf_program.o
clang -DHAVE_ATTR_TEST=0 -o monitor-exec -lelf -I/home/mod/code/linux/kernel/linux-5.4.77/samples/bpf -I/home/mod/code/linux/kernel/linux-5.4.77/tools/lib -I/home/mod/code/linux/kernel/linux-5.4.77/tools/perf -I/home/mod/code/linux/kernel/linux-5.4.77/tools/include -L/usr/lib64 -lbpf
/home/mod/code/linux/kernel/linux-5.4.77/samples/bpf/bpf_load.c loader.c

my kernel information is:
[mod@mod hello_world]$ uname -a
Linux mod 5.4.77-1-lts #1 SMP Tue, 10 Nov 2020 22:42:03 +0000 x86_64 GNU/Linux

kernel_src I use is:
https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.77.tar.xz

clang version is:
[mod@mod hello_world]$ clang --version
OHOS (34024) clang version 9.0.0 (llvm-project c20cd5feb33c9df88918ffe9a0df76499befaa46) (based on LLVM 9.0.0)
Target: x86_64-unknown-linux-gnu
Thread model: posix

I see same error on https://blog.raymond.burkholder.net/index.php?/archives/1000-eBPF-Basics.html,but
'[mod@mod hello_world]$ llvm-objdump -S -no-show-raw-insn bpf_program.o

bpf_program.o: file format ELF64-BPF

Disassembly of section tracepoint/syscalls/sys_enter_execve:

0000000000000000 bpf_prog:
0: r1 = 33
1: *(u16 *)(r10 - 8) = r1
2: r1 = 7236284523806213712 ll
4: *(u64 *)(r10 - 16) = r1
5: r1 = 4764857262830019912 ll
7: *(u64 *)(r10 - 24) = r1
8: r1 = r10
9: r1 += -24
10: r2 = 18
11: call 6
12: r0 = 0
13: exit
[mod@mod hello_world]$ '

call 6 is correct.

when I run monitor-exec with root,The program is blocked in the function load_bpf_file("bpf_program.o").I
test the solutions on other issues,but don't work,what shout I do?