ros-realtime / linux-real-time-kernel-builder Goto Github PK

Hi

I've got the RPi 4 patched via rt-kernel-docker-builder and the corresponding instruction and everything went as described, except that RPi4's CPU cores are not operating at 1000MHz. What can I print/debug in the issue in order to investigate the problem?

After executing:

ubuntu@ubuntu:~$ echo '1' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/stats/reset
ubuntu@ubuntu:~$ cpufreq-info -s -m

I've got the following output:
600 MHz:32.79%, 700 MHz:66.47%, 800 MHz:0.30%, 900 MHz:0.00%, 1000 MHz:0.00%, 1.10 GHz:0.00%, 1.20 GHz:0.00%, 1.30 GHz:0.15%, 1.40 GHz:0.00%, 1.50 GHz:0.30% (441)

After .config-fragment is merged the consistency check shall be done to make sure that the kernel is configured correctly e.g.

$ ./scripts/config -s PREEMPT_RT
y

As discussed in the RTWG meeting, the RTWG is happy to host the Raspberry Pi 4 + RT + ROS image builder (https://github.com/shuhaowu/ros-realtime-rpi4-image) that I worked on. I'm also happy to contribute it upstream under an open source license. I'm not sure what the process should be, as this probably hasn't been done before, but I have a few items:

• Figure out how to transfer the repo: I can transfer the repo to one of the organization members (who?), who can then transfer it to the ros-realtime organization.
• Licensing: Apache license.
• Future maintenance: As I mentioned in the RTWG meeting, I am happy to continue maintain the code via code reviews and changes. While I don't use it actively on a live robot, I'm using the same image builder for similar projects. My preference for the time being, if the RTWG is OK with it, is to retain at least some sort of committer access to this.
• CI/CD/Release: In my repo, I have it working with Github CI. The build takes 15 minutes without compression using the Github builder. The image without compression is about 8GB and with compression is about 1.7GB (with galactic installed, iirc). Compression takes longer than the build using Github's builder, but only about 5 minutes on my workstation (Ryzen 7 3700X). We'll need to figure out host this. Github release might be ok if we compress, as it says we can host 2GB files with no bandwidth limits.

Other items like what images we should build, what tools should we include preferably should be answered in a separate issue, perhaps in the final repo once it is created. I expected that we need to make a few changes to the builder, as I've only tested it with a single build configuration for now. I also expect that we need to change the docs a bit as the docs right now only target people who develop this project instead of the users.

cc: @LanderU @carlossvg

Hi,

I get following error during the "Build a Linux Real-Time kernel using docker" procedure:

user@5179afff0dbc:~/linux_build/linux-raspi$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg
sh ./scripts/package/mkdebian
dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch)  -b -nc -uc
dpkg-buildpackage: info: source package linux-5.4.218-rt79+
dpkg-buildpackage: info: source version 5.4.218-rt79+-1
dpkg-buildpackage: info: source distribution focal
dpkg-buildpackage: info: source changed by user 
dpkg-buildpackage: info: host architecture arm64
 dpkg-source --before-build .
dpkg-source: info: using options from linux-raspi/debian/source/options: --extend-diff-ignore=^arch/arm64/boot/dts/broadcom/bcm283x-rpi-csi1-2lane.dtsi$ --extend-diff-ignore=^arch/arm64/boot/dts/broadcom/bcm283x-rpi-lan7515.dtsi$ --extend-diff-ignore=^arch/arm64/boot/dts/overlays$ --diff-ignore --tar-ignore
 debian/rules build
make KERNELRELEASE=5.4.218-rt79+ ARCH=arm64 	KBUILD_BUILD_VERSION=1 -f ./Makefile
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  CHK     include/generated/compile.h
  CC      lttng/probes/lttng-probe-random.o
lttng/probes/lttng-probe-random.c:18:10: fatal error: trace/events/random.h: No such file or directory
   18 | #include <trace/events/random.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[5]: *** [scripts/Makefile.build:270: lttng/probes/lttng-probe-random.o] Error 1
make[4]: *** [scripts/Makefile.build:519: lttng/probes] Error 2
make[3]: *** [Makefile:1768: lttng] Error 2
make[2]: *** [debian/rules:6: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make[1]: *** [scripts/Makefile.package:83: bindeb-pkg] Error 2
make: *** [Makefile:1503: bindeb-pkg] Error 2

Hi,

I've been trying various combinations of kernels and patches without any luck. I've manually run the getpatch and tried to find the suggested combinations; the docker image builds correctly but the make process inevitable dies with a message like:

make[3]: *** [Makefile:1722: fs] Error 2
make[2]: *** [debian/rules:6: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make[1]: *** [scripts/Makefile.package:77: deb-pkg] Error 2
make: *** [Makefile:1457: deb-pkg] Error 2

and the upstream errors might include:

fs/proc/kmsg.c: In function ‘kmsg_poll’:
fs/proc/kmsg.c:43:18: error: implicit declaration of function ‘printk_wait_queue’; did you mean ‘bit_waitqueue’? [-Werror=implicit-function-declaration]
   43 |  poll_wait(file, printk_wait_queue(), wait);
      |                  ^~~~~~~~~~~~~~~~~
      |                  bit_waitqueue
fs/proc/kmsg.c:43:18: warning: passing argument 2 of ‘poll_wait’ makes pointer from integer without a cast [-Wint-conversion]
   43 |  poll_wait(file, printk_wait_queue(), wait);
      |                  ^~~~~~~~~~~~~~~~~~~
      |                  |
      |                  int
In file included from fs/proc/kmsg.c:13:
./include/linux/poll.h:48:70: note: expected ‘wait_queue_head_t *’ {aka ‘struct wait_queue_head *’} but argument is of type ‘int’
   48 | static inline void poll_wait(struct file * filp, wait_queue_head_t * wait_address, poll_table *p)

I'd have a lot more faith in trying to understand understand these errors if the example from the README worked:

<raspi release> is in a form of 5.4.0-1034-raspi, see Ubuntu raspi Linux kernels
<RT patch> is in a form of 5.4.106-rt54, see RT patches

But using these values generates:

docker build --build-arg UNAME_R=5.4.0-1034-raspi --build-arg RT_PATCH=5.4.106-rt54 -t rtwg-image .
...
#8 6.587 E: Unable to find a source package for linux-image-5.4.0-1034-raspi

Is it possible to have examples in the README that can find the source packages AND successfully compile and patch a kernel? Thanks, apologies if I'm missing something important.

psrecord is a really useful tool to measure cpu and memory usage. right now its not included by default in the build but I'd really like it to be since I am using it over on the autoware_reference_system 😄

if there is pushback, not a big deal we can just install it later with python3 -m pip install psrecord

The files mentioned in the README do not exist on a fresh installation of ubuntu on RPi4

$ sudo ln -s -f /boot/vmlinuz-5.4.101-rt53 /boot/vmlinuz
$ sudo ln -s -f /boot/vmlinuz-5.4.0-1034-raspi /boot/vmlinuz.old
$ sudo ln -s -f /boot/initrd.img-5.4.101-rt53 /boot/initrd.img
$ sudo ln -s -f /boot/initrd.img-5.4.0-1034-raspi /boot/initrd.img.old

What I see is

ubuntu@ubuntu:~$ ls /boot
config-5.11.0-1007-raspi  dtb                    dtb-5.4.140-rt64  grub                          initrd.img-5.4.140-rt64       System.map-5.4.140-rt64    vmlinuz-5.4.140-rt64
config-5.11.0-1019-raspi  dtb-5.11.0-1007-raspi  dtbs              initrd.img-5.11.0-1007-raspi  System.map-5.11.0-1007-raspi  vmlinuz-5.11.0-1007-raspi
config-5.4.140-rt64       dtb-5.11.0-1019-raspi  firmware          initrd.img-5.11.0-1019-raspi  System.map-5.11.0-1019-raspi  vmlinuz-5.11.0-1019-raspi

LTTNG version is set to 2.12 and not configurable in Dockerfile. Besides that, it is not possible to set it via GitHub actions

Which Ubuntu 20.04 image for RaspberryPi 4B should I install as pre-requisite for the rt_preempt kernel patch?

With Raspberry Pi Imager v.1.6.2 I could choose Ubuntu Server 20.04.3 LTS(RPi 2/3/4/400) 64-bit server OS with long-term support for arm64 architectures, (2021-08-26) but not the Ubuntu 20.04 Desktop version.

The following link in the documentation on https://github.com/ros-realtime/rt-kernel-docker-builder Download and install Ubuntu 20.04 image

https://ubuntu.com/download/raspberry-pi/thank-you?version=20.04&architecture=arm64+raspi
does not work ("File not found"):
https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04-preinstalled-arm64+raspi.img.xz

Docker build fails to clone kernel with a message:

error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function

while processing tags here https://github.com/ros-realtime/linux-real-time-kernel-builder/blob/master/Dockerfile#L94 it could happen that more than one result will be returned and the wrong tag will be picked, it shall be sorted by version first.

Just in case, if it can be of use to someone, in order to avoid the manual step "adjust vmlinuz and initrd.img" I used the build command:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LOCALVERSION=-raspi KDEB_PKGVERSION=5.4.99 -j nproc deb-pkg

After that the initial command "sudo dpkg -i *.deb" do automatically the complete installation.

Thanks to share, thanks for your work,
BR

Hello,

I get the following error when trying to build the kernel. Is it normal?

Container builded with below command.

docker build --no-cache --build-arg UBUNTU_VERSION=focal --build-arg KERNEL_VERSION=5.4.0 --build-arg UNAME_R=5.4.0-1079-raspi --build-arg RT_PATCH=5.4.221-rt79 --build-arg LTTNG_VERSION=2.12 -t rtwg-image .

This solves several issues related to kernel versions mismatch:

• #9

This work is being done here: https://github.com/razr/linux-real-time-kernel-builder/commits/get-from-git

I followed the guide and this is what I saw:
docker build fail if you specify the kernel version and the RT_PATCH version. if you run the build command without this two parameters it build the image.
then I run the docker image and tried to run the make command and I got the following error:

user@user-VirtualBox:/Documents/rt-kernel-docker-builder$ docker run -t -i rtwg-image bash
user@5a3e5f043b21:/linux_build$ ls -la
total 178432
drwxr-xr-x 1 user user 4096 Aug 4 16:04 .
drwxr-xr-x 1 user user 4096 Aug 4 16:04 ..
-rw-rw-r-- 1 root root 85 Aug 4 15:40 .config-fragment
drwxr-xr-x 1 user user 4096 Aug 4 16:05 linux-raspi-5.4.0
-rw-r--r-- 1 user user 11583830 Jul 20 17:01 linux-raspi_5.4.0-1041.45.diff.gz
-rw-r--r-- 1 user user 3546 Jul 20 17:01 linux-raspi_5.4.0-1041.45.dsc
-rw-r--r-- 1 user user 170244619 Apr 3 2020 linux-raspi_5.4.0.orig.tar.gz
-rw-r--r-- 1 user user 859109 Jul 9 18:57 patch-5.4.129-rt61.patch
user@5a3e5f043b21:~/linux_build$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j nproc deb-pkg
make: *** No rule to make target 'deb-pkg'. Stop.

any suggestion how to solve it?

Hey!

I'm checking out this work, and looks like something is not well configured or we're missing some sources. When I try to build the docker image with the instructions it ends into an error fetching the build deps for the linux image. See the error attached:

E: Unable to find a source package for linux-image-5.4.0-1022-raspi
The command '/bin/sh -c apt-get update && apt-get build-dep -q -y linux linux-image-${UNAME_R}     && apt-get install -q -y     libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf     fakeroot     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Is failling on this line: https://github.com/ros-realtime/rt-kernel-docker-builder/blob/master/.gitpod.Dockerfile#L49

And also will file at: https://github.com/ros-realtime/rt-kernel-docker-builder/blob/master/.gitpod.Dockerfile#L62

The artifact name is always the same: RPI4 RT Kernel deb packages. Depending on the parameters we could change the name dynamically to make it more meaningful, e.g. 5.15.35-rt41-raspi-kernel to distinguish the builds

    - run: | 
        echo "artifactName=`ls /home/user/linux_build/linux-image-* | cut -d '-' -f 3-4`-kernel" >> $GITHUB_ENV
      - uses: actions/upload-artifact@v2
        with:
          name: '${{ env.artifactName }}
          path: ~/linux_build/*.deb

Hello,
I'm starting doing my little project using ROS2 (humble) to control motor, servo motors, etc, on a Raspberry 3B+.

I have immediately noticed that using a general kernel, I have jitter on the controller of my micro servo motor. So, I wonder if it is possible reduce/remove this jitter, which is pretty annoying, using a RT kernel.

In case it is physically impossible, is there any way to have a more stable scheduler to have a more consistent PWM signal ?

not sure if its just a problem for my setup but in order to get the symlink commands to work properly I had to use the full paths to the files rather than the commands shown in the README.

# was this
$ sudo ln -s -f vmlinuz-5.4.101-rt53 vmlinuz
# should be this
$ sudo ln -s -f /boot/vmlinuz-5.4.101-rt53 /boot/vmlinuz

ill make a PR and we can review it there

Any plans to support one of the newer rpi5 linux versions?

Ubuntu 22.04 is out, but it seems like it has a PREEMPT_RT kernel already precompiled? See https://ubuntu.com/engage/an-introduction-to-real-time-linux-part-i

cc: @carlossvg @LanderU @razr

@LanderU wants to have one Dockerfile for both architectures

so I recently ran into this problem while trying to setup ros2_tracing to track system performance. essentially I was unable to sudo apt install lttng-modules-dkms as it said the RT kernel was not officially supported.

I did some digging online and came across this thread that looks to be exactly the same issue.

It goes on to suggest a solution of compiling lttng from source....which ideally i would like to avoid.

As lttng is a pretty universal tracer tool I was wondering if an alternative proposal could happen where we add it to the Dockerfile here so that we can compile it right into the kernel for use later on.

I found this README for lttng-modules that looks like it has instructions to add it directly to a kernel build.

lttng-modules-README.md

### Kernel built-in support

It is also possible to build these modules as part of a kernel image. Simply
run the [`scripts/built-in.sh`](scripts/built-in.sh) script with the path to
your kernel source directory as an argument.  It will symlink the
lttng-modules directory in the kernel sources and add an include in the kernel
Makefile.

Then configure your kernel as usual and enable the `CONFIG_LTTNG` option.


### Required kernel config options

Make sure your target kernel has the following config options enabled:

  - `CONFIG_MODULES`: loadable module support (not strictly required
     when built into the kernel),
  - `CONFIG_KALLSYMS`: see files in [`wrapper`](wrapper); this is
     necessary until the few required missing symbols are exported to GPL
     modules from mainline,
  - `CONFIG_HIGH_RES_TIMERS`: needed for LTTng 2.x clock source,
  - `CONFIG_TRACEPOINTS`: kernel tracepoint instrumentation
     (enabled as a side-effect of any of the perf/ftrace/blktrace
     instrumentation features).
  - `CONFIG_KPROBES` (5.7+): use kallsyms for kernel 5.7 and newer.

• Remove master
• Create rolling and foxy branches