ros-realtime / roscon-2023-realtime-workshop Goto Github PK

default values for 3-1, 3-2, 4-2:
execution time of DataLogger: 6ms
execution time of ObjectDetector: 3ms

4-1: SingleThreadedExecutor: timer

• variable execution time of DataLogger [3ms, X ms], x choose correctly (12,20) period 16ms, object 3

They should detect if another environment is sourced and if so, prompt the user to start a new terminal.
If not, source the current environment and launch the application

• Create a detailed list of objectives for the audience.
• Come up with technical requirements to accomplish these objectives
• Come up with a description/story board of how the session should look like

how to run the exercises: step by step instructions. Inlcuding what they have to change.

• What model of access point can I buy that can support ~150 devices (assume 25 RPI4 + 100 laptops connected at the same time)
• See if we can have anything host some code/files so we can download them to laptops during the workshop without internet access

• Create a detailed list of objectives for the audience.
• Come up with technical requirements to accomplish these objectives
• Come up with a description/story board of how the session should look like

• - The wording "Replace the double with a std::atomic<double>, and remove the locks": needs to point user to the file path.
• - File paths should be absolute.
• - static_assert(std::atomic<double>::is_always_lock_free); is already there in single_data.h in exercise 2-1. Consider in the text, putting the sentence closer together. Consider telling user to add the static assert line immediately above the atomic.
• - Consider making the "remove the lock" sentence more visible.
• - After exercise 2-1 is complete, tell the user what to expect.
• - Consider telling user to load their own exercise2-1.perfetto file.
• - Don't use stress-ng and use the single stress.sh file.
• - What do I do at the end of exercise 2-1? I have the file and what am I suppposed to look for?
• - After seeing the results of 2-1, consider telling people to switch to nanosecond view if they see a single column at 1us.
• - run-exercise2-2.sh command needs to be ./
• - Consider putting latency view in focus over timeline view
• - In exercise 2-2: tell users (stress in bold) to not remove the lock after replace with atomic, and simply compile first to see if the static_assert succeeds.
• - "This tends to be the case for larger amounts of data." -> This is almost always the case for data greater than 64/128bits depending on CPU architectures
• - "A priority inheritance mutex implementation is available as a cactus_rt::mutex. Change the std::mutex to a cactus_rt::mutex." -> Consider saying remove the atomic and static assert before this step.
• - " periodically pops all data from the data queue at 100 Hz". Consider referring to PopData()
• - "unction is meant to simulate the ROS thread acquiring the lock and needing to do some work. " -> maybe something about simulating contention, as it's not that realistic to do processing while holding a lock
• - Exercise 2-3 which slice do I visualize
• - When letting people know which slice to visualize, mention the thread name too, and possibly als the time scale.
• - Exercise 2-3 baseline: what should I expect as the audience?
• Repeat yourself with the path in 2-3
• 2-3: queue -> moodycamel::ReaderWriterQueue instead of omitting the type

application setup:

system.cpp:

• publisher_node
• actuation_node

sensor.cpp:

• one node with

  • object detector subscription
  • data logger subscription

• main.cpp
  MultiThreadedExecutor exe;
  exe.add_node(publisher_node)
  exe.add_node(actuation_node)
  exe.add_node(sensor)

demonstrate limitations of vanilla ROS Executor

option A:

• use SingleThreadedExecutor

option B:

• use MultiThreadedExecutor

demonstration with tracer:

pub1     p        p
pub2     p        p

          |        |
sub1      xx       xxxxxx     

          |        |
sub2      ..ooo    ......ooo

delay:    short     long

simplification:

• not fully loaded system, short execution time of publisher =>
  timestamp assigned in publisher
  (almost equal to) =
  timestamp when message is received by subscriptions
• user experience: sometimes sub2 callback gets delayed (by callback sub1)

visualization of End-To-End Latency:

• measure end-to-end latencies (publisher -> object detector -> actuation)
• user expericence: large variation

real-time-executor

sub2 is real-time subscription

visualization:

pub1     p        p
pub2     p        p

          |        |
sub1      ...xx    ...xxxxxx     

          |        |
sub2      ooo      ooo

delay:    short     short

End-To-End-Latency

• user experience: always short latency (small variation)

when building exercise 3-2, the following packages were missing (dependency of rclcpp)

• test_msgs (repository: https://github.com/ros2/rcl_interfaces/tree/humble)
• test_interface_files repo: https://github.com/ros2/test_interface_files/tree/humble/
• ament_cmake_google_benchmark, repo: https://github.com/ament/ament_cmake.git
• https://github.com/ros2/performance_test_fixture.git
• here I could not find the benchmark package:

 --- stderr: performance_test_fixture                                           
CMake Error at CMakeLists.txt:20 (find_package):
  By not providing "Findbenchmark.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "benchmark", but CMake did not find one.

  Could not find a package configuration file provided by "benchmark" with
  any of the following names:

    benchmarkConfig.cmake
    benchmark-config.cmake

  Add the installation prefix of "benchmark" to CMAKE_PREFIX_PATH or set
  "benchmark_DIR" to a directory containing one of the above files.  If
  "benchmark" provides a separate development package or SDK, be sure it has
  been installed.
---
Failed   <<< performance_test_fixture [1.08s, exited with code 1]

https://github.com/ros2/performance_test_fixture/blob/humble/CMakeLists.txt

The newest docker image is uploaded to Github. There's also updated READMEs that details how to use this. Please follow the instructions and let me know if there's anything wrong with it, and if anything doesn't work for you.

Please also test the VS code setup if you use vscode (otherwise feel free to skip). Some people might be tempted to use it and it should just work.

cc: @nightduck @JanStaschulat @stephanie-eng

This may only impact us developers who have to launch it more than once, but there is a persistent issue where calling stop on the previous container also removes it but does so asynchronously. This causes conflicts when attempting to restart it, but the user is still prompted to run docker/shell (which will fail, because there's no container running).

oren@oren-mc1040:~/git/roscon-2023-realtime-workshop$ docker/start
+ docker stop roscon-2023-realtime-workshop
roscon-2023-realtime-workshop
+ docker rm roscon-2023-realtime-workshop
Error response from daemon: removal of container roscon-2023-realtime-workshop is already in progress
++ pwd
++ id -u
++ id -g
+ docker run -d --device=/dev/dri --volume=/tmp/.X11-unix:/tmp/.X11-unix --volume=/home/oren/git/roscon-2023-realtime-workshop:/code --ulimit=rtprio=98 --ulimit=memlock=-1 --cap-add=SYS_NICE --init --rm -p 3100:3100 -e DISPLAY= -e QT_X11_NO_MITSHM=1 -e HOST_UID=1000 -e HOST_GID=1000 --name=roscon-2023-realtime-workshop roscon-2023-realtime-workshop
docker: Error response from daemon: Conflict. The container name "/roscon-2023-realtime-workshop" is already in use by container "77bbd844afd149378336d2d5e3173f1ec3f066fb9aee0159993098d5e3d20e54". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
+ set +x
Started container roscon-2023-realtime-workshop. To enter the container, run the command:

  docker/shell

After installing docker, I need to prefix all docker commands with sudo

without sudo,

david@home:~/roscon-2023-realtime-workshop$ docker/fetch ~/Downloads/docker-image.tar.gz 
Trying to import /home/david/Downloads/docker-image.tar.gz to roscon-2023-realtime-workshop. This should take about a minute.
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/load?quiet=0": dial unix /var/run/docker.sock: connect: permission denied

With sudo docker/fetch ~/Downloads/docker-image.tar.gz it seemed to succeed

david@home:~/roscon-2023-realtime-workshop$ sudo docker/fetch ~/Downloads/docker-image.tar.gz
Trying to import /home/david/Downloads/docker-image.tar.gz to roscon-2023-realtime-workshop. This should take about a minute.
Loaded image: roscon-2023-realtime-workshop:latest
Image has been imported. Try running the following command to start the container:

  docker/start

But when I tried to run
sudo docker/start from roscon repo folder, and get the following message

david@home:~/roscon-2023-realtime-workshop$ sudo docker/start
+ docker stop roscon-2023-realtime-workshop
Error response from daemon: No such container: roscon-2023-realtime-workshop
+ docker rm -f roscon-2023-realtime-workshop
Error response from daemon: No such container: roscon-2023-realtime-workshop
+ docker container inspect roscon-2023-realtime-workshop
++ pwd
++ id -u
++ id -g
+ docker run -d --device=/dev/dri --volume=/tmp/.X11-unix:/tmp/.X11-unix --volume=/home/david/roscon-2023-realtime-workshop:/code --ulimit=rtprio=98 --ulimit=memlock=-1 --cap-add=SYS_NICE --init --rm -p 3100:3100 -e DISPLAY=:0 -e QT_X11_NO_MITSHM=1 -e HOST_UID=0 -e HOST_GID=0 --name=roscon-2023-realtime-workshop roscon-2023-realtime-workshop
d746b71e0686ac35ba28926c95860b40ab16849954b25607c7fc3892fe2ed366
+ set +x
Started container roscon-2023-realtime-workshop. To enter the container, run the command:

  docker/shell
david@home:~/roscon-2023-realtime-workshop$ sudo docker/shell
Error response from daemon: No such container: roscon-2023-realtime-workshop

Hi, here's what i see when I run the commands recommended

chief-of-mischief@gaurang-legion:~/Desktop$ docker version | grep -A2 Client
Client: Docker Engine - Community
Cannot connect to the Docker daemon at unix:///home/chief-of-mischief/.docker/desktop/docker.sock. Is the docker daemon running?
 Version:           24.0.6
 API version:       1.43
chief-of-mischief@gaurang-legion:~/Desktop$ systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset>
     Active: active (running) since Mon 2023-10-16 14:10:14 EDT; 2min 6s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 7574 (dockerd)
      Tasks: 29
     Memory: 34.7M
        CPU: 614ms
     CGroup: /system.slice/docker.service
             └─7574 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/cont>

Oct 16 14:10:14 gaurang-legion systemd[1]: Starting Docker Application Containe>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.2729658>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.2733908>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.3874492>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.7583824>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.7982099>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.7983349>
Oct 16 14:10:14 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:14.8803635>
Oct 16 14:10:14 gaurang-legion systemd[1]: Started Docker Application Container>
Oct 16 14:10:22 gaurang-legion dockerd[7574]: time="2023-10-16T14:10:22.6777447>
chief-of-mischief@gaurang-legion:~/Desktop$ sudo usermod -aG docker $USER
chief-of-mischief@gaurang-legion:~/Desktop$ docker version | grep -A2 Client
Cannot connect to the Docker daemon at unix:///home/chief-of-mischief/.docker/desktop/docker.sock. Is the docker daemon running?
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
chief-of-mischief@gaurang-legion:~/Desktop$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Oct 16 14:10 /var/run/docker.sock
chief-of-mischief@gaurang-legion:~/Desktop$ 

Is this okay or should I make changes?

@JanStaschulat assign me

Hi yall

I'm on a Ubuntu 22.04 host. Not using VS Code / dev containers.

When I get to docker/start, I get an error message, but it seems to exit normally (see line 2):

$ docker/start
roscon-2023-realtime-workshop
Error response from daemon: No such container: roscon-2023-realtime-workshop
++ pwd
++ id -u
++ id -g
+ docker run -d --device=/dev/dri --volume=/tmp/.X11-unix:/tmp/.X11-unix --volume=/home/morten/workspace/roscon-2023-realtime-workshop:/code --ulimit=rtprio=98 --ulimit=memlock=-1 --cap-add=SYS_NICE --init --rm -p 3100:3100 -e DISPLAY=:0.0 -e QT_X11_NO_MITSHM=1 -e HOST_UID=1001 -e HOST_GID=1001 --name=roscon-2023-realtime-workshop roscon-2023-realtime-workshop
20c5f8de0c5479caa10f811502a21daef74cda6e674caad4fc74031d568e40b9
+ set +x
Started container roscon-2023-realtime-workshop. To enter the container, run the command:

  docker/shell

I'm then able to run docker/shell and I see the expected prompt. However, opening http://localhost:3100/ doesn't work (won't load).

Remove snippets of code from 3-1, 3-2, 4-1, and 4-2, and put the complete code in a solutions folder.

Impartial code should still build without errors

4-1 high priority node ( real-time threads, multiple Executors)

Goal:

• Learn, how to prioritize entire ROS node
• real-time behavior for a timer/subscription, which are in the same ROS node

User Experience:

• setup A: baseline : plain vanilla executor, timer/subscription is delayed
• setup B: with prioritization: plain vanilla executor, timer/subscription is never delayed

Visualization:

• setup A: user views tracing and can sees delays
• setup B: user views tracing output and does not see any delays

exercise for user:

• baseline (provided)
• execute setup A
• publisher is already with real-time priority
• user does the same thing for sensor_node
• execute setup B
• compare tracing results (save previous logout, rename)

4-2 processing chain (callback-groups, real-time threads, multiple Executors)

Goal:

• Learn, how to prioritize multiple callbacks distributed over multiple nodes within one process.
• mixed-criticality use-case, real-time and non real-time is distributed over multiple nodes.

User Experience:

• setup A: baseline : plain vanilla executor, subscription is delayed, high jitter on end-to-end latency
• setup B: with prioritization: plain vanilla executor, timer/subscription on high priority path not delayed, low jitter on end-to-end latency

Visualization:

• setup A: user views tracing and can sees high jitter on end-to-end latency of high-priority path
• setup B: user views tracing output and observes low jitter on end-to-end latency of high-priority path

exercise for user:

• baseline (provided)
• execute setup A and visualize trace (observe high jitter)
• publisher_node has setup with callback_group/real-time thread
• user applies this to sensor_node/ObjectDetector subscription and actuation_node/subscription
• execute setup B
• compare tracing results (save previous logout, rename)

• Exercise 4 preparation for testing (readme and slides), review session
• Exercise 2 preparation for testing (readme and slides), review session
• Writing emails to panelists and attendees
• Minor adjustment to timing on website
• Ahead of time Raspberry Pi setup instructions
• Schedule Exercise 1 and Exercise 3 testing time (Sunday)
• Publisher executor rename
• Run script for exercise 2 should be uniform

Until @shuhaowu releases the new docker and raspberry pi image, the workaround is:

# Start up docker container
cd exercise3-2
sudo rosdep init
rosdep update
sudo apt update
rosdep install --from-paths src --ignore-src

Then exercise3-2 will build