We need some instrumentation to understand how our code is performing. Not only do we need profiling instrumentation but we need some metrics to care about! Some general areas I want to know about are:

• receiving video stream in streamrs and encoding it for sending to client browser
• receiving video stream in inference server and preprocessing it for model inference
• capturing video and serving it / sending it to downstream services.

Some tools of interest are:

• flamegraph-rs
• tokio-console
• also perf, of course, maybe with some Brendan Gregg inspiration?

Currently the streamrs crate does the bare minimum and sends h264 frames to streamrs which in turn streams each frame (mjpeg) to client browsers by encoding every frame as jpeg! This is certainly not the end all be all of video encoding and decoding. Without worrying too much about what the camera source is running (we currently assume raspberry pi with the awesome built-in libcamera-vid or raspicam-vid) we should explore serving video from streamrs to clients with

• sending h264 over websockets
• DASH
• HLS

Currently, the camera source runs this command or something similar to send data to streamrs or the inference server:

libcamera-vid -n -t 0 -o - | gst-launch-1.0 fdsrc fd=0 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=$HOST port=$PORT

This is convenient, however the camera should really be running a server (like gstreamer-rs rtsp-server). The reason why the camera feed currently shoots frames to a udp sink is that I was unhappy with the latency I noticed in the live stream via an RTSP server to the streamrs http server / camera feed receiver. Wanting to get something going, I was pretty happy with the latency I saw just chucking frames over udp, so here we are.

There are a few problems with this approach:

• security / secure transport
• network bandwidth usage: the camera will 5ever send frames to the sink, even when streamrs doesn't require it)
• multiplexing the stream to many clients: currently I run a proxy server (nginx) that solves this but how do we handle:
• multiple camera sources, one sink: how can one instance of streamrs process and serve feeds from multiple cameras?
  one way to handle this is to use sticky load balancing based on camera source IP, but this seems like it could be more troublesome than just making client requests to an RTSP/RTP server.