volumetricperformance / raspberry-capture-kit Goto Github PK

Document steps to configure a desktop OS to run the streaming script

For macOS, Windows 10, ubuntu. Describe setting up.

• Python environment
• gstreamer/gobject
• librealsense

In working with artists and students we are finding significant barrier in connecting to the Raspberry Pi over the local network.

Consider using a cloud based console that uses websockets to communicate between the web console and the device.

The web console is already web sockets based so "all" that needs to be done is some kind of match making service.

Once the Raspberry Pi is connected to internet it gets an access code and displays this.

Performers then use the access code to connect and send the relevant commands (start stream to specific url, stop stream, restart, shutdown) and display the progress.

What are the steps to get get RPI this running on rpi.

Starting from a clean image document the steps setup python, gstreamer, librealsense for the code to run.

There is currently no feedback / error reporting from gstreamer, making diagnostics hard. Is there a way to periodically print out stats from gstreamer?

Can we use the "clip" creation process in Twitch to make recordings?
https://help.twitch.tv/s/article/how-to-use-clips?language=en_US#create

It is possible to compile opencv with gstreamer support so we can write to it using cv2.VideoWriter.

This would be a neater approach then using Gst.Buffer, we can skip the step of converting the images np array to a string.

Examples:
https://github.com/madams1337/python-opencv-gstreamer-examples

Reference for cross compiling opencv to have gstreamer support so above examples work.
https://raspberrypi.stackexchange.com/questions/116993/cross-compile-opencv-with-gstreamer-on-raspberry-pi-3-4

The depthkit client assumes the depth data is hsv encoded as rgb. Depth data should be mapped to a 0-1 range and then passed in as Hue (h) in the hsv-rgb conversion.

See realsense-hsv-viewer.py as a reference:
https://github.com/volumetricperformance/realsense-rtmp-twitch/blob/master/realsense-hsv-viewer.py

The current baleno based captive portal does not work in Wifi environments that have WPA3 networks. More and more enterprise and educational networks have these.

The work around is to connect a keyboard and display to the Raspberry and use the nmtui command to configure the wifi. This is cumbersome.

Consider integrating Berrylan instead
https://github.com/nymea/berrylan

Build a simple web frontend for configuration, control and and status. This means we can use a laptop, or mobile device to interact with the rpi system.

Web features:

• set rtmp endpoint
• start / stop stream
• connection diagnostics
• hardware diganostics (camera available, mic available)

RPI screen display

• large font IP address to point to
• realsense camera stream when running

We are seeing that considerable barriers towards getting the performance kit networked and connected to the internet.

Consider adding a travel router to the kit of hardware
https://www.tp-link.com/us/home-networking/wifi-router/tl-wr902ac/

In doing so move the wifi/networking configuration from the "headless" raspberry pi to the travel router.

• The raspberry pi connects to the router via ethernet.
• The laptop connects to the router via ethernet.
• The router is configured to bridge to whatever wifi is available.

The current stream is video only, most performers will also want audio.

Demonstrate how to set up a gstream that capture audio and video.
Use a USB microphone as audio source.

Reference for setting up microphone on a rpi.
https://pimylifeup.com/raspberrypi-microphone/