Go fully headless about wolf HOT 3 CLOSED

Hey! 👋

I am the dev of https://github.com/Smithay/smithay - a wayland compositor framework written in rust. More than that, I am currently developing a new desktop environment at System76, so I would consider myself pretty knowledgeable in the linux desktop space.

I have been trying to build a re-implementation of sunshine for a couple of months to implement exactly, what has been proposed here: A headless wayland-server that can be fired up on demand with only serving the nv_gamestream protocol as a frontend. The idea being to utilize my gaming machine as some sort of local "GeForce Now" replacement for all my end-devices.

I recently stumbled upon wolf, which seems much more fleshed out than my attempt (currently I have implemented a bunch of the HTTP/S-interface, pairing logic and I am a little stuck at the RTSP part, with moonlight timing out). A super barebones wayland-compositor was done in an attempt to get Sunshine running headlessly with wayland and the nvidia-driver, but that bugs out in sunshines twisted EGL-code. In fact the whole code base was bad enough in readability for me, that I started to build a more integrated solution (not-crossplatform, headless, etc).

I would be super interested in joining forces or at least aid with my knowledge in this goal.

Some information:

video

Wayland only carries screen contents, so a wayland-server only gives you a video-feed.
I can easily help with that. I am currently trying to bridge the gap between my rust-code and the cxx-library to initialize the gstreamer-pipeline with an appsrc, that I could control from the rust-side of things.

If the languages are an issue here, we could implement a gstreamer-source connecting to the wayland-compositor as a separate process. Performance should be identical, but it seems you are trying to cut back on processes and complexity. (Which I generally agree on, just not for audio, see below.)

audio

For audio, you could technically fake a libalsa-library (similar to how alsa compatibility for pulseaudio/pipewire setups work), but that seems a little over the top. You still need to connect that back to your gamestream-server. (Audio never goes through wayland.)

If you need a network protocol anyway, you can "just" use one of the existing ones and get good compatibility.

The pulseaudio configuration format is actually quite simple (and pipewire should also get better in the future: https://www.collabora.com/news-and-blog/blog/2022/10/27/from-lua-to-json-refactoring-wireplumber-configuration-system/)

Pipewire requires more components than a single server and clients still use the pulseaudio protocol anyway. So my suggestion would be to setup a very simple pulseaudio server with a single null-sink and setup the gstreamer-audio-pipeline with pulsesrc to pickup the monitor of said sink. That should cause every application to pick up this single sink, completely decouple for any other running session.

input

While input goes through the wayland-protocol, it does so only for mouse and keyboard, not for gamepads.. (reasoning being, there are a bunch of "special" gamepads - e.g. vr controllers - making it difficult to build a unified protocol.)

Games tend to just open nodes in /dev/input*, which cannot be easily emulated by fuse or container namespaces. Instead one would need to use cuse (character device in userspace), which requires the same or worse permissions (to load the kernel module) than accessing /dev/uinput.

It gets worse, as /dev/uinput created devices will likely get picked up by any application having access to /dev/input. So there is no good way to separate different instances of streams/compositors running or even real desktop environments running, when using /dev/uinput. Theoretically each container could get it's own UID and we could run udev with rules to setup the created devices with only permissions for the matching UID of the container, but... this seems to be prone to errors (and introduces a new component udev).

The only alternative I am aware of, is what libcamera does to fix this problem for video nodes in /dev/v4l2* for webcams. They provide a wrapper, that you can inject via LD_PRELOAD, that hi-jacks the open-call (and others), checks for a camera path and emulates the expected results (otherwise passing the call through). (Code: https://github.com/kbingham/libcamera/blob/master/src/v4l2/v4l2_compat.cpp)

So it should theoretically be possible to do the same for input-devices. The biggest problem is, that I don't know, if just setting LD_PRELOAD will be enough for more complex applications such as Steam. Steam runs games in it's own sandbox for example, which might clear some environment variables and might make it harder to hack into. So someone should probably build a proof-of-concept for this one.

For perfect compatibility there is essentially only /dev/uinput.

Feel free to shoot any questions, I am super happy to help, even if I should end up not being involved with the code. I am a big fan of this project!

from wolf.

Here are a couple of ideas, feel free to jump on our discord channel to discuss this.

Sure thing, done.

Honestly, I'm happy to keep using pulseaudio. It's very simple to setup, it works reliably in Docker and I think we never had an issue with it in GOW.

👍

If we could implement this as a plugin it would be awesome!

I could also implement a wayland-compositor as a plugin (gstreamer rust-bindings are pretty nice), but I am not sure, if that is the best approach. It might make dynamic configuration a little harder to deal with. I have an example compositor already, that just pumps dmabufs into a gstreamer appsrc. That would be the easiest way to get going imo.

I'm more than happy to switch to another language

I am okay with whatever you guys feel like, but I am of course more knowledgeable when it comes to rust, given that is my current day job. If we continue using C++, there are ways to interface with rust and the api-boundary for the compositor would be relatively small anyway imo. But if there is interest in switching the whole project to Rust, I am more than happy to upload my current implementation and immediately start contributing.

Wolf doesn't need multiple containers to serve multiple users, we can already send different gstreamer pipelines to different users just by using a different set of ports.

I think I was just a little confused by the multi-container approach games-on-whales takes today. But of course, if we have a container running as root with /dev/input and /dev/uinput mapped, we can just allocate a range of user-ids for the newly created input devices. (Though this should be user configurable to avoid overlapping with user ids in the base system.)

Preloading a little library is also a viable option, if we want to get rid of that entirely. That solution would also not require a privileged container and could even run in a rootless container.

from wolf.

Hello @Drakulix, thanks for all the details, it's incredibly helpful!

I'll be honest, I've just finished the audio/video streaming part; I'm still in the middle of documenting and polishing the code, so I haven't got time to properly look into this.

Here are a couple of ideas, feel free to jump on our discord channel to discuss this.

Audio

Honestly, I'm happy to keep using pulseaudio. It's very simple to setup, it works reliably in Docker and I think we never had an issue with it in GOW.

Video

I think that Gstreamer can be the perfect glue here. If we could implement this as a plugin it would be awesome!
I don't mind the language, I've started in C++ (without knowing it) so that I could understand better Moonlight and Sunshine codebases. It paid off in the long run because it allowed me to properly debug and understand all the little details in order to make everything work.
I'm more than happy to switch to another language, I'd probably write Wolf in a different language if I could start from scratch now with the knowledge that I've got!

Input

I was really hoping that we could move away from udev with Wayland, it's source of all kind of problems.
If this is not handled by the protocol, I would start with using it (in the same way that Sunshine does) but with a little twist:
Why don't we create proper uids and create/assing devices to it? This should trivially separate different users, and it should probably be the "right" way of doing it.

EDIT: I just saw that you mentioned this already, but I'm a bit confused. Wolf doesn't need multiple containers to serve multiple users, we can already send different gstreamer pipelines to different users just by using a different set of ports.
In this Docker container we could have as many "real" users as we want; when creating the fake device in udev we could probably do that as a different user, right?

If possible I would try to remove udev from the equation in future, we can compromise in the meantime 😅

from wolf.