Feature Idea: Deduped Steam Libraries about wolf HOT 12 OPEN

Not sure if this is something that we could add to Wolf or it just needs to be properly setup on the host..

I'm not sure if a headless steam manage steamcmd implements the local network transfer stuff. Perhaps with a full fledged steam client we can just copy the files in?

The steam image runs the full client, so everything should be available as is

from wolf.

It could be something that Wolf doesn't do, but its a lot of state that specifically is useful to wolf, and seems to have advantages to be tied in to its lifecycle. Integrating some sort of plan here to avoid wasted diskspace and redownloading games for each streaming device will also make users less likely to just throw a single drive in mounts and put steam library on there. But maybe that would just work well enough, it feels like it's asking for race conditions messing everything up in a multi user environment.

So for the goal of 1) reducing disk usage 2) reducing need to redownload:

1. Something needs to run "duperemove" after certain periods.

This part of the feature is fairly easy to do outside, just have a cron job that dedupes the wolf folder once a day. However it makes a lot of sense to me if if Wolf schedules it when a container shuts down, to dedupe files added during that session.

2. This one is a lot tricker to set up, and is more software than just configuration. Just running a steam client on the host doesn't do anything unless all games which will be installed by clients are pre installed there. And there's no always running steam for hosts which are fully headless. The idea here would be to have a steam container that isn't tied to a client, and some plumbing so that after clients install games in their steam containers, it installs them so that a different client won't have to download from the net again.

| The steam image runs the full client, so everything should be available as is

Right, but the steamcmd comment wasn't about a image spun up for a client. Those do already transfer from local network, and if you have Steam running on the host they will transfer from there, though not as fast as could be done between containers on a shared internal bridge.

Building on that, if wolf can have with a container that's always on, and can be scripted to install games, it could be seamless that people don't have to download games multiple times if multiple moonlight devices play the same game on the wolf server. Another thing that could potentially be scripted externally. But here there is a risk partly installed games being transferred depending on implementation Wolf doing it when a client shuts down would make it somewhat atomic.

SteamCMD is easier to script which is why I wonder if it can do the local network transfers. Another image like is spin up for clients would for sure work to serve games to the others, but I'm not sure sosohow to script it installing games. And wolf probably would need a way to connect to it's display to administer it (set it to allow all users on the local network to transfer games at least)

Been thinking about a potentially easier path for 2) even if it does have some disadvantages:

Another approach could be adding a LanCache container to wolfs network. I stopped running one at home because I had flakiness with steam deciding to bypass the container, but I hope it would be more reliable in the virtual network.Another downside is that update churn can cause extra disk space spent on no longer relevant files. It also wouldn't dedupe as cleanly as a "main" steam container approach because it stores compressed chunks and not the specific files. On the plus side it wouldn't need any sort of system to recognize what clients installed and install them on a master container. It would be a lot less moving parts, just some docker network design.

There would still be advantages to having LanCache run by wolf and not set up externally. Mostly so it can be part of an internal bridge, because networking between containers is much faster that way than on the default bridge, DNS is handled better, and wolf can ensure DNS is set on the steam containers to use LanCache without a user having to mess with their local networks settings.

from wolf.

The more I read about this the more I think this is getting outside the scope of Wolf but it's a very interesting discussion and you brought up a lot of good points, there's lots to digest so here are a few random counter points:

• RE duperemove: I don't like having to implement filesystem specific code (nor forcing such a personal choice to users), this should be scripted outside Wolf. We could easily add a pre/post session script hook to facilitate that.
• The solution of having a Steam "main" account has the main drawback that you need an account that owns all the possible games that the "secondary" accounts will need, AFAIU SteamCMD needs a valid login (that possibly bought the game already) to download/update games.
  • This really feels like a hack and easily error prone, having to orchestrate all that seems like asking for troubles..
• LanCache + Deduplication (btrfs/zfs/any other fs that supports this) seems to really be the right solution for this. With LanCache you download only once, on-demand and most importantly transparently to all clients (and not just Steam); with deduplication you avoid any wasted disk space and just share the same blocks between clients.
  • This should be setup outside Wolf, based on user preference and actual requirements; there's no one-size fits all solution for this.
  • As for the networking and internal bridges concerns this should be already possible by overriding the default docker params in Wolf config.toml (you can manually override NetworkingConfig, NetworkMode and so on already in there).
    • Besides, this is something that different users will want to tweak (ex: a LanCache somewhere in the network that might be shared with other machines/desktops or a local one in the same machine that runs Wolf)

I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code. I always wanted to checkout LanCache ony my homelab this might be the right time to actually do it!

from wolf.

Thanks for reading and engaging!

I don't like having to implement filesystem specific code (nor forcing such a personal choice to users),
duperemove, isn't filesystem specific it's a external utility that uses linux ioctl s that some filesystems (btrfs, xfs, in progress with zfs) support. Simply exits if the files it's pointed at don't support the ioctl.

I do like the idea of hooks and they will be useful for other stuff. I'm still tempted to say it should be something on by default for people but I can live with it not being :)

You need an account that owns all the possible games that the "secondary" accounts

mm yeah. I suppose mostly assuming it was different client devices but same steam account, but this would be a hard part. Hmm.

I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code.

I think with Lancache even if it's just publishing a docker compose example of it would massively help people out. Running it on a different machine will be much slower without multi gig networking.

As for the networking and internal bridges concerns this should be already possible by overriding the default docker params in Wolf config.toml (you can manually override NetworkingConfig, NetworkMode and so on already in there).

I think something may still need to be done internally for dns, to make it easy to keep the traffic to the internal docker network if running one there.

I might still explore scripting the main steam container some, because I don't trust lancache to work reliably :) If it ends up feeling easy I might have you take a look and see if you want to integrate. Or just figure out how to interface it with the hooks you add.

from wolf.

I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers.

Perhaps a user based collection. Or manual selection. Any thoughts for me on forking, vs preparing a PR or pointers on where I might want to start in the code?

from wolf.

I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers.

This is a valid point and I've been thinking about it for quite some time, I believe this is part of a more general feature: share sessions (aka: co-op for multiple users or as a specific case for a single user share the container between devices).

I think there are different possible approaches to this:

• Change Wolf so that it supports a notion of users and will "assign" containers to them. This will practically have to be manually configured in the config.toml or, possibly, configured at runtime over a webui
• Make a GUI that when you start a connection with the Moonlight clients will show up the list of connected/possible users (like the Netflix first page) and then let you pick the games for that user and ultimately starts and controls the app container.

Whilst the second approach might be a bit more complex it feels that it's the best way of achieving this:

• A completely independent game and user launcher so that Wolf can be focused just on the Moonlight protocol and streaming
• From the user perspective you don't have to open up a browser to set things up (some Moonlight clients aren't even able to do that: xbox, switch, ...) but you have a nice and easy interface that you can interact with your joypad
• Wolf could send messages over this GUI, ex: when first downloading a game it could show the progress bar instead of a black page like it is right now

Thoughts?

from wolf.