hello, caching on blizzard is having some issues, several gaming lounges reported the issue.
As soon as i stopped the container that was running blizzard cache and stopped the dns re-route downloads worked fine
issues : blizzard app is stuck on 0% and does not update , game updates also stuck or run for a while and then hang with a download speed reaching 0-25b/s
im running those commands :

sudo docker run --name cache-steam --restart=always -d -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 192.168.88.68:80:80 steamcache/generic:latest

sudo docker run --name cache-blizzard --restart=always -d -v /cache/blizzard/data:/data/cache -v /cache/blizzard/logs:/data/logs -p 192.168.88.69:80:80 steamcache/generic:latest

sudo docker run --name steamcache-dns --restart=always -p 192.168.88.68:53:53/udp -e CACHE_MAX_AGE=95d -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.88.68 -e BLIZZARDCACHE_IP=192.168.88.69 -e DISABLE_RIOT=true -e DISABLE_FRONTIER=true -e DISABLE_ORIGIN=true -e DISABLE_WINDOWS=true steamcache/steamcache-dns:latest

note that the caching was working fine for the past couple of months, suddenly the blizzard issue came up in several gaming lounges

Using generic now, defined al zones for steam and other platform direct at my sophos firewall to route to the steamcache:generic.
Works fine at start. Now 2 Probem:

1. At battle.net i got only a very reduced speed, like 1 Mbit/s (6Mbit/s peer). Docker runs, but the cpu and the memory is not the problem i see at docker stats.
2. At steam a i can install games at my library, the steam-store at the steamsotware generates "error -137" and does not work.

Any suggestions on this probs?

Describe the issue you are having

The Blizzard client is having issues with the bnet cache docker.

• Downloading the games slowly, about 3% of available network speed,
• Reporting false "HITS" in the logs when downloading into a clean directory/fresh install
• Doesn't appear retrieve the cached files. This is believed as the router appears to be reporting the same download speed as the client on cached games. More testing to be done to confirm.
• Lots of reported log errors (timeouts).

Other cache dockers (Steam and Origin) download at full network speeds (3.2mb/s). and caches the files downloaded and retrieved correctly. (Origin is also reporting false hits too when downloading fresh cache files, but not a major issue right now)

Using Ubuntu 18.10 with latest docker-ce and latest docker pulls.

Below is a Grafana parsing of the logs, the client download speed and disk usage.

How are you running the container(s)?

Blizzard / Battle.net

docker run \
  --restart unless-stopped -d \
  --name blizzard-cache \
  -v /home/data/steamcache/blizzard/data:/data/cache \
  -v /home/data/steamcache/blizzard/logs:/data/logs \
  -p 192.168.1.212:80:80 \
  steamcache/generic:latest

DNS Configuration

DNS

docker run \
  --restart unless-stopped -d \
  --name steamcache-dns \
  -p 192.168.1.210:53:53/udp \
  -e UPSTREAM_DNS=1.1.1.1 \
  -e STEAMCACHE_IP=192.168.1.210 \
  -e ORIGINCACHE_IP=192.168.1.211 \
  -e BLIZZARDCACHE_IP=192.168.1.212 \
  -e RIOTCACHE_IP=192.168.1.213 \
  -e FRONTIERCACHE_IP=192.168.1.214 \
  -e WSUSCACHE_IP=192.168.1.215 \
  steamcache/steamcache-dns:latest

HTTPS

docker run \
  --restart unless-stopped -d \
  --name sniproxy \
  -p 443:443 \
  steamcache/sniproxy:latest

Output of container(s)

error.log tail

2018/11/07 02:54:06 [error] 20#20: *14622 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.1:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:08 [error] 16#16: *14646 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.4:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:10 [error] 14#14: *14473 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", upstream: "http://137.221.64.4:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:11 [error] 15#15: *14552 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.5:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:17 [error] 24#24: *14602 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", upstream: "http://137.221.64.3:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"

Access.log tail

192.168.1.120 / - - - [07/Nov/2018:02:55:14 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "us.cdn.blizzard.com" "bytes=147858222-148124461"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "MISS" "level3.blizzard.com" "bytes=158507822-158774061"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=158774062-159040301"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=159040302-159306541"

I am going to attent a lanparty (12 persons) where we have to share a slow ADSL connection (6Mbit down at best).
Steamcache is running as a Docker service on an Ubuntu 16.04 installation on a I5-2500, 8GB, 2GB WD SATA harddrive.

• Can I "load" Steamcache with complete game installations by installing them through the Steamcache server on my fast internet connection at home (same IP settings)?
• Will the cached data expire after some time?
• Will the steamcache performance deteriorate when the amount of cached data increases?

Hi, @mintopia @mikeweyandt @GotenXiao @astrolox @JasonRivers,

I was trying to use generic image with steamcache and steamcache-dns.

I installed the generic image on another server, it seemed to work fine, but now blizzard is not updating

Any idea? Is it working fien for you guys?

Best Regards,

Hi, I'm preparing my cache downloading all games but, after few hours of download, Steam stops the download. When analysing the cache-dns log I find many timeout messages but I can't figure it out why.

here is a part of the log

2018/06/27 16:25:46 [error] 13#13: *315735 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:25:51 [error] 13#13: *315742 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:25:56 [error] 13#13: *315746 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:01 [error] 13#13: *315753 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:06 [error] 13#13: *315758 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:11 [error] 13#13: *315765 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:16 [error] 13#13: *315770 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:21 [error] 13#13: *315777 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:26 [error] 13#13: *315781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:31 [error] 13#13: *315788 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:36 [error] 13#13: *315793 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:41 [error] 13#13: *315800 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:46 [error] 13#13: *315808 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:51 [error] 13#13: *315815 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:56 [error] 13#13: *315822 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:01 [error] 13#13: *315827 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:06 [error] 13#13: *315835 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:11 [error] 13#13: *315841 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:16 [error] 13#13: *315849 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:21 [error] 13#13: *315854 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:26 [error] 13#13: *315860 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:31 [error] 13#13: *315865 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:36 [error] 13#13: *315871 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:41 [error] 13#13: *315874 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:46 [error] 13#13: *315882 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:51 [error] 13#13: *315885 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:56 [error] 13#13: *315892 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:28:01 [error] 13#13: *315897 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:28:06 [error] 13#13: *315903 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 17:23:15 [error] 13#13: *319046 unexpected status code 200 in slice response while reading response header from upstream, client: 10.0.0.5, server: , request: "GET /depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1 HTTP/1.1", subrequest: "/depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b", upstream: "http://146.66.155.12:80/depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1", host: "valve1202.steamcontent.com"
2018/06/28 07:38:01 [error] 14#14: *319486 upstream timed out (110: Operation timed out) while reading upstream, client: 10.0.0.5, server: , request: "GET /depot/578081/chunk/9f0bb3ed6db976c99e2ca5517eea9a91352f1b0f?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1 HTTP/1.1", upstream: "http://146.66.155.6:80/depot/578081/chunk/9f0bb3ed6db976c99e2ca5517eea9a91352f1b0f?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1", host: "valve1206.steamcontent.com"

Can you help me please.

thanks

I am redirecting the data created by the lancache docker (see my command line below).

sudo docker run --name lancache --restart=always -d -v /srv/lancache/data:/data/cache -v /srv/lancache/logs:/data/logs -p 10.0.0.12:80:80 steamcache/generic:latest

Are the /data/cache and the /data/logs the only directories I have to redirect?
Can I just redirect /data to /srv/lancache/data (or another directory on the computer running the cache) and get the same result?

Hello,

I am having trouble deploying the Docker container the way it is documented.
When I run this command:
docker run --restart unless-stopped --name cache-steam -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 10.0.2.15:80:80 steamcache/generic:latest

As its documented under: https://hub.docker.com/r/steamcache/generic/
I only recieve this error Message:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
2017/12/12 12:36:38 [emerg] 10#10: mkdir() "/data/cache/cache" failed (13: Permission denied)
nginx: [emerg] mkdir() "/data/cache/cache" failed (13: Permission denied)
nginx: configuration file /etc/nginx/nginx.conf test failed

When I try and run the Docker passing only "-v /cache/steam:/data" like this:
docker run --restart unless-stopped --name cache-steam -v /cache/steam:/data -p 10.0.2.15:80:80 steamcache/generic:latest

the container actually starts successfully, created the "/cache/steam/data" and "/cache/steam/logs" directorys and even caches Steam downloads, but the internal Docker folder /data is not linked to /cache/steam on the Server itself.

The internal directorys look like this:
docker exec cache-steam /bin/sh -c "ls -al /data/cache"
total 12
drwxr-xr-x 3 nginx nginx 4096 Dec 12 12:56 .
drwxr-xr-x 4 root root 4096 Dec 12 12:27 ..
drwx------ 2 nginx root 4096 Dec 12 12:56 cache

e.g: /data/cache/cache

Did I miss a step in the configuration or is the Documentation simply outdated?

Cheers,
Finn

So I have searched online and I can't find the cause of my issue at all, I have all my containers up and running but I cannot figure out why when I run the :

docker exec -it steam-cache /scripts/watchlog.sh

commannd it just returns this every time:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: "/scripts/watchlog.sh": stat /scripts/watchlog.sh: no such file or directory": unknown

Following are the containers I have running:

sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cd9394c087f5 steamcache/steamcache-dns:latest "bash /scripts/boots…" 8 hours ago Up 3 seconds 192.168.1.142:53->53/udp steamcache-dns
24192243e997 steamcache/sniproxy:latest "sniproxy -f" 9 hours ago Up 8 hours 0.0.0.0:443->443/tcp sniproxy
f20a8c4a43c8 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.146:80->80/tcp uplay-cache
23744b50368b steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.145:80->80/tcp windows-cache
1c3a35a7db33 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.144:80->80/tcp blizzard-cache
de3c152902be steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.143:80->80/tcp origin-cache
2ad98fb607f9 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.142:80->80/tcp steam-cache

Any help would be much appreciated.

Describe the issue you are having

Checking the heartbeat in wget is fine with no loops, checking it in chrome generates a request loop

wget http://cacheip/steamcache-heartbeat -S -O -

I am running a combined Steamcache/Lancache setup which works like a charm.
My Origin games are cached when I install them so a second installation will be taken care of by the local cache, like designed.

However, if the Origin client on my game PC needs to be updated, it keeps telling me to work offline and is unable to connect to the Origin cloud and download the client update files.
If I stop the Origin cache I can perform the Origin client update without any problem.

Not a big issue for me but less experienced users on my network will be confused by these extra steps...

hey there caching is working awesome but it will cache all the things on 80 http not just steam can someone help me to get rid of

As the title says, windows updates are no longer caching. In fact they ajust hange when routed through generic cache.

Describe the issue you are having

Its not an issue direct, its more a question on how you handle the log file watching.
I like to analyze the log files in production use to see how much files are served from the cache and how much not.
Are there any tool you have for this purpose?

How are you running the container(s)?

The docker-compose file from the docker-compose Repository.
One container of the generic cache for every service.
Logfiles under /data/{service}/logs/access.log

DNS Configuration

No problem, everything works fine

Output of container(s)

Hi,

IMO this docker would be much more user friendly if it were support the HTTPS by default and wasn't required another docker container for that.

Describe the issue you are having

Steamcache can still get stuck in a loop when requesting it's own IP

Steps to reproduce

Within container

wget http://127.0.0.1/looop -S -O -

Expected response

Nginx should detect the X-SteamCache-Processed-By header and drop the request

We're using the the Generic Steamcache in our environment, and we've run into an issue this morning where the steam client is failing to download. Took a peek in the error log, and this is what I am seeing:

2018/03/06 16:10:33 [error] 10#10: *95612 unexpected status code 302 in slice response while reading response header from upstream, client: 10.16.10.158, server: , request: "GET /client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795 HTTP/1.1", subrequest: "/client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795", upstream: "http://162.254.192.11:80/client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795", host: "client-download.steampowered.com"

Any ideas on how I could mitigate this? My research on this showed that I need to increase the proxy buffer size in nginx, which I can totally do, but it means modifying the container.

As an added note, the error the user gets is as follows:
"Steam needs to be online to update. Please confirm your network connection and try again"

Tracked via uklans/cache-domains#32

Speaking to @astrolox about how to go about testing the cache images.

Initial thoughts:

1. Attempt to download file directly, hash and save.
2. Clear cache data directory
3. Attempt to download file through cache. Ensure we have a cache miss, compare hash to original download
4. Attempt to download file again through cache, Ensure we have a cache hit, compare hash to original and previous version.

This only covers a simple case though. It would be good for advanced options for testing range requests, etag headers, etc.

Other people's thoughts and comments?

Hello,
I dont have any experience with docker and steamcache.
After reading all differents threats, I install docker and start to pull steamcache/generic + steamcache-dns on my ubuntu cache server.

I type this command and hit enter:
docker run --restart=always --name cache-steam -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 192.168.1.104:80:80 steamcache/generic:latest

I got the prompt:
nginx: the configuration file /etc/nginx.nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is sucessful
after the last line, I'm stuck inside the prompt. I tried to hit Ctrl + P, Ctrl + Q, Ctrl PQ, Ctrl + C, i'm still stuck inside. any idea how to deal with it ?

If someone had a white page how to proceed in every step pls.

THank you

Hi @mintopia

I was having an issue with generic image (blizzard client not updating), and then i noticed i was running sniproxy with generic image which obviously caused a conflict since they both use 443 TCP port.

Can i run lancache without listening to 443 port? AFAIK the sniproxy should be used to redirect https traffic to avoid the MITM scenario.

Please advise,

Hey guys,

at first -> cool project!

But i have a problem with loggin in to steam. It hangs at "Connection with Steam-Account ***"
In the log of the Docker-Container 'steam-cache' there are many logs of the ngnix with MISS:

10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/129705e028733989abfdef19c458c9e83261cdbd?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 316464 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-" 10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/bd1a68c7a0a939fcebe3c03c05240669320a611a?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 465264 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-" 10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/a1fa4bf66bb639230f1799f0fe966ba1fa52dc3d?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 27593 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-"

Maybe i am not alone with this problem?

Issue

I have steamcache and steamcache-dns running for my Xbox One downloads. The caching process is working, except I'm seeing slow download speeds on the initial download of un-cached data. I have gigabit internet from my ISP. Without caching, the download goes at 250mbps. With caching, it seems to be about 1/3 of that. Once a file is in cache, my Xbox will download it at about 250mbps once again. This result has been very consistent over the past 3 days. I understand that once the cache is primed, then the benefits become apparent. My thread here is all about the initial download though.

Read Dead Redemption 2 (80GB), for example, would take 3.5 to download through steamcache. That download would normally take under 1 hour without caching. I expected some overhead, but this seems unusual. Is this typical?

Also, I used Wireshark to inspect traffic and I don't see any obvious issues there. I saw that one of the links my Xbox was fetching was this link. I can open that in Firefox and begin downloading at 240mbps to 400mbps. That means my OS can fetch files from Xbox's servers at good speeds.

So... fetching is fast. Serving cached data is fast. But downloading fresh data is very slow.

Setup

steamcache was setup using this command:
sudo docker run --restart unless-stopped --name gamecache -v /mnt/gamecache:/data/cache -v /mnt/gamecache:/data/logs -p 192.168.1.223:80:80 steamcache/generic:latest
I am running this from a virtual machine. The VM is running Ubuntu 18 x64. Host is Windows 10. CPU i7 7700HQ - 4 cores at 2.8ghz. 16GB RAM. The VM has use of 2 cores and 4GB of RAM (will be more in production later). The VM and the cache/data folder are stored on an external SSD. I experimented with running Ubuntu natively but I didn't see a difference in speeds.
Here's how my system resources look while downloading:

Disk Z is where the VM and game cache are stored. At this same time, the Xbox itself reports about 55mbps.

DNS Configuration

sudo docker run --name steamcache-dns -p 192.168.1.223:53:53/udp -e STEAMCACHE_IP=192.168.1.223 -e USE_GENERIC_CACHE=true -e LANCACHE_IP=192.168.1.223 -e UPSTREAM_DNS=8.8.8.8 steamcache/steamcache-dns:latest

Output of container(s)

Not sure how to get this. When I run the docker commands at first, I see output from the DNS setup command, but I'm not sure how to get back to it after a reboot.

In the Steamcache guide on ARS Technica (https://arstechnica.com/gaming/2017/01/building-a-local-steam-caching-server-to-ease-the-bandwidth-blues/) there is a script on how to check if the cache functions properly.

sudo docker exec steamcache /scripts/watchlog.sh

Is a similar script available to check on the cache function in the Generic cache?

A nice feature additional to the toolchain from the generic / steamcache itself would be to prefill the Cache with actual content beforehand a bigger event.

The way i solve it until now is to predownload a "most common" set of games through the gameservices. depending on the Client this uses much time for micromanaging to delete the local installs.

How about a tool that uses functions from the steamcmd tool, to download a set of specific game IDs, and delete them afterwards and/or if possible for other plattforms aswell.

Any plans of adding PlayStation Network compatibility too?

(edited by admin)

For steam, /server-status shouldn't be cached.
Contains load information about that CDN server, and may result in lower performance if it remains cached.
example content:
"status" { "csid" "3" "load" "22" "cell" "31" }

Can we expect Epic Games support in the future? (Fortnite, Unreal Tournament, UE4, etc.)

Thanks!

Hello,
I want to use this with my QNAP NAS.

QNAP has a Container Station for handling docker images:
https://www.qnap.com/solution/container_station/en/

How do I need to configure it so it will work ?

trying to set up a multi-purpose cache system with each having its own IP address so that later down the track I can just assign said IP addresses to more NIC and not have to play with docker anymore.

I have steamcache-dns working fins and so is sliproxy, but steamcache and the generic version give me permissions issues when they start up and in portainer they lost their internal IP addresses after about 30 seconds

I have tried setting all the directories to both nginx and dockerroot and it was root by default...

the only way I have managed to get around the issue myself has been to manually recreate all the containers myself in portainer with the same settings that were given by your page.

tbh: I'm not sure if it works yet, but I was just wondering what I was doing wrong for the cli lines to not work.

thanks

Requests come through when the updated League of Legends launches (verified by tailing the logs), but something causes it to fail to patch.

Not sure as of this point in time, but the new patcher may be forcing HTTPS and validating certificates.

I've got a steamcache-dns, steamcache and steamcache-generic docker setup going, and it's working really well for everything except Battle.net.

I have a 100/20 connection, but with caching enabled, I only get 2-3mbps from Blizzard's servers, and I'm not sure why. I also get intermittent disconnections while downloading from Battle.net.

Just wondering if anyone else is also seeing this?

While downloading League of legends it downloads until a certain point and than gives a error 004.
When i delete the DNS it downloads again.

I'm having some difficulty getting GTAV to download from the cache. The game was already downloaded to my caching server, but now the launcher keeps stating that it lost connection to the download servers. This can be reproduced on any computer I try to download it from. Avoiding the cache allows the game to download normally. Interestingly, I did check the error log for the Rockstar cache as it tried to connect to these servers and I would get errors that mention "invalid argument" and "unexpected status code 500." I'm attaching the error log as well as the access log.

access.log
error.log

It seems the majority of the requests to steam miss because steam responds in HTTP 302 most of the time (which, in code, isn't cached).

Is there anything I can do to fix this on my end to make this more useful? Caching for blizzard works wonderfully, but that is mostly HTTP 200.

This project has a bunch of other redirects that would be great if integrated into steamcache as well.

Using that project directly is surely an option but steamcache is super easy to get up and running with its docker support and IMHO seems much simpler to me.

I am mainly referring to these: https://github.com/steamcache/generic#dns-entries

Sorry, not a network expert and had a bit of trouble with the docs.

I am looking to cache Steam + Battle.NET + Origin using the generic image.

I previously had mine working great on a raspberrypi using a NAS as storage, pushing full 100Mbit

I moved to a VM setup now, and i cant get the generic cache nor the sniproxy to act right.

My Blizzard docker steamcache/generic gets me the below in error.log

2018/07/18 03:56:59 [error] 19#19: *172 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /tpr/bnt001/config/b5/49/b549cde7f3a27e4aed8b09f3661dfa41 HTTP/1.0", upstream: "http://10.0.0.83:80/tpr/bnt001/config/b5/49/b549cde7f3a27e4aed8b09f3661dfa41", host: "level3.blizzard.com"

Hi,
Im running ubunto 18.04 desktop edition on vmWARE(using windows os)with a 100 GB given space.

when i run this code:
sudo docker run
--restart unless-stopped -d
--name steam-cache
-v /home/user/cache/steam/data:/data/cache
-v /home/user/cache/steam/logs:/data/logs
-p 192.168.1.50:80:80
steamcache/generic:latest

**Result:**latest: Pulling from steamcache/generic

4fe2ade4980c: Pull complete
e11b023b07a0: Pull complete
aaf7d5908de5: Pull complete
8ae3a5ca453f: Pull complete
b494cbfdaa4c: Pull complete
489e1e313017: Pull complete
Digest: sha256:04d20e341efbe344dc8cda6dea56e90a8801cc1194277d0afb289879943ea4ef
Status: Downloaded newer image for steamcache/generic:latest
4660032cc496b4dd3cfac2753ebf7c38c068bb4262324b7e6a667c641f34b35f
docker: Error response from daemon: driver failed programming external connectivity on endpoint steam-cachee (921438246a6a4ba7fc0233335883bde17af1c1b30c186f3478f7108ab4166d1f): Error starting userland proxy: listen tcp 192.168.1.50:80: bind: cannot assign requested address.

PS: my router ip is 192.168.1.1 ( and there is no one on my network having this ip)

user is my 'username'

Please help ! 👍

I would like to know if this repo works flawless as I see so many docker steamcache and dont know which to pick. I would like something to cache all games from any editors. Thank you

Hello,

before using steamcache I had download speeds from 45-50MB/s .

Now with steamcache (run in docker) the initial download speed is at max 8-9 MB/s per second.

What could the issue be here?

Hi,

I'm trying to run the steamcache generic with the following comand;

docker run \
  --restart unless-stopped \
  --name steam-cache \
  -v /home/sniper/Lan/steamcache/steam/data:/data/cache \
  -v /home/sniper/Lan/steamcache/steam/logs:/data/logs \
  -p 80:80 \
  steamcache/generic:latest

But I get the error:
Error response from daemon: driver failed programming external connectivity on endpoint steam-cache (189994c288542e25e5360c614a0035be230385b6fec688dd96f7d77c2a764387): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use. ERRO[0000] error waiting for container: context canceled�

How can I fix it?

thanks

My caching server suddenly started getting upstream time outs. I didn't make any changes to my configuration so I'm not sure what's causing it

I have Steamcache's sniproxy/generic/steamdns installed in containers on my Synology NAS.

Error I'm getting:

2018/08/08 09:49:44 [error] 27#27: *1036 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.0.3, server: , request: "GET /depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947 HTTP/1.1", upstream: "http://162.254.197.12:80/depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947", host: "valve502.steamcontent.com"

If I try testing it manually by going to the upstream link: http://162.254.197.12:80/depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947

I get unauthorized.

But if I include the hostname it works:

http://valve502.steamcontent.com/depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947

Is it possible to run this generic lancache on the same machine which run steamcache?
I get a bind x.x.x.x:80 failed:port is already allocated error message.

Hi,

It seems the the generic steamcache don't work. It doesn't cache the Hearthstone nor the Windows Update.
I use these settings:

version: '3'
services:

  steamcache:
    image: steamcache/generic
    container_name: steamcache
    restart: always
    volumes:
      - /cache/data:/data/cache
    network_mode: bridge
    ports:
      - "81:80"
    environment:
      - CACHE_MEM_SIZE=500m
      - CACHE_DISK_SIZE=100g
      - CACHE_MAX_AGE=14d
  
  steamcache-dns:
    image: steamcache/steamcache-dns
    container_name: steamcache-dns
    depends_on:
      - steamcache
    restart: always
    network_mode: bridge
    ports:
      - "54:53/udp"
    environment:
      - USE_GENERIC_CACHE=true
      - LANCACHE_IP=192.168.0.100
      - UPSTREAM_DNS=1.1.1.1

  sniproxy:
    image: steamcache/sniproxy
    container_name: sniproxy
    depends_on:
      - steamcache
    restart: always
    network_mode: bridge
    ports:
      - "444:443"

The log shows:

Checking permissions (This may take a long time if the permissions are incorrect on large caches)...
Done. Starting caching server.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

Hi,
could you add RSI Downloader (Star Citizen?) Its always huge GB to download.

Thanks.

The generic image appears to suffer from the same issue described here, in the steamcache image.

Would it be possible to have this implemented for the generic image as well in order to improve caching?

Currently, a game has to be downloaded 6+ times in order to get enough saturation of chunks to guarantee the local cache is used.

DNS issue please ignore.

Tracked via uklans/cache-domains#33