steam cache keeps missing some chunks about generic HOT 18 OPEN

I've mentioned this issue to Valve (with regards to the hostname being in the path, which is preventing caching) and had this response:

Regarding the “host name in the path” requests – that is related to Open Cache proxies. ISPs are setting these up to cache Steam download traffic; we direct clients to use them when they match the IP filter list supplied by their API.

The good news is, there's an update in the Steam Beta client which, when rolled out to non-beta, with an update to the cache domains list, should resolve this and some of the other Valve issues.

from generic.

it seems like it wants to redownload chunks when they are requested from different cdn's

$ grep e0b46aed321a384cbd314b01cb2e87303bf704c8 steam/logs/access.log
192.168.1.2 / - - - [10/Dec/2019:03:12:04 +0000] "GET /depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache46-fra1.steamcontent.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:20:38 +0000] "GET /cache65-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:25:15 +0000] "GET /cache68-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:31:34 +0000] "GET /cache65-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-fi001-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:46:03 +0000] "GET /cache69-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
192.168.1.2 / - - - [10/Dec/2019:03:53:49 +0000] "GET /cache59-fra1.steamcontent.com/depot/70401/chunk/e0b46aed321a384cbd314b01cb2e87303bf704c8 HTTP/1.1" 200 77200 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-fi002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"

from generic.

just tested on the monolithic container, same behaviour

from generic.

Something weird going on here, steam Appears to be doing a get for CDN/depot instead of the usual /depot

Could you upload your entire access.log somewhere (gist.github.com maybe) and tell us which game you were downloading this on

from generic.

[steam] 10.81.0.19 / - - - [10/Dec/2019:04:45:44 +0000] "GET /depot/271241/patch/3951236313252844930/7962357758666437985 HTTP/1.1" 200 187632 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache16-lhr1.steamcontent.com" "-"

As a reference this is the sort of thing we usually see. So we are attempting to narrow down. If it is a particular game/dept/country or cdn node for example

from generic.

depot 70401 is Recettear: An Item Shop's Tale
https://steamdb.info/app/70400/

from generic.

it does the same thing for other games. I'm in Italy. this is my access log with the monolithic container
https://gist.github.com/1dcbb2fec321638176a5727133befc4b

from generic.

this is the linux steam client by the way

from generic.

can you try changing the download region in the steam settings -> downloads -> download region for me?

from generic.

tried Germany, UK London and US Miami and it looks the same. see the bottom of this access log

https://gist.github.com/e8e82a467e8280857afee67dede4f20d

from generic.

I also tested the windows steam client today, with the same result. other services seem to cache fine (I tested blizzard launcher and it served overwatch fine). valve must've changed something in their cdn infrastructure. I'm guessing we're gonna have to have some way to strip that leading cache path or something

from generic.

Thanks for confirming that the region choice changes nothing. It looks like your part of the world resolves to a cqloud cdn. We are attempting to understand why they behave differently. Its possible its a misconfig at the value end

from generic.

[EDIT] Added more infos

Same here, i'm from italy and i experience exactly the same problem with your monolithic docker. I'm not sure if i had to open another issue on monolithic or if i can continue here.

Docker was freshly installed so it's the latest available (yesterday)

My configuration:
docker run --restart unless-stopped --name lancache --detach -v /data/cache:/data/cache -v /data/logs:/data/logs -p 192.168.246.180:80:80 lancachenet/monolithic:latest

docker run --restart unless-stopped --name lancache-sniproxy -p 192.168.246.180:443:443 lancachenet/sniproxy:latest

DNS redirected from pfsensezeropingheroes/lancache-dns-pfsense

I changed region to UK without any change.

speeds are not consistent download to download

First download at DSL speed
second download, begin at gigabit speed but then at DSL speed with some spikes
third download, begin at DSL speed with Spikes at Gigabit speed

some examples:

[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache34-lhr1.steamcontent.com/depot/230291/chunk/cb1f19fa78e552a58008de7c799d72bf1fa6817a HTTP/1.1" 200 458320 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache34-lhr1.steamcontent.com/depot/230291/chunk/cb1f19fa78e552a58008de7c799d72bf1fa6817a HTTP/1.1" 200 458320 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache29-lhr1.steamcontent.com/depot/230291/chunk/47e04b1e588fcc832c98132a45f7037d4dbfcaf8 HTTP/1.1" 200 213264 "-" "Valve/Steam HTTP Client 1.0" "HIT" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:05 +0000] "GET /cache29-lhr1.steamcontent.com/depot/230291/chunk/5c574103bd3bcf27f18ce685ef1e0ac4252bf731 HTTP/1.1" 200 18400 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"
[steam] 192.168.246.110 / - - - [07/Jan/2020:12:26:06 +0000] "GET /cache8-lhr1.steamcontent.com/depot/230291/chunk/70ab0794caf6878f6daf4f5a10a53be2b5b8853a HTTP/1.1" 200 280912 "-" "Valve/Steam HTTP Client 1.0" "MISS" "qn-10075-ve002-2.steam-content-dnld-1.eu-c1-cdn.cqloud.com" "-"

Even if the game is always the same, the cache continues to grow download after download.

from generic.

Further to this - The changes to the Steam client are now in the Live client.

If the relevant parties in this issue could retest with the latest lancachenet/monolithic container, the latest lancachenet/dns container, and the latest uklans/cache-domains list, and see if this issue has been resolved?

Many thanks

from generic.

I'll test in the next days.. now little holiday :)

This issue is about generic, are the same updates working also in monolithic?

from generic.

The Valve client change will work with both Monolithic and Generic, so test with whichever you are comfortable using.

from generic.

seems to work fine now, at least on the windows client. I had to download the game twice to fully warm up the cache (maybe it was using some different chunks the 2nd time around), but it's definitely hitting all the chunks it has seen once

from generic.

Hello, Having the same issue today with Doom (not doom eternal) game, still from Italy.
Error says Downloaded data is damaged.
Basically rebooting steam solved it as it restarted downloading the last chunk. So I just wanted to sort of warning you this is still somehow happening

from generic.