[Store] slice bounds out of range [:1029] with capacity 1024 about thanos HOT 4 OPEN

@hczhu-db What version of Thanos are you running? I think we used to see this issue. However, I think after several rounds of fixes around some race conditions when fetching chunks, I haven't seen this issue for a while.

We are running 0.34.1 with our own changes on top of it. Those changes are mainly for observability (like new metrics and logs) and Compactor skipping overlapping block files.

@jnyi

from thanos.

I encountered the same panic and found that the chunk pool returns a slice shorter than required.

panic: runtime error: slice bounds out of range [:2297] with capacity 2129

goroutine 4034779 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks(0xc0914ad920, {0x35f4b28, 0xc091ff2a00}, {0xc093280000, 0x317, 0x317?}, {0xc0914b5968, 0x2, 0x2}, 0x29, ...)
        /thanos/thanos/pkg/store/bucket.go:3542 +0x1118
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).load.func4()
        /thanos/thanos/pkg/store/bucket.go:3476 +0x11b
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 4034046
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x96

from thanos.

@hczhu-db What version of Thanos are you running?
I think we used to see this issue. However, I think after several rounds of fixes around some race conditions when fetching chunks, I haven't seen this issue for a while.

from thanos.

I also got the following error interleaving with slice bounds out of range error. More weirdly, the same range query can work or trigger either error if it's run multiple times.

MinTime: 1708874139848 MaxTime: 1711214139848: rpc error: code = Unknown desc = receive series from 01HSED6EP817R1VPXQCS73ZGDA: load chunks: populate chunk: allocate chunk bytes: pool exhausted

from thanos.