Comments (2)
tghartland
commented on July 23, 2024
I was looking at the file downsample.go yesterday which is why I couldn't find any related diff.
Now that I'm looking in compact.go, there is this commit: 68bef3f.
Before this change, the function sy.Metas() was used in the function call of downsampleBucket(...) for both the first and second pass.
The change to add in filtering moves that call to before the first pass, filters the list, and then passes filteredMetas to both function calls.
sy.SyncMetas(ctx) is still called before the second pass, but that does not affect the filteredMetas list and so the same list of blocks is passed to both the first and second pass downsamplings.
I will test and prepare a fix to regenerate the filtered list after the sync preceding the second pass.
from thanos.
tghartland
commented on July 23, 2024
Regenerating the filtered list works, a single downsampled block is produced as expected.
First pass:
ts=2024-06-27T14:19:10.888463617Z caller=compact.go:447 level=info msg="start first pass of downsampling"
ts=2024-06-27T14:19:11.080666564Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=192.18201ms duration_ms=192 cached=88 returned=10 partial=0
ts=2024-06-27T14:19:13.953316251Z caller=downsample.go:371 level=info msg="downloaded block" id=01J19GEH5AKEHPGAVAQXS2E0TD duration=2.872277655s duration_ms=2872
ts=2024-06-27T14:19:15.617337241Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=119.378994ms duration_ms=119 cached=88 returned=88 partial=0
ts=2024-06-27T14:20:04.643177789Z caller=streamed_block_writer.go:178 level=info msg="finalized downsampled block" mint=1719187200072 maxt=1719360000000 ulid=01J1D116NT0KF4NJ5NJQ10DHVG resolution=300000
ts=2024-06-27T14:20:04.643249129Z caller=downsample.go:399 level=info msg="downsampled block" from=01J19GEH5AKEHPGAVAQXS2E0TD to=01J1D116NT0KF4NJ5NJQ10DHVG duration=48.619147549s duration_ms=48619
ts=2024-06-27T14:20:10.787352179Z caller=downsample.go:433 level=info msg="uploaded block" id=01J1D116NT0KF4NJ5NJQ10DHVG duration=4.577254287s duration_ms=4577
Second pass:
ts=2024-06-27T14:20:10.984754236Z caller=compact.go:479 level=info msg="start second pass of downsampling"
ts=2024-06-27T14:20:11.195070096Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=210.268301ms duration_ms=210 cached=89 returned=11 partial=0
ts=2024-06-27T14:20:11.195155859Z caller=compact.go:491 level=info msg="re-filtered list of blocks"
ts=2024-06-27T14:20:11.19546523Z caller=compact.go:508 level=info msg="downsampling iterations done"
No duplicate block is created.
I don't have enough data in this bucket to test if it would do the 5m->1h downsampling in the second pass, but I expect it would.
Without the fix, that would have needed to wait until the next loop of downsampling.
Preparing an MR.
from thanos.
Related Issues (20)
- Regression in thanos v0.35.1 HOT 3
- Thanos Receiver: Router/Ingestor setup no longer returns `thanos_receive_write_timeseries_*` and `thanos_receive_write_samples_*` metrics with thanos v0.35.1 HOT 3
- Extend Thanos bucket rewrite to support filtered archiving of existing blocks
- Support additional aggregates for downsampling
- Store Announced LabelSets Unexpected
- Warning in Grafana 11+: Thanos Receive dashboard depends on Angular
- compactor: Thanos Compactor compaction level number being set to 93+ for 1 half of HA Prometheus (S3 Storage) HOT 3
- Query: Network bandwidth usage upward of 500MB/s between Querier and configured stores HOT 17
- Codespace doesn't seem to be working
- AWS S3 objectstorage is not working for ap-south-2 region inspite of endpoint and region is mentioned and getting default to dualstack endpoint of us-east-1 HOT 7
- Thanos receive running on ext4 FS experiencing compaction failures HOT 10
- query: If i choose time window shorter than 6 months, i don't see downsampled metrics HOT 5
- compact: Thanos Compactor doesn't delete blocks which are marked for deletion HOT 1
- [Bug] Gaps in sum and avg aggregations when joining histogram quantile with pod labels
- store: disk usage continually increasing over time HOT 6
- Thanos store does not show data in downsampled block
- Receive: compaction failure causing query returning irrelevant time series HOT 2
- found duplicate storeEndpoints producer (sidecar or ruler)
- thanos compact panic on GetActiveAndPartialBlockIDs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.