Comments (2)
Maybe you could try the version from main
in your tests and check whether daa34a5 helps? 🤔
from thanos.
@GiedriusS I have tried with main-2024-04-03-f80fd94, I am still seeing following error context deadline exceeded
. I've also tried to configure otel collector timeout. Does it mean I will need to increase aysnc workers --receive.forward.async-workers=x
with number of async workers higher than 5?
2024-04-05T04:19:49.099Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n", "errorCauses": [{"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n"}], "dropped_items": 461230}
from thanos.
Related Issues (20)
- make test fails with GO 1.23 HOT 2
- Proxy: prune Storage Gateway if other stores can provide the data quicker
- Provide support for Perses dashboards in Mixins. HOT 4
- Thanos receive - failed to handle request, internal server error HOT 1
- Thanos is displaying 6h data point data , when we have 1h downscaling HOT 3
- Excessive egress data (gcp cloud storage) after update from 0.34.0 to 0.35.0 or 0.36.0 HOT 6
- Thanos compactor msg="retriable error" HOT 2
- MantineUI: Split router pages according to component template label
- MantineUI: Migrate query tenancy features to new MantineUI
- MantineUI: Port over parse_expr endpoint and make it work with explain/analyze HOT 1
- MantineUI: Migrate Blocks UI to new framework HOT 1
- Mantine UI: Test scalability
- MantineUI: Sharing links seems broken HOT 1
- MantineUI: Make new UI default
- query: "Duplicate store address is provided" HOT 1
- Thanos Receiver: Error logs "try lock file: open <>.tmp-for-creation/index_tmp_po: no such file or directory" HOT 5
- Native Histograms: Downsampling support HOT 1
- Query distributed mode does not fetch from storage gateways HOT 4
- compact: Penalty deduplication very slow during compaction HOT 3
- Skript HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.