Comments (11)
LordJeffrey
commented on May 28, 2024
Other errors involving "replications":
ERRO[0098] Get http://:8091/pools/default/buckets/presence/stats/replications%2Fa67eb4ce35e01b1573cc1e2261b1d2f2%2Fpresence%2Fpresence%2Fdocs_opt_repd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
from couchbase_exporter.
blakelead
commented on May 28, 2024
Hi @LordJeffrey ,
I'll treat this issue ASAP.
In the mean time, I have a question: when you say that you don't get any stats, do you mean no XDCR stats or no metrics at all ?
from couchbase_exporter.
blakelead
commented on May 28, 2024
I had time to actually test the exporter with Couchbase community 3.0.1 using vagrant, and unfortunately I did not reproduce your errors.
Here's what I did:
I installed Couchbase 3.0.1 on a vagrant ubuntu/trusty64 VM with Docker:
sudo docker run -d --name cb -p 8091-8094:8091-8094 -p 11210:11210 couchbase:community-3.0.1I connected to the Couchbase node in the browser and used default configuration. I then created a remote cluster and initiated a replication between 2 of the examples buckets (beer-sample and default) to test XDCR metrics collection.
I downloaded the version 0.5.2 of the exporter and started it on my machine with the following configuration file:
web:
listenAddress: :9191
telemetryPath: /metrics
db:
user: admin
password: mypassword
uri: http://192.168.10.10:8091
log:
level: debug
format: text
scrape:
cluster: true
node: true
bucket: true
xdcr: trueThe logs I get when requesting the exporter are as follows:
> ./couchbase_exporter
DEBU[0000] Get http://192.168.10.10:8091/pools (6.018162ms)
INFO[0000] Couchbase version: 3.0.1-1444-rel-community
INFO[0000] Community version: true
WARN[0000] Version 3.0.1-1444-rel-community may not be supported by this exporter
DEBU[0000] /Users/aabdelhak/Projets/go/src/github.com/blakelead/couchbase_exporter/metrics/cluster-default.json loaded
DEBU[0000] Cluster exporter registered
DEBU[0000] /Users/aabdelhak/Projets/go/src/github.com/blakelead/couchbase_exporter/metrics/node-default.json loaded
DEBU[0000] Node exporter registered
DEBU[0000] /Users/aabdelhak/Projets/go/src/github.com/blakelead/couchbase_exporter/metrics/bucket-default.json loaded
DEBU[0000] Bucket exporter registered
DEBU[0000] /Users/aabdelhak/Projets/go/src/github.com/blakelead/couchbase_exporter/metrics/bucketstats-default.json loaded
DEBU[0000] Bucketstats exporter registered
DEBU[0000] /Users/aabdelhak/Projets/go/src/github.com/blakelead/couchbase_exporter/metrics/xdcr-default.json loaded
DEBU[0000] XDCR exporter registered
INFO[0000] Listening at :9191
DEBU[0004] Get http://192.168.10.10:8091/pools/default/tasks (4.702954ms)
DEBU[0004] Get http://192.168.10.10:8091/nodes/self (15.828967ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default (21.458141ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets (51.181884ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets (54.048682ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_written (59.044683ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fchanges_left (61.957526ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Frate_received_from_dcp (65.587845ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_filtered (69.969096ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_failed_cr_source (73.142982ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fbandwidth_usage (76.05548ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_rep_queue (80.47983ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fwtavg_meta_latency (82.655471ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fnum_checkpoints (85.58867ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdata_replicated (91.066479ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_checked (100.958467ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fwtavg_docs_latency (103.820349ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Ftime_committing (106.791178ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_received_from_dcp (111.049308ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fdocs_opt_repd (113.51766ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Frate_replicated (115.636506ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fsize_rep_queue (118.916487ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats/replications%2F6735d4da4ef0f0758e89ea83f322f3a5%2Fbeer-sample%2Fdefault%2Fnum_failedckpts (123.003632ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/beer-sample/stats (114.714329ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/default/stats (119.81162ms)
DEBU[0004] Get http://192.168.10.10:8091/pools/default/buckets/gamesim-sample/stats (124.265292ms)
There is something that you could check: the exporter has a 10 second http timeout. Is there any lag in communication between the exporter and Couchbase cluster ?
from couchbase_exporter.
LordJeffrey
commented on May 28, 2024
For me, I wasn't getting any stats. This is very good that you tested this and got stats -- hope I didn't waste your time. I'm going to try to redo all my steps to make sure I have everything right and try again. Thanks so much!
from couchbase_exporter.
LordJeffrey
commented on May 28, 2024
Also, thanks for deleting my comments :)
from couchbase_exporter.
blakelead
commented on May 28, 2024
I'm thankful that you are using my exporter so don't worry, you're not wasting my time :)
Don't hesitate if you have more info about your issue.
from couchbase_exporter.
LordJeffrey
commented on May 28, 2024
Doing some testing today. It seems it IS reporting the new stats, as well as other stats, but it looks like I'm simply/actually timing out. I get this occasionally, at random:
(hostname)8091/pools/default/buckets/presence/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
ERRO[0336] Could not unmarshal bucketstats data for bucket mwi
I get this error for various different stats at random. The most common type involves "repliations":
/pools/default/buckets/presence/stats/replications%2Fa67eb4ce35e01b1573cc1e2261b1d2f2%2Fpresence%2Fpresence%2Fdocs_filtered: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I'm going to look through the documentation to see if there is a way to increase the timeout time. If there isn't, could you post a way? It could be my system is heavily used enough that I'll need more time to deliver the data.
from couchbase_exporter.
LordJeffrey
commented on May 28, 2024
One thing that seems to sort of work is setting this in "couchbase_exporter.go":
// custom server used to set timeouts
httpSrv := &http.Server{
Addr: *listenAddr,
ReadTimeout: 20 * time.Second,
WriteTimeout: 30 * time.Second,
}
I say it only "sort of" works because I still get the timeout errors (after 10 seconds), but the client request I do in my browser doesn't go on forever. With the default settings, if I get the timeout alerts, the client tries again and again causing a loop of errors every 10 seconds. With these settings, it stops after 10 seconds, prints the stats it has, and yes I still get errors. Not sure how all this works, any insight appreciated.
from couchbase_exporter.
blakelead
commented on May 28, 2024
I'll investigate on that and you are right I should parameterize the timeouts.
I'll do that ASAP!
from couchbase_exporter.
LordJeffrey
commented on May 28, 2024
Sweet. I'm going to be out all next week, so no rush (if you were rushing, haha). Cheers.
from couchbase_exporter.
blakelead
commented on May 28, 2024
Hi @LordJeffrey,
I can't reproduce the timeouts you have but I added 2 new parameters in version 0.6.0:
- web.timeout for the exporter itself
- db.timeout for couchbase requests -> I think this is the one you need to increase
I hope this will solve your problem when you get back :)
from couchbase_exporter.
Related Issues (20)
- [0.6.0] environment variable for scraping XDCR doesn't work HOT 1
- [examples] Rework Grafana dashboard template
- Change Licence to Apache 2.0 HOT 1
- Cluster configuration HOT 1
- Memcached moxi exporter HOT 2
- Support Couchbase 6.0 HOT 4
- Support for Couchbase Sync-Gateway +2.5 metrics HOT 2
- Multi URI to same cluster
- Switch to using https for connection to Couchbase cluster HOT 10
- json error HOT 7
- Issue with docker image run HOT 4
- Incorrect binary in linux amd64 release HOT 2
- Is TYPE correct for cb_bucket_ep_oom_errors and cb_bucket_ep_tmp_oom_errors ? HOT 9
- couchbase_export build failed HOT 3
- multiple xdcr crashes the exporter HOT 3
- No Binaries added to Release Assets for 0.9.2 HOT 1
- [ci] migrate to Github Actions
- Difference between this and Couchbase Official Prometheus Exporter HOT 3
- Do not force installing exporters on each node in cluster HOT 3
- can't identify some couchbase cluster metrics HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from couchbase_exporter.