Comments (5)
brian-brazil
commented on May 14, 2024
Could you take a heap dump showing what's taking all the ram?
from blackbox_exporter.
RRAlex
commented on May 14, 2024
Sure could, but I'm not too familiar with the Go ecosystem, how would one proceed? :)
from blackbox_exporter.
brian-brazil
commented on May 14, 2024
https://golang.org/pkg/net/http/pprof/
from blackbox_exporter.
RRAlex
commented on May 14, 2024
After a bit of fighting, I think I've got the master branch running with pprof.
So, using ...
go tool pprof -text ./blackbox_exporter http://localhost:6060/debug/pprof/heap
... soon after starting blackbox-exporter, I get:
Fetching profile from http://localhost:6060/debug/pprof/heap
Saved profile in /home/user/pprof/pprof.blackbox_exporter.localhost:6060.inuse_objects.inuse_space.001.pb.gz
4620.17kB of 4620.17kB total ( 100%)
Dropped 223 nodes (cum <= 23.10kB)
flat flat% sum% cum cum%
1024.03kB 22.16% 22.16% 1024.03kB 22.16% internal/golang.org/x/net/http2/hpack.addDecoderNode
518.02kB 11.21% 33.38% 518.02kB 11.21% crypto/x509.(*CertPool).AddCert
514kB 11.13% 44.50% 514kB 11.13% crypto/tls.(*block).reserve
514kB 11.13% 55.63% 514kB 11.13% net/http.(*Transport).dialConn
513.12kB 11.11% 66.73% 513.12kB 11.11% net/textproto.init.1
512.69kB 11.10% 77.83% 512.69kB 11.10% encoding/pem.Decode
512.28kB 11.09% 88.92% 512.28kB 11.09% compress/flate.generateFixedLiteralEncoding
512.02kB 11.08% 100% 512.02kB 11.08% reflect.unsafe_NewArray
0 0% 100% 512.28kB 11.09% compress/flate.init
0 0% 100% 512.28kB 11.09% compress/gzip.init
0 0% 100% 2056.73kB 44.52% crypto/tls.(*Conn).Handshake
0 0% 100% 2056.73kB 44.52% crypto/tls.(*Conn).clientHandshake
0 0% 100% 514kB 11.13% crypto/tls.(*Conn).readHandshake
0 0% 100% 514kB 11.13% crypto/tls.(*Conn).readRecord
0 0% 100% 514kB 11.13% crypto/tls.(*block).readFromUntil
0 0% 100% 2056.73kB 44.52% crypto/tls.(*clientHandshakeState).doFullHandshake
0 0% 100% 1030.71kB 22.31% crypto/x509.(*CertPool).AppendCertsFromPEM
0 0% 100% 1030.71kB 22.31% crypto/x509.(*Certificate).Verify
0 0% 100% 512.02kB 11.08% crypto/x509.ParseCertificate
0 0% 100% 1030.71kB 22.31% crypto/x509.initSystemRoots
0 0% 100% 512.02kB 11.08% crypto/x509.parseCertificate
0 0% 100% 1030.71kB 22.31% crypto/x509.systemRootsPool
0 0% 100% 512.02kB 11.08% encoding/asn1.Unmarshal
0 0% 100% 512.02kB 11.08% encoding/asn1.UnmarshalWithParams
0 0% 100% 512.02kB 11.08% encoding/asn1.parseField
0 0% 100% 512.02kB 11.08% encoding/asn1.parseSequenceOf
0 0% 100% 1024.03kB 22.16% internal/golang.org/x/net/http2/hpack.init
0 0% 100% 1024.03kB 22.16% internal/golang.org/x/net/http2/hpack.init.1
0 0% 100% 2049.44kB 44.36% main.init
0 0% 100% 513.12kB 11.11% mime/multipart.init
0 0% 100% 2056.73kB 44.52% net/http.(*Transport).dialConn.func3
0 0% 100% 514kB 11.13% net/http.(*Transport).getConn.func4
0 0% 100% 2049.44kB 44.36% net/http.init
0 0% 100% 513.12kB 11.11% net/textproto.init
0 0% 100% 512.02kB 11.08% reflect.MakeSlice
0 0% 100% 4620.17kB 100% runtime.goexit
0 0% 100% 2049.44kB 44.36% runtime.main
0 0% 100% 1030.71kB 22.31% sync.(*Once).Do
And 30 minutes later:
Fetching profile from http://localhost:6060/debug/pprof/heap
Saved profile in /home/user/pprof/pprof.blackbox_exporter.localhost:6060.inuse_objects.inuse_space.004.pb.gz
17959.79kB of 17959.79kB total ( 100%)
Dropped 241 nodes (cum <= 89.80kB)
flat flat% sum% cum cum%
4110.52kB 22.89% 22.89% 4110.52kB 22.89% crypto/tls.(*block).reserve
3592.50kB 20.00% 42.89% 4104.52kB 22.85% net/http.(*Transport).dialConn
2053.50kB 11.43% 54.32% 2053.50kB 11.43% bytes.makeSlice
1026.50kB 5.72% 60.04% 4622.01kB 25.74% crypto/tls.(*Conn).readHandshake
1024.05kB 5.70% 65.74% 1024.05kB 5.70% reflect.unsafe_NewArray
1024.03kB 5.70% 71.44% 1024.03kB 5.70% internal/golang.org/x/net/http2/hpack.addDecoderNode
518.02kB 2.88% 74.33% 518.02kB 2.88% crypto/x509.(*CertPool).AddCert
513.12kB 2.86% 77.18% 513.12kB 2.86% net/textproto.init.1
512.69kB 2.85% 80.04% 512.69kB 2.85% encoding/pem.Decode
512.28kB 2.85% 82.89% 512.28kB 2.85% compress/flate.generateFixedLiteralEncoding
512.23kB 2.85% 85.74% 512.23kB 2.85% net/http.(*Transport).getIdleConnCh
512.19kB 2.85% 88.60% 512.19kB 2.85% runtime.malg
512.09kB 2.85% 91.45% 1024.33kB 5.70% main.probeHTTP
512.02kB 2.85% 94.30% 512.02kB 2.85% encoding/asn1.parseBigInt
512.02kB 2.85% 97.15% 512.02kB 2.85% encoding/asn1.parsePrintableString
512.02kB 2.85% 100% 512.02kB 2.85% syscall.anyToSockaddr
0 0% 100% 2056.01kB 11.45% bufio.(*Reader).Peek
0 0% 100% 2056.01kB 11.45% bufio.(*Reader).fill
0 0% 100% 2053.50kB 11.43% bytes.(*Buffer).Write
0 0% 100% 2053.50kB 11.43% bytes.(*Buffer).grow
0 0% 100% 512.28kB 2.85% compress/flate.init
0 0% 100% 512.28kB 2.85% compress/gzip.init
0 0% 100% 8213.31kB 45.73% crypto/tls.(*Conn).Handshake
0 0% 100% 2056.01kB 11.45% crypto/tls.(*Conn).Read
0 0% 100% 8213.31kB 45.73% crypto/tls.(*Conn).clientHandshake
0 0% 100% 5651.52kB 31.47% crypto/tls.(*Conn).readRecord
0 0% 100% 512.50kB 2.85% crypto/tls.(*Conn).writeRecord
0 0% 100% 3598.02kB 20.03% crypto/tls.(*block).readFromUntil
0 0% 100% 512.50kB 2.85% crypto/tls.(*block).resize
0 0% 100% 7700.81kB 42.88% crypto/tls.(*clientHandshakeState).doFullHandshake
0 0% 100% 1030.71kB 5.74% crypto/x509.(*CertPool).AppendCertsFromPEM
0 0% 100% 1030.71kB 5.74% crypto/x509.(*Certificate).Verify
0 0% 100% 2048.09kB 11.40% crypto/x509.ParseCertificate
0 0% 100% 1030.71kB 5.74% crypto/x509.initSystemRoots
0 0% 100% 1024.03kB 5.70% crypto/x509.parseCertificate
0 0% 100% 1030.71kB 5.74% crypto/x509.systemRootsPool
0 0% 100% 2048.09kB 11.40% encoding/asn1.Unmarshal
0 0% 100% 2048.09kB 11.40% encoding/asn1.UnmarshalWithParams
0 0% 100% 2048.09kB 11.40% encoding/asn1.parseField
0 0% 100% 1536.07kB 8.55% encoding/asn1.parseSequenceOf
0 0% 100% 1024.03kB 5.70% internal/golang.org/x/net/http2/hpack.init
0 0% 100% 1024.03kB 5.70% internal/golang.org/x/net/http2/hpack.init.1
0 0% 100% 2049.44kB 11.41% main.init
0 0% 100% 1024.33kB 5.70% main.main.func2
0 0% 100% 512.02kB 2.85% main.probeHTTP.func1
0 0% 100% 1024.33kB 5.70% main.probeHandler
0 0% 100% 513.12kB 2.86% mime/multipart.init
0 0% 100% 512.02kB 2.85% net.(*Dialer).Dial
0 0% 100% 512.02kB 2.85% net.(*netFD).dial
0 0% 100% 512.02kB 2.85% net.Dial
0 0% 100% 512.02kB 2.85% net.dial
0 0% 100% 512.02kB 2.85% net.dialSerial
0 0% 100% 512.02kB 2.85% net.dialSerial.func1
0 0% 100% 512.02kB 2.85% net.dialSingle
0 0% 100% 512.02kB 2.85% net.dialTCP
0 0% 100% 512.02kB 2.85% net.internetSocket
0 0% 100% 512.02kB 2.85% net.socket
0 0% 100% 512.23kB 2.85% net/http.(*Client).Do
0 0% 100% 512.23kB 2.85% net/http.(*Client).doFollowingRedirects
0 0% 100% 512.23kB 2.85% net/http.(*Client).send
0 0% 100% 1024.33kB 5.70% net/http.(*ServeMux).ServeHTTP
0 0% 100% 512.23kB 2.85% net/http.(*Transport).RoundTrip
0 0% 100% 512.02kB 2.85% net/http.(*Transport).dial
0 0% 100% 8213.31kB 45.73% net/http.(*Transport).dialConn.func3
0 0% 100% 512.23kB 2.85% net/http.(*Transport).getConn
0 0% 100% 4104.52kB 22.85% net/http.(*Transport).getConn.func4
0 0% 100% 1024.33kB 5.70% net/http.(*conn).serve
0 0% 100% 2056.01kB 11.45% net/http.(*noteEOFReader).Read
0 0% 100% 2056.01kB 11.45% net/http.(*persistConn).readLoop
0 0% 100% 1024.33kB 5.70% net/http.HandlerFunc.ServeHTTP
0 0% 100% 2049.44kB 11.41% net/http.init
0 0% 100% 2056.01kB 11.45% net/http.noteEOFReader.Read
0 0% 100% 512.23kB 2.85% net/http.send
0 0% 100% 1024.33kB 5.70% net/http.serverHandler.ServeHTTP
0 0% 100% 513.12kB 2.86% net/textproto.init
0 0% 100% 1024.05kB 5.70% reflect.MakeSlice
0 0% 100% 17447.60kB 97.15% runtime.goexit
0 0% 100% 2049.44kB 11.41% runtime.main
0 0% 100% 512.19kB 2.85% runtime.newproc.func1
0 0% 100% 512.19kB 2.85% runtime.newproc1
0 0% 100% 512.19kB 2.85% runtime.startTheWorldWithSema
0 0% 100% 512.19kB 2.85% runtime.systemstack
0 0% 100% 1030.71kB 5.74% sync.(*Once).Do
0 0% 100% 512.02kB 2.85% syscall.Getsockname
After ~8 more hours:
Fetching profile from http://localhost:6060/debug/pprof/heap
Saved profile in /home/delve/pprof/pprof.blackbox_exporter.localhost:6060.inuse_objects.inuse_space.007.pb.gz
246.06MB of 254.07MB total (96.85%)
Dropped 323 nodes (cum <= 1.27MB)
flat flat% sum% cum cum%
70.75MB 27.85% 27.85% 70.75MB 27.85% crypto/tls.(*block).reserve
63.19MB 24.87% 52.72% 68.19MB 26.84% net/http.(*Transport).dialConn
22.02MB 8.67% 61.38% 37.02MB 14.57% crypto/x509.parseCertificate
19.05MB 7.50% 68.88% 19.05MB 7.50% bytes.makeSlice
16.54MB 6.51% 75.39% 71.73MB 28.23% crypto/tls.(*Conn).readHandshake
14.50MB 5.71% 81.10% 14.50MB 5.71% reflect.unsafe_NewArray
7MB 2.76% 83.86% 7MB 2.76% math/big.nat.make
4.50MB 1.77% 85.63% 4.50MB 1.77% crypto/aes.(*aesCipherGCM).NewGCM
3.50MB 1.38% 87.01% 3.50MB 1.38% net/http.(*Transport).tryPutIdleConn
3.50MB 1.38% 88.38% 3.50MB 1.38% runtime.malg
3.50MB 1.38% 89.76% 3.50MB 1.38% crypto/aes.NewCipher
3MB 1.18% 90.94% 3MB 1.18% net/http.(*Transport).getIdleConnCh
3MB 1.18% 92.12% 3MB 1.18% crypto/x509/pkix.(*Name).FillFromRDNSequence
2MB 0.79% 92.91% 2MB 0.79% net/http.cloneTLSClientConfig
2MB 0.79% 93.70% 2MB 0.79% net.sockaddrToTCP
2MB 0.79% 94.49% 8MB 3.15% encoding/asn1.parseBigInt
1.50MB 0.59% 95.08% 9.50MB 3.74% crypto/tls.aeadAESGCM
1.50MB 0.59% 95.67% 1.50MB 0.59% encoding/asn1.parsePrintableString
1MB 0.39% 96.06% 5.50MB 2.17% main.probeHTTP
0.50MB 0.2% 96.26% 3MB 1.18% net.socket
0.50MB 0.2% 96.45% 1.51MB 0.59% crypto/x509.(*Certificate).Verify
0.50MB 0.2% 96.65% 6.50MB 2.56% crypto/x509.parsePublicKey
0.50MB 0.2% 96.85% 25MB 9.84% encoding/asn1.parseField
0 0% 96.85% 27.61MB 10.87% bufio.(*Reader).Peek
0 0% 96.85% 27.61MB 10.87% bufio.(*Reader).fill
0 0% 96.85% 19.05MB 7.50% bytes.(*Buffer).Write
0 0% 96.85% 19.05MB 7.50% bytes.(*Buffer).grow
0 0% 96.85% 4.50MB 1.77% crypto/cipher.NewGCM
0 0% 96.85% 4.50MB 1.77% crypto/cipher.NewGCMWithNonceSize
0 0% 96.85% 142.77MB 56.19% crypto/tls.(*Conn).Handshake
0 0% 96.85% 27.61MB 10.87% crypto/tls.(*Conn).Read
0 0% 96.85% 142.77MB 56.19% crypto/tls.(*Conn).clientHandshake
0 0% 96.85% 82.79MB 32.59% crypto/tls.(*Conn).readRecord
0 0% 96.85% 8.01MB 3.15% crypto/tls.(*Conn).writeRecord
0 0% 96.85% 62.74MB 24.70% crypto/tls.(*block).readFromUntil
0 0% 96.85% 8.01MB 3.15% crypto/tls.(*block).resize
0 0% 96.85% 124.26MB 48.91% crypto/tls.(*clientHandshakeState).doFullHandshake
0 0% 96.85% 9.50MB 3.74% crypto/tls.(*clientHandshakeState).establishKeys
0 0% 96.85% 51.02MB 20.08% crypto/x509.ParseCertificate
0 0% 96.85% 25MB 9.84% encoding/asn1.Unmarshal
0 0% 96.85% 25MB 9.84% encoding/asn1.UnmarshalWithParams
0 0% 96.85% 16MB 6.30% encoding/asn1.parseSequenceOf
0 0% 96.85% 2MB 0.79% main.init
0 0% 96.85% 5.50MB 2.17% main.main.func2
0 0% 96.85% 3MB 1.18% main.probeHTTP.func1
0 0% 96.85% 5.50MB 2.17% main.probeHandler
0 0% 96.85% 6MB 2.36% math/big.(*Int).SetBytes
0 0% 96.85% 6MB 2.36% math/big.nat.setBytes
0 0% 96.85% 3MB 1.18% net.(*Dialer).Dial
0 0% 96.85% 2.50MB 0.98% net.(*netFD).dial
0 0% 96.85% 3MB 1.18% net.Dial
0 0% 96.85% 3MB 1.18% net.dial
0 0% 96.85% 3MB 1.18% net.dialSerial
0 0% 96.85% 3MB 1.18% net.dialSerial.func1
0 0% 96.85% 3MB 1.18% net.dialSingle
0 0% 96.85% 3MB 1.18% net.dialTCP
0 0% 96.85% 3MB 1.18% net.internetSocket
0 0% 96.85% 4MB 1.57% net/http.(*Client).Do
0 0% 96.85% 4MB 1.57% net/http.(*Client).doFollowingRedirects
0 0% 96.85% 4MB 1.57% net/http.(*Client).send
0 0% 96.85% 5.50MB 2.17% net/http.(*ServeMux).ServeHTTP
0 0% 96.85% 4MB 1.57% net/http.(*Transport).RoundTrip
0 0% 96.85% 3MB 1.18% net/http.(*Transport).dial
0 0% 96.85% 142.77MB 56.19% net/http.(*Transport).dialConn.func3
0 0% 96.85% 4MB 1.57% net/http.(*Transport).getConn
0 0% 96.85% 68.19MB 26.84% net/http.(*Transport).getConn.func4
0 0% 96.85% 5.50MB 2.17% net/http.(*conn).serve
0 0% 96.85% 27.61MB 10.87% net/http.(*noteEOFReader).Read
0 0% 96.85% 31.11MB 12.24% net/http.(*persistConn).readLoop
0 0% 96.85% 3.50MB 1.38% net/http.(*persistConn).readLoop.func2
0 0% 96.85% 5.50MB 2.17% net/http.HandlerFunc.ServeHTTP
0 0% 96.85% 2MB 0.79% net/http.init
0 0% 96.85% 27.61MB 10.87% net/http.noteEOFReader.Read
0 0% 96.85% 4MB 1.57% net/http.send
0 0% 96.85% 5.50MB 2.17% net/http.serverHandler.ServeHTTP
0 0% 96.85% 14.50MB 5.71% reflect.MakeSlice
0 0% 96.85% 249.57MB 98.23% runtime.goexit
0 0% 96.85% 2MB 0.79% runtime.main
0 0% 96.85% 3.50MB 1.38% runtime.newproc.func1
0 0% 96.85% 3.50MB 1.38% runtime.newproc1
0 0% 96.85% 4.50MB 1.77% runtime.startTheWorldWithSema
0 0% 96.85% 4.50MB 1.77% runtime.systemstack
EDIT:
netstat -natp in the blackbox container (~8 hours from start) shows 6k+ ESTABLISHED connections.
2 IP stand out (4k & 2k respectively).
Both are Grafana with HTTPS certificates endpoints (nginx doesn't seem affected)!
There is probably a bug with this setup that keeps the connection established and thus eats more and more ram that is never freed. I can't tell what or why isn't blackbox-exporter cutting the connection after its test, but that sure has to explain why ram is slowly running out if it keeps those open forever...
from blackbox_exporter.
brian-brazil
commented on May 14, 2024
This was likely #100
from blackbox_exporter.
Related Issues (20)
- tls: failed to parse certificate from server: x509: unsupported elliptic curve
- Upgrade grpc dependency HOT 1
- When using DNS as the probe protocol, can the hashed value of the resolving result be added to the metric?
- Probe using proxy_url and skip_resolve_phase_with_proxy doesn't work as wanted
- Allow to cache HTTP clients (useful for slow oauth2 probes)
- Provide further breakdown of "processing" phase metric (for oauth2 probes)
- Request for FIPS-Compliant Image HOT 1
- Issue with Proxy Configuration in Blackbox Exporter HOT 1
- Feature request: add possibility to define discoveries consumed by prometheus
- Feature request - cmdline flag to disable/change url for debug pprof endpoints HOT 3
- blackbox http host check can not work normally HOT 1
- Release v0.25.0 - Container Image HOT 1
- Inconsistent support of IPv6 literals HOT 3
- Feature request: probe to prometheus.io is not helpful, should make options or pulldowns HOT 2
- packaged deb version of blackbox_exporter
- Implement DHCP HOT 2
- Recent Probes List - add timestamp or log id as label and column for better identification
- Twistlock CVE's Found 1/24/23 HOT 2
- Cut a new release? HOT 1
- Same blackbox.yml to monitor different proxies
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blackbox_exporter.