Continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.
Home Page: https://parca.dev/
License: Apache License 2.0
![allcontributors[bot] avatar](https://avatars.githubusercontent.com/in/23186?v=4 "allcontributors[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![github-merge-queue[bot] avatar](https://avatars.githubusercontent.com/u/9919?v=4 "github-merge-queue[bot]")
![pre-commit-ci-lite[bot] avatar](https://avatars.githubusercontent.com/in/257897?v=4 "pre-commit-ci-lite[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
Sometimes I click on a blob in the timeline and it opens up a page saying "failed to fetch any source profiles".
I have seen three different log messages that all seem to be associated with (different variants of?) this issue:
2019/11/28 18:03:48 : decompressing profile: gzip: invalid header
2019/11/28 18:04:46 : decompressing profile: unexpected EOF
2019/11/28 18:07:49 : parsing profile: unrecognized profile format
I guess it's always possible that the scrape fails, but it would be better if the UI didn't show a blob at that time.
curl localhost:10902/api/v1/labels
{"status":"success"}
Only status and no data returned. Seems this bug is caused by https://github.com/conprof/conprof/blob/master/api/api.go#L138. Need more investigation.
Assign myself. Will handing the response similar to Thanos https://github.com/thanos-io/thanos/blob/master/pkg/api/api.go#L219
Without any params we got weird messages like failed to parse series labels with error 1:1: parse error: unexpected end of input
It appears the batch merge logic has missed an edge case:
2020/12/14 12:44:47 http: panic serving 10.0.12.3:32822: runtime error: invalid memory address or nil pointer dereference
goroutine 6428 [running]:
net/http.(*conn).serve.func1(0xc001fec140)
/usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x255f9c0, 0x3feabc0)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/google/pprof/profile.equalValueType(...)
/go/pkg/mod/github.com/google/[email protected]/profile/merge.go:478
github.com/google/pprof/profile.(*Profile).compatible(0xc0001c4500, 0xc0001c4280, 0x0, 0x0)
/go/pkg/mod/github.com/google/[email protected]/profile/merge.go:459 +0x54
github.com/google/pprof/profile.combineHeaders(0xc001d2d280, 0xb, 0x10, 0x7f13193ad098, 0x10, 0x1)
/go/pkg/mod/github.com/google/[email protected]/profile/merge.go:409 +0xd1
github.com/google/pprof/profile.Merge(0xc001d2d280, 0xb, 0x10, 0xc0004683c0, 0x0, 0x0)
/go/pkg/mod/github.com/google/[email protected]/profile/merge.go:43 +0x6c
github.com/conprof/conprof/api.(*API).mergeProfiles(0xc00002d170, 0x2f28f00, 0xc001c9f4a0, 0x311d3e00, 0xed76803b3, 0x4018ae0, 0x311d3e00, 0xed7695533, 0x4018ae0, 0xc000c88320, ...)
/app/api/merge.go:111 +0x3c5
github.com/conprof/conprof/api.(*API).profileByParameters(0xc00002d170, 0x2f28f00, 0xc001c9f4a0, 0x29b5c12, 0x5, 0x0, 0x0, 0xc00113a180, 0x71, 0xc0024b77b7, ...)
/app/api/api.go:184 +0x5a5
github.com/conprof/conprof/api.(*API).MergeProfiles(0xc00002d170, 0xc001ff8400, 0x29b4bb0, 0x4)
/app/api/merge.go:126 +0x165
github.com/conprof/conprof/api.(*API).Query(0xc00002d170, 0xc001ff8400, 0xc000877370, 0x40d9f0, 0xc001ffe360, 0x90, 0x90, 0x28ad600)
/app/api/api.go:271 +0x9bb
github.com/conprof/conprof/api.Instr.func1.1(0x2f1fec0, 0xc0003177a0, 0xc001ff8400)
/app/api/error.go:80 +0x36
net/http.HandlerFunc.ServeHTTP(0xc0000588c0, 0x2f1fec0, 0xc0003177a0, 0xc001ff8400)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/thanos-io/thanos/pkg/server/http/middleware.RequestID.func1(0x2f1fec0, 0xc0003177a0, 0xc001ff8300)
/go/pkg/mod/github.com/thanos-io/[email protected]/pkg/server/http/middleware/request_id.go:40 +0x20a
net/http.HandlerFunc.ServeHTTP(0xc0025c5e80, 0x2f1fec0, 0xc0003177a0, 0xc001ff8300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x7f131a19c798, 0xc001fe9180, 0xc001ff8300)
/go/pkg/mod/github.com/!n!y!times/[email protected]/gzip.go:336 +0x225
net/http.HandlerFunc.ServeHTTP(0xc001c97200, 0x7f131a19c798, 0xc001fe9180, 0xc001ff8300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x7f131a19c798, 0xc001fe9130, 0xc001ff8300)
/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:196 +0xe9
net/http.HandlerFunc.ServeHTTP(0xc001c97b90, 0x7f131a19c798, 0xc001fe9130, 0xc001ff8300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x7f131a19c798, 0xc001fe90e0, 0xc001ff8300)
/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:100 +0xda
net/http.HandlerFunc.ServeHTTP(0xc001c9e180, 0x7f131a19c798, 0xc001fe90e0, 0xc001ff8300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1(0x7f131a19c798, 0xc001fe9090, 0xc001ff8300)
/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:163 +0xe9
net/http.HandlerFunc.ServeHTTP(0xc001c9e930, 0x7f131a19c798, 0xc001fe9090, 0xc001ff8300)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1(0x2f1fa40, 0xc000c860e0, 0xc001ff8300)
/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:68 +0x11c
net/http.HandlerFunc.ServeHTTP(...)
/usr/local/go/src/net/http/server.go:2042
github.com/conprof/conprof/api.Instr.func1.2(0x2f1fa40, 0xc000c860e0, 0xc001ff8200, 0x0, 0x0, 0x0)
/app/api/error.go:91 +0x33f
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000233da0, 0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:387 +0xc7e
net/http.(*ServeMux).ServeHTTP(0xc000322940, 0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
main.cors.func1(0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/app/main.go:193 +0x9f
net/http.HandlerFunc.ServeHTTP(0xc000322b40, 0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xc000322b00, 0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc0001dc000, 0x2f1fa40, 0xc000c860e0, 0xc001ff8200)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc001fec140, 0x2f28e40, 0xc001fd7040)
/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c
It's probably best if we didn't use an empty profile as the first profile, but instead the actual first sample.
I like the idea about continuous profiling
I hope you can implement heap profiling visualization
like a stacked time-series data like this
but more detailed to each type of allocation and maybe with a link to source code
If the fetched profile is nil https://github.com/conprof/conprof/blob/master/api/api.go#L334, then it will panic when rendering top and flamegraph.
Referring to this file: https://github.com/conprof/conprof/blob/master/deployments/manifests/configsecret.yaml
There's nothing secret there, nor is base64 a particularly hard cypher.
Wouldn't it be easier to put it in a ConfigMap?
We have an API that exposes what endpoints have been discovered in the all subcommand, but not in the sampler subcommand. We should fix this.
cc @metalmatze
Hey, it’s me again :)
I’ve been thinking about my storage concerns recently (#11 ) and come up with a few ideas. (So far I’ve been successfuly using my naive file based solution with a few locking optimizations, running it with a one week retention / 1 TB of data without tracing on).
Firstly, some stuff_assumptions_opinions which you maybe can confirm/deny, or I’ll have to benchmark to check:
So overall I propose two possible approaches:
With approach no. 1 we could use something like gocloud/blob to provide a few file storage backends (file-based, s3, etc.) basically for free.
Both approaches also come without any locking.
What do you think about this? If you have any initial cautions, answers, or think it’s an outright bad direction, please tell me! (Don’t be afraid to be harsh)
While during scrape we download profiles that are potentially gzipped already and write them directly to the chunk, there is huge potential to compress profiles within chunk as well. Multiple profiles around a similar time period should be very compressible.
We could try something like:
Definitely benchmark and we result. The fact that profiles are gzipped might be a blocker for multi-profile compression.
If try to parse trace profile directly from HTTP response we get this error back:
failed to parse target's trace profile: failed to read event 3 argument at offset 0 (failed to read trace at offset 13621: read 1, error EOF)
Right now the workaround is to read bytes into buffer and only then parse the profile:
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return errors.Wrap(err, "failed to read body")
}
_, err = trace.Parse(io.TeeReader(bytes.NewBuffer(b), w), "")
if err != nil {
return errors.Wrap(err, "failed to parse target's trace profile")
}
This is sort of blocked on golang/go#38971 as right now the package is internal and shouldn't be called from anywhere except for Go internals.
I think we can try to work on a fix, if they expose the package.
Some web page html header with this info would be nice, especially if you have so many tabs opened ;p
I was able to trigger a crash by passing in a valid query that was no longer in the DB.
This line deferences the s value which could be nil from the line above.
Reproduction Steps
Found the following panic while trying to run the project locally.
Output of go version
➜ go version
go version go1.13.8 linux/amd64
Run the quickstart:
➜ git clone [email protected]:conprof/conprof.git
➜ GO111MODULE=on GOPROXY=https://proxy.golang.org go install -v
➜ conprof all --config.file examples/conprof.yaml --web.listen-address 0.0.0.0:10902 --storage.tsdb.path ./data
The error appears too if you run make test (github.com/conprof/conprof/config).
Researched the error a bit, and it starts appearing after #73.
level=info ts=2020-10-13T19:36:54.660385024Z caller=head.go:648 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2020-10-13T19:36:54.660454504Z caller=head.go:662 msg="On-disk memory mappable chunks replay completed" duration=2.54µs
level=info ts=2020-10-13T19:36:54.660471325Z caller=head.go:668 msg="Replaying WAL, this may take a while"
level=info ts=2020-10-13T19:36:54.66093177Z caller=head.go:720 msg="WAL segment loaded" segment=0 maxSegment=32
level=info ts=2020-10-13T19:36:54.661276244Z caller=head.go:720 msg="WAL segment loaded" segment=1 maxSegment=32
level=info ts=2020-10-13T19:36:54.661857341Z caller=head.go:720 msg="WAL segment loaded" segment=2 maxSegment=32
level=info ts=2020-10-13T19:36:54.662097643Z caller=head.go:720 msg="WAL segment loaded" segment=3 maxSegment=32
level=info ts=2020-10-13T19:36:54.664607202Z caller=head.go:720 msg="WAL segment loaded" segment=4 maxSegment=32
level=info ts=2020-10-13T19:36:54.664955677Z caller=head.go:720 msg="WAL segment loaded" segment=5 maxSegment=32
level=info ts=2020-10-13T19:36:54.665107558Z caller=head.go:720 msg="WAL segment loaded" segment=6 maxSegment=32
level=info ts=2020-10-13T19:36:54.665204339Z caller=head.go:720 msg="WAL segment loaded" segment=7 maxSegment=32
level=info ts=2020-10-13T19:36:54.66526041Z caller=head.go:720 msg="WAL segment loaded" segment=8 maxSegment=32
level=info ts=2020-10-13T19:36:54.665333451Z caller=head.go:720 msg="WAL segment loaded" segment=9 maxSegment=32
level=info ts=2020-10-13T19:36:54.665408112Z caller=head.go:720 msg="WAL segment loaded" segment=10 maxSegment=32
level=info ts=2020-10-13T19:36:54.665473143Z caller=head.go:720 msg="WAL segment loaded" segment=11 maxSegment=32
level=info ts=2020-10-13T19:36:54.665548043Z caller=head.go:720 msg="WAL segment loaded" segment=12 maxSegment=32
level=info ts=2020-10-13T19:36:54.665618744Z caller=head.go:720 msg="WAL segment loaded" segment=13 maxSegment=32
level=info ts=2020-10-13T19:36:54.665693005Z caller=head.go:720 msg="WAL segment loaded" segment=14 maxSegment=32
level=info ts=2020-10-13T19:36:54.665773756Z caller=head.go:720 msg="WAL segment loaded" segment=15 maxSegment=32
level=info ts=2020-10-13T19:36:54.668818861Z caller=head.go:720 msg="WAL segment loaded" segment=16 maxSegment=32
level=info ts=2020-10-13T19:36:54.669494709Z caller=head.go:720 msg="WAL segment loaded" segment=17 maxSegment=32
level=info ts=2020-10-13T19:36:54.669672471Z caller=head.go:720 msg="WAL segment loaded" segment=18 maxSegment=32
level=info ts=2020-10-13T19:36:54.670020305Z caller=head.go:720 msg="WAL segment loaded" segment=19 maxSegment=32
level=info ts=2020-10-13T19:36:54.670192557Z caller=head.go:720 msg="WAL segment loaded" segment=20 maxSegment=32
level=info ts=2020-10-13T19:36:54.67044766Z caller=head.go:720 msg="WAL segment loaded" segment=21 maxSegment=32
level=info ts=2020-10-13T19:36:54.670698213Z caller=head.go:720 msg="WAL segment loaded" segment=22 maxSegment=32
level=info ts=2020-10-13T19:36:54.670933656Z caller=head.go:720 msg="WAL segment loaded" segment=23 maxSegment=32
level=info ts=2020-10-13T19:36:54.671190889Z caller=head.go:720 msg="WAL segment loaded" segment=24 maxSegment=32
level=info ts=2020-10-13T19:36:54.671472342Z caller=head.go:720 msg="WAL segment loaded" segment=25 maxSegment=32
level=info ts=2020-10-13T19:36:54.671965558Z caller=head.go:720 msg="WAL segment loaded" segment=26 maxSegment=32
level=info ts=2020-10-13T19:36:54.672251951Z caller=head.go:720 msg="WAL segment loaded" segment=27 maxSegment=32
level=info ts=2020-10-13T19:36:54.672640416Z caller=head.go:720 msg="WAL segment loaded" segment=28 maxSegment=32
level=info ts=2020-10-13T19:36:54.673203792Z caller=head.go:720 msg="WAL segment loaded" segment=29 maxSegment=32
level=info ts=2020-10-13T19:36:54.673518056Z caller=head.go:720 msg="WAL segment loaded" segment=30 maxSegment=32
level=info ts=2020-10-13T19:36:54.67387757Z caller=head.go:720 msg="WAL segment loaded" segment=31 maxSegment=32
level=info ts=2020-10-13T19:36:54.674372576Z caller=head.go:720 msg="WAL segment loaded" segment=32 maxSegment=32
level=info ts=2020-10-13T19:36:54.674400986Z caller=head.go:725 msg="WAL replay completed" checkpoint_replay_duration=118.471µs wal_replay_duration=13.79712ms total_replay_duration=13.945822ms
panic: reflect.StructOf: StructOf does not allow unexported fields [recovered]
panic: reflect.StructOf: StructOf does not allow unexported fields
goroutine 1 [running]:
gopkg.in/yaml%2ev2.handleErr(0xc0011a38b0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/yaml.go:249 +0x9a
panic(0x2179c80, 0x2d3e970)
/usr/lib/go-1.13/src/runtime/panic.go:679 +0x1b2
reflect.runtimeStructField(0xc0001e2660, 0x18, 0x202f4e3, 0x21, 0x2e363e0, 0x229c260, 0x0, 0x0, 0x0, 0x0, ...)
/usr/lib/go-1.13/src/reflect/type.go:2754 +0x1c1
reflect.StructOf(0xc000a47000, 0x1a, 0x27, 0x0, 0x0)
/usr/lib/go-1.13/src/reflect/type.go:2362 +0x2356
github.com/prometheus/prometheus/discovery.getConfigType(0x2e363e0, 0x2628ba0, 0x0, 0x0)
/home/dgzlopes/go/pkg/mod/github.com/prometheus/[email protected]/discovery/registry.go:105 +0x6ea
github.com/prometheus/prometheus/discovery.UnmarshalYAMLWithInlineConfigs(0x232d180, 0xc0002320f0, 0xc00018c1a0, 0xdf8475800, 0xdf8475800)
/home/dgzlopes/go/pkg/mod/github.com/prometheus/[email protected]/discovery/registry.go:125 +0x1ba
github.com/conprof/conprof/config.(*ScrapeConfig).UnmarshalYAML(0xc0002320f0, 0xc00018c1a0, 0x4107af0, 0x0)
/home/dgzlopes/conprof/config/config.go:190 +0x93
gopkg.in/yaml%2ev2.(*decoder).callUnmarshaler(0xc000b30120, 0xc0003890a0, 0x7f2e11e9d520, 0xc0002320f0, 0xc0002320f0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:270 +0xac
gopkg.in/yaml%2ev2.(*decoder).prepare(0xc000b30120, 0xc0003890a0, 0x232d180, 0xc0006600e8, 0x196, 0x8, 0x232d180, 0xc00018c101, 0xc0006600e8)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:313 +0x197
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc000b30120, 0xc0003890a0, 0x232d180, 0xc0006600e8, 0x196, 0x196)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:364 +0xe3
gopkg.in/yaml%2ev2.(*decoder).sequence(0xc000b30120, 0xc000389030, 0x210aae0, 0xc00018c020, 0x197, 0x210aae0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:609 +0x25c
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc000b30120, 0xc000389030, 0x210aae0, 0xc00018c020, 0x197, 0xc00018c020)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:374 +0x183
gopkg.in/yaml%2ev2.(*decoder).mappingStruct(0xc000b30120, 0xc00036dab0, 0x2387fa0, 0xc00018c020, 0x199, 0x1fa0ff6)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:767 +0xa6c
gopkg.in/yaml%2ev2.(*decoder).mapping(0xc000b30120, 0xc00036dab0, 0x2387fa0, 0xc00018c020, 0x199, 0x2387fa0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:626 +0xa94
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc000b30120, 0xc00036dab0, 0x2387fa0, 0xc00018c020, 0x199, 0x0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:372 +0x1af
gopkg.in/yaml%2ev2.(*decoder).document(0xc000b30120, 0xc00036d6c0, 0x2387fa0, 0xc00018c020, 0x199, 0xa0ed50)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:384 +0x7c
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc000b30120, 0xc00036d6c0, 0x2387fa0, 0xc00018c020, 0x199, 0x199)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/decode.go:360 +0x247
gopkg.in/yaml%2ev2.unmarshal(0xc000222990, 0x83, 0x90, 0x22b26e0, 0xc00018c020, 0x1, 0x0, 0x0)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/yaml.go:148 +0x35d
gopkg.in/yaml%2ev2.UnmarshalStrict(...)
/home/dgzlopes/go/pkg/mod/gopkg.in/[email protected]/yaml.go:89
github.com/conprof/conprof/config.Load(0xc0002223f0, 0x83, 0x83, 0x283, 0xc0002223f0)
/home/dgzlopes/conprof/config/config.go:116 +0x99
github.com/conprof/conprof/config.LoadFile(0x7ffe20552d9a, 0x15, 0xc000050300, 0x2, 0x2)
/home/dgzlopes/conprof/config/config.go:130 +0xa4
main.runSampler(0xc00052fe80, 0x2d92160, 0xc0007e8090, 0xc0002ce000, 0x7ffe20552d9a, 0x15, 0xc000050240, 0xc0002ce000, 0x0)
/home/dgzlopes/conprof/sampler.go:100 +0x20f
main.runAll(0xc00052fe80, 0xc00043a3c0, 0x2d92160, 0xc0007e8090, 0x7ffe20552de7, 0x6, 0x7ffe20552d9a, 0x15, 0x49ab483a10000, 0xc000050240, ...)
/home/dgzlopes/conprof/all.go:71 +0x12b
main.registerAll.func1(0xc00052fe80, 0xc00043a3c0, 0x2d92160, 0xc0007e8090, 0xc0007fc000, 0x0, 0x0, 0x0, 0x2dfffa0, 0x2dfffa0)
/home/dgzlopes/conprof/all.go:40 +0x92
main.main()
/home/dgzlopes/conprof/main.go:119 +0xe88
I would like to proxy conprof, so that the URL on the outside is like http://example.com/conprof.
Is there something like Prometheus --web.external-url flag?
Hello, bro.
I want to start the cluster for HA, How I should do?
Hey,
I ran conprof with this config:
scrape_configs:
- job_name: conprof
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_conprof_scrape]
action: keep
regex: true
- source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
scrape_interval: 1m
scrape_timeout: 1mand I attached a 500Gig persistent volume which it uses to write it's data to. The config targets 10 pods and it took only 3-4 hours until the 500Gig were completely full. Is this expected / what do you recommend to configure so that conprof doesn't need that much storage?
conprof-0 conprof level=debug ts=2020-04-28T04:21:13.165040571Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.2.38:8080/debug/pprof/trace
conprof-0 conprof level=debug ts=2020-04-28T04:21:14.18123509Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.2.38:8080/debug/pprof/trace msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
Same for cmdline profile:
conprof-0 conprof level=debug ts=2020-04-28T04:23:02.305811774Z caller=scrape.go:370 component=scrape-manager scrape_pool=conprof msg="scraping profile" url=http://10.200.0.177:8080/debug/pprof/cmdline
conprof-0 conprof level=debug ts=2020-04-28T04:23:02.310718078Z caller=scrape.go:505 component=scrape-manager scrape_pool=conprof target=http://10.200.0.177:8080/debug/pprof/cmdline msg="Scrape failed" err="failed to parse target's pprof profile: parsing profile: unrecognized profile format"
Might be a consequence of running on arm
I started up conprof built with -race and almost immediately got complaints:
or/github.com/conprof/db/tsdb/chunkenc/byte.go:172 +0x48f
github.com/conprof/db/tsdb.(*memSeries).append()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:2130 +0x198
github.com/conprof/db/tsdb.(*Head).processWALSamples()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:372 +0x3fd
github.com/conprof/db/tsdb.(*Head).loadWAL.func5()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:466 +0x66
Previous write at 0x00c000f62010 by goroutine 55:
runtime.slicecopy()
/Users/josh/go/1.15/src/runtime/slice.go:246 +0x0
github.com/conprof/db/tsdb/wal.(*Reader).next()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/wal/reader.go:134 +0xe04
github.com/conprof/db/tsdb/wal.(*Reader).Next()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/wal/reader.go:45 +0x44
github.com/conprof/db/tsdb.(*Head).loadWAL.func6()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:474 +0x9d
Goroutine 34 (running) created at:
github.com/conprof/db/tsdb.(*Head).loadWAL()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:465 +0x549
github.com/conprof/db/tsdb.(*Head).Init()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:713 +0xdf0
github.com/conprof/db/tsdb.open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:660 +0xb84
github.com/conprof/db/tsdb.Open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:530 +0xe4
main.runAll()
/Users/josh/x/conprof/all.go:46 +0x20c
main.registerAll.func1()
/Users/josh/x/conprof/all.go:41 +0x1d4
main.main()
/Users/josh/x/conprof/main.go:131 +0x1f4e
Goroutine 55 (running) created at:
github.com/conprof/db/tsdb.(*Head).loadWAL()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:472 +0x70d
github.com/conprof/db/tsdb.(*Head).Init()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:713 +0xdf0
github.com/conprof/db/tsdb.open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:660 +0xb84
github.com/conprof/db/tsdb.Open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:530 +0xe4
main.runAll()
/Users/josh/x/conprof/all.go:46 +0x20c
main.registerAll.func1()
/Users/josh/x/conprof/all.go:41 +0x1d4
main.main()
/Users/josh/x/conprof/main.go:131 +0x1f4e
==================
==================
WARNING: DATA RACE
Read at 0x00c000f6ff40 by goroutine 34:
runtime.slicecopy()
/Users/josh/go/1.15/src/runtime/slice.go:246 +0x0
github.com/conprof/db/tsdb/chunkenc.(*bytesAppender).Append()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/chunkenc/byte.go:172 +0x48f
github.com/conprof/db/tsdb.(*memSeries).append()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:2130 +0x198
github.com/conprof/db/tsdb.(*Head).processWALSamples()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:372 +0x3fd
github.com/conprof/db/tsdb.(*Head).loadWAL.func5()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:466 +0x66
Previous write at 0x00c000f6ff45 by goroutine 55:
[failed to restore the stack]
Goroutine 34 (running) created at:
github.com/conprof/db/tsdb.(*Head).loadWAL()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:465 +0x549
github.com/conprof/db/tsdb.(*Head).Init()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:713 +0xdf0
github.com/conprof/db/tsdb.open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:660 +0xb84
github.com/conprof/db/tsdb.Open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:530 +0xe4
main.runAll()
/Users/josh/x/conprof/all.go:46 +0x20c
main.registerAll.func1()
/Users/josh/x/conprof/all.go:41 +0x1d4
main.main()
/Users/josh/x/conprof/main.go:131 +0x1f4e
Goroutine 55 (running) created at:
github.com/conprof/db/tsdb.(*Head).loadWAL()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:472 +0x70d
github.com/conprof/db/tsdb.(*Head).Init()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/head.go:713 +0xdf0
github.com/conprof/db/tsdb.open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:660 +0xb84
github.com/conprof/db/tsdb.Open()
/Users/josh/x/conprof/vendor/github.com/conprof/db/tsdb/db.go:530 +0xe4
main.runAll()
/Users/josh/x/conprof/all.go:46 +0x20c
main.registerAll.func1()
/Users/josh/x/conprof/all.go:41 +0x1d4
main.main()
/Users/josh/x/conprof/main.go:131 +0x1f4e
==================
Maybe a cause of #36?
I would like to proxy conprof, so that the URL on the outside is like http://example.com/conprof.
Can we add something like Prometheus --web.external-url flag?
#39 was closed, and the UI now paints, but if I try to run a query it makes a request like /api/v1/query_range?from=1614161843691&to=1614165535095&query=%7Bjob%3D%22foo%22%7D to the root of the webserver.
Recently fgprof is added to Thanos in endpoint /debug/fgprof. I am wondering how to enable scraping this endpoint at the same time with profile.
fgprof is similar to pprof profile, so I can configure something like below to enable fgprof, but this cannot enable profile at the same time.
scrape_configs:
- job_name: 'conprof'
scrape_interval: 1m
scrape_timeout: 1m
profiling_config:
pprof_config:
profile:
path: /debug/fgprof
enabled: true
static_configs:
- targets: ['localhost:6060']
I got duplicate series for heap{job="conprof"}. Should they be the same series?
Not sure if this is a candidate for memory profiling in Rust:
https://github.com/koute/memory-profiler
- Gathers every allocation and deallocation, along with full stack traces
...- Can dynamically stream the profiling data to another machine instead of saving it locally, which is useful for profiling on memory-constrained systems
Output is currently in JSON and Heaptrack formats so would need a new format. Not sure what else?
I restarted conprof and it has been silent for 25 minutes; if it was listening on /debug/pprof I could find out what it is doing. Also it should respond to /metrics.
Attempting to set some profiles on and some off results in no profiling and no diagnostic.
E.g.
profiling_config:
pprof_config:
allocs:
enabled: false
block:
enabled: false
cmdline:
enabled: false
goroutine:
enabled: true
path: "/debug/pprof/goroutine"
heap:
enabled: true
path: "/debug/pprof/heap"
mutex:
enabled: false
profile:
enabled: false
threadcreate:
enabled: false
trace:
enabled: false
(I have a PR to fix this, will post soon)
This was the result of me starting conprof with a freshly-created EBS volume.
I worked round it by running chown -R nobody /conprof on that volume as root, but is there a better way?
excerpts from config:
volumeClaimTemplates:
- metadata:
name: storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: gp2
volumeMode: Filesystem
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
parameters:
fsType: ext4
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
Seems to be related to go modules in some way? /shrug
go version go1.12.4 linux/amd64
GO111MODULE=on go get -u github.com/conprof/conprof
go: finding github.com/conprof/conprof latest
go: finding github.com/shurcooL/vfsgen v0.0.0-20181202132449-6a9ea43bcacd
go: finding github.com/opentracing/opentracing-go v1.1.0
go: finding github.com/prometheus/procfs v0.0.0-20190522114515-bc1a522cf7b1
go: finding github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6
go: finding github.com/oklog/run v1.0.0
go: finding github.com/mwitkow/go-conntrack latest
go: finding github.com/spaolacci/murmur3 v1.1.0
go: finding github.com/shurcooL/httpfs v0.0.0-20181222201310-74dc9339e414
go: finding github.com/shurcooL/vfsgen latest
go: finding github.com/shurcooL/httpfs latest
go: finding golang.org/x/tools v0.0.0-20190521203540-521d6ed310dd
go: finding gopkg.in/check.v1 latest
go: finding github.com/prometheus/common v0.4.1
go: finding github.com/prometheus/client_golang v0.9.3
go: finding github.com/prometheus/procfs latest
go: finding golang.org/x/net v0.0.0-20190520210107-018c4d40a106
go: finding github.com/ianlancetaylor/demangle latest
go: finding github.com/conprof/tsdb v0.8.1-0.20190522132737-ee822c001f9a
go: finding github.com/google/pprof v0.0.0-20190515194954-54271f7e092f
go: finding github.com/prometheus/prometheus v0.0.0-20190328180107-4d60eb36dcbe
go: finding golang.org/x/sync latest
go: finding golang.org/x/sys v0.0.0-20190522044717-8097e1b27ff5
go: finding golang.org/x/tools latest
go: finding golang.org/x/net latest
go: finding github.com/stretchr/objx v0.2.0
go: finding github.com/google/pprof latest
go: finding golang.org/x/sys latest
go: finding github.com/conprof/tsdb v0.8.0
go: finding golang.org/x/crypto latest
go: finding github.com/alecthomas/units latest
go: finding github.com/alecthomas/template latest
go: finding golang.org/x/text v0.3.2
go: finding github.com/prometheus/common v0.4.0
go: finding github.com/prometheus/client_model latest
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.2
go: finding github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084
go: finding github.com/sirupsen/logrus v1.4.2
go: finding golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
go: finding golang.org/x/sys v0.0.0-20190412213103-97732733099d
go: finding golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e
go: finding github.com/prometheus/tsdb v0.7.1
go: finding github.com/dgryski/go-sip13 latest
go: finding github.com/kr/logfmt latest
go: finding github.com/kisielk/errcheck v1.2.0
go: finding golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: finding golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563
go: finding github.com/prometheus/tsdb v0.8.0
go: finding github.com/biogo/store latest
go: finding github.com/hashicorp/yamux latest
go: finding github.com/hashicorp/memberlist v0.1.4
go: finding github.com/miekg/dns v1.1.13
go: finding github.com/jackc/fake latest
go: finding github.com/golang/groupcache latest
go: finding contrib.go.opencensus.io/exporter/ocagent v0.5.0
go: finding github.com/sean-/seed latest
go: finding github.com/google/btree v1.0.0
go: finding github.com/evanphx/json-patch v4.2.0+incompatible
go: finding github.com/opentracing-contrib/go-stdlib latest
go: finding github.com/go-ini/ini v1.42.0
go: finding github.com/jmespath/go-jmespath latest
go: finding labix.org/v2/mgo latest
go: finding github.com/cockroachdb/cockroach-go latest
go: finding github.com/oklog/oklog v0.3.2
go: finding github.com/onsi/gomega v1.5.0
go: finding gopkg.in/tomb.v1 latest
go: finding github.com/codahale/hdrhistogram latest
go: finding k8s.io/api latest
go: finding github.com/petermattis/goid latest
go: finding github.com/json-iterator/go v1.1.6
go: finding google.golang.org/api v0.5.0
go: finding github.com/getsentry/raven-go v0.2.0
go: finding github.com/cenk/backoff v2.1.1+incompatible
go: finding github.com/hashicorp/consul v1.5.1
go: finding k8s.io/klog v0.3.2
go: finding github.com/shopspring/decimal latest
go: finding github.com/mitchellh/reflectwalk v1.0.1
go: finding k8s.io/kube-openapi latest
go: finding google.golang.org/grpc v1.21.0
go: finding github.com/jtolds/gls v4.20.0+incompatible
go: finding github.com/smartystreets/goconvey latest
go: finding github.com/hashicorp/go-rootcerts v1.0.0
go: finding cloud.google.com/go v0.39.0
go: finding github.com/mitchellh/go-homedir v1.1.0
go: finding github.com/peterbourgon/g2s latest
go: finding github.com/lib/pq v1.1.1
go: finding github.com/modern-go/concurrent latest
go: finding honnef.co/go/tools latest
go: finding k8s.io/apimachinery latest
go: finding github.com/certifi/gocertifi latest
go: finding github.com/influxdata/influxdb v1.7.6
go: finding github.com/montanaflynn/stats v0.5.0
go: finding github.com/elastic/gosigar v0.10.3
go: finding github.com/smartystreets/assertions latest
go: finding github.com/hashicorp/go-msgpack v0.5.5
go: finding google.golang.org/genproto latest
go: finding github.com/gophercloud/gophercloud v0.1.0
go: finding github.com/google/go-cmp v0.3.0
go: finding github.com/prometheus/prometheus v2.5.0+incompatible
go: finding github.com/sasha-s/go-deadlock v0.2.0
go: finding github.com/grpc-ecosystem/grpc-opentracing latest
go: finding github.com/facebookgo/clock latest
go: finding github.com/pascaldekloe/goe v0.1.0
go: finding github.com/samuel/go-zookeeper latest
go: finding github.com/Azure/go-autorest v12.1.0+incompatible
go: finding github.com/grpc-ecosystem/grpc-gateway v1.9.0
go: finding google.golang.org/appengine v1.6.0
go: finding github.com/StackExchange/wmi latest
go: finding github.com/hashicorp/serf v0.8.3
go: finding github.com/golang/glog latest
go: finding google.golang.org/api v0.4.0
go: finding golang.org/x/time latest
go: finding golang.org/x/oauth2 latest
go: finding github.com/lightstep/lightstep-tracer-go v0.16.0
go: finding launchpad.net/gocheck latest
go: finding github.com/hashicorp/go-immutable-radix v1.1.0
go: finding github.com/aws/aws-sdk-go v1.19.39
go: finding github.com/knz/strtime latest
go: finding github.com/cockroachdb/cockroach v19.1.1+incompatible
go: finding github.com/rcrowley/go-metrics latest
go: finding github.com/kr/pty v1.1.4
go: finding github.com/hashicorp/mdns v1.0.1
go: finding github.com/hashicorp/go-checkpoint v0.0.0-20171009173528-1545e56e46de
go: finding golang.org/x/lint latest
go: finding github.com/ory/dockertest v3.3.4+incompatible
go: finding github.com/elazarl/go-bindata-assetfs v0.0.0-20160803192304-e1a2a7ec64b0
go: finding github.com/gogo/googleapis v1.2.0
go: finding github.com/google/gofuzz v1.0.0
go: finding github.com/hashicorp/go-discover v0.0.0-20190403160810-22221edb15cd
go: finding github.com/hashicorp/vault-plugin-secrets-kv v0.0.0-20190318174639-195e0e9d07f1
go: finding github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869
go: finding github.com/patrickmn/go-cache v0.0.0-20180527043350-9f6ff22cfff8
go: finding gopkg.in/fsnotify/fsnotify.v1 v1.4.7
go: finding github.com/bmizerany/assert latest
go: finding github.com/hashicorp/go-bexpr v0.1.0
go: finding github.com/coreos/etcd v3.3.13+incompatible
go: finding github.com/patrickmn/go-cache v2.1.0+incompatible
go: finding github.com/shirou/gopsutil v0.0.0-20181107111621-48177ef5f880
go: finding github.com/hashicorp/raft v1.0.1-0.20190409200437-d9fe23f7d472
go: finding k8s.io/client-go v11.0.0+incompatible
go: finding github.com/go-test/deep v1.0.1
go: finding github.com/hashicorp/go-discover latest
go: finding github.com/prometheus/client_golang v0.0.0-20180328130430-f504d69affe1
go: finding github.com/hashicorp/go-memdb v0.0.0-20180223233045-1289e7fffe71
go: finding github.com/hashicorp/go-hclog v0.0.0-20180402200405-69ff559dc25f
go: finding github.com/keybase/go-crypto v0.0.0-20180614160407-5114a9a81e1b
go: finding github.com/coredns/coredns v1.1.2
go: finding gopkg.in/ory-am/dockertest.v3 v3.3.4
go: finding github.com/hashicorp/vault-plugin-secrets-kv v0.5.1
go: finding github.com/hashicorp/hcl v0.0.0-20180906183839-65a6292f0157
go: finding github.com/hashicorp/go-hclog v0.9.2
go: finding k8s.io/apimachinery v0.0.0-20190223001710-c182ff3b9841
go: finding github.com/hashicorp/go-memdb v1.0.3
go: finding github.com/duosecurity/duo_api_golang v0.0.0-20190308151101-6c680f768e74
go: finding github.com/NYTimes/gziphandler v1.1.1
go: finding github.com/keybase/go-crypto latest
go: finding github.com/hashicorp/raft v1.1.0
go: finding github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932
go: finding github.com/hashicorp/go-version v0.0.0-20170202080759-03c5bf6be031
go: finding github.com/packethost/packngo v0.1.0
go: finding github.com/opencontainers/runc v0.1.1
go: finding github.com/hashicorp/hil v0.0.0-20160711231837-1e86c6b523c5
go: finding github.com/duosecurity/duo_api_golang latest
go: finding github.com/Jeffail/gabs v1.1.0
go: finding github.com/imdario/mergo v0.3.7
go: finding github.com/bitly/go-hostpool latest
go: finding github.com/fatih/structs v0.0.0-20180123065059-ebf56d35bba7
go: finding github.com/coredns/coredns v1.5.0
go: finding github.com/gopherjs/gopherjs latest
go: finding github.com/hashicorp/go-version v1.2.0
go: finding go.opencensus.io v0.21.0
go: finding github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5
go: finding github.com/google/go-querystring v1.0.0
go: finding github.com/hashicorp/hil latest
go: finding github.com/Jeffail/gabs v1.4.0
go: finding github.com/fatih/structs v1.1.0
go: finding github.com/hashicorp/consul/api v1.1.0
go: finding github.com/softlayer/softlayer-go latest
go: finding github.com/linode/linodego v0.7.1
go: finding github.com/gregjones/httpcache latest
go: finding github.com/jarcoal/httpmock v1.0.4
go: finding github.com/cockroachdb/cmux latest
go: finding github.com/hashicorp/consul/sdk v0.1.1
go: finding github.com/cenkalti/backoff v2.1.1+incompatible
go: finding github.com/Nvveen/Gotty latest
go: finding github.com/Microsoft/go-winio v0.4.3
go: finding github.com/armon/go-metrics latest
go: finding github.com/prometheus/common v0.0.0-20180326160409-38c53a9f4bfc
go: finding github.com/circonus-labs/circonusllhist v0.0.0-20161110002650-365d370cc145
go: finding github.com/nicolai86/scaleway-sdk v1.11.1
go: finding github.com/linode/linodego v0.9.0
go: finding github.com/envoyproxy/go-control-plane v0.0.0-20180919002855-2137d9196328
go: finding github.com/renier/xmlrpc latest
go: finding golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c
go: finding github.com/vmware/govmomi v0.20.1
go: finding github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6
go: finding github.com/docker/go-connections v0.3.0
go: finding github.com/Microsoft/go-winio v0.4.12
go: finding github.com/hashicorp/go-plugin v0.0.0-20180331002553-e8d22c780116
go: finding github.com/SAP/go-hdb v0.12.0
go: finding github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78
go: finding k8s.io/api v0.0.0-20190325185214-7544f9db76f6
go: finding github.com/lyft/protoc-gen-validate v0.0.0-20180911180927-64fcb82c878e
go: finding golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
go: finding github.com/hashicorp/vic v1.5.1-0.20190403131502-bbfe86ec9443
go: finding golang.org/x/exp latest
go: finding github.com/Azure/go-ansiterm latest
go: finding github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
go: finding golang.org/x/net v0.0.0-20190403144856-b630fd6fe46b
go: finding golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3
go: finding github.com/denisenkom/go-mssqldb v0.0.0-20180620032804-94c9c97e8c9f
go: finding github.com/hashicorp/go-plugin v1.0.0
go: finding github.com/envoyproxy/go-control-plane v0.8.0
go: finding github.com/SAP/go-hdb v0.14.1
go: finding github.com/onsi/gomega v1.4.2
go: finding github.com/joyent/triton-go latest
go: finding github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5
go: finding github.com/gogo/protobuf v0.0.0-20171007142547-342cbe0a0415
go: finding github.com/golang/mock v1.3.1
go: finding github.com/gotestyourself/gotestyourself v2.2.0+incompatible
go: finding github.com/tent/http-link-go latest
go: finding golang.org/x/sys v0.0.0-20190508220229-2d0786266e9c
go: finding golang.org/x/tools v0.0.0-20190328211700-ab21143f2384
go: finding github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
go: finding github.com/abdullin/seq latest
go: finding gopkg.in/DataDog/dd-trace-go.v0 v0.6.1
go: finding k8s.io/client-go v10.0.0+incompatible
go: finding github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
go: finding github.com/mitchellh/hashstructure v0.0.0-20170609045927-2bca23e0e452
go: finding github.com/denisenkom/go-mssqldb latest
go: finding github.com/coreos/go-systemd v0.0.0-20190212144455-93d5ec2c7f76
go: finding github.com/lyft/protoc-gen-validate v0.0.14
go: finding github.com/jstemmer/go-junit-report latest
go: finding go.uber.org/multierr v1.1.0
go: finding github.com/coreos/pkg latest
go: finding github.com/SermoDigital/jose v0.0.0-20180104203859-803625baeddc
go: finding golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
go: finding google.golang.org/appengine v1.2.0
go: finding github.com/gophercloud/gophercloud v0.0.0-20190307220656-fe1ba5ce12dd
go: finding github.com/googleapis/gax-go/v2 v2.0.4
go: finding golang.org/x/tools v0.0.0-20190311212946-11955173bddd
go: finding github.com/envoyproxy/protoc-gen-validate v0.0.14
go: finding github.com/estesp/manifest-tool v0.9.0
go: finding github.com/coreos/go-systemd latest
go: finding github.com/SermoDigital/jose v0.9.1
go: finding github.com/digitalocean/godo v1.15.0
go: finding github.com/prometheus/procfs v0.0.0-20180408092902-8b1c2da0d56d
go: finding github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0
go: finding github.com/mitchellh/gox v1.0.1
go: finding google.golang.org/grpc v0.0.0-20180920234847-8997b5fa0873
go: finding github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501
go: finding k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6
go: finding github.com/gorilla/websocket v1.4.0
go: finding go.uber.org/zap v1.9.1
go: finding gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d
go: finding github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46
go: finding github.com/hashicorp/vic v1.5.0
go: finding github.com/circonus-labs/circonus-gometrics v0.0.0-20161109192337-d17a8420c36e
go: finding github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5
go: finding github.com/go-openapi/spec v0.19.0
go: finding golang.org/x/net v0.0.0-20170114055629-f2499483f923
go: finding github.com/openzipkin/zipkin-go-opentracing v0.3.4
go: finding go.uber.org/zap v1.10.0
go: finding k8s.io/gengo latest
go: finding github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1
go: finding cloud.google.com/go v0.37.4
go: finding gopkg.in/asn1-ber.v1 latest
go: finding cloud.google.com/go v0.28.0
go: finding github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible
go: finding github.com/tmc/grpc-websocket-proxy latest
go: finding github.com/hashicorp/vault v0.10.3
go: finding github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a
go: finding github.com/mattn/go-isatty v0.0.8
go: finding github.com/go-openapi/jsonpointer v0.19.0
go: finding github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680
go: finding github.com/grpc-ecosystem/grpc-gateway v1.8.3
go: finding github.com/openzipkin/zipkin-go-opentracing v0.3.5
go: finding github.com/rogpeppe/fastuuid v1.1.0
go: finding gopkg.in/square/go-jose.v2 v2.3.1
go: finding sigs.k8s.io/structured-merge-diff v0.0.0-20190426204423-ea680f03cc65
go: finding github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367
go: finding github.com/stretchr/testify v0.0.0-20151208002404-e3a8ff8ce365
go: finding github.com/mailru/easyjson latest
go: finding github.com/Shopify/sarama v1.21.0
go: finding github.com/onsi/ginkgo v1.8.0
go: finding github.com/armon/circbuf latest
go: finding sigs.k8s.io/structured-merge-diff latest
go: finding github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c
go: finding github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: finding github.com/shirou/w32 v0.0.0-20160930032740-bb4de0191aa4
go: finding github.com/Shopify/sarama v1.22.1
go: finding github.com/docker/go-units v0.3.3
go: finding github.com/digitalocean/godo v1.10.0
go: finding golang.org/x/tools v0.0.0-20190425150028-36563e24a262
go: finding google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
go: finding github.com/shirou/w32 latest
go: finding github.com/PuerkitoBio/purell v1.0.0
go: finding github.com/hashicorp/raft-boltdb v0.0.0-20150201200839-d1e82c1ec3f1
go: finding github.com/jonboulle/clockwork v0.1.0
go: finding github.com/docker/go-units v0.4.0
go: finding github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c
go: finding golang.org/x/crypto v0.0.0-20190228161510-8dd112bcdc25
go: finding github.com/google/go-github v17.0.0+incompatible
go: finding k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92
go: finding github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: finding github.com/PuerkitoBio/purell v1.1.1
go: finding github.com/hashicorp/raft-boltdb latest
go: finding github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff
go: finding github.com/hashicorp/vault v1.1.2
go: finding github.com/aws/aws-sdk-go v1.14.17
go: finding github.com/xiang90/probing latest
go: finding github.com/mholt/caddy v0.11.5
go: finding k8s.io/kube-openapi v0.0.0-20190306001800-15615b16d372
go: finding golang.org/x/image v0.0.0-20190227222117-0694c2d4d067
go: finding istio.io/gogo-genproto v0.0.0-20190124151557-6d926a6e6feb
go: finding github.com/asaskevich/govalidator v0.0.0-20180319081651-7d2e70ef918f
go: finding github.com/coreos/etcd v3.3.11+incompatible
go: finding github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802
go: finding github.com/DataDog/zstd v1.3.6-0.20190409195224-796139022798
go: finding golang.org/x/image latest
go: finding github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87
go: finding github.com/mholt/caddy v1.0.0
go: finding github.com/asaskevich/govalidator latest
go: finding github.com/DataDog/dd-trace-go v0.6.1
go: finding istio.io/gogo-genproto latest
go: finding github.com/miekg/dns v1.1.6
go: finding github.com/BurntSushi/xgb latest
go: finding github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9
go: finding github.com/hashicorp/go-msgpack v0.5.4
go: finding github.com/posener/complete v1.2.1
go: finding github.com/ryanuber/go-glob v0.0.0-20170128012129-256dc444b735
go: finding github.com/go-openapi/swag v0.19.0
go: finding github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7
go: finding github.com/DataDog/zstd v1.4.0
go: finding github.com/Azure/azure-sdk-for-go v30.0.0+incompatible
go: finding github.com/go-openapi/jsonreference v0.19.0
go: finding github.com/DataDog/dd-trace-go v1.14.0
go: finding github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d
go: finding github.com/DataDog/zstd v1.3.5
go: finding golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a
go: finding github.com/opencontainers/image-spec v1.0.1
go: finding github.com/golang/lint latest
go: finding github.com/opencontainers/go-digest v1.0.0-rc1
go: finding github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go: finding github.com/hashicorp/net-rpc-msgpackrpc latest
go: finding golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea
go: finding github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
go: finding github.com/munnerz/goautoneg latest
go: finding github.com/hashicorp/go-hclog v0.9.1
go: finding github.com/google/uuid v1.1.1
go: finding google.golang.org/genproto v0.0.0-20190508193815-b515fa19cec8
go: finding github.com/xdg/stringprep v1.0.0
go: finding github.com/coreos/go-semver v0.2.0
go: finding github.com/gocql/gocql v0.0.0-20180617115710-e06f8c1bcd78
go: finding github.com/DataDog/datadog-go v0.0.0-20160329135253-cc2f4770f4d6
go: finding golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e
go: finding github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd
go: finding github.com/go-ldap/ldap v3.0.2+incompatible
go: finding go.etcd.io/bbolt v1.3.2
go: finding github.com/eapache/go-xerial-snappy latest
go: finding github.com/coreos/go-semver v0.3.0
go: finding golang.org/x/sys v0.0.0-20190312061237-fead79001313
go: finding gopkg.in/ini.v1 v1.42.0
go: finding github.com/DataDog/datadog-go v2.2.0+incompatible
go: finding github.com/emicklei/go-restful v2.9.5+incompatible
go: finding github.com/modern-go/reflect2 v0.0.0-20180320133207-05fbef0ca5da
go: finding github.com/googleapis/gnostic v0.0.0-20170426233943-68f4ded48ba9
go: finding github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c
go: finding github.com/go-openapi/swag v0.17.0
go: finding github.com/pierrec/lz4 v0.0.0-20190327172049-315a67e90e41
go: finding github.com/gocql/gocql latest
go: finding github.com/coreos/bbolt v1.3.2
go: finding github.com/denverdino/aliyungo latest
go: finding go.uber.org/atomic v1.4.0
go: finding github.com/dnstap/golang-dnstap v0.0.0-20170829151710-2cf77a2b5e11
go: finding github.com/jackc/pgx v3.4.0+incompatible
go: finding github.com/xdg/scram latest
go: finding github.com/go-ldap/ldap v3.0.3+incompatible
go: finding github.com/farsightsec/golang-framestream v0.0.0-20181102145529-8a0cb8ba8710
go: github.com/golang/[email protected]: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding golang.org/x/net v0.0.0-20181005035420-146acd28ed58
go: finding golang.org/x/sys v0.0.0-20190129075346-302c3dd5f1cc
go: finding github.com/mitchellh/copystructure v0.0.0-20160804032330-cdac8253d00f
go: finding github.com/mattn/go-colorable v0.1.2
go: finding github.com/mitchellh/go-testing-interface v0.0.0-20171004221916-a61a99592b77
go: finding github.com/go-openapi/jsonreference v0.17.0
go: finding github.com/davecgh/go-spew v0.0.0-20151105211317-5215b55f46b2
go: finding github.com/jefferai/jsonx v0.0.0-20160721235117-9cc31c3135ee
go: finding gopkg.in/resty.v1 v1.9.1
go: finding github.com/golang/snappy v0.0.0-20170215233205-553a64147049
go: finding golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6
go: finding github.com/go-openapi/jsonpointer v0.17.0
go: finding github.com/golang/mock v1.2.0
go: finding github.com/smartystreets/goconvey v0.0.0-20190306220146-200a235640ff
go: finding gotest.tools v2.2.0+incompatible
go: finding github.com/ugorji/go/codec v0.0.0-20190309163734-c4a1c341dc93
go: finding github.com/dnaeon/go-vcr v0.0.0-20180814043457-aafff18a5cc2
go: finding github.com/oklog/run v0.0.0-20180308005104-6934b124db28
go: finding k8s.io/api v0.0.0-20181204000039-89a74a8d264d
go: finding github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: finding github.com/containerd/continuity v0.0.0-20181203112020-004b46473808
go: finding honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a
go: finding golang.org/x/text v0.0.0-20160726164857-2910a502d2bf
go: finding golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846
go: finding github.com/Azure/go-autorest v10.15.3+incompatible
go: finding github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb
go: finding golang.org/x/crypto v0.0.0-20190404164418-38d8ce5564a5
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2
go: finding github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed
go: finding github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492
go: finding github.com/go-sql-driver/mysql v0.0.0-20180618115901-749ddf1598b4
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09
go: finding k8s.io/klog v0.0.0-20181108234604-8139d8cb77af
go: finding gopkg.in/mgo.v2 v2.0.0-20160818020120-3f83fa500528
go: finding github.com/PuerkitoBio/purell v1.1.0
go: finding github.com/hashicorp/go-retryablehttp v0.0.0-20180531211321-3b087ef2d313
go: finding golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c
go: finding github.com/google/martian v2.1.0+incompatible
go get: error loading module requirements
Hi, I have some use cases about collecting profiles from short-lived jobs and querying these profiles from the conprof TSDB. For example, profiles collected from go benchmarks.
@brancz mentioned that this can be achieved with the existing profile gRPC client to write the profiles to the remote store.
So I propose to add this feature to the conprof sampler or another separate conprof xxx subcommand. This should read a profile from a file, attach specified label sets and send a request to the store.
Source view (for me, at least) just lists function names with source file name, and I guess that's because the server hasn't downloaded the source code.
Thinking out loud about possible implementation...
__meta_kubernetes_pod_container_image.Right now it is next to impossible to add additional profiling endpoints such as fgprof. You need to modify an existing profile configuration that you don't care about like threadcreate:
scrape_configs:
- job_name: 'kubernetes-pods'
scrape_interval: 1m
scrape_timeout: 1m
profiling_config:
pprof_config:
threadcreate:
path: /debug/fgprof
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
target_label: __param_seconds
replacement: 3
- action: labelmap
regex: __meta_kubernetes_pod_label_app_kubernetes_io_(.+)
replacement: "app_kubernetes_io_$1"
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
action: replace
target_label: containerA proposal could be to instead of having hardcoded profile name values under pprof_config use map[string]PprofProfilingConfig instead and merge what the user passes on top of the default. That way there is a reasonable default as well as the possibility to extend.
Hi,
I was wondering if this tool will solve the following use case:
To know the values of redis keys (which are in memory) at any point in time.
Thanks.
It would be nice if conprof didn't depend on graphviz.
Potential pure go candidate to replace it: https://github.com/wwmoraes/dot
Cannot get this to compile and work as it depends on: github.com/coreos-inc/tectonic-operators for the pprof handlers.
I would be keen to add multi arch support for arm/arm64. We might as well do ppc64le / s390x .
For quay you would need to create empty repos like I did for systemd_exporter:
https://quay.io/repository/povilasv/systemd-exporter-linux-amd64
https://quay.io/repository/povilasv/systemd-exporter-linux-s390x
https://quay.io/repository/povilasv/systemd-exporter-linux-ppc64le
https://quay.io/repository/povilasv/systemd-exporter-linux-armv7
https://quay.io/repository/povilasv/systemd-exporter-linux-ppc64le
This is how it looks in quay:
https://quay.io/repository/povilasv/systemd-exporter?tab=tags
What do you think?
FYI I have a working k8s cluster which has arm64/arm nodes in it for testing.
We would like to port Go's trace web UI to conprof.
PR started in #56
The --config.file flag should not be necessary when --target is specified on the sampler. This seems like a bug.
Steps are not clear in readme how to build conprof tool and use it with any golang application. I will be very thankful if anyone can guide me. I followed whatever written in readme and getting below error.
I tried to run with 4 hour retention, but disk space is growing continuously over the 4 days since I started it.
containers:
- args:
- all
- --storage.tsdb.path=/conprof
- --storage.tsdb.retention.time=4h
- --config.file=/etc/conprof/conprof.yaml
- --http-address=:8080
image: quay.io/conprof/conprof:master-2021-02-25-3457edb
This is the entire logfile; looks like it got stuck? Scraping and UI are still working.
level=info ts=2021-02-26T11:20:10.847835811Z caller=repair.go:56 msg="Found healthy block" mint=1614163347593 maxt=1614297600000 ulid=01EZEBARVJ1ZAQ7FH3NT53RPJ7
level=info ts=2021-02-26T11:20:10.849056671Z caller=repair.go:56 msg="Found healthy block" mint=1614319200177 maxt=1614326400000 ulid=01EZES1X15CG6G2TYRKDYBYWFE
level=info ts=2021-02-26T11:20:10.850114138Z caller=repair.go:56 msg="Found healthy block" mint=1614326400177 maxt=1614333600000 ulid=01EZEZXM8TJ8QK54N513R3GK57
level=info ts=2021-02-26T11:20:10.851410046Z caller=repair.go:56 msg="Found healthy block" mint=1614297600177 maxt=1614319200000 ulid=01EZEZXN4BFF9YQA6DPJCRGYR1
level=info ts=2021-02-26T11:20:10.895048464Z caller=head.go:645 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-02-26T11:20:12.532123782Z caller=head.go:659 msg="On-disk memory mappable chunks replay completed" duration=1.637044896s
level=info ts=2021-02-26T11:20:12.532162065Z caller=head.go:665 msg="Replaying WAL, this may take a while"
level=info ts=2021-02-26T11:20:12.535973672Z caller=head.go:691 msg="WAL checkpoint loaded"
level=info ts=2021-02-26T11:20:13.443256827Z caller=head.go:717 msg="WAL segment loaded" segment=65 maxSegment=68
level=info ts=2021-02-26T11:20:14.131505966Z caller=head.go:717 msg="WAL segment loaded" segment=66 maxSegment=68
level=info ts=2021-02-26T11:20:14.39420013Z caller=head.go:717 msg="WAL segment loaded" segment=67 maxSegment=68
level=info ts=2021-02-26T11:20:14.394447946Z caller=head.go:717 msg="WAL segment loaded" segment=68 maxSegment=68
level=info ts=2021-02-26T11:20:14.394472708Z caller=head.go:722 msg="WAL replay completed" checkpoint_replay_duration=3.828929ms wal_replay_duration=1.858470671s total_replay_duration=3.499393489s
level=info ts=2021-02-26T11:20:14.397532391Z caller=http.go:58 service=http/server component=all msg="listening for requests and metrics" address=:8080
2021/02/26 11:29:05 Serving web UI on http://localhost:0
2021/02/26 11:29:10 Serving web UI on http://localhost:0
2021/02/26 11:29:46 Serving web UI on http://localhost:0
2021/02/26 12:52:16 Serving web UI on http://localhost:0
2021/02/26 12:52:34 Serving web UI on http://localhost:0
2021/02/26 12:52:43 Serving web UI on http://localhost:0
stat /blocksconvert: no such file or directory
2021/02/26 12:52:47 Serving web UI on http://localhost:0
2021/02/26 12:55:07 Serving web UI on http://localhost:0
2021/02/26 12:55:16 Serving web UI on http://localhost:0
2021/02/26 12:55:22 Serving web UI on http://localhost:0
level=info ts=2021-02-26T13:00:02.437044897Z caller=compact.go:506 msg="write block" mint=1614333600000 maxt=1614340800000 ulid=01EZF6SCJMVQDY6YG0K0W9PH0E duration=944.120386ms
level=info ts=2021-02-26T13:00:02.444210154Z caller=head.go:824 msg="Head GC completed" duration=873.327µs
level=info ts=2021-02-26T13:00:02.506216244Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=65 to_segment=67 mint=1614340800000
level=info ts=2021-02-26T13:00:02.76080776Z caller=head.go:921 msg="WAL checkpoint complete" first=65 last=67 duration=254.726227ms
level=info ts=2021-02-26T15:00:04.335355547Z caller=compact.go:506 msg="write block" mint=1614340801478 maxt=1614348000000 ulid=01EZFDN5NQ9WFCMW42WEZVPJAF duration=952.343087ms
level=info ts=2021-02-26T15:00:04.345607506Z caller=head.go:824 msg="Head GC completed" duration=1.010266ms
level=info ts=2021-02-26T15:00:04.427835244Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=68 to_segment=69 mint=1614348000000
level=info ts=2021-02-26T15:00:04.645748315Z caller=head.go:921 msg="WAL checkpoint complete" first=68 last=69 duration=218.049752ms
level=info ts=2021-02-26T17:00:04.0260127Z caller=compact.go:506 msg="write block" mint=1614348001478 maxt=1614355200000 ulid=01EZFMGWXQP7W0T7TFECA810KX duration=642.574481ms
level=info ts=2021-02-26T17:00:04.033045337Z caller=head.go:824 msg="Head GC completed" duration=817.604µs
level=info ts=2021-02-26T17:00:04.097944143Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=70 to_segment=71 mint=1614355200000
level=info ts=2021-02-26T17:00:04.320727401Z caller=head.go:921 msg="WAL checkpoint complete" first=70 last=71 duration=222.924636ms
level=info ts=2021-02-26T17:00:12.240036867Z caller=compact.go:447 msg="compact blocks" count=3 mint=1614319200177 maxt=1614340800000 ulid=01EZFMGXV19ZVJDV1CCBZHA5M0 sources="[01EZES1X15CG6G2TYRKDYBYWFE 01EZEZXM8TJ8QK54N513R3GK57 01EZF6SCJMVQDY6YG0K0W9PH0E]" duration=7.919002189s
level=info ts=2021-02-26T17:00:12.284015895Z caller=db.go:1189 msg="Deleting obsolete block" block=01EZF6SCJMVQDY6YG0K0W9PH0E
level=info ts=2021-02-26T17:00:12.31475957Z caller=db.go:1189 msg="Deleting obsolete block" block=01EZES1X15CG6G2TYRKDYBYWFE
level=info ts=2021-02-26T17:00:12.342587672Z caller=db.go:1189 msg="Deleting obsolete block" block=01EZEZXM8TJ8QK54N513R3GK57
2021/02/26 17:20:18 Serving web UI on http://localhost:0
level=info ts=2021-02-26T19:00:04.562363712Z caller=compact.go:506 msg="write block" mint=1614355201478 maxt=1614362400000 ulid=01EZFVCM5QP139FV65F10F2709 duration=1.179309374s
level=info ts=2021-02-26T19:00:04.571055029Z caller=head.go:824 msg="Head GC completed" duration=1.029109ms
level=info ts=2021-02-26T19:00:04.640246921Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=72 to_segment=73 mint=1614362400000
level=info ts=2021-02-26T19:00:04.886541637Z caller=head.go:921 msg="WAL checkpoint complete" first=72 last=73 duration=246.432558ms
level=info ts=2021-02-26T21:00:06.244124985Z caller=compact.go:506 msg="write block" mint=1614362401478 maxt=1614369600000 ulid=01EZG28BDQQZ32GK80V68SFEPW duration=2.860323305s
level=info ts=2021-02-26T21:00:06.252746555Z caller=head.go:824 msg="Head GC completed" duration=872.273µs
level=info ts=2021-02-26T21:00:06.29288143Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=74 to_segment=75 mint=1614369600000
level=info ts=2021-02-26T21:00:06.527013328Z caller=head.go:921 msg="WAL checkpoint complete" first=74 last=75 duration=234.275547ms
level=info ts=2021-02-26T23:00:02.238456384Z caller=compact.go:506 msg="write block" mint=1614369601117 maxt=1614376800000 ulid=01EZG940TN35QRGSXXAFCMDGJ3 duration=745.328747ms
level=info ts=2021-02-26T23:00:02.246877818Z caller=head.go:824 msg="Head GC completed" duration=845.64µs
level=info ts=2021-02-26T23:00:02.337550991Z caller=checkpoint.go:94 msg="Creating checkpoint" from_segment=76 to_segment=77 mint=1614376800000
level=info ts=2021-02-26T23:00:02.564973884Z caller=head.go:921 msg="WAL checkpoint complete" first=76 last=77 duration=227.610707ms
level=info ts=2021-02-26T23:00:04.89628212Z caller=compact.go:447 msg="compact blocks" count=3 mint=1614340801478 maxt=1614362400000 ulid=01EZG941W5PTWXWKVTJT5M2FZB sources="[01EZFDN5NQ9WFCMW42WEZVPJAF 01EZFMGWXQP7W0T7TFECA810KX 01EZFVCM5QP139FV65F10F2709]" duration=2.330881761s
Goroutine profile https://share.polarsignals.com/5ea1195/
(excerpt:
goroutine 55 [semacquire, 3776 minutes]:
sync.runtime_Semacquire(0xc00044191c)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc00044191c)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/conprof/db/tsdb.(*Block).Close(0xc000441900, 0xc0015b3620, 0xc0015b32e8)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/block.go:326 +0x6f
github.com/conprof/db/tsdb.(*DB).deleteBlocks(0xc000150460, 0xc0015b3620, 0xc0015b346e, 0x41c3b80)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:1168 +0x7c5
github.com/conprof/db/tsdb.(*DB).reloadBlocks(0xc000150460, 0x0, 0x0)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:1043 +0x11e5
github.com/conprof/db/tsdb.(*DB).compactBlocks(0xc000150460, 0x177e05b0f00, 0x0)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:912 +0x188
github.com/conprof/db/tsdb.(*DB).Compact(0xc000150460, 0x0, 0x0)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:844 +0x40a
github.com/conprof/db/tsdb.(*DB).run(0xc000150460)
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:747 +0x27e
created by github.com/conprof/db/tsdb.open
/go/pkg/mod/github.com/conprof/[email protected]/tsdb/db.go:686 +0x791
Would be amazing.
Help wanted 🤗
Currently gRPC requests are uncompressed, we should allow compressing them with snappy. Something like: https://github.com/un000/grpc-snappy
Mac OS x uses a case insensitive file system by default. Since version/ and VERSION are the same you can't clone this repo on Mac OS X.
I think the best way to keep consistent (and maintained) Go pprof UI AND add additional context and options, I would create another page with embedded directly pprof UI. This way we can have both worlds: maintainability and customizations.
The alternative is to... create our own profiles UI. I think this will be amazing long term to support more languages (Rust, Java etc), but not for now. I-framing will also mean that swapping UI will be easy.
WDYT?
This may not be safe in every environment, but when delpoyed in Kubernetes for example it can be nice to reload configuration automatically as it is provisioned atomically anyways by secrets/configmaps.
It would be great to have the possibility to reload the configuration file, just like we can do with prometheus:
curl -X POST http://localhost:8080/-/reload
I see we have some methods, but it's not exposed:
https://github.com/conprof/conprof/blob/master/scrape/manager.go#L84
It would be great if we could support having both the local storage + UI and at the same time being able to send the data to another system like Polar Signals :-)
This would permit us to have a local setup and at the same time a centralized (big picture) one.
The current Query Range is misleading as we plan (maybe?) to maintain Prometheus-like API structure for metadata, add remote read but allow query range with a different response format than official query_range and no PromQL support.
It feels like even our search UI needs remote read, not Query capabilities. What do you think? This assumes react do protobuf formatting just fine (seems so: https://medium.com/@singhandresh09/using-protobuf-in-react-native-7cc8286389f5)
There are various existing pprof libraries for various runtimes. Most resulting from the google cloud profiler, these can be used to build conprof client libraries:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.