Comments (4)
GiedriusS
commented on August 25, 2024
Could you add a goroutine dump when this happens through /debug/pprof?
from thanos.
george-angel
commented on August 25, 2024
This isn't full blown 0 available, but running in dev with 17/20 or 15/20 available:
goroutine profile: total 55
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x48a265 0x488098 0x4e760e 0x4e75f6 0x4e738d 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x48a264 syscall.Syscall+0x24 /usr/local/go/src/syscall/syscall_linux.go:69
# 0x488097 syscall.read+0x37 /usr/local/go/src/syscall/zsyscall_linux_amd64.go:721
# 0x4e760d syscall.Read+0x2ad /usr/local/go/src/syscall/syscall_unix.go:181
# 0x4e75f5 internal/poll.ignoringEINTRIO+0x295 /usr/local/go/src/internal/poll/fd_unix.go:736
# 0x4e738c internal/poll.(*FD).Read+0x2c /usr/local/go/src/internal/poll/fd_unix.go:160
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
Another replica:
goroutine profile: total 79
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
5 @ 0x43e6ee 0x44eb65 0xf6be1c 0xf9fd49 0xf9fd36 0xf9efbf 0xf9dd1a 0xf9ee25 0x1715f34 0x155d442 0x16c63e6 0x1dbee16 0x1dbebf7 0x4719a1
# 0xf6be1b google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader+0x7b /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:330
# 0xf9fd48 google.golang.org/grpc/internal/transport.(*Stream).RecvCompress+0xc8 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:345
# 0xf9fd35 google.golang.org/grpc.(*csAttempt).recvMsg+0xb5 /go/pkg/mod/google.golang.org/[email protected]/stream.go:1071
# 0xf9efbe google.golang.org/grpc.(*clientStream).RecvMsg.func1+0x1e /go/pkg/mod/google.golang.org/[email protected]/stream.go:928
# 0xf9dd19 google.golang.org/grpc.(*clientStream).withRetry+0x139 /go/pkg/mod/google.golang.org/[email protected]/stream.go:777
# 0xf9ee24 google.golang.org/grpc.(*clientStream).RecvMsg+0x124 /go/pkg/mod/google.golang.org/[email protected]/stream.go:927
# 0x1715f33 github.com/thanos-io/thanos/pkg/tracing/interceptors.(*monitoredClientStream).RecvMsg+0x53 /app/pkg/tracing/interceptors/client.go:76
# 0x155d441 github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredClientStream).RecvMsg+0x61 /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/client.go:62
# 0x16c63e5 github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesClient).Recv+0x45 /app/pkg/store/storepb/rpc.pb.go:1015
# 0x1dbee15 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0xf5 /app/pkg/store/proxy_merge.go:608
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
5 @ 0x43e6ee 0x44eb65 0xf9c04c 0x4719a1
# 0xf9c04b google.golang.org/grpc.newClientStreamWithParams.func4+0x8b /go/pkg/mod/google.golang.org/[email protected]/stream.go:392
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
4 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
3 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
3 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fceea6 0x1fceae8 0x47d19f 0x1fce9ad 0x1fce8ad 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fceea5 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).init+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:170
# 0x1fceae7 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fce9ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fce8ac github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd9d24 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd9d23 github.com/thanos-io/promql-engine/execution/binary.(*vectorOperator).init.func1+0x43 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/vector.go:181
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
Another:
goroutine profile: total 71
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf6be1c 0xf9fd49 0xf9fd36 0xf9efbf 0xf9dd1a 0xf9ee25 0x1715f34 0x155d442 0x16c63e6 0x1dbee16 0x1dbebf7 0x4719a1
# 0xf6be1b google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader+0x7b /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:330
# 0xf9fd48 google.golang.org/grpc/internal/transport.(*Stream).RecvCompress+0xc8 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:345
# 0xf9fd35 google.golang.org/grpc.(*csAttempt).recvMsg+0xb5 /go/pkg/mod/google.golang.org/[email protected]/stream.go:1071
# 0xf9efbe google.golang.org/grpc.(*clientStream).RecvMsg.func1+0x1e /go/pkg/mod/google.golang.org/[email protected]/stream.go:928
# 0xf9dd19 google.golang.org/grpc.(*clientStream).withRetry+0x139 /go/pkg/mod/google.golang.org/[email protected]/stream.go:777
# 0xf9ee24 google.golang.org/grpc.(*clientStream).RecvMsg+0x124 /go/pkg/mod/google.golang.org/[email protected]/stream.go:927
# 0x1715f33 github.com/thanos-io/thanos/pkg/tracing/interceptors.(*monitoredClientStream).RecvMsg+0x53 /app/pkg/tracing/interceptors/client.go:76
# 0x155d441 github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredClientStream).RecvMsg+0x61 /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/client.go:62
# 0x16c63e5 github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesClient).Recv+0x45 /app/pkg/store/storepb/rpc.pb.go:1015
# 0x1dbee15 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0xf5 /app/pkg/store/proxy_merge.go:608
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
4 @ 0x43e6ee 0x44eb65 0xf9c04c 0x4719a1
# 0xf9c04b google.golang.org/grpc.newClientStreamWithParams.func4+0x8b /go/pkg/mod/google.golang.org/[email protected]/stream.go:392
3 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
3 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
2 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
2 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
2 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fceea6 0x1fceae8 0x47d19f 0x1fce9ad 0x1fce8ad 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fceea5 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).init+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:170
# 0x1fceae7 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fce9ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fce8ac github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fd9d24 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fd9d23 github.com/thanos-io/promql-engine/execution/binary.(*vectorOperator).init.func1+0x43 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/vector.go:181
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x4719a1
from thanos.
MichaHoffmann
commented on August 25, 2024
Likely #7618 I guess! Can you try 0.36.1?
from thanos.
george-angel
commented on August 25, 2024
Unfortunately v0.36.1 still exhibits above behaviour.
from thanos.
Related Issues (20)
- sidecar: Do not crash when Object Storage is not accessible HOT 1
- The result of "Use Deduplication" is not as expected HOT 9
- Receiver samples limit should include native histograms
- Docs version dropdown overflows the page
- Rule: Support QueryOffset
- Add support to the Info field in the body of query API responses
- Expose more rueidis clientOpts
- Index Cache: Change fetchMulti interface to return slice rather than map HOT 3
- Sidecar: After a block upload was delayed, all subsequent block uploads were delayed as well HOT 4
- Can we filter out the blocks that are already fully compressed and then downsample them. This way downsampled blocks will not be compressed.There shouldn't be any other problems with this, right? HOT 1
- query/sidecar: same store.limits flags on both components
- Thanos update causing compactor to err on sync before retention HOT 3
- Thanos querier rate/increase function creating huge spikes/incorrect results when deduplication is enabled
- Make TLS versions and ciphers configurable HOT 1
- FT add feature flag for 'promql-experimental-functions'. HOT 3
- Improvement of OpenSSF Scorecard Score HOT 1
- Thanos Query: gaps in deduplicated data
- Segmentation Violation when connecting to Kafka
- Thanos-1x.github.io HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.