Comments (9)
welcome
commented on September 22, 2024
Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.
from signoz.
nityanandagohain
commented on September 22, 2024
What is the status of clickhouse ? is it running properly, also share the logs of clickhouse.
The error that you pasted is a warn log, try checking for error logs
from signoz.
srikanthccv
commented on September 22, 2024
Please try adding the timeout: 10s to the clickhousetraces exporter and check if that resolves the issue.
override-values.yaml
otelCollector:
config:
exporters:
clickhousetraces:
timeout: 10s
from signoz.
afreakk
commented on September 22, 2024
Please try adding the
timeout: 10sto the clickhousetraces exporter and check if that resolves the issue.override-values.yaml
otelCollector: config: exporters: clickhousetraces: timeout: 10s
Im having the same issue, the timeout might have made a difference, will update later
(why is the timeout not default 10s on the signoz chart?)
{
"level": "error",
"ts": 1720077675.7976239,
"caller": "clickhousetracesexporter/writer.go:417",
"msg": "Could not write a batch of spans to index table: ",
"kind": "exporter",
"data_type": "traces",
"name": "clickhousetraces",
"error": "read: read tcp 10.242.91.83:37124->172.20.41.107:9000: use of closed network connection",
"errorVerbose": "read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.242.91.83:37124->172.20.41.107:9000: use of closed network connection",
"stacktrace": "github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"
}
chi-signoz-clickhouse-cluster-0-0-0 is at the same time reporting this error
2024.07.04 07:24:04.916899 [ 5626 ] {879dba04-79ce-4c55-ae28-3955192b2272} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:10.242.85.225]:9000 -> [::ffff:10.242.91.83]:35826). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000caa69a1 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000caa733e in /usr/bin/clickhouse
3. DB::TCPHandler::runImpl() @ 0x000000001292120f in /usr/bin/clickhouse
4. DB::TCPHandler::run() @ 0x0000000012933eb9 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
9. ? @ 0x00007fc777623609
10. ? @ 0x00007fc777548353
2024.07.04 07:24:04.917118 [ 5626 ] {879dba04-79ce-4c55-ae28-3955192b2272} <Warning> TCPHandler: Client has gone away.
from signoz.
srikanthccv
commented on September 22, 2024
How much resoruces were provisioned and how much data do you ingest?
from signoz.
afreakk
commented on September 22, 2024
on the collector its
cpu: 100m
memory: 300Mi
on the chi-signoz-clickhouse-cluster-0-0 its
cpu: 1950m
memory: 5400Mi
sending about 15k spans per hour, except when something breaks down
It broke down again now though, same clickhouselogsexporter","error":"StatementSend:read: read tcp 10.242.87.250:33846->172.20.41.107:9000: use of closed network connection" error.
from signoz.
srikanthccv
commented on September 22, 2024
First I will tell you why it worked in the past but now you see some errors. There was a bug in the exporter code that didn't respect the context cancellation.
use of closed network connection
This happens because signoz otel collector is unable to complete the export within the default timeout of 5s. There can be several reasons for it
- The batches are very big
- The collector is unable to marshal the received batch data and complete the export
- The ClickHouse is processing slowly
If the resource usage is not high and the timeout doesn't help, then it is most probably the ClickHouse server or the disk latency.
from signoz.
afreakk
commented on September 22, 2024
It seems it still ends up in that state, even with timeout 10s.
Althougm maybe a bit less often
{"level":"info","ts":1720336895.264984,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41784->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41784->172.20.82.18:9000: use of closed network connection","interval":"15.513534214s"}
{"level":"error","ts":1720336896.7878253,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41794->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41794->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336896.7879827,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41794->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41794->172.20.82.18:9000: use of closed network connection","interval":"37.837615401s"}
{"level":"info","ts":1720336898.687397,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:41802->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41802->172.20.82.18:9000: use of closed network connection","interval":"9.755125534s"}
{"level":"info","ts":1720336899.1848605,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:41804->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41804->172.20.82.18:9000: use of closed network connection","interval":"18.265298984s"}
{"level":"error","ts":1720336899.2210565,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41808->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41808->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336899.2213676,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41808->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41808->172.20.82.18:9000: use of closed network connection","interval":"28.790728919s"}
{"level":"info","ts":1720336899.965375,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:50748->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:50748->172.20.82.18:9000: use of closed network connection","interval":"3.110373954s"}
{"level":"error","ts":1720336900.9423711,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41834->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41834->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336900.942526,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:41834->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41834->172.20.82.18:9000: use of closed network connection","interval":"24.962824412s"}
{"level":"info","ts":1720336901.4670222,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"StatementSend:read: read tcp 10.16.86.193:41848->172.20.82.18:9000: use of closed network connection","interval":"15.550329113s"}
{"level":"info","ts":1720336901.8047156,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:50756->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:50756->172.20.82.18:9000: use of closed network connection","interval":"5.230311643s"}
{"level":"info","ts":1720336907.5629592,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"read: read tcp 10.16.86.193:56478->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56478->172.20.82.18:9000: use of closed network connection","interval":"13.149593587s"}
{"level":"error","ts":1720336908.217097,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46116->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46116->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336908.21724,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46116->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46116->172.20.82.18:9000: use of closed network connection","interval":"33.058460955s"}
{"level":"error","ts":1720336909.3671706,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46118->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46118->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336909.3673365,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46118->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46118->172.20.82.18:9000: use of closed network connection","interval":"16.938410889s"}
{"level":"error","ts":1720336909.378309,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46124->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46124->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336909.3784513,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46124->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46124->172.20.82.18:9000: use of closed network connection","interval":"26.934317592s"}
{"level":"info","ts":1720336909.9043813,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:41818->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:41818->172.20.82.18:9000: use of closed network connection","interval":"4.849795304s"}
{"level":"info","ts":1720336913.0772662,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:46138->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46138->172.20.82.18:9000: use of closed network connection","interval":"5.345100406s"}
{"level":"error","ts":1720336913.2071788,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46150->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46150->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336913.2073214,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:46150->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46150->172.20.82.18:9000: use of closed network connection","interval":"19.629389678s"}
{"level":"info","ts":1720336913.6752946,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:46158->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:46158->172.20.82.18:9000: use of closed network connection","interval":"24.704749948s"}
{"level":"info","ts":1720336917.0366595,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:54960->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54960->172.20.82.18:9000: use of closed network connection","interval":"10.992716496s"}
{"level":"info","ts":1720336918.444767,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:54966->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54966->172.20.82.18:9000: use of closed network connection","interval":"20.676556328s"}
{"level":"error","ts":1720336920.77996,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:54976->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54976->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336920.7801387,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:54976->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54976->172.20.82.18:9000: use of closed network connection","interval":"15.595929665s"}
{"level":"error","ts":1720336920.877413,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:54980->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54980->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336920.877559,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:54980->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:54980->172.20.82.18:9000: use of closed network connection","interval":"32.919785399s"}
{"level":"info","ts":1720336924.687212,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:37442->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:37442->172.20.82.18:9000: use of closed network connection","interval":"28.203340541s"}
{"level":"info","ts":1720336924.755754,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:37448->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:37448->172.20.82.18:9000: use of closed network connection","interval":"5.059568537s"}
{"level":"info","ts":1720336927.018638,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"StatementSend:read: read tcp 10.16.86.193:37462->172.20.82.18:9000: use of closed network connection","interval":"37.845894832s"}
{"level":"info","ts":1720336927.4518843,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:37468->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:37468->172.20.82.18:9000: use of closed network connection","interval":"39.097404099s"}
{"level":"info","ts":1720336928.4248295,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:37470->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:37470->172.20.82.18:9000: use of closed network connection","interval":"8.838054959s"}
{"level":"error","ts":1720336935.9074938,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42796->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42796->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336935.907649,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42796->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42796->172.20.82.18:9000: use of closed network connection","interval":"41.707383905s"}
{"level":"error","ts":1720336936.3073008,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42810->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42810->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336936.3075063,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42810->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42810->172.20.82.18:9000: use of closed network connection","interval":"41.300123095s"}
{"level":"error","ts":1720336938.0131812,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42824->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42824->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336938.0133417,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42824->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42824->172.20.82.18:9000: use of closed network connection","interval":"42.737298763s"}
{"level":"error","ts":1720336938.9953308,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42834->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42834->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336938.9955454,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42834->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42834->172.20.82.18:9000: use of closed network connection","interval":"19.20243964s"}
{"level":"error","ts":1720336942.8375888,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42864->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42864->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336942.837732,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:42864->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42864->172.20.82.18:9000: use of closed network connection","interval":"32.84002695s"}
{"level":"error","ts":1720336944.6270814,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56748->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56748->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336944.6272407,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56748->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56748->172.20.82.18:9000: use of closed network connection","interval":"42.669300948s"}
{"level":"info","ts":1720336945.852604,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"read: read tcp 10.16.86.193:42828->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:42828->172.20.82.18:9000: use of closed network connection","interval":"5.4352701s"}
{"level":"error","ts":1720336946.3144207,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56764->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56764->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336946.3146195,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56764->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56764->172.20.82.18:9000: use of closed network connection","interval":"19.216683434s"}
{"level":"error","ts":1720336946.3774953,"caller":"clickhousetracesexporter/writer.go:417","msg":"Could not write a batch of spans to index table: ","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56774->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56774->172.20.82.18:9000: use of closed network connection","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:417\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:436\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:59\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:89\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:159\ngo.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37\ngo.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:99\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43"}
{"level":"info","ts":1720336946.3776388,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"traces","name":"clickhousetraces","error":"read: read tcp 10.16.86.193:56774->172.20.82.18:9000: use of closed network connection","errorVerbose":"read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!sig!noz/[email protected]/proto/reader.go:62\n - read tcp 10.16.86.193:56774->172.20.82.18:9000: use of closed network connection","interval":"36.771074789s"}
{"level":"info","ts":1720336947.081959,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"StatementSend:read: read tcp 10.16.86.193:54982->172.20.82.18:9000: use of closed network connection","interval":"6.951402223s"}
{"level":"info","ts":1720336947.0887554,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"logs","name":"clickhouselogsexporter","error":"StatementSend:read: read tcp 10.16.86.193:37474->172.20.82.18:9000: use of closed network connection","interval":"7.105925408s"}
This is how the namespace's resource usage looks before it fails
Also, were using aws-EKS, so there shouldnt be any disk failures
from signoz.
afreakk
commented on September 22, 2024
When i bumped to newest clickhouse version, that last issue went aaway.
from signoz.
Related Issues (20)
- [EPIC] Correlations HOT 2
- Requests return 403 unexpectedly HOT 2
- Display the trace ID HOT 1
- Take to logs of a service on clicking on charts of Error, latency, RPS HOT 2
- Show the pod CPU & memory usage vs limits in K8s HOT 1
- Update favicon and SigNoz logo inside tenants HOT 1
- Webhook altert chanel use a GET rather than a POST HOT 1
- Revamping help center HOT 2
- Cleaning up Sign in flow HOT 8
- Design for correlation components HOT 3
- Traces UI when hitting refresh, it displays different set of span datas
- Deprecate Clarity and Segment
- Dashboard support Folder like Grafana HOT 1
- Support for panel level variable HOT 5
- [FE]: Dashabord names not visible in light mode
- [EPIC] Data rendering limitations in product
- Dashboards → Time Series Panel → More data points, the slower the render HOT 1
- Dashboards → Bar chart → Same as time series chart, more series/data points, the slower the render HOT 1
- Dashboards → Pie chart panel → More data points, the slower the render HOT 1
- Dashboard → Variables → Dependent Variables can lead to multiple Query range API triggers and combined with the above issues, it will further degrade the performance. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from signoz.