[out_loki] A lot of warning 'Tenant ID is overwritten A -> B' if tenant_id_key is used about fluent-bit HOT 2 OPEN

Why you used fluentbit not promtail to send data to Loki, I think promtail will good to use when with Loki

from fluent-bit.

@YevhenLodovyi, are your log entries separated into different chunks, e. g. by making sure that they get re-emitted with a tag that contains the actual value of the tenant_id? Have a look at #2935 (comment), that describes the issue quite well. It would be helpful if you provided the lua script and the rest of the fluent-bit config.

If your chunks are well aligned:

We are possibly encountering the same issue, where a race condition seems to be involved. I guess it was introduced (or at least not fixed) by PR #6931 where dynamic_tenant_id gets shared within the thread. I guess due to usage of coroutines, the value changes between loki_compose_payload and flb_http_add_header.

But I have to admit that I don't see a context switch in between. Maybe @leonardo-albertovich could shed some light on it?

Though we are quite sure that the value changes in between, as we captured the traffic using tcpdump and the chunks are well aligned and contain log messages from (in our case) distinct namespaces only, but are tagged with the wrong X-Scope-OrgID header.

We have configured Tenant_id_key to customer and the request looks like this:

POST /loki/api/v1/push HTTP/1.1
Host: loki-gateway.logging.svc:80
Content-Length: 944
User-Agent: Fluent-Bit
Content-Type: application/json
X-Scope-OrgID: customer1
Connection: keep-alive

{"streams":[{"stream":{"job":"fluent-bit","node":"worker11","namespace":"customer2-namespace"},"values":[["1715604287815933056","{\"stream\":\"stdout\",\"logtag\":\"F\",\"message\":\"INFO      trace_generator_slow - generate_slow_traces - SlowOperation created in customer2-namespace - Trace ID: 95f6274c96752cf944547abda39e24b1, Span ID: 5eaddc539554b1ec - 13/05/2024 Monday 12:44:47\",\"namespace_name\":\"customer2-namespace\",\"host\":\"worker11\",\"kubernetes\":{\"pod_name\":\"trace-generator-slow-job-swzcs\",\"namespace_name\":\"customer2-namespace\",\"container_name\":\"trace-generator-slow\",\"labels\":{\"batch.kubernetes.io/controller-uid\":\"f935d0ea-6e29-4be8-a379-0126cb2d2b8d\",\"batch.kubernetes.io/job-name\":\"trace-generator-slow-job\",\"controller-uid\":\"f935d0ea-6e29-4be8-a379-0126cb2d2b8d\",\"job-name\":\"trace-generator-slow-job\"}},\"customer\":\"customer2\",\"cluster\":\"cluster-name\"}"]]}]}

If this is a separate topic, I'll file an additional issue.

Update: We recompiled fluent-bit with FLB_OUTPUT_SYNCHRONOUS flag set in out_loki.c. The issue still persists. Either the cause is something else, or we misinterpreted the meaning of the flag.

from fluent-bit.