Comments (5)
Hi @singhbaljit.
This is an intended behavior that is based on MQTT spec and OpenTelemetry semantic conventions for messaging systems:
- If EMQX receives trace context in a published message, e.g., traceparent/tracestate User-property for MQTT v5.0, it must be sent unaltered when forwarding the Application Message to a Client to conform with MQTT specification 3.3.2.3.7.
- https://github.com/open-telemetry/semantic-conventions/blob/main/docs/messaging/messaging-spans.md#context-propagation :
A message may traverse many different components and layers in one or more intermediaries when it is propagated from the producer to the consumer(s). To be able to correlate consumer traces with producer traces using the existing context propagation mechanisms, all components must propagate context down the chain.
Messaging systems themselves may trace messages as the messages travels from producers to consumers. Such tracing would cover the transport layer but would not help in correlating producers with consumers. To be able to directly correlate producers with consumers, another context that is propagated with the message is required.
A message creation context allows correlating producers with consumers of a message and model the dependencies between them, regardless of the underlying messaging transport mechanism and its instrumentation.
The message creation context is created by the producer and should be propagated to the consumer(s). Consumer traces cannot be directly correlated with producer traces if the message creation context is not attached and propagated with the message.
A producer SHOULD attach a message creation context to each message. If possible, the message creation context SHOULD be attached in such a way that it cannot be changed by intermediaries.
from emqx.
In other words, traces emitted by EMQX provide extra level of details and cover intermediary level (messaging system itself).
One can disable OpenTelemetry traces in EMQX and EMQX will still perfectly participate in distributed tracing by simply propagating trace context from publishers to subscribers (as described above).
from emqx.
@SergeTupchiy thank you for responding promptly, and thank you for referencing the documentation. It was wrong for me to call it "incorrect". EMQX is following the MQTT v5 spec correctly: that is, the user properties are not mutated. However, I do still think this is functionally incorrect. Here's effectively what the trace map looks like:
from emqx.
I also don't think the OpenTelemetry document is saying to forward traceparent
unmutated. It is only saying that the context must be propagated, which means trace id must be unmutated; the span id could/should change. This is exactly what happens in Spring Boot Kafka (Micrometer).
I do think the MQTT v5 spec and the W3C context propagation for MQTT are in contradiction. I can raise this issue there as well. But do you think it possible to add another configuration to enable overriding the traceparent
?
from emqx.
Messaging systems themselves may trace messages as the messages travels from producers to consumers. Such tracing would cover the transport layer but would not help in correlating producers with consumers.
A producer SHOULD attach a message creation context to each message. If possible, the message creation context SHOULD be attached in such a way that it cannot be changed by intermediaries.
Doesn't it say that any tracing done by messaging system itself is internal and should not affect producer/consumer correlation?
Mutating trace id breaks any correlation and makes any distributed tracing impossible. This is quite obvious, so I don't think that this is the only point of the Open Telemetry document statements mentioned above.
Yes, it is technically feasible to override traceparent
in User Properties sent out from EMQX to subscribers (so that subscribers can link to EMQX send_published_message
spans) and make this behaviour configurable. It may be considered for development as a new feature (if not rejected), but it's definitely not a bug.
from emqx.
Related Issues (20)
- STOMP GW does not send heartbeat to client HOT 9
- EMQX cluster cannot restart after persistence HOT 1
- After STOMP enables authentication, authentication fails when the account password carries a colon
- runq_overload everyday for few minutes HOT 4
- Receive Maximum Not Sent in Bridge CONNACK HOT 4
- ~10ms latency on publishing and receiving message on the same machine on windows HOT 6
- webhook监听上下线事件时,事件时序有误,原因不明(connected and disconnected events may out of order) HOT 3
- Authentication fails periodically and restart fixes it HOT 9
- For my project required kafka and kerberos integration with EMQX opensource,So any how is it possible to do these integration with Emqx OPenSource version HOT 2
- Get acknowledgement from subscriber(s) after publish messages HOT 2
- Restored retained message have no payload HOT 2
- error messages received and MQTT broker keep running up and down, very HOT 4
- Several protocol violations or bugs in EMQX HOT 31
- Upgrade to Openssl 3.0 or higher HOT 7
- LDAP server treated as down due to wrong error returned than expected? HOT 7
- EMQX Cluster error msg: failed_to_kick_session_on_remote_node HOT 5
- Bug: v5.7.0 /api/v5/monitor API return 500 Error becasue of the incompatible conf changes HOT 4
- MemoryDB certificate verification fails after upgrade to version 5.7.0 HOT 8
- How to automatically create a configuration file after the container starts HOT 4
- how to configure different ACLs for different listeners HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emqx.