Comments (14)
I do agree that it's a usability problem.
It's easy to forget about this limitation and then people run into issues like:
- un-filtered search -> get results, look at a trace. copy paste a tag
- search for the tag
- no results ??
this makes jaeger look like an unreliable piece of software and people don't want to use it.
from jaeger.
This is a hack that will work for very sparse tags like jaeger-debug-id, but will not work in other cases, e.g. when searching by a tag like "error=true" or http.status_code, because "retrieve all traces" becomes impossible due to volume.
from jaeger.
I see value in having this hack with the limit parameter (in addition to the current behavior) so that it still retrieves results for low cardinality tags.
from jaeger.
I have a problem and I'm not sure if it's the same as what's discussed here. (the descriptions of current and desired logic don't mention how the limit comes into play) but what i'm seeing is that much fewer results are being returned when doing a tag search, i have to "artificially" raise the limit to get the amount i want. (e.g. with limit 20 i may get 1 result, with limit 200 i get 26 results), but the problem is only when doing tag searching, it's fine if i don't have a tag clause in the query.
from jaeger.
this is a known (and hard to solve) issue in Cassandra storage implementation
from jaeger.
So, this may be hard to solve, but I want to suggest that its critical to usability: its a non-obvious limitation that will cause lots of head-scratching and push-back from users of a deployment.
Can you perhaps detail the Cassandra limitations that drive this behaviour somewhere? Also, what is the recommended backend? We went with Cassandra because the Uber blog post suggests Uber is running Cassandra :)
from jaeger.
"its critical to usability" - fwiw Zipkin lives with the same limitation for years. You need to bump the number if you need more exotic searches. We could rename it from LIMIT to something amorphous like "search depth" in the UI.
I wouldn't say Cassandra is the recommended backend, it's mostly an operational preference for people. But Elastic doesn't have that LIMIT problem because of how ES itself implements it (fanout to all nodes where each node returns LIMIT results). The benefit of Cassandra is higher throughput.
The main issue for say a query with two tags is that we're maintaining exact-match Cassandra indices, e.g. {service-name}-{tag-key}-{tag-value} => {trace-id}
. So if you do a search by two tags, we execute two queries, both with the LIMIT provided in the input, and then intersect the resulting sets of trace IDs. Cassandra 3.4+ supports SASI indices that we thought would address this issue (they sort of work similar to ES and you need to fan out request to all nodes in the cluster), but their performance turned out to be even worse than ES, and not just on writes, but on reads (update: it is possible we didn't use it correctly).
We've discussed a possible hack of repeating queries for each tag by gradually increasing LIMIT for each query until the intersection is also of LIMIT size. Never had a chance to try to implement it, not even sure how well it would work.
So in summary - we have no plans to fix this just yet. Silver lining - we're looking into other solutions based on aggregations that could make the whole point of searching for individual traces less important.
from jaeger.
Hi, this is causing a lot of inconsistent results, and not giving me what I want. See the behaviour here: https://youtu.be/m7qZJIyCmGY
Essentially, this is giving me only the last 3-4 traces and the inconsistency b/w queries is worrying especially because all the spans might not yet be added to the traces being shown and I'd want to see older traces.
Could we atleast show a warning that the results will be off and point to this ticket maybe? Quite frankly, I wouldn't have been able to find this issue (thanks @Dieterbe for the pointer).
from jaeger.
@gouthamve, thanks to you and the others on this thread for calling out this issue.
This is definitely a severe issue, and it's great to know the extent it's affecting you.
I'll break the problems described into two broad categories:
- Challenges with search in Cassandra
- Challenges with results containing incomplete traces
For # 1, we have two tracks for addressing it. For the longer-term, we're currently prototyping a more robust (and expressive) search, and we expect to be able go live with it by the end of the year. It should be able to address 1 as well as lay the ground work for looking at aggregated data. In the shorter term, we're looking at ways to keep users more informed about the limitations of the Cassandra search. To this end, we created Inform users of jaegertracing/jaeger#166 when Cassandra is the backing store.
The UI ticket (ui-243) is definitely not a solution, but would you say it would have been helpful to be aware that it's a known issue?
Determining a resolution to # 2 is still a work in progress. One of the main challenges is, it's impossible to know, with 100% certainty, when a trace is complete. One approach we're considering is to show the number of spans associated with a trace when viewing search results and to update that number, in real time. The idea being if that number goes up while a user is viewing the search results, then the trace is probably not complete. But, whether this is the right approach or not is still TBD. I wish I had better news, on this front.
Lastly, your feedback is super useful; thanks again for letting us know this came up in a severe fashion.
from jaeger.
Re: 2 - is there a separate ticket for that? I have some thoughts but don't think this is the right ticket.
from jaeger.
@rbtcollins Great! Currently, we don't have a ticket for issues around incomplete traces. Can you start one to capture your thoughts?
from jaeger.
I can see three things we could do here (higher priority first):
- Make it clear in the documentation the limitation of Cassandra search and recommend Elasticsearch to people who are interested in the search use case (as opposed to data-mining driven navigation)
- For Cassandra users, implement jaegertracing/jaeger-ui#243
- Consider a different Cassandra storage implementation that uses SASI indices (we had an earlier experience with SASI that resulted in unusable query latency, but I think we did it incorrectly. There has been improvements in Zipkin on that front, including using a new tokenizer specifically implemented in Cassandra to support Zipkin tag search).
We should discuss it at the next project call next Friday.
from jaeger.
Is there any progress regarding this? ES users still can't specify tags alongside minDuration.
from jaeger.
this might be fixed by #1477, once released
from jaeger.
Related Issues (20)
- [Bug]: All Operation search not functioning under Azure CosmosDB-for-Cassandra storage HOT 3
- Bring code coverage back to 95% HOT 1
- [Bug]: Trace Id mismatch between backend Cassandra tables HOT 8
- [Bug]: Jaeger all in one connection refused when setting `collector.otlp.grpc.host-port` HOT 5
- Extract archive storage integration test from ES and generalize to all storage backends HOT 3
- [Bug]: mkdir /badger/key: permission denied HOT 2
- [Feature]: Enable IAM based auth to ES for AWS clients HOT 2
- [jaeger-v2] Deployment options HOT 7
- [Bug]: Cassandra second node is stuck on CrashLoopBackOff HOT 2
- [jaeger-v2] Rethink storage configuration to align with OTEL
- [jaeger-v2] Do away with storage.ArchiveFactory HOT 3
- [Bug]: Unstable Jaeger Deployment with Cassandra ; Cassandra STS is failing HOT 3
- Jaeger's LFX Mentorship - Term 02 - 2024 June - August HOT 10
- [jaeger-v2] Storage backend integration tests HOT 14
- [Bug]: internal.span.format field error in zipkin receiver
- [jaeger-v2] Build jaeger-remote-storage for gRPC integration test HOT 5
- [Feature]: Possible impact of changing Cassandra table compaction strategy HOT 1
- [Bug]: Does Jaeger-all-in-one does not support OpenSearch 2.11? HOT 4
- [Bug]: Operation based sampling strategies are not being returned for ratelimiting service strategies HOT 6
- Create GitHub Action to block PRs from ${fork}/main branch HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jaeger.