Loki is not designed for high cardinality indexes about logs-benchmark HOT 11 OPEN

@ankitnayan did you reach out to the Loki team at Grafana then before creating your benchmark if you didn't understand how to tune Loki correctly? Or for that matter to even understand what Loki is and what it isn't?

This is a prime example of how not to benchmark; a lack of understanding of the systems at test, comparing apples to oranges, and arbitrary tests which are unlikely to be actual real world workflows.

from logs-benchmark.

@stevehipwell we got a reply from Grafana on the above link.

from logs-benchmark.

+1 on @stevehipwell comment. If you are going to include Loki, please use it in the way it's intended. Happy to give tips on more representitive testing.

Shameless plug: See also Effective troubleshooting with Grafana Loki - query basics https://www.youtube.com/watch?v=UiiZ463lcVA

from logs-benchmark.

@ankitnayan if you don't understand how something works then how can you include it in a benchmark let alone make that benchmark fair?

from logs-benchmark.

Another +1 to @stevehipwell comment. The whole Loki architecture is built upon being fast to query and not needing to index. Really misleading to create this 'benchmark'

Would be more interesting to compare a real use case of querying

from logs-benchmark.

I failed to find any resources on the capacity planning of loki and the performance of different types of queries. I see a closed issue at grafana/loki#3182. It would be great if someone can publish docs on resources needed. If someone is willing to run this on a generic dataset, we would be happy to collaborate.

from logs-benchmark.

@stevehipwell I would appreciate some efforts in ingesting and filtering logs in Loki over a dataset rather than asking to move Loki out of comparison. I really thought about it multiple times before posting the blog. I wanted to encourage collaboration on a good dataset and bring out the best configs from all the log management tools.

We also shared that Loki did not run successfully for even low-cardinality data. Quoting the query from the blog

logs count with method GET per 10 min over the entire data (low cardinality timeseries)

Get first 100 logs with method GET (logs based on filters)

If you think the users don't use Loki for the queries mentioned in the blog. Let me know the right set of queries to benchmark it against. If you think it is supposed to perform poorly in some scenarios, share the use cases. We have already mentioned the high-cardinality usecase.

Also please do share resource usage and perf numbers and scale of your setup.

I am here to improve the stated results in the benchmark for the overall benefit of the users.

from logs-benchmark.

I am sensing rage rather than contribution. No worries, all PRs to the setup and discussions around perf and setup rather than texts are welcome.

from logs-benchmark.

@ankitnayan not rage, more incredulity. Instead of holding your hand up over this and either removing Loki until you understand how it could be compared or reaching out to Grafana you're asking me, a random person, to help you with your benchmark.

from logs-benchmark.

Done. I will wait for a reply before reaching out at non-public channels
https://community.grafana.com/t/how-to-tune-loki-for-better-performance-in-a-benchmark/80709

from logs-benchmark.

Does loki OOM under the benchmark?

from logs-benchmark.