compairing 'peak' performance about frameworkbenchmarks HOT 3 CLOSED

In most/all cases, the lowest performance is at the lowest level of concurrency we tested (8) because in that scenario the platform/framework doesn't get to demonstrate its capacity to process new requests while I/O is happening with other requests.

That said, you can view the data we captured at various concurrency levels in the line charts and data tables. In the data tables, the peak performance is highlighted in dark red.

Most frameworks reach a plateau and then level off as concurrency rises. However, if you look at the line charts, especially on dedicated hardware, there are a couple that decline after reaching their peak.

Is the data we've provided in the line charts and tables not what you're looking for?

from frameworkbenchmarks.

The tables and line charts are fine, but not what the eye is guided too - I feel what we see on the first glance should be the 'most correct' interpretation of the data and peak performance seems not to be the most correct performance.

That said with the given set of concurrency levels you are entirely right, the lowest performance is currently with the lowest concurrency for most frameworks, comparing those would be further from the 'optimal interpretation' of the data then comparing peak levels.

Also please don't see this is blind criticism, I love the work you've done and think it's a great idea to have a tool for framework benchmarks - it was a ease to add additional frameworks which means this is nicely extendable too. So with a fresh set of eyes I like to address the issues I see on the benchmark with the hope to improve ont he relevance of the results.

Cheers,
Heinz

from frameworkbenchmarks.

Hi @Licenser. I am going to close this issue up. I think this was addressed in our conversation about concurrency levels. But let me know if you feel otherwise. I also have as a goal to add more flexibility to the tables and charts, but that will take me some time.

from frameworkbenchmarks.