bbc2 / lang-bench Goto Github PK

A few variants of the primes algorithm might be worth exploring:

• An algorithm where the prime testing function stops looking for a divider after reaching the square root of the tested number. This will introduce the square root calculus in the algorithm allowing to compare the performance of float calculus instead of just integers.
  We are assuming here that most square root implemented algorithms are more or less the same which might not be true. Also it is possible that in most languages the square calculus just call the C implementation, which means that any comparisons of performance between those languages might not be relevant.
• An algorithm where we always test all the dividers even if we found one. This would remove a conditional branching from the algorithm which might influence the performances and could give an interesting insight on how branching can affect the performances of different languages.
• An algorithm where the array of primes is pre allocated instead of dynamically allocated. This should remove the influence of the different languages strategies about dynamic allocations (if their implementation was using dynamic allocation).

These variants highly influence the algorithm complexity so comparing the performances of a language across those makes no sense. However, studying the relative speed of the language compared to the others across the different algorithms might be interesting.

Add clear instructions to be able to run the project easily on a new system (even though this is still "hazardous material").

This should be the reference in terms of execution speed.

We need to find a suitable way of storing the results.

Accessibility:

• The results should all be stored at the same place. Git repository? Some file hosting service?
• The results should be stored in a structured way that allows to easily fetch some pertinent sub groups of results: benchmark version (git tag/hash)? target type (server/laptops/desktop computers)? Algorithms? ...

Reliability / reproducibility:

• The results should be dated.
• The results should contain the git hash.
• The results should contain target information: CPU, RAM, OS...
• The results should contain context information: battery level (can influence CPU clock), external CPU usage monitoring (measure CPU usage 1 minute before running the benchmark for example), external Disk usage (which could also indicate external activity like a surprise antivirus scan).

Optimization:

• There should be a way to run only the tests that have never been run for this target. We could also give a target commit that would be the minimum version of the benchmark under which the a ran test would not considered as done.

I think this would be a good start. Any suggestions?