jgpc42 / lein-jmh Goto Github PK

Is it possible to print the (sample) variance for a benchmark? I am trying to compute a confidence interval for the ratio of the means of two experiments based on the "Rigorous Benchmarking in Reasonable Time" paper.

All invocations of this task using just lein jmh are currently broken,
because of the default argument "" that is passed to
leiningen.jmh/jmh. "" cannot be read with read-string.

Try lein jmh in lein-jmh’s own sample project for repro.

The file extension of the jmh.edn file would point to it being edn,
but it doesn’t seem to be edn. For example, the sample file is clearly
not valid edn, as it has an anonymous function literal and a symbol that
resolves to a static method of a Java class (Thread/sleep).
edn is a data format that isn’t
tied to a particular language (I’m sure you know this).

As far as I understand, currently jmh.edn may contain arbitrary
Clojure code but must evaluate to a map. I’m not sure if the file should
be called jmh.edn then; certain expectations come with the extension:
purely declarative, no Clojure constructs.

Me, I would welcome a pure edn benchmark file.

Possible changes: call the default file jmh.clj; actually make it an
edn file (perhaps with the proviso that unqualified symbols will be in
clojure.core); document clearly what the contents of the file may or
may not be.

This is a feature request for your consideration.

lein-jmh does a very fine job as a thin wrapper around jmh-clojure, but
I believe the default data-oriented presentation of the benchmark
results is not easy to use for a human reader.

JMH itself presents benchmark results in a table-like format, where
differences between benchmarks really can be seen at a glance.

Please consider providing an option for outputting results in a similar
human-readable at-a-glance format. Thank you!

Using JMH in a managed manner with CLI tools has more friction than using the lein plugin.
Would supporting a tool API help with that?

I would not mind working on it if you're interested

Thanks

Hi there,

It seems like this plugin isn't using the :source-paths value from the project.clj file.

I've run into an issue with one of my projects where I can't access any functions from my namespaces in the jmh.edn file. Whenever I add a function from my a namespace in my project, lein jmh fails with:

java.lang.RuntimeException: value did not resolve to a fn: jmh-test-more-src.more-core/more-gen-fn

The project uses a maven style layout, where I've provided source-paths. (and java-source-paths but that doesn't seem related based on my repro below)

I've got a repo with a minimal reproduction here: https://github.com/lfn3/lein-jmh-source-path-issue-repro