This example project is my attempt at answering stackoverflow question How to profile methods in Scala?
The recommended approach to benchmarking Scala code is via sbt-jmh
"Trust no one, bench everything." - sbt plugin for JMH (Java Microbenchmark Harness)
This approach is taken by many of the major Scala projects, for example,
- Scala programming language itself
- Dotty (Scala 3)
- cats library for functional programming
- Metals language server for IDEs
Simple wrapper timer based on System.nanoTime
is not a reliable method of benchmarking:
System.nanoTime
is as bad asString.intern
now: you can use it, but use it wisely. The latency, granularity, and scalability effects introduced by timers may and will affect your measurements if done without proper rigor. This is one of the many reasons whySystem.nanoTime
should be abstracted from the users by benchmarking frameworks
Furthermore, considerations such as JIT warmup, garbage collection, system-wide events, etc. might introduce unpredictability into measurements:
Tons of effects need to be mitigated, including warmup, dead code elimination, forking, etc. Luckily, JMH already takes care of many things, and has bindings for both Java and Scala.
Based on Travis Brown's answer here is an example of how to setup JMH benchmark for Scala
- Add jmh to
project/plugins.sbt
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.3.7")
- Enable jmh plugin in
build.sbt
enablePlugins(JmhPlugin)
- Add to
src/main/scala/bench/VectorAppendVsListPreppendAndReverse.scala
package bench import org.openjdk.jmh.annotations._ @State(Scope.Benchmark) @BenchmarkMode(Array(Mode.AverageTime)) class VectorAppendVsListPreppendAndReverse { val size = 1_000_000 val input = 1 to size @Benchmark def vectorAppend: Vector[Int] = input.foldLeft(Vector.empty[Int])({ case (acc, next) => acc.appended(next)}) @Benchmark def listPrependAndReverse: List[Int] = input.foldLeft(List.empty[Int])({ case (acc, next) => acc.prepended(next)}).reverse }
- Execute benchmark with
sbt "jmh:run -i 10 -wi 10 -f 2 -t 1 bench.VectorAppendVsListPreppendAndReverse"
The results are
Benchmark Mode Cnt Score Error Units
VectorAppendVsListPreppendAndReverse.listPrependAndReverse avgt 20 0.024 ± 0.001 s/op
VectorAppendVsListPreppendAndReverse.vectorAppend avgt 20 0.130 ± 0.003 s/op
which seems to indicate prepending to a List
and then reversing it at the end is order of magnitude faster than keep appending to a Vector
.