Recommend mitigations for benchmark instability introduced by GHCs SpecConstr. about tasty-bench HOT 5 CLOSED

@AndreasPK I'm sorry, I don't quite follow, what exactly can SpecConstr specialise here?

Would passing GHC.Exts.SPEC help?

from tasty-bench.

I checked Core of bytestring benchmarks.

benchIndices_$s$wbenchLoop2
  :: ByteString -> Word64# -> State# RealWorld -> State# RealWorld
benchIndices_$s$wbenchLoop2
  = \ (x_s3Me :: ByteString)
      (ww_s3Mh :: Word64#)
      (eta_s3Mj :: State# RealWorld) ->
      case ww_s3Mh of wild_X1 {
        __DEFAULT ->
          case seq#
                 (case x_s3Me of { BS ww1_s3NJ ww2_s3NK ww3_s3NL ->
                  case $weta1 ww1_s3NJ ww2_s3NK ww3_s3NL of wild2_i3hw {
                    Nothing -> Nothing;
                    Just x1_i3hx -> case x1_i3hx of { I# ipv_i3hc -> wild2_i3hw }
                  }
                  })
                 eta_s3Mj
          of
          { (# ipv_i3sR, ipv1_i3sS #) ->
          benchIndices_$s$wbenchLoop2
            x_s3Me (subWord64# wild_X1 1##64) ipv_i3sR
          };
        0##64 -> eta_s3Mj
      }
end

AFAIU the effect of specialization is that the function has been specialized to inline instance NFData (Maybe a). I think it's a good effect: indeed we'd rather have less overhead caused by benchLoop.

from tasty-bench.

@AndreasPK what do you think about 9c86990? It seems to resolve the issue for bytestring: now everything is as specialized as possible.

from tasty-bench.

@AndreasPK what do you think about 9c86990? It seems to resolve the issue for bytestring: now everything is as specialized as possible.

Using SPEC isn't something I had considered but should work. I that's the only way tasty-bench runs into the SpecConstr issue that's probably good enough.

from tasty-bench.

Thanks for the report and for the analysis, @AndreasPK! I'll merge 9c86990 then.

from tasty-bench.