Statistics on simulation frequency dumped only for Treadle about chiseltest HOT 5 CLOSED

Actually two issues in one here:

1. How statistics should be printed (also see #68): from a regressions / automated test suite perspective, printing stuff during unit tests is probably not too useful. But I agree that statistics are probably useful. What do you think about the idea in the other thread about the test(...) {...} method returning some kind of TestResult object, with a statistics field? This could either be printed by the user, or automatically printed if some environment config/flag-like thingy is set.

2. More of an issue specifically with the Verilator backend, it's largely based on legacy code from ages ago. Splitting up elaboration / FIRRTL time isn't too bad since that's within the control of testbench core, but getting Verilator compile vs. simulation time might involve nontrivial plumbing.

from chiseltest.

@ducky64 Keeping an eye on turnaround times and keeping them low is a big part of being productive. Can we automate this task for the user? I think I would prefer unit-tests that enforce non-functional performance requirements. I made one such pull request that fails if elaboration happens twice. There should be another unit-test that fails if Verilator binary is not re-used when it can be reused. A unit-test that fails if ccache is not used by default with Verilator if it is installed.

from chiseltest.

Those tests sound reasonable, though they seem to require progressively deeper hooks into how Verilator runs. If you think it's possible to write those tests cleanly, go for it, though the Verilator backend is largely a black box of legacy code to me. There's ideas on how we could clean it up (eg, using standardized interfaces like VPI / DPI, maybe also JNI for performance since IPC is slow), but no timeline.

Having compute performance as part of the regressions is much trickier, since it both depends on the host machine as well as the internals of Verilator that we have basically no control over. I'm open to ideas, but I don't see any obvious solutions.

from chiseltest.

I'm glad to hear that you welcome contributions to switch from IPC to JNI. It's meaningless to talk about peek/poke performance of Treadle vs. Verilator when it's IPC that's dominating.

I don't know how to do compute performance as part of the regression tests, but a first, much simpler step is to check for non-functional requirements, like only elaborating once.

from chiseltest.

We won't support these printouts. getStepCount can be used to implement something similar in the user code: https://github.com/ucb-bar/chiseltest/blob/7165c349c7b69e28690a9de88e20f8a3f7c812e4/src/test/scala/chiseltest/tests/GetStepCountTest.scala

from chiseltest.