Code coverage support (hpc) about rules_haskell HOT 6 CLOSED

@SebastianKG can we close? Reassigning to you in the meantime.

from rules_haskell.

I'll write up some documentation, then close. We have the functionality, but no one knows how to use it except me.

from rules_haskell.

The documentation leaves a few questions for me. When the coverage is enable by specifying expected_expression_coverage, what are the differences between this and no coverage except for the report and possibly test failure? Does it compile in a different mode? Are there performance considerations? Could latent bugs be triggered?

How does one generate a coverage report, or is it not yet possible?

This probably doesn't belong here but I'm not sure % coverage as an integer is a sufficient metric to prevent regressions. If you have 100 tests and someone adds one it probably won't change the number. A better metric could be number of uncovered "things" like expressions, if conditions, etc. Is there a mechanism for making people update the number when they improve it? Maybe if the actual coverage is not exactly equal to the number it would be better than if the actual coverage was less than the number for a failure condition.

from rules_haskell.

@ghorn great questions, thanks for reviewing this work.

When the coverage is enable by specifying expected_expression_coverage, what are the differences between this and no coverage except for the report and possibly test failure? Does it compile in a different mode? Are there performance considerations? Could latent bugs be triggered?

I didn't want to get too technical in the documentation, but it would be good to cover the basics of what the user should expect. Since there are additional steps during compilation, and since the resulting files are instrumented to generate coverage checklists at runtime, the user should expect both slightly slower compilation and slightly slower execution. This only occurs when running bazel coverage, and hasn't changed the performance of run/build/test. I'll add this information to the doc.

How does one generate a coverage report, or is it not yet possible?

Not possible yet, but definitely would be awesome. It isn't really possible to output a file from a run of bazel coverage (nor from bazel test), so I would have to find another makeshift solution for getting the user that coverage report. hpc does support html output, so that part of the process would be easy.

This probably doesn't belong here but I'm not sure % coverage as an integer is a sufficient metric to prevent regressions. If you have 100 tests and someone adds one it probably won't change the number. A better metric could be number of uncovered "things" like expressions, if conditions, etc. Is there a mechanism for making people update the number when they improve it? Maybe if the actual coverage is not exactly equal to the number it would be better than if the actual coverage was less than the number for a failure condition.

This is really interesting feedback. I've always seen coverage used as a percentage, but I see some advantages to a raw count of uncovered expressions. As you said, I think this is a problem for a different issue, but I'll keep it in mind.

from rules_haskell.

Myy PR got merged while I was writing all that. 😅 will still add information about coverage performance in a secondary PR. Closing this issue though, now that there is some documentation out there.

from rules_haskell.

@Fuuzetsu check out the new documentation for information on how to use the new functionality https://rules-haskell.readthedocs.io/en/latest/haskell-use-cases.html#checking-code-coverage

from rules_haskell.