jmid / mutaml Goto Github PK

Hello, It seems that the instrumentation is launched before the preprocessor, generating errors in some cases.
Here is a minimalistic example with sedlex preprocessing :

$ git clone https://github.com/jmid/mutaml.git
$ cd mutaml
$ opam install .
$ cd ..
$ git clone https://github.com/epatrizio/mutaml_exp.git
$ cd mutaml_exp
$ dune runtest --instrument-with mutaml
File "lib/dune", line 4, characters 2-18:
4 |   (pps sedlex.ppx))
      ^^^^^^^^^^^^^^^^
Running mutaml instrumentation on "lib/lib.ml"
Randomness seed: 374585493   Mutation rate: 50   GADTs enabled: true
Created 4 mutations of lib/lib.ml
Writing mutation info to lib/lib.muts
File "lib/lib.ml", line 12, characters 4-24:
12 |   | "hello" -> token buf
         ^^^^^^^^^^^^^^^^^^^^
Error: Sedlex: 'when' guards are not supported

What happens is that mutaml inserts some when guards on a branch from a match%sedlex construct. Sedlex then complains because it can not handle such guards.
I don't know if it's possible to tell dune to instrument the code only after preprocessing ? Otherwise, should the instrumentation tool (mutaml) be careful and not modify code that contains preprocessing annotations ?
Thanks a lot!

Yesterday when I was instrumenting dune_engine I ran into a compiler warning only when building with the instrumentation.

To reproduce:

1. Clone the https://github.com/ocaml/dune repository.
2. Dune doesn't need any libraries, so the opam switch for mutaml should be fine. You can even plop dune inside mutaml and build it directly from there so dune can find the mutaml lib incase it is not inside the switch.
3. In https://github.com/ocaml/dune/blob/main/src/dune_engine/dune add another (instrumentation) field at the end for mutaml.
4. Do dune build @src/dune_engine/check --instrument-with mutaml and you will get an error (fatal warning).

File "src/dune_engine/process.ml", line 673, characters 8-17:
673 |       | WEXITED n -> Error (Failed n)
              ^^^^^^^^^
Error (warning 12 [redundant-subpat]): this sub-pattern is unused.

My workaround yesterday was to just do the build in --profile release so that compiler warnings were non-fatal. I thought I would report this anyway. This isn't quite a compiler error that you've said needs reporting, but I thought I'd mention it anyway in case there is something you can do about it.

I fell like redundant cases can be removed if they are detected. No idea if this is simple to do.

The ocaml-ci tests are failing on Alpine because it doesn't understand diff --color - which is unfortunately hard-coded for now.
We should add, e.g., an environment variable to be able to adjust it.

To avoid running all the tests for all the mutations, mutaml-runner could set an environment variable MUTAML_MUTATED_FILES containing a list of files that have mutations active. This would allow to condition a test so that it only runs if the code that it tests is part of the files that are mutated.

There are some run time dependencies that are not met when the test runner is run within mutaml-runner.

Here is what I tried:

$ mutaml-runner _build/default/test/expect-tests/dune_engine/.dune_engine_test.inline-tests/inline_test_runner_dune_engine_test.exe
Could not read file mutaml-mut-files.txt - _build/default/mutaml-mut-files.txt: No such file or directory

$ mutaml-runner "dn build @test/expect-tests/dune_engine/runtest --profile release --instrument-with mutaml --always-show-command-line --force"
Could not read file mutaml-mut-files.txt - _build/default/mutaml-mut-files.txt: No such file or directory

This is after

$ dn build @test/expect-tests/dune_engine/runtest --instrument-with mutaml --profile release

(I have to use release since some fatal warnings are being triggered).

(dn is just a bash alias for dune).

It would be great to get this working for expect tests, I've been wanting to try it out on the dune codebase.

Hey!

I couldn't find a current version for mutaml either in tags/releases tab as well as link to tar.gz of particular version. Would it be possible to create some release (0.1 or any other)? It blocks us from using mutaml internally.

I would be happy to help with that and further issues as well!

Thanks.

When running mutaml-runner it would be interesting to select mutation files from the command line (mutaml-runner --muts src/foo.muts "…") rather than having them selected from the contents of the mutaml-mut-files.txt file.

The reason this would be useful is that some test executables may exercise some parts of the code-base only. There is no point in trying mutations in files unrelated to the tests that are being run.

Right now I work around this by editing the mutaml-mut-files.txt file, running one test executable, re-editing to select a different set of mutations. running a different test executable, etc. Being able to run the following instead would be useful.

mutaml-runner --muts src/foo.muts "dune exec test/test_foo.exe"
mutaml-runner --muts src/bar_blah.muts,src/bar_blob.muts "dune exec test/test_bar.exe"

(Even better would be to pass mutaml-runner a mapping of test executables to source files and have it consume that. But it is significantly more work, and the --muts parameter would already solve the core issue.)

Hey.

What are these three dependencies:

• "conf-timeout"
• "conf-which"
• "conf-diffutils"

I found them in opam, but didn't find the source repo and usages of them in mutaml. Can you please explain to me what they are actually doing and point out to repo?

I would like to improve the dune support for mutaml. Could you give a run down of the current issues affecting it? I can see what we can do on the Dune side to accommodate your use case.

While testing on my old macBook I realized its diff didn't support --color.
It would be nice:tm: to settle on a diff command usable across both Linux, macOS, and *BSD

Ideally, a diff command should

• support colored output - for prettier, 202* output
• support some form of --labeling - to avoid distracting users with temporary file names

In comparison, dune uses git diff over diff:
https://github.com/ocaml/dune/blob/6c573436428c2ebbb6f435b1c712777a81dc8a42/src/dune_engine/print_diff.ml#L107-L131

Hi, I'd like to thank you for creating this tool! Is it possible to do mutations only on a subset of modules in a library? I'd like to try it on our project, but for the sake of performance I'd like to run it only on one or two important modules and not to alter the library structure of the project.