zimmski / go-mutesting Goto Github PK

Via the argument "--match" it should be possible to apply the mutation process only on certain functions.

Go get fails for me:

$ go get -t -v github.com/zimmski/go-mutesting/...
github.com/zimmski/go-mutesting (download)
github.com/jessevdk/go-flags (download)
github.com/zimmski/go-tool (download)
github.com/zimmski/osutil (download)
github.com/stretchr/testify/vendor/github.com/pmezard/go-difflib/difflib
github.com/zimmski/go-mutesting/mutator
github.com/jessevdk/go-flags
golang.org/x/tools/go/buildutil
github.com/zimmski/go-mutesting/astutil
github.com/zimmski/go-tool/importing
golang.org/x/tools/go/ast/astutil
github.com/stretchr/testify/vendor/github.com/davecgh/go-spew/spew
github.com/zimmski/go-mutesting/mutator/expression
github.com/zimmski/go-mutesting/mutator/branch
github.com/zimmski/go-mutesting/mutator/statement
github.com/zimmski/go-mutesting/example
github.com/zimmski/go-mutesting/example/sub
github.com/stretchr/testify/assert
golang.org/x/tools/go/loader
github.com/zimmski/go-mutesting
github.com/zimmski/osutil
github.com/zimmski/go-mutesting/test
# github.com/zimmski/osutil
\Local\Temp\go-build052092062\github.com\zimmski\osutil\_obj\_cgo_main.o:_cgo_main.c:(.data+0x0): undefined reference to `stdout'
\Local\Temp\go-build052092062\github.com\zimmski\osutil\_obj\_cgo_main.o:_cgo_main.c:(.data+0x8): undefined reference to `stderr'
collect2.exe: error: ld returned 1 exit status

We need to test the cmd and simple.sh so we can make automatically sure that the project really works.

Trying to run go get -t -v github.com/zimmski/go-mutesting/...

The following happens:

github.com/zimmski/osutil
# github.com/zimmski/osutil
../../../../github.com/zimmski/osutil/capture.go:79: cannot assign to _Cmacro_stderr()
../../../../github.com/zimmski/osutil/capture.go:79: cannot assign to _Cmacro_stdout()
../../../../github.com/zimmski/osutil/capture.go:103: cannot assign to _Cmacro_stderr()
../../../../github.com/zimmski/osutil/capture.go:103: cannot assign to _Cmacro_stdout()

What can I do to solve it?

PIT is definitely the best mutation tester out there. Add every feature, mutator and idea in the issue tracker.

I tried go-mutesting on an operational transformation algorithm I'm implementing (package github.com/mstone/focus/ot) and got this panic while running:

go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" github.com/mstone/focus/ot

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x448bd5]

goroutine 16 [running]:
runtime.panic(0x5dd0c0, 0x71caf3)
    /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
go/ast.(*Ident).Pos(0x0, 0x57f2c0)
    /usr/lib/go/src/pkg/go/ast/ast.go:423 +0x5
go/ast.(*BinaryExpr).Pos(0xc2080682d0, 0xc2080a4798)
    /usr/lib/go/src/pkg/go/ast/ast.go:441 +0x3f
go/printer.(*printer).binaryExpr(0xc20804e000, 0xc208068300, 0x0, 0x6, 0x1)
    /usr/lib/go/src/pkg/go/printer/nodes.go:647 +0x399
go/printer.(*printer).expr1(0xc20804e000, 0x7f46a38d5ae0, 0xc208068300, 0x0, 0x1)
    /usr/lib/go/src/pkg/go/printer/nodes.go:686 +0xf81
go/printer.(*printer).expr(0xc20804e000, 0x7f46a38d5ae0, 0xc208068300)
    /usr/lib/go/src/pkg/go/printer/nodes.go:893 +0x4d
go/printer.(*printer).controlClause(0xc20804e000, 0xc2080a4d00, 0x0, 0x0, 0x7f46a38d5ae0, 0xc208068300, 0x0, 0x0)
    /usr/lib/go/src/pkg/go/printer/nodes.go:996 +0x12c
go/printer.(*printer).stmt(0xc20804e000, 0x7f46a38d5b18, 0xc20806e3c0, 0x100)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1151 +0x29f2
    /usr/lib/go/src/pkg/go/printer/nodes.go:942 +0xfe
go/printer.(*printer).adjBlock(0xc20804e000, 0x33, 0xb, 0xc208068510)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1514 +0x683
go/printer.(*printer).funcDecl(0xc20804e000, 0xc208068540)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1538 +0x2b5
go/printer.(*printer).decl(0xc20804e000, 0x7f46a38d59c8, 0xc208068540)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1548 +0xa4
go/printer.(*printer).declList(0xc20804e000, 0xc20804a400, 0x13, 0x20)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1589 +0x1b0
go/printer.(*printer).file(0xc20804e000, 0xc20804c500)
    /usr/lib/go/src/pkg/go/printer/nodes.go:1597 +0x18b
go/printer.(*printer).printNode(0xc20804e000, 0x5ce020, 0xc20804c500, 0x0, 0x0)
    /usr/lib/go/src/pkg/go/printer/printer.go:1087 +0x8b2
go/printer.(*Config).fprint(0xc2080a57d8, 0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0xc208068150, 0x0, 0x0)
    /usr/lib/go/src/pkg/go/printer/printer.go:1226 +0xad
go/printer.(*Config).Fprint(0xc2080a57d8, 0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0x0, 0x0)
    /usr/lib/go/src/pkg/go/printer/printer.go:1284 +0x96
go/printer.Fprint(0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0x0, 0x0)
    /usr/lib/go/src/pkg/go/printer/printer.go:1291 +0xb0
main.saveAST(0xc208024720, 0xc2080b00c0, 0x57, 0xc20800e040, 0x7f46a38d5ba0, 0xc20804c500, 0x0, 0x0, 0xc208024500, 0x0, ...)
    /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:396 +0x202
main.mainCmd(0xc20800e010, 0x3, 0x3, 0xa)
    /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:257 +0x1dfc
main.main()
    /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:352 +0x69

goroutine 19 [finalizer wait]:
runtime.park(0x417dc0, 0x721070, 0x71fb89)
    /usr/lib/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x721070, 0x71fb89)
    /usr/lib/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/lib/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/lib/go/src/pkg/runtime/proc.c:1445

goroutine 22 [chan receive]:
github.com/zimmski/go-mutesting/mutator/expression.(*MutatorRemoveTerm).Mutate(0x721570, 0x7f46a38d60a8, 0xc2080682d0, 0xc2080b0240)
    /home/mistone/go/src/github.com/zimmski/go-mutesting/mutator/expression/remove.go:76 +0x1b6
github.com/zimmski/go-mutesting.(*mutateWalk).Visit(0xc2080b6020, 0x7f46a38d60a8, 0xc2080682d0, 0x0, 0x0)
    /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:71 +0x115
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d60a8, 0xc2080682d0)
    /usr/lib/go/src/pkg/go/ast/walk.go:52 +0x58
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d60a8, 0xc208068300)
    /usr/lib/go/src/pkg/go/ast/walk.go:147 +0x1ead
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d6078, 0xc20806e3c0)
    /usr/lib/go/src/pkg/go/ast/walk.go:230 +0x43bc
go/ast.walkStmtList(0x7f46a38d5bd0, 0xc2080b6020, 0xc20806e580, 0x3, 0x4)
    /usr/lib/go/src/pkg/go/ast/walk.go:32 +0xd1
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5ec8, 0xc208068510)
    /usr/lib/go/src/pkg/go/ast/walk.go:224 +0x41e7
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5e68, 0xc208068540)
    /usr/lib/go/src/pkg/go/ast/walk.go:342 +0xccb
go/ast.walkDeclList(0x7f46a38d5bd0, 0xc2080b6020, 0xc20804a400, 0x13, 0x20)
    /usr/lib/go/src/pkg/go/ast/walk.go:38 +0xd1
go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5ba0, 0xc20804c500)
    /usr/lib/go/src/pkg/go/ast/walk.go:351 +0x2cb6
github.com/zimmski/go-mutesting.func·001()
    /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:51 +0x71
created by github.com/zimmski/go-mutesting.MutateWalk
    /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:54 +0xdb

To anyone who is reading this. i was a long time away from open source projects and have now a little more time on my hand. If you are interested in go-mutesting, please help me get this project back on track. first step is to have a CI again for the repository. If you can help with that i would be eternally grateful.

When trying to build, I get this error message:

$ go build cmd/go-mutesting/main.go
# github.com/zimmski/osutil
../osutil/capture.go:79: cannot assign to _Cmacro_stderr()
../osutil/capture.go:79: cannot assign to _Cmacro_stdout()
../osutil/capture.go:103: cannot assign to _Cmacro_stderr()
../osutil/capture.go:103: cannot assign to _Cmacro_stdout()

After some digging I believe the problem is this:
golang/go#25221

Here's a workaround for building it for MacOS (darwin):

docker run -it --name go-mutesting golang:1.9 /bin/bash -c "go get -t -v github.com/zimmski/go-mutesting/...; cd /go/src/github.com/zimmski/go-mutesting; GOOS=darwin go build cmd/go-mutesting/main.go"
docker cp go-mutesting:/go/src/github.com/zimmski/go-mutesting/main go-mutesting
docker rm go-mutesting

And for Linux:

docker run -it --name go-mutesting golang:1.9 /bin/bash -c "go get -t -v github.com/zimmski/go-mutesting/...; cd /go/src/github.com/zimmski/go-mutesting; go build cmd/go-mutesting/main.go"
docker cp go-mutesting:/go/src/github.com/zimmski/go-mutesting/main go-mutesting
docker rm go-mutesting

• Rename simple.sh to all.sh and document that it runs all tests of the current directory
• Add more ENV variables so that the exec commands know which package and directory they are in
• Do a new simple.sh which runs only the tests of the current file's package
• Document simple.sh

/example has a LOT of duplicate types of mutations. We only need a few here to handle the tests of /cmd/go-mutesting/main_test.go and to show some mutations to demonstrate go-mutesting in general.

By reducing the content we would also largely reduce the testing time.

Have a look at other tools and papers for ideas.

After a mutation is generated it is normaly directly tested against the test suite. Filters should be implemented that lay between generation and testing. A filter then verifies that the mutation is valid and worthy to be tested if not the mutation is not tested. These mutations are unlike a "skipped mutation" not "skipped" but "filtered" which means the mutation does not reach a exec command and they are not included in the mutation score.

A fine example ist the validation if a mutation only concerns a logging statement which is (in almost all cases) not tested, and also must not be tested. Meaning, the mutation is a false positive and will be ignored by the user. A filter could automated this detection.

Example:

for key, val := range thing {
  doSomething(key, val)
}

Gets mutated into:

for key, val := range thing {
}

This will never compile because key and val are declared and never used, thus producing a false negative. It should probably mutate it into:

for _, _ := range thing {
}

That said, I'm not that familiar with go/ast et. al. so this could be very hard.

When focusing tests on behavior and not necessarily on error handling or logging, it would be nice to be able to skip mutations for certain function calls like fmt.Errorf or klog.Infof to focus mutation tests on the behavior logic.

Even though the repo is not maintained, it's the only place with issues enabled, hence reporting here.

Sometimes there is an initial failure in the test suite. The consequence of such an error is that all mutations are successful which is not the intended outcome.

If it is detected that the original test suite has failing tests, we need to abort the mutation process.

During IdentifiersInStatement we are reusing nodes which often leads to compile errors for the mutations. This for example happens if the node has comment or some position information attached to it. The mutated code is then not correctly formatted, and cannot be compiled.

This would be fixed if we could just clone the node and trim it from its comments and position information.

main.go is a mess. Everything that is not cmd argument related should be put into github.com/zimmski/go-mutesting. This would make it also possible to test a lot of functionality without need for parsing cmd arguments.

When a command is executed and STRG+C is pressed (or the SIGNAL is send) the program exits immidiatelly which is bad because such commands can leave temporary files or the mutated files changed. We should always allow a command to exit.

Since the switch to go/types, go-mutesting throws an error on files using functions defined in other files (I've also had errors with imports).

I've only just discovered the project, so I may have missed something, but here's what I get:

With the last version:

As you can see the conf.Check throws an error (https://github.com/zimmski/go-mutesting/blob/master/cmd/go-mutesting/main.go#L274).

Without the last merge:

Everytime I try to run the command go get -t -v github.com/zimmski/go-mutesting/... I get the following error :

go get github.com/zimmski/go-mutesting/...: no matching versions for query "latest"

it looks like is something related to this issue, but I was not able to make it work. It looks like is something related to the ... at the end of the command.
Any ideas?
Thanks a lot!

If we do "go-mutesting --test-recursive some/package/..." we will go through every Go file and mutate it. The "--test-recursive" option here would maybe indicate to the user that we will run ALL test cases of "some/package". However, we only run the test cases of the package the current Go file belongs to plus all subpackages. This seems confusing and we should give a better usability for this case. Especially since one needs to add a specific exec command to test for this. What would be interesting is maybe to add a "test" (needs better wording) that represents the test command instead of being forced to write a whole exec command.

Add every feature, mutator and idea in the issue tracker.

Is there a way to disable testing of the various 3rd party imports/dependencies? I receive the following errors for an AWS dependency, and the command never fully completes successfully.

build: importGo github.com/aws/aws-xray-sdk-go/internal/logger: exit status 1
error writing go.mod: open /Users/xyz/go/pkg/mod/github.com/aws/[email protected]/go.mod298498081.tmp: permission denied

)
/Users/xyz/go/pkg/mod/github.com/aws/[email protected]/xray/aws.go:22:2: could not import github.com/aws/aws-sdk-go/aws/client (go/build: importGo github.com/aws/aws-sdk-go/aws/client: exit status 1

The AST should be cleaned from all comments, empty lines and odd formatting (i.e. usage of ";") before taking a checksum. This would lead a uniform coding style and could lead to finding more duplications. At least for the code in "example" it would find some.

This should be also done with the content of the original file, so that we can identify if a mutation produces no changes.

All cmd tests are simply looking at the statistic output, this is slightly problematic. All cmd tests should also compare the content of the generated mutations.

Actual:

$ go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" --exec-timeout 10 github.com/zimmski/go-mutesting/example 
SKIP...
< SKIP 11 more >
The mutation score is NaN (0 passed, 0 failed, 12 skipped, total is 12)

Expected:

$ go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" --exec-timeout 10 github.com/zimmski/go-mutesting/example 
PASS...
< 11 more things >
The mutation score is 0.750000 (6 passed, 2 failed, 0 skipped, total is 8)

Looks like there's four more mutators in master now than there were when the README was written, but that still doesn't explain everything being skipped.

• "Bacterio: Java mutation testing tool: A framework to evaluate quality of tests cases" https://dl.acm.org/citation.cfm?id=2473512

E.g. SIGINT, see https://golang.org/pkg/os/signal/ all signals should shutdown the mutation process and especially clean up before exiting the program.

Hi,

I've got an error while executing the following command:

> go-mutesting --verbose --debug .
Enable mutator "branch/case"
Enable mutator "branch/else"
Enable mutator "branch/if"
Enable mutator "expression/comparison"
Enable mutator "expression/remove"
Enable mutator "statement/remove"
Save mutations into "/tmp/go-mutesting-389248154"
Mutate "config.go"
/tmp/sql/config.go:23:10: undeclared name: newConfigError
/tmp/sql/config.go:27:10: undeclared name: newConfigError
/tmp/sql/config.go:31:10: undeclared name: newConfigError
Could not load package of file "config.go": couldn't load packages due to errors: /tmp/sql

newConfigError is in my scenario a function defined in the same package, but in another file.

Playing with the code, I found out here that you are only loading one file if importPath is equal to ".".
As only one file is loaded (instead of all go files in the module), every entities not present in the same file raised exception.

Replacing conf.CreateFromFilenames(dir, fileAbs) with conf.Import(buildPkg.ImportPath) in parse.go solved my problem, but I'm not sure that the best thing to do as I don't really know the inner logic...

• Tag new version with v1.0
• List changes in the tagged release
• Announce on golang-nuts
• Maybe send it to golang-weekly?

I think it is confusing and unhandy for some users to define the --exec option. A default execution command, which must be built-in to work around other problems, of simple.sh would be a nice addition.

Since this would allow an invocation of go-mutesting without arguments we have to also make the following changes:

• allow the binary to do mutation testing with the built-in execution command -> document it
• rewrite the help part of the documentation

Also #25 has an interesting idea. A mutation can be a simple struct which could include only two things: a function to mutate and a function to reset. Making the outside function responsible for the real mutation.

This would drastically reduce the mutator code to simply one function per mutator i.e. "give me all the mutations to this Ast.Node".

Remove statements that are

• no comments -> this would be useless
• no declarations -> would lead to "cannot compile" errors

This means most of the time just assignments and calls.

The mutation files should be generated using a Makefile target.

A mutation can be only killed if a test case kills it. If there are no test cases, there is no need to run the mutation at all. This should be an error, and not skipped.

Hi,

Empty structs instantiations do not mutate correctly and produce code that does not compile.

Example 1:

return http.Header{}, err

will be mutated to

_, _ = http.Header, err

The {} are missing which prevents the mutated code from compiling.

Example 2:

hdr["Hash"] = []string{hash}`

will be mutated to

_, _, _ = hdr, string, hash

The mutated code is broken.

Regards.

At the moment a condition like "if i >= 0" will be substituted by "if false" which is fine for a lot of cases but we can do better. The substitution "if i > 0" has its charm since we can find cases where boundaries where not tested.

Described at http://pitest.org/quickstart/mutators/#CONDITIONALS_BOUNDARY

Hi. Will be better if in black-list file we can write comment about excluded hash. For example

0d764298eb31fa4a23cff2c91d5cdd08 //Can't be done, because ctx.Header(k, v) remove k from map if v is empty.

If u is agree, i will create PR.

Add every feature, mutator and idea in the issue tracker.
Allowing mutation testing via source is a neat feature which is for example missing in go-mutesting.

The problem with using MD5 checksums for blacklisting mutations (blacklisting false-positives) is that a simple change in the source code (e.g. because of a new feature) will change ALL checksums and will therefore make the checksum of the blacklist obsolete. A more smart mechansim must be implemented to make blacklisting really possible.

The current solution is to sequentially test one mutation after another. This is extremely inefficient since most of the time is spend in the execution command. A parallel solution is needed.

• Make it possible to define a "worker amount" via the binary
• Make the generation of the mutations concurrent (no need to add workers here since the mutation generation is not the bottleneck)
• Create a version of simple.sh which runs tests in a temporary folder -> isolate the test execution

Since mutation testing does also generate a lot of false positives, and I do not see any way (yet) to make less of them, it is important to include a blacklist feature like an "ignore" list.

This depends on #1 which then can be used to add checksums of to be ignored mutations on startup.

Add everything that cannot be done yet with go-mutesting

Every generated mutation should be md5-summed so we can check if the mutation did already exists e.g. made by another mutator. It is just wasted time to test the mutation again.

Mutators are currently tested with assets from the FS which have to be changed for every addition. Would be better to keep only diffs of the assets (difference to the original) and keep the diffs in the source code. Or, something else.

I do not think that this is important, since the README is pretty usable as it is. This issue is just needed to complete the documentation.

When I run go-mutesting main.go which uses some of the AWS functionality, it is unable to download them and the mutation does not run.

• Run a normal test without modifications with coverage enabled
• Mark things that are not covered as mutations that can never be killed
• Only look with mutators through covered lines
• Every mutator updates the coverage information -> a mutation that could not be killed -> no need to test here with other mutators
• This makes it also necessary to order the mutators e.g. branch before statement before expression

Returns should be treated differently. They should have their own mutator since they cannot be left out, but their parameters can be adapted.

The migration to go/loader added a lot of execution time during testing. However, it also allows for a perfect type checking.

We could at least remove some of the loading time by loading all files at once, instead of loading each file separately. This would remove the need for parsing and checking the same packages again and again.