Running working tests with mutations causes errors and no report being generated about ooze HOT 9 CLOSED

Hey there @hnord-vdx. Thanks for reporting.

Indeed the 10 minute mark seems quite specific; it seems like you're reaching a timeout somewhere. Do you happen to have any timeouts configured in any of your test running scripts?

The error output you're getting seems to be coming from something aborting the execution of the mutation tests.

Keep in mind that, although your all your tests pass, mutation tests run them over and over for each mutation it generates; so it will likely take more than 10 minutes.

Do you mind sharing how (which command) you're running the mutation tests?

from ooze.

Hi @gtramontina

I first thought the problem lied with docker test when we were running the test suite, but it still fails after 10 minutes (when we ignored those files). Perhaps it has to do with the tests for our http handlers.. I will look into that.

The command I used is: go test -v -tags=mutation

from ooze.

You're probably running into timeouts because of the mutations. For example, if you're calling sync.WaitGroup.Add with a fixed amount, a mutation would increase that amount by 1, with the effect being that the wait group will never complete, resulting in a timeout from the testing framework.

You could try using ooze.WithTestCommand with something like go test -count=1 -timeout=10s ./... to shorten the timeout. Ideally your original unit tests run very fast, so 10 seconds should be plenty.

from ooze.

@hnord-vdx do you mind sharing the Ooze settings you're using (sorry, should've asked earlier)?

One thing to note is that if you setup mutation tests to run in parallel, and you have tests that depend on state or network (as you mentioned HTTP handlers), the mutation test runs can interfere with one another, causing false negatives (tests failing for reasons that are not the mutation).

@blizzy78 That's a very good guess¹! Although the 10 minute mark is a bit suspicious to me. Mutation tests can easily take longer than 10 minutes to run. @hnord-vdx Are you hitting this timeout while running the mutation tests locally or on a third-party build pipeline (github actions/travis-ci/…)?

from ooze.

I get the same error:

goroutine 4265 [chan send]:
github.com/gtramontina/ooze/internal/future.(*DeferredFuture[...]).Resolve.func1.1()
        /home/knutzehe/go/pkg/mod/github.com/gtramontina/[email protected]/internal/future/deferred.go:28 +0x2a
created by github.com/gtramontina/ooze/internal/future.(*DeferredFuture[...]).Resolve.func1
        /home/knutzehe/go/pkg/mod/github.com/gtramontina/[email protected]/internal/future/deferred.go:27 +0xbb
exit status 2
FAIL    github.com/hknutzen/Netspoc/go  600.621s

when applying Oooze to my project https://github.com/hknutzen/Netspoc
I added a file Netspoc/go/mutation_test.go

//go:build mutation

package main_test

import (
	"testing"

	"github.com/gtramontina/ooze"
	"github.com/gtramontina/ooze/viruses/comparisoninvert"
)

func TestMutation(t *testing.T) {
	ooze.Release(
		t,
		ooze.Parallel(),
		ooze.WithTestCommand("go test -count=1 -timeout=30s ./..."),
		ooze.WithViruses(comparisoninvert.New()),
	)
}

and ran this command in directory Netspoc/go/
go test -tags=mutation

Without Ooze, the 3583 tests run in parallel in about 0.8s

from ooze.

@hknutzen I'll clone your repo and try to diagnose locally.

One thing that always draws my attention is to these somewhat round numbers (600.621s ~= 10 minutes).

Are you hitting this timeout while running the mutation tests locally or on a third-party build pipeline (github actions/travis-ci/…)?

☝️

from ooze.

Argh! 🤦 How come this wasn't the first think that came to mind?! 🤦😅
Go testing library's default timeout is 10 minutes! If you run your mutation tests with go test -tags=mutation -timeout=1h, for example, you should get past this issue.

This is how Ooze's own mutation tests are executed:

Have a try and let me know! cc/ @hknutzen @hnord-vdx

from ooze.

Ok thanks, with go test -tags=mutation -timeout=1h I get a result:

????????????????????????????????????????????????????????????????????????????????
?? ?? Total:     2485                    ??
?? ?? Killed:    2478                    ??
?? ?? Survived:     7                    ??
????????????????????????????????????????????????????????????????????????????????
?? ?? Score:     1.00 (minimum: 1.00)    ??
????????????????????????????????????????????????????????????????????????????????
FAIL
exit status 1
FAIL    github.com/hknutzen/Netspoc/go  1064.948s

It found one real mutant:

?? ?? Mutant survived: pkg/pass1/check-identical-services.go ?? Comparison Invert
????????????????????????????????????????????????????????????????????????????????
?? --- pkg/pass1/check-identical-services.go (original)
?? +++ pkg/pass1/check-identical-services.go (mutated with 'Comparison Invert')
?? @@ -97,7 +97,7 @@
??                                               return false
??                                       }
??                                       if l1[k] != l2[k] {
?? -                                             return name < l2[k]
?? +                                             return name >= l2[k]
??                                       }
??                               }

The other 6 mutants are not relevant. They are found in test/tstdata/testdata.go, which is only used for testing.

from ooze.

That's good news, then? 😄 I'm assuming @hnord-vdx's issue is the same, so I'll close this issue.

As for the non-relevant mutations, you can ignore them using the IgnoreSourceFiles option.

from ooze.