Implement "previous failures first" mode about phpunit HOT 18 CLOSED

+1 any updates?

from phpunit.

Any updates?

from phpunit.

Is it possible to set this in phpunit.xml?
-edit Never mind

cacheResult="true"
executionOrder="defects,reverse"

from phpunit.

I'm a bit 0/-1 on that functionality. While implement a fix for sth, it is good to not break anything else.
And if you know which exactly component is problematic and needs fixing, just filter to run tests only for it, it's already possible

from phpunit.

@GrahamCampbell you should be able to filter on data providers:

Without:

$ ../bin/test.sh --filter "CurrencyRateTest::testNormalizeException"
~/work/project/laravel ~/work/project/laravel
Configuration cache cleared!
    PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.7 with Xdebug 2.9.5
Configuration: /Users/petah/work/project/laravel/phpunit.xml
Random seed:   1594163298


CurrencyRateTest testNormalizeException with data set #5.
CurrencyRateTest testNormalizeException with data set #4.
CurrencyRateTest testNormalizeException with data set #1.
CurrencyRateTest testNormalizeException with data set #3.
CurrencyRateTest testNormalizeException with data set #2.
CurrencyRateTest testNormalizeException with data set #0.                                                              6 / 6 (100%)

Time: 1.47 seconds, Memory: 38.00 MB

OK (6 tests, 12 assertions)

With:

$ ../bin/test.sh --filter "CurrencyRateTest::testNormalizeException.*2"
~/work/project/laravel ~/work/project/laravel
Configuration cache cleared!
    PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.7 with Xdebug 2.9.5
Configuration: /Users/petah/work/project/laravel/phpunit.xml
Random seed:   1594163303


CurrencyRateTest testNormalizeException with data set #2.                                                                   1 / 1 (100%)

Time: 1.01 seconds, Memory: 34.00 MB

OK (1 test, 2 assertions)

from phpunit.

Proposal: a command:
phpunit --run-failed
which reads the junit log file defined in phpunit.xml or from the passed --log-junit option.
The presence of the log file is a prerequisite for using the feature, because writing one by default would impact performance.

from phpunit.

This would have to be limited to tests that have no dependency (@depends) in the current architecture of PHPUnit.

from phpunit.

IMHO starting from PHPUnit 7.3, because of #3147, this should be possible via --cache-result --order-by=defect

from phpunit.

@reinholdfuereder Yes! That is exactly what the feature is for.

To help protect against unexpected breakages, you can activate the @depends checker to keep tests in as good an order as possible: --cache-result --order-by=depends,defects

from phpunit.

OK, great!

@sebastianbergmann So maybe this issue can be already resolved/closed as (mostly) fixed? (Although one or the other bug might be still around and documentation might need to be added/updated/completed; but that might be already tracked elsewhere, I guess...)

from phpunit.

@reinholdfuereder The combi of functionality from #3147 + #3092 was explicitly aimed at providing this. I am not familiar with JUnit --run-failed and cannot find it in their docs. If anybody has a link that'd be great.

The only missing bit of functionality I still want myself is to only run failed and not just sorting defects to the front of a normal run. I'd have to implement a --filter based on 'status after the previous run(s)'. Totally up for doing that; will have a look when I'm settled at my new job.

from phpunit.

FWIW you can pretty easily write a listener to rerun failed tests:

class OutputListener implements \PHPUnit\Framework\TestListener
{
    private $failures = [];

    public function addError(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        $this->failures[get_class($test)] = true;
    }

    public function addFailure(PHPUnit_Framework_Test $test, PHPUnit_Framework_AssertionFailedError $e, $time)
    {
        $this->failures[get_class($test)] = true;
    }

    public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        $content = 'vendor/bin/phpunit --filter ' . implode('\|', array_keys($this->failures));
        $script = __DIR__ . '/../rerun-failed-tests.sh';
        file_put_contents($script, $content . PHP_EOL);
    }
}

from phpunit.

@keradus
As a software developer
I want to rerun failed tests first THEN all tests
so that I can have both a quick red-green cycle and the safety of full coverage

Sounds like something you'd want? :)

from phpunit.

from opening message:

The PHPUnit test runner should be enhanced to read a list of previously failed tests from an XML logfile and run these tests before the other tests.

maybe not with xml, but with --order-by=defects we have this already possible.
imho, nothing left here and this feature request can be closed as done

from phpunit.

Agreed :)

from phpunit.

yes, should be solved with --cache-result --order-by=depends,defects.

from phpunit.

And if you know which exactly component is problematic and needs fixing, just filter to run tests only for it, it's already possible

Can't filter on data providers though? I only wanna rerun number 624, and not the other two thousand...

from phpunit.

Oh, cool. Thanks @Petah. 🍻

from phpunit.