fiunchinho / phpunit-randomizer Goto Github PK

A PHPUnit extension that allows you to execute your test cases in a random order

License: MIT License

PHP 100.00%

phpunit-randomizer's Introduction

phpunit-randomizer

PHPUnit has now similar features to this plugin already built-in, so you wouldn't need to use this plugin. To execute your tests in a random order just

vendor/bin/phpunit --order-by=random

You can pass a known seed to get the same order (useful when tests have failed in a certain order, so you can re-run the suite)

SEED=$(head -200 /dev/urandom | cksum | cut -f1 -d " ")
echo "Running tests with seed $SEED"
vendor/bin/phpunit --order-by=random --random-order-seed=$SEED

A PHPUnit extension that allows you to execute your test cases in a random order. This way you can identify tests that depend on other tests because they share some state, like object state, or even database state. PHPUnit has an option to execute test cases in process isolation, but that takes a lot of time when you have many tests.

With this library, you don't have to modify any PHPUnit code. Just install it, and use the binary from the vendor folder (vendor/fiunchinho/phpunit-randomizer/bin/phpunit-randomizer) to run your tests, instead of using the default phpunit command.

Installation

composer require --dev fiunchinho/phpunit-randomizer

Usage

The executable binary is under the bin folder and it works exactly the same as the default phpunit file, accepting the same arguments, except for one: The order argument.

$ vendor/bin/phpunit-randomizer -h

When you execute your tests using this library, you will get the same results as using a regular PHPUnit installation unless you use the --order rand parameter. Try executing the following:

$ vendor/bin/phpunit-randomizer --order rand

If you look to the output, you'll see that it says that the tests have been randomized using a random seed. If you don't specify any seed (like the example above), the seed will be calculated randomly every time you execute your tests. But if you want to repeat an specific order that is interesting to you (because it failed, for example), you can re-run that order specifing the seed in your order argument.

For example, let's try to execute the example tests (that print the name of the test case so we can see the order in which they are executed) twice, and see how the order differ.

Here you can see how the order of the test cases is different in the two executions. Also, you can see the seed used in both cases (8639 in the first one, 4674 in the second). Now, if we want to repeat the order of the second execution, we can select the same seed.

As you can see, the order is exactly the same.

PHPUnit

PHPUnit Documentation

phpunit-randomizer's People

Contributors

Stargazers

Watchers

Forkers

codeyellowbv eltonoliveira hasumedic gruenspar marc-casoliva prymas007 qsnipp bruli hendan lolli42 sanmai

phpunit-randomizer's Issues

Run again with same seed (randomized order)

Hi,
Awesome extension of phpunit!
It works like a charm.

What you think about adding functionality for being able to run the same randomized order (seed) again?

To compare, here's the equivalent feature in RSpec:
https://relishapp.com/rspec/rspec-core/docs/command-line/order-new-in-rspec-core-2-8

Fatal error

PHP Fatal error: Uncaught Declaration of PHPUnitRandomizer\TestRunner::doRun(PHPUnit_Framework_Test $suite, array $arguments = Array) should be compatible with PHPUnit_TextUI_TestRunner::doRun(PHPUnit_Framework_Test $suite, array $arguments, $exit)

XX/vendor/symfony/phpunit-bridge/DeprecationErrorHandler.php:57
XX/vendor/fiunchinho/phpunit-randomizer/src/PHPUnitRandomizer/TestRunner.php:4
XX/vendor/fiunchinho/phpunit-randomizer/src/PHPUnitRandomizer/Command.php:56
XX/vendor/fiunchinho/phpunit-randomizer/src/PHPUnitRandomizer/Command.php:13

thrown in XX/vendor/fiunchinho/phpunit-randomizer/src/PHPUnitRandomizer/TestRunner.php on line 4

Default to regular order, enable random with flag

Just throwing an idea out:
What do you think about defaulting to regular order, and enabling random with a flag.

That way, people can use this package as a wrapper around phpunit, and run it in the same order as usual, and enable random order whenever you want to do that.

Again, with inspiration from RSpec, I suggest we use "--order rand":
https://relishapp.com/rspec/rspec-core/docs/command-line/order-new-in-rspec-core-2-8

option --order is ambiguous

Getting an error while running

vendor/bin/phpunit-randomizer --order rand
PHPUnit 7.4.3 by Sebastian Bergmann and contributors.

option --order is ambiguous

Any idea?

PHPUnit 6.0

PHPUnit 6.0 has been released.

One of the large changes is that classes do not use the _ namespace separator, but rather the PSR \ namespace separator. (See https://github.com/sebastianbergmann/phpunit/blob/master/ChangeLog-6.0.md).

This breaks the usages of PHPUnit-randomizer where the old classes are imported. Would you accept a PR which uses the new classes, and release a new version of your randomizer? Most of the namespaced classes are in use since somewhere around PHPUnit 5.4.

setUpBeforeClass and tearDownAfterClass are not being executed

For some reason, PHPUnit is not executing setUpBeforeClass and tearDownAfterClass when using the randomizer.
This is really important, because many tests that rely on these methods will fail.

Consider the @depends annotation

The annotation @Depends is not taken into account while it's necessary to run my tests.

You cannot execute a test B which depends on a test A before the test A.

It is possible to fix that behavior or is it intentional ?

PHPUnit version

Looking at the composer file, it seems that phpunit-randomizer is tied to version 4.2 of PHPUnit.
Can we safely raise the required version or there is a reason behind that?
As you might know, PHPUnit started to release new versions more frequently, so it would be great if we can use the latest version of it

You don't need this - There is a default PHPUnit feature --order-by=random

For everybody that finds this repo and wants to try it out: you don't need this.

You can run the following:

vendor/bin/phpunit --order-by=random

To the author: It would be great if you could change the readme of this project to say it's not needed anymore, because PHPUnit has an --order-by=random feature.

Best,

Ruud

Adding the project to Packagist

We need to start using semantical versioning and register the project in packagist.

Add a license

Please consider adding a license to this package. ATM it is unclear how this package may be used.