Comments (6)
Yes, this is a bit related to #17. Perhaps the set of tests could be cached in .pytest_cache
somehow, and then this could be bisected on. The problem with bisection is that in the worst test isolation failures, cutting the list of tests in the middle fixes them, since it's a test in the first half leaking state into the second half.
Fixing #17 would at least allow manual bisection
from pytest-randomly.
I remember an approach in the context of bisecting for multiple errors: In the bisection step, you can reorganize the partitions to find a suitable split.
For example, assume we have a (full) list of tests F. A test f_x fails if f_p is executed before, f_x and f_p in F. Therefore, we want to identify a subset that contains f_x and f_p.
We start with a bisection and create 2 lists with half of the elements:
H1, H2 where H1 + H2 = F
Now, there are 4 cases (assuming F fails):
- H1 fails, H2 fails => this most likely means we have more issues. We can just continue with H1 and ignore H2 (can be investigated later)
- H1 fails, H2 is ok => continue with H1
- H1 is ok, H2 fails => continue with H2
- H1 is ok, H2 is ok => continue with a new partition
So, both H1 and H2 are ok and f_x is in one list and f_p in the other list. Therefore, we create new partitions. We split up H1 and H2 in half (and get quarters):
Q1, Q2, Q3, Q4 where Q1 + Q2 = H1 and Q3 + Q4 = H2
We create all possible pairs of these 4 lists and use these pairs as input to the previous step:
- (Q1+Q3, Q2+Q4)
- (Q1+Q4, Q2+Q3)
- We can skip (Q1+Q2, Q3+Q4), because we tested it already.
Now, there must be at least one partition where f_x and f_p are both in one half of the selected tests. Continue with the typical approach until we cannot reduce anymore.
It could happen that all these pairs are ok. Then, we would have at least one addition test f_p2 that is required to be executed before f_x. We could either further split up our partitions and rearrange them or at some point we might consider switching to a less efficient algorithm and take out subsets of certain size from F. Worst case: all tests in F before the last test f_l test must be executed to make f_l fail. We can verify this worst case only by testing all subsets that remove for each subset a single different test from F.
from pytest-randomly.
That sounds like a great process @thbde . I think it's a challenge to implement since the test process would need restarting repeatedly for each arrangement of the partitions. We'd need to preserve a bunch of state in .pytest_cache
and encourage the user to keep reinvoking pytest- --randomly-minimize-failure
(or whatever the flag is), with extra output to describe where we're at. Or perhaps pytest could reinvoke itself at the end of the run, though that sounds a bit hacky.
from pytest-randomly.
Just stumbled over this issue, and FWIW, some time ago I wrote a small plugin that does something like this (though not as involved as described).
It does not use random ordering (though you can run it together with pytest-randomly
, so that the first run will be random), but inverse ordering. It just runs the tests backwards and forwards, finds tests that fail in one of the runs, bisects the results and repeats until a dependency is found. Each test run is done in a separate process to minimize dependencies between runs.
from pytest-randomly.
Thanks for the note.
Since my last comment, I’ve also used the tool detect-test-pollution. It works well with pytest-randomly, as you can provide it a file with a list of test ID’s. So pytest-randomly can be used to find a problematic order, output the list of ID’s, then you can work on minimizing it with detect-test-pollution.
I'm gonna close this issue since external tools exist.
from pytest-randomly.
Thanks - detect-test-pollution looks interesting, I wasn't aware of it!
pytest-find-dependencies is a bit more restricted, as it always runs all tests and tries to find dependencies there, though it also works well with pytest-randomly which will shuffle the initial test run. I actually use it in CI to ensure independent tests.
from pytest-randomly.
Related Issues (20)
- Incompatible with `pytest-mypy-plugins` (3.10 version) HOT 2
- Support for enabling/disabling test seeding in specific tests. HOT 5
- AttributeError: 'Config' object has no attribute 'cache' with cacheprovider disabled HOT 4
- hashlib.md5 doesn't work when FIPS is enabled HOT 3
- Test suite failures mostly on tests checking ordering HOT 1
- Test numpy on Python 3.11
- 3.12.0: pytest is failing in two units HOT 1
- Support for time getters HOT 1
- Shuffle inside modules only mode HOT 2
- Disabling entry points from dependencies HOT 3
- sdist is missing tests HOT 6
- Random seed appears to also seed other random() calls HOT 10
- Test failures with pytest 7.3.0: Possible off-by-one line fragment mismatch? HOT 3
- Feature enhancement to allow turning plugin on and off HOT 4
- Support numpy.random.default_rng HOT 4
- Add default support for baker random seeding HOT 1
- RFE: please start making github releases HOT 1
- provide unique seed to each test HOT 7
- 3.15.0: pytest fails in 3 units HOT 3
- Integration with pytest-rerunfailures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-randomly.