Comments (6)
Hi @thbde
Thanks for pointing outt the research!
However, in this case, the commit when a flawed test is introduced may be different to the commit for which the test fails. This leads to higher effort for the root cause detection.
Yes it does.
Running the tests in reverse order is a simple and effective approach to detect simple order issues.
Yes it is. Django's test framework includes a --reverse
flag for this reason.
You know I would've thought this would exist in pytest core, but it doesn't.
You can make a plugin to allow --reverse
to reverse the tests with just a few lines:
def pytest_addoption(parser):
parser._addoption(
"--reverse",
action="store",
... # todo: fill in remaining arguments
)
def pytest_collection_modifyitems(session, config, items):
if config.getoption("reverse"):
items[:] = items[::-1]
Given this is part of Django's test runner, and it's such a small amount of code, it might be suitable for pytest core. Since it's not randomness-based, I don't think it's appropriate here.
I'm interested to see what the pytest core devs say, if you open an issue on pytest core (please tag me, and refer here).
from pytest-randomly.
I just read the paper and found it so interesting I decided it's really worth having this ability in pytest. I've create a plugin pytest-reverse to release a full version of that code snippet. Please test it!
And please do open an issue on pytest core, I think this would be a small and valuable addition.
from pytest-randomly.
I was a bit busy, I will open the issue as suggested. Thank you for your fast feedback.
Regarding your positive reaction to the paper, maybe a short summary and a link to the paper could be a nice addition to the README. This would provide additional arguments and resources to for the interested user of randomly. I can open a PR with some text if you want.
Regarding the place of the option (here or core): reverse might be a special version of randomness :) (at least in contrast to in-order). But I am also happy with a core option.
from pytest-randomly.
Yes please, and on pytest-reverse ! It's also worth mentioning pytest-reverse on https://docs.pytest.org/en/latest/flaky.html
from pytest-randomly.
I was busy with some other tasks, but here we go. I also added a link to pytest-reverse.
I tried to keep the amount of text low. Please share your thoughts if you prefer more explanation and reasoning like in my initial post here.
Feel free to close this issue after accepting the PR.
from pytest-randomly.
Thank you! 🙏
Good work with the references section.
from pytest-randomly.
Related Issues (20)
- Incompatible with `pytest-mypy-plugins` (3.10 version) HOT 2
- Support for enabling/disabling test seeding in specific tests. HOT 5
- AttributeError: 'Config' object has no attribute 'cache' with cacheprovider disabled HOT 4
- hashlib.md5 doesn't work when FIPS is enabled HOT 3
- Test suite failures mostly on tests checking ordering HOT 1
- Test numpy on Python 3.11
- 3.12.0: pytest is failing in two units HOT 1
- Support for time getters HOT 1
- Shuffle inside modules only mode HOT 2
- Disabling entry points from dependencies HOT 3
- sdist is missing tests HOT 6
- Random seed appears to also seed other random() calls HOT 10
- Test failures with pytest 7.3.0: Possible off-by-one line fragment mismatch? HOT 3
- Feature enhancement to allow turning plugin on and off HOT 4
- Support numpy.random.default_rng HOT 4
- Add default support for baker random seeding HOT 1
- RFE: please start making github releases HOT 1
- provide unique seed to each test HOT 7
- 3.15.0: pytest fails in 3 units HOT 3
- Integration with pytest-rerunfailures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-randomly.