Comments (3)
What if there was a pytest fixture or context manager? Then we could have
@pytest.mark.parametrize('seed', [9001, 1337])
def test_old_bad_seeds(seed, randomly):
randomly.reseed(seed)
# test that has failed in the past with those exact seeds
Tbh I'm not that excited about this though. I think if you find a failing test with a particular seed, it's not the seed that should be turned into a new test case but the actual underlying data that caused the failure. For example if seed 9001 caused your data generator to create a user with a really long name, add a test with a user with a fixed really long name.
Fixing the seed in a test run is pretty fragile, as the data generator could change in the future, meaning the seed values don't mean the same thing, so the tests could stop testing the specific bugs they exposed. For example, new versions of faker
come with new data, remapping the data returned for a given seed.
Do you have a use case that prevents making tests with fixed test data, or at makes it impractical?
from pytest-randomly.
Fixing the seed in a test run is pretty fragile, as the data generator could change in the future, meaning the seed values don't mean the same thing, so the tests could stop testing the specific bugs they exposed
Yeap, that seems reasonable. hypothesis
spoiled me here.
I will stick to the data examples.
from pytest-randomly.
Btw hypothesis
has @example
for this - https://hypothesis.readthedocs.io/en/latest/reproducing.html#providing-explicit-examples . But that can't have an analog in pytest-randomly
because we don't touch data generation :)
from pytest-randomly.
Related Issues (20)
- Incompatible with `pytest-mypy-plugins` (3.10 version) HOT 2
- Support for enabling/disabling test seeding in specific tests. HOT 5
- AttributeError: 'Config' object has no attribute 'cache' with cacheprovider disabled HOT 4
- hashlib.md5 doesn't work when FIPS is enabled HOT 3
- Test suite failures mostly on tests checking ordering HOT 1
- Test numpy on Python 3.11
- 3.12.0: pytest is failing in two units HOT 1
- Support for time getters HOT 1
- Shuffle inside modules only mode HOT 2
- Disabling entry points from dependencies HOT 3
- sdist is missing tests HOT 6
- Random seed appears to also seed other random() calls HOT 10
- Test failures with pytest 7.3.0: Possible off-by-one line fragment mismatch? HOT 3
- Feature enhancement to allow turning plugin on and off HOT 4
- Support numpy.random.default_rng HOT 4
- Add default support for baker random seeding HOT 1
- RFE: please start making github releases HOT 1
- provide unique seed to each test HOT 7
- 3.15.0: pytest fails in 3 units HOT 3
- Integration with pytest-rerunfailures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-randomly.