pytest-dev / pytest-randomly Goto Github PK
View Code? Open in Web Editor NEW:game_die: Pytest plugin to randomly order tests and control random.seed
License: MIT License
:game_die: Pytest plugin to randomly order tests and control random.seed
License: MIT License
No response
No response
https://docs.pytest.org/en/6.2.x/cache.html#stepwise
Using --stepwise
doesn't seem to work with pytest-randomly at current.
If I have tests A through D that all fail with 'assert 0', but a seed that runs them with D first, it stops after the failure of D but assumes that A through C passed even though they didn't run.
It seems to base its success/fail tracking on the test order pre-shuffle.
Cannot import importlib_metadata.
As discussed in #19, it's sometimes undesirable that we reset the random seed to the exact same value for fixtures and for tests - if both use the same data generation strategy (e.g. factory boy) they can end up generating the exact same objects which is a less useful test than the randomness they are trying to employ.
What we could do is use multiple random states based upon the seed we've been given (e.g. seed, seed + 1, seed +2) for test setup, test running, test teardown, etc. Or we could just advance the random generator with the same seed an arbitrary amount e.g. generate 100 random numbers before the test setup, but 0 random numbers before the test.
importlib-metadata
3.6.0 changed the behaviour of entry_points()
to no longer return a dictionary of entry points but instead return an object with a deprecated dict-like interface. As of importlib-metadata
3.9.0 a DeprecationWarning is emitted when calling the dictionary methods.
pytest-randomly
calls .get()
here:
pytest-randomly/src/pytest_randomly.py
Lines 151 to 156 in 1f0ef49
This results in the following warning:
venv/lib/python3.7/site-packages/pytest_randomly.py:153: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
e.load() for e in entry_points().get("pytest_randomly.random_seeder", [])
If warnings are turned into errors, this causes an internal error in pytest.
(venv) domdf@ryzen:/tmp/pytest_randomly_deprecation$ PYTHONDEVMODE=1 PYTHONWARNINGS=error pytest test_foo.py
============================================================================================ test session starts ============================================================================================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/_pytest/main.py", line 267, in wrap_session
INTERNALERROR> config.hook.pytest_sessionstart(session=session)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/_pytest/terminal.py", line 713, in pytest_sessionstart
INTERNALERROR> config=self.config, startdir=self.startdir
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pytest_randomly.py", line 170, in pytest_report_header
INTERNALERROR> _reseed(config)
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/pytest_randomly.py", line 153, in _reseed
INTERNALERROR> e.load() for e in entry_points().get("pytest_randomly.random_seeder", [])
INTERNALERROR> File "/tmp/pytest_randomly_deprecation/venv/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 302, in get
INTERNALERROR> flake8_bypass(self._warn)()
INTERNALERROR> DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
See python/importlib_metadata#278 and python/importlib_metadata#289 for more context.
This is currently only an issue on Python 3.7 and earlier, as pytest-randomly
uses the standard library on newer versions.
A configuration or decorator to tests that can be used to change the seeding behaviour randomly adds, so we could specific specific tests that always run on a specific seed and are deterministic and others that are not.
We currently can do this by separating them in different test suits, would be handy to have this able to do this for specific tests.
Is this something we might want ? In case it is i could implement this if we have intention of adding this.
Thanks !
I get this while packaging it for gentoo
================================================================================================================= FAILURES ==================================================================================================================
_________________________________________________________________________________________________________ test_entrypoint_injection _________________________________________________________________________________________________________
testdir = <Testdir local('/var/tmp/portage/dev-python/pytest-randomly-3.3.1-r1/temp/pytest-of-portage/pytest-0/test_entrypoint_injection0')>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f655c41dba8>
def test_entrypoint_injection(testdir, monkeypatch):
"""Test that registered entry points are seeded"""
class _FakeEntryPoint:
"""Minimal surface of Entry point API to allow testing"""
def __init__(self, name, obj):
self.name = name
self._obj = obj
def load(self):
return self._obj
entry_points = []
def fake_entry_points():
return {
"pytest_randomly.random_seeder": entry_points,
"ignore": lambda x: 1 / 0,
}
monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points)
reseed = Mock()
entry_points.append(_FakeEntryPoint("test_seeder", reseed))
# Need to run in-process so that monkeypatching works
testdir.runpytest("--randomly-seed=1")
> assert reseed.call_args == ((1,),)
E assert None == ((1,),)
E +None
E -((1,),)
../../../../work/pytest-randomly-3.3.1/tests/test_pytest_randomly.py:654: AssertionError
----------------------------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------------------------
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.6.10, pytest-5.4.2, py-1.8.0, pluggy-0.13.1
Using --randomly-seed=1
rootdir: /var/tmp/portage/dev-python/pytest-randomly-3.3.1-r1/temp/pytest-of-portage/pytest-0/test_entrypoint_injection0
plugins: randomly-3.3.1, aiohttp-0.3.0, mock-3.1.0, travis-fold-1.3.0, forked-1.1.3, xdist-1.32.0, instafail-0.4.1.post0
collected 0 items
=========================================================================================================== no tests ran in 0.03s ===========================================================================================================
============================================================================================================= warnings summary ==============================================================================================================
tests/test_pytest_randomly.py: 36 tests with warnings
/usr/lib64/python3.6/site-packages/_pytest/compat.py:333: PytestDeprecationWarning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/latest/deprecations.html#terminalreporter-writer for more information.
return getattr(object, name, default)
-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================================================================================================== short test summary info ==========================================================================================================
FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - assert None == ((1,),)
================================================================================================ 1 failed, 31 passed, 36 warnings in 28.68s =================================================================================================
It would be cool if this plugin could use PYTHONHASHSEED
for the source of randomness so that tests are even more predictable. Setting it is probably outside the scope of the plugin since from inside the python interpreter it requires a restart which isn't such a great thing (though python-libfaketime
gets away with it), however some tools like tox
set it.
Just so you know, backports.entry-points-selectable
specifies Python markers for its pytest-mypy
dependency that are not being parsed correctly by Poetry, resulting in errors while resolving dependencies for pytest-randomly 3.6.0. See python-poetry/poetry#3862 (comment) and the previous comments. I'll temporarily pin pytest-randomly to 3.5.0 ๐
I've just updated to the latest version of pytest-randomly
(3.3
) and found that it doesn't work for me.
Steps:
pip install pytest pytest-randomly==3.3
pytest
This yields the following error output:
$ pytest
======================= test session starts =======================
platform linux -- Python 3.6.10, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
Using --randomly-seed=1586964821
rootdir: /tmp/temp
plugins: randomly-3.3.0
collected 0 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/_pytest/main.py", line 246, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/_pytest/main.py", line 257, in pytest_collection
INTERNALERROR> return session.perform_collect()
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/_pytest/main.py", line 453, in perform_collect
INTERNALERROR> self.config.pluginmanager.check_pending()
INTERNALERROR> File ".../.virtualenvs/temp/lib/python3.6/site-packages/pluggy/manager.py", line 277, in check_pending
INTERNALERROR> % (name, hookimpl.plugin),
INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_configure_node' in plugin <module 'pytest_randomly' from '.../.virtualenvs/temp/lib/python3.6/site-packages/pytest_randomly.py'>
======================= no tests ran in 0.02s =======================
If I downgrade to pytest-randomly<3.3
then my tests are run as expected.
I'm currently using pytest
5.4.1
, though this also reproduces with pytest
5.3.5
.
Once you have tests that fail only when run in a particular order, it'd be incredibly useful to have sort of support for automatically determining what the minimum set of tests are required to trigger the failure.
Is it possible to use pytest-randomly
but selectively disable shuffling of functions in classes?
Similar to https://github.com/jbasko/pytest-random-order#disable-shuffling-in-module-or-class
This could allow incremental tests, as described in https://docs.pytest.org/en/stable/example/simple.html#incremental-testing-test-steps but still allow some randomness.
In incremental tests where previous functions may depend on prior steps (organized in functions in a class) random order would expectedly fail.
3.8.10
6.2.5
3.10.1
pytest fails to run tests if cacheprovider
plugin is disabled during test execution.
$ cat test_pass.py
def test_pass1():
assert True
def test_pass2():
assert True
$ pytest -p no:cacheprovider test_pass.py -v
================================================================================= test session starts =================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
rootdir: /tmp/pytest-of-mandeepsandhu/mytest
collected 2 items
test_pass.py::test_pass1 PASSED [ 50%]
test_pass.py::test_pass2 PASSED [100%]
================================================================================== 2 passed in 0.00s ==================================================================================
pytest-randomly
and repeat step 2$ pip install --user pytest-randomly
...
Successfully installed pytest-randomly-3.10.1
$ pytest -p no:cacheprovider test_pass.py -v
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/_pytest/main.py", line 265, in wrap_session
INTERNALERROR> config._do_configure()
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 982, in _do_configure
INTERNALERROR> self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/hooks.py", line 308, in call_historic
INTERNALERROR> res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/mandeepsandhu/.local/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 116, in pytest_configure
INTERNALERROR> assert config.cache is not None
INTERNALERROR> AttributeError: 'Config' object has no attribute 'cache'
randomly
disabled, works:$ pytest -p no:cacheprovider -p no:randomly test_pass.py -v
================================================================================= test session starts =================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
rootdir: /tmp/pytest-of-mandeepsandhu/mytest
collected 2 items
test_pass.py::test_pass1 PASSED [ 50%]
test_pass.py::test_pass2 PASSED [100%]
================================================================================== 2 passed in 0.00s ==================================================================================
Packages:
pytest-3.2.2
xdist-1.20.0
randomly-1.2.1
Example code:
import pytest
import random
def gen_param():
a = random.random()
b = random.random()
c = a + b
return a, b, c
@pytest.mark.parametrize('a,b,c', [gen_param() for _ in range(10)])
def test_sum(a, b, c):
assert a + b == c
Example result:
Different tests were collected between gw1 and gw0. The difference is:
--- gw1
+++ gw0
@@ -1,10 +1,10 @@
-test_it.py::test_sum[0.21119735007187512-0.03478699051186407-0.2459843405837392]
-test_it.py::test_sum[0.19989965451085068-0.21530345609429247-0.41520311060514314]
-test_it.py::test_sum[0.5682066547612487-0.7243829926261657-1.2925896473874143]
-test_it.py::test_sum[0.5138857769400398-0.9866435513079722-1.500529328248012]
-test_it.py::test_sum[0.32391650283278506-0.39646296915151646-0.7203794719843015]
-test_it.py::test_sum[0.9573539653252039-0.46631807929040026-1.4236720446156041]
-test_it.py::test_sum[0.18758435224247982-0.4081118220534776-0.5956961742959574]
-test_it.py::test_sum[0.8300722136940875-0.24370118062201607-1.0737733943161034]
-test_it.py::test_sum[0.45416992471686735-0.5539633757267955-1.0081333004436628]
-test_it.py::test_sum[0.6404127883887936-0.07517291369462298-0.7155857020834165]
+test_it.py::test_sum[0.4235467615256703-0.6336556280381637-1.0572023895638338]
+test_it.py::test_sum[0.08598091323183876-0.9197414141632071-1.0057223273950457]
+test_it.py::test_sum[0.6499835837722387-0.08942031974171283-0.7394039035139516]
+test_it.py::test_sum[0.5982265644051936-0.4014341639946195-0.9996607283998131]
+test_it.py::test_sum[0.6108773740309141-0.39536962117174335-1.0062469952026576]
+test_it.py::test_sum[0.13520942528376823-0.36746285760417974-0.502672282887948]
+test_it.py::test_sum[0.8469134601088156-0.34936702626625926-1.196280486375075]
+test_it.py::test_sum[0.5828050759610505-0.028386017512678552-0.611191093473729]
+test_it.py::test_sum[0.1425962119341786-0.5579729193825124-0.700569131316691]
+test_it.py::test_sum[0.6183292075112786-0.5376259380555282-1.1559551455668067]
From what I could gather, it's possible to fix it simply by adding
def pytest_configure(config):
_reseed(config)
to pytest_randomly.py
. But I've never written a single plugin for pytest and I've read only some excerpts from the documentation, so I may be wrong.
pytest-randomly 1.2.0 seems to have broken test collection when using pytest-xdist.
[tox]
skipsdist = true
usedevelop = false
envlist =
py{27,36}-randomly{11,12}
[testenv]
deps =
pytest
pytest-xdist
randomly11: pytest-randomly<1.2.0
randomly12: pytest-randomly==1.2.0
commands = pytest -n auto {posargs}
def test_1():
assert 1 + 1 == 2
def test_2():
assert 2 + 2 == 4
$ tox
py27-randomly11 create: /Users/andy/tmp/pytest-randomly-xdist/.tox/py27-randomly11
py27-randomly11 installdeps: pytest, pytest-xdist, pytest-randomly<1.2.0
py27-randomly11 installed: apipkg==1.4,execnet==1.4.1,py==1.4.34,pytest==3.1.2,pytest-randomly==1.1.2,pytest-xdist==1.17.1
py27-randomly11 runtests: PYTHONHASHSEED='4070452905'
py27-randomly11 runtests: commands[0] | pytest -n auto
======================================================== test session starts ========================================================
platform darwin -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1497720106
rootdir: /Users/andy/tmp/pytest-randomly-xdist, inifile:
plugins: xdist-1.17.1, randomly-1.1.2
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
..
===================================================== 2 passed in 0.67 seconds ======================================================
py27-randomly12 create: /Users/andy/tmp/pytest-randomly-xdist/.tox/py27-randomly12
py27-randomly12 installdeps: pytest, pytest-xdist, pytest-randomly==1.2.0
py27-randomly12 installed: apipkg==1.4,execnet==1.4.1,py==1.4.34,pytest==3.1.2,pytest-randomly==1.2.0,pytest-xdist==1.17.1
py27-randomly12 runtests: PYTHONHASHSEED='4070452905'
py27-randomly12 runtests: commands[0] | pytest -n auto
======================================================== test session starts ========================================================
platform darwin -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1497720111
rootdir: /Users/andy/tmp/pytest-randomly-xdist, inifile:
plugins: xdist-1.17.1, randomly-1.2.0
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
============================================================== ERRORS ===============================================================
_______________________________________________________ ERROR collecting gw1 ________________________________________________________
Different tests were collected between gw3 and gw1. The difference is:
--- gw3
+++ gw1
@@ -1,2 +1,2 @@
+test_it.py::test_2
test_it.py::test_1
-test_it.py::test_2
====================================================== 1 error in 0.62 seconds ======================================================
ERROR: InvocationError: '/Users/andy/tmp/pytest-randomly-xdist/.tox/py27-randomly12/bin/pytest -n auto'
py36-randomly11 create: /Users/andy/tmp/pytest-randomly-xdist/.tox/py36-randomly11
py36-randomly11 installdeps: pytest, pytest-xdist, pytest-randomly<1.2.0
py36-randomly11 installed: apipkg==1.4,execnet==1.4.1,py==1.4.34,pytest==3.1.2,pytest-randomly==1.1.2,pytest-xdist==1.17.1
py36-randomly11 runtests: PYTHONHASHSEED='4070452905'
py36-randomly11 runtests: commands[0] | pytest -n auto
======================================================== test session starts ========================================================
platform darwin -- Python 3.6.1, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1497720116
rootdir: /Users/andy/tmp/pytest-randomly-xdist, inifile:
plugins: xdist-1.17.1, randomly-1.1.2
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
..
===================================================== 2 passed in 0.72 seconds ======================================================
py36-randomly12 create: /Users/andy/tmp/pytest-randomly-xdist/.tox/py36-randomly12
py36-randomly12 installdeps: pytest, pytest-xdist, pytest-randomly==1.2.0
py36-randomly12 installed: apipkg==1.4,execnet==1.4.1,py==1.4.34,pytest==3.1.2,pytest-randomly==1.2.0,pytest-xdist==1.17.1
py36-randomly12 runtests: PYTHONHASHSEED='4070452905'
py36-randomly12 runtests: commands[0] | pytest -n auto
======================================================== test session starts ========================================================
platform darwin -- Python 3.6.1, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1497720122
rootdir: /Users/andy/tmp/pytest-randomly-xdist, inifile:
plugins: xdist-1.17.1, randomly-1.2.0
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
collecting 0 items / 3 errors
============================================================== ERRORS ===============================================================
_______________________________________________________ ERROR collecting gw1 ________________________________________________________
Different tests were collected between gw0 and gw1. The difference is:
--- gw0
+++ gw1
@@ -1,2 +1,2 @@
+test_it.py::test_1
test_it.py::test_2
-test_it.py::test_1
_______________________________________________________ ERROR collecting gw2 ________________________________________________________
Different tests were collected between gw0 and gw2. The difference is:
--- gw0
+++ gw2
@@ -1,2 +1,2 @@
+test_it.py::test_1
test_it.py::test_2
-test_it.py::test_1
_______________________________________________________ ERROR collecting gw3 ________________________________________________________
Different tests were collected between gw0 and gw3. The difference is:
--- gw0
+++ gw3
@@ -1,2 +1,2 @@
+test_it.py::test_1
test_it.py::test_2
-test_it.py::test_1
====================================================== 3 error in 0.69 seconds ======================================================
ERROR: InvocationError: '/Users/andy/tmp/pytest-randomly-xdist/.tox/py36-randomly12/bin/pytest -n auto'
______________________________________________________________ summary ______________________________________________________________
py27-randomly11: commands succeeded
ERROR: py27-randomly12: commands failed
py36-randomly11: commands succeeded
ERROR: py36-randomly12: commands failed
Hi, I am one of the developers of mimesis
and a huge fan of your library.
I really want to provide a native integration of mimesis
and pytest-randomly
just like the one you have with faker
and factoryboy
. I really like that the reseed happens before each test, so the results are significantly better than the regular one time seed.
What needs to be done?
random
is used internally to expose global random
instance: https://github.com/lk-geimfari/mimesis/pull/471/files#diff-02fe14a63fc506efde39da2f898b5e0fR127 so it would be easy to seed itfaker
and others, if that's fineRelated: lk-geimfari/mimesis#469
I would like to hear your opinion. Thanks!
Many thanks for the awesome plugin, really help a lot.
I wonder is it possible to get random seed value during pytest running ?
e.g.
def test_get_ramdom_seed():
print(seed)
Thanks in advance!
INTERNALERROR> File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py36/lib/python3.6/site-packages/pytest_randomly.py", line 144, in _reseed
INTERNALERROR> np_random.seed(seed)
INTERNALERROR> File "mtrand.pyx", line 243, in numpy.random.mtrand.RandomState.seed
INTERNALERROR> File "_mt19937.pyx", line 166, in numpy.random._mt19937.MT19937._legacy_seeding
INTERNALERROR> File "_mt19937.pyx", line 180, in numpy.random._mt19937.MT19937._legacy_seeding
INTERNALERROR> ValueError: Seed must be between 0 and 2**32 - 1
looks like some sort of seed truncation is needed:
def _numpy_seed(seed):
return seed if 0 <= seed <= 2**32-1 else random.Random(seed).getrandbits(32)
I'm working on a project which uses random generators to make sure tests which generate things like database entries don't create duplicates. I've combined this with pytest-randomly and "--randomly-seed=${GITHUB_RUN_ID}" --randomly-dont-reset-seed
to have reproducible randomness: if the CI pipeline fails I can easily re-run the exact same thing locally. This is working fine, but when combined with pytest-xdist it looks like each worker is getting the same initial seed, and so the tests are back to generating duplicate values.
Is there some way to send a reproducible but different seed to each test when combined with python-xdist?
pytest-randomly
seems to conflict with pytest-pylint
when used with the randomly flag. I have attached a full console log that demonstrates the problem. Please let me know if there is anything else I can provide to help figure out what's going on. I've spent some time looking through pytest_collection_modifyitems
, but I'm not sure what the root cause is.
~/t/randomly-debug
โฏ ls -a
. .. sample
~/t/randomly-debug
โฏ tree sample
sample
โโโ __init__.py
โโโ test
โโโ test_randomly.py
1 directory, 2 files
~/t/randomly-debug
โฏ cat sample/__init__.py
~/t/randomly-debug
โฏ cat sample/test/test_randomly.py
"""docstring"""
def test_randomly_interaction():
"""docstring"""
return True
~/t/randomly-debug
โฏ virtualenv venv
New python executable in /Users/dylan.lloyd/t/randomly-debug/venv/bin/python2.7
Also creating executable in /Users/dylan.lloyd/t/randomly-debug/venv/bin/python
Please make sure you remove any previous custom paths from your /Users/dylan.lloyd/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
~/t/randomly-debug
โฏ ./venv/bin/pip install pylint==1.6.0 pytest==3.0.7 pytest-pylint==0.6.0
Collecting pylint==1.6.0
Using cached pylint-1.6.0-py2.py3-none-any.whl
Collecting pytest==3.0.7
Using cached pytest-3.0.7-py2.py3-none-any.whl
Collecting pytest-pylint==0.6.0
Using cached pytest_pylint-0.6.0-py2-none-any.whl
Collecting six (from pylint==1.6.0)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting isort>=4.2.5 (from pylint==1.6.0)
Using cached isort-4.2.15-py2.py3-none-any.whl
Collecting mccabe (from pylint==1.6.0)
Using cached mccabe-0.6.1-py2.py3-none-any.whl
Collecting colorama (from pylint==1.6.0)
Using cached colorama-0.3.9-py2.py3-none-any.whl
Collecting astroid<1.5.0,>=1.4.5 (from pylint==1.6.0)
Using cached astroid-1.4.9-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python2.7/site-packages (from pytest==3.0.7)
Collecting py>=1.4.29 (from pytest==3.0.7)
Using cached py-1.4.34-py2.py3-none-any.whl
Collecting wrapt (from astroid<1.5.0,>=1.4.5->pylint==1.6.0)
Collecting lazy-object-proxy (from astroid<1.5.0,>=1.4.5->pylint==1.6.0)
Installing collected packages: six, isort, mccabe, colorama, wrapt, lazy-object-proxy, astroid, pylint, py, pytest, pytest-pylint
Successfully installed astroid-1.4.9 colorama-0.3.9 isort-4.2.15 lazy-object-proxy-1.3.1 mccabe-0.6.1 py-1.4.34 pylint-1.6.0 pytest-3.0.7 pytest-pylint-0.6.0 six-1.11.0 wrapt-1.10.11
~/t/randomly-debug
โฏ ./venv/bin/pytest sample/test/test_randomly.py --pylint sample/__init__.py
=================================== test session starts ====================================
platform darwin -- Python 2.7.12, pytest-3.0.7, py-1.4.34, pluggy-0.4.0
rootdir: /Users/dylan.lloyd/t/randomly-debug, inifile:
plugins: pylint-0.6.0
collected 3 items
sample/test/test_randomly.py ..
sample/__init__.py .
================================= 3 passed in 0.11 seconds =================================
~/t/randomly-debug
โฏ ./venv/bin/pip install pytest-randomly==1.2.1
Collecting pytest-randomly==1.2.1
Using cached pytest_randomly-1.2.1-py2.py3-none-any.whl
Requirement already satisfied: pytest in ./venv/lib/python2.7/site-packages (from pytest-randomly==1.2.1)
Requirement already satisfied: setuptools in ./venv/lib/python2.7/site-packages (from pytest->pytest-randomly==1.2.1)
Requirement already satisfied: py>=1.4.29 in ./venv/lib/python2.7/site-packages (from pytest->pytest-randomly==1.2.1)
Installing collected packages: pytest-randomly
Successfully installed pytest-randomly-1.2.1
~/t/randomly-debug
โฏ ./venv/bin/pytest sample/test/test_randomly.py --pylint sample/__init__.py
=================================== test session starts ====================================
platform darwin -- Python 2.7.12, pytest-3.0.7, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1509722578
rootdir: /Users/dylan.lloyd/t/randomly-debug, inifile:
plugins: randomly-1.2.1, pylint-0.6.0
collected 3 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/main.py", line 132, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/main.py", line 141, in pytest_collection
INTERNALERROR> return session.perform_collect()
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/main.py", line 602, in perform_collect
INTERNALERROR> config=self.config, items=items)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/dylan.lloyd/t/randomly-debug/venv/lib/python2.7/site-packages/pytest_randomly.py", line 119, in pytest_collection_modifyitems
INTERNALERROR> current_module = item.module
INTERNALERROR> AttributeError: 'PyLintItem' object has no attribute 'module'
=============================== no tests ran in 0.01 seconds ===============================
~/t/randomly-debug
โฏ ./venv/bin/pytest sample/test/test_randomly.py --pylint sample/__init__.py --randomly-dont-reorganize
================================================================================= test session starts ==================================================================================
platform darwin -- Python 2.7.12, pytest-3.0.7, py-1.4.34, pluggy-0.4.0
Using --randomly-seed=1509722745
rootdir: /Users/dylan.lloyd/t/randomly-debug, inifile:
plugins: randomly-1.2.1, pylint-0.6.0
collected 3 items
sample/test/test_randomly.py ..
sample/__init__.py .
=============================================================================== 3 passed in 0.10 seconds ===============================================================================
I have some fixtures that create resources in certain modules. I'd like to make sure other modules aren't using them and silently passing.
Is there a way to randomize the module order? They appear to run in sorted order.
Sometimes I ran tests with some random values that pass.
When the seed changes during the next test run I might run into some errors.
Consider this imaginary example:
import random
def will_fail(x):
return x / random.randint(0, 10)
def test_will_fail():
assert isinstance(will_fail(5), float)
The test will pass for some seeds, but later I will run into the situation when this test will fail.
So, I will fix the code, like this:
def will_fail(x):
return x / random.randint(1, 10)
And I want to be sure that this won't happen again.
I can manually run random.seed
and set seed values from the previous failing tests.
I see this as a huge API feature, that will cover these cases.
It may look something like this:
@pytest.mark.seed('your-seed') # we had failures with this seed in the past!
@pytest.mark.seed('your-other-seed') # this seed is also breaking our code
def test_will_fail():
...
By doing this we can change the global random seed for a single test. We can do it multiple times. And then return back to normal tests execution.
This API is inspired by hypothesis
: https://hypothesis.readthedocs.io/en/latest/reproducing.html?highlight=seed
As we have discussed in #135 (comment) there's an issue with optional dependencies.
Switching to pipenv
will include:
requirements.in
and requirements.txt
Pipfile
and Pipfile.lock
tox
worksThat's it.
Later I will move to #135
As the plugin behaves now, tests are randomized when the plugin is installed. I would like to change this behavior, such that when the plugin is installed, an additional option flag (--randomly-reorganize
) comes available to randomize tests.
The reason for me to want it like this, is that I do not always want to randomize tests, and I would like to explicitly pass an argument when I want to randomize tests.
Another minor reason why this might be better, is that an inverted option (e.g. --disable-x
or --no-y
) is arguably slightly more difficult to grasp then it's non-inverted option (e.g. --enable-x
or --y
)
Did this topic come up already?
3.9.9
6.2.5
3.10.2
On a system where FIPS is enabled, pytest-randomly fails on line 275
hasher = hashlib.md5()
with the following error
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
This appears to be common among tools that use hashlib
. The following workaround seems to fix the problem, edit pytest_randomly/__init__.py
line 275 to be:
hasher = hashlib.md5(usedforsecurity=False)
I found this idea for a fix here: s3tools/s3cmd#1005 (comment). This issue was related to another package using hashlib.md5
In #463 , numpy tests were disabled for Python 3.11, because it didn't support Python 3.11. Once it does, re-enable installing numpy and running its tests.
Hi,
I have a question about how the --randomly-seed
option works.
Recently we had a test failure on our CI system, and since we use pytest-randomly
I grabbed the seed value from the output on the CI job and ran the test on my machine.
What I noticed was that despite using the same --randomly-seed
value, the tests ran in a different order on my machine. And in fact the tests all passed on my machine.
My question then is, is it expected that the test order be different on different machines for the same --randomly-seed
value?
Thanks,
Dan
shuffle with the same seed isn't stable for item subsets and supersets:
import random
def shuffle(seed, v):
v = list(v)
random.Random(x=seed).shuffle(v)
return v
assert (
shuffle(1, "abcd") == ["d", "a", "c", "b"]
and shuffle(1, "abc") == ["b", "c", "a"]
)
eg if you remove tests due to "--lf" or "--sw", or add/remove tests
As a developer, I want to execute the tests in reverse order to reveal test dependencies.
Given that the natural order of execution is O={t1, ..., tn}, maybe provide a command line switch that allows to run all tests on reverse order reverse(O)={tn, ..., t1}.
Motivation:
An empirical study [0] on the test independence assumption (i.e., the result of a test execution does not depend on other test executions) shows, that the reverse strategy can already reveal quite a large amount of test dependencies. A reverse order can reveal more than 70% of all cases for manually created tests and more than 60% of all cases for generated tests, compared to the best detection strategy.
The authors of the study finally suggest to use the randomized strategy. However, the suggestions has limitations in practice. The authors evaluated 10 reruns for random order executions with different seeds. This may lead to a 10x increase in test costs (such as waiting times) and may not be practical. Alternatively to 10 reruns in a row, we could execute the tests in random order only once per commit and observe over time. For each 10 commits, we reach the same threshold of 10 random executions. However, in this case, the commit when a flawed test is introduced may be different to the commit for which the test fails. This leads to higher effort for the root cause detection.
Running the tests in reverse order is a simple and effective approach to detect simple order issues.
(I am also unable to create a PR for this issue because I do not fully understand the logic of the conditionals in the pytest_collection_modifyitems function. It would be helpful for a reader to have some comments on why the logic flow is organized in the current way.)
[0] Sai Zhang, Darioush Jalali, Jochen Wuttke, Kฤฑvanรง Muลlu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically revisiting the test independence assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 385โ396. doi:https://doi.org/10.1145/2610384.2610404
3.9.9
6.2.5
3.11.0
Hello,
When running the test suite after updating the package to 3.11.0 on GNU Guix, I see these failures:
$ pytest -vv
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=511009415
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source, configfile: pyproject.toml
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collected 36 items
tests/test_pytest_randomly.py::test_xdist[3] PASSED [ 2%]
tests/test_pytest_randomly.py::test_faker PASSED [ 5%]
tests/test_pytest_randomly.py::test_doctests_reordered FAILED [ 8%]
tests/test_pytest_randomly.py::test_test_functions_reordered_when_randomness_in_module FAILED [ 11%]
tests/test_pytest_randomly.py::test_test_functions_reordered FAILED [ 13%]
tests/test_pytest_randomly.py::test_failing_import PASSED [ 16%]
tests/test_pytest_randomly.py::test_factory_boy PASSED [ 19%]
tests/test_pytest_randomly.py::test_files_reordered FAILED [ 22%]
tests/test_pytest_randomly.py::test_doctests_in_txt_files_reordered FAILED [ 25%]
tests/test_pytest_randomly.py::test_it_reuses_the_same_random_seed_per_test PASSED [ 27%]
tests/test_pytest_randomly.py::test_passing_nonsense_for_randomly_seed PASSED [ 30%]
tests/test_pytest_randomly.py::test_files_reordered_when_seed_not_reset FAILED [ 33%]
tests/test_pytest_randomly.py::test_numpy_doesnt_crash_with_large_seed PASSED [ 36%]
tests/test_pytest_randomly.py::test_it_runs_before_stepwise FAILED [ 38%]
tests/test_pytest_randomly.py::test_using_last_seed PASSED [ 41%]
tests/test_pytest_randomly.py::test_it_reports_a_header_when_set PASSED [ 44%]
tests/test_pytest_randomly.py::test_entrypoint_injection FAILED [ 47%]
tests/test_pytest_randomly.py::test_it_works_with_the_simplest_test_items PASSED [ 50%]
tests/test_pytest_randomly.py::test_without_cacheprovider PASSED [ 52%]
tests/test_pytest_randomly.py::test_faker_fixture PASSED [ 55%]
tests/test_pytest_randomly.py::test_numpy PASSED [ 58%]
tests/test_pytest_randomly.py::test_class_test_methods_reordered FAILED [ 61%]
tests/test_pytest_randomly.py::test_fixtures_dont_interfere_with_tests_getting_same_random_state PASSED [ 63%]
tests/test_pytest_randomly.py::test_it_reports_a_header_when_not_set PASSED [ 66%]
tests/test_pytest_randomly.py::test_using_last_explicit_seed PASSED [ 69%]
tests/test_pytest_randomly.py::test_it_resets_the_random_seed_at_the_end_of_test_classes PASSED [ 72%]
tests/test_pytest_randomly.py::test_xdist[0] PASSED [ 75%]
tests/test_pytest_randomly.py::test_xdist[1] PASSED [ 77%]
tests/test_pytest_randomly.py::test_the_same_random_seed_per_test_can_be_turned_off PASSED [ 80%]
tests/test_pytest_randomly.py::test_xdist[2] PASSED [ 83%]
tests/test_pytest_randomly.py::test_it_resets_the_random_seed_at_the_start_of_test_classes PASSED [ 86%]
tests/test_pytest_randomly.py::test_entrypoint_missing PASSED [ 88%]
tests/test_pytest_randomly.py::test_classes_reordered FAILED [ 91%]
tests/test_pytest_randomly.py::test_fixtures_get_different_random_state_to_tests PASSED [ 94%]
tests/test_pytest_randomly.py::test_xdist[4] PASSED [ 97%]
tests/test_pytest_randomly.py::test_works_without_xdist PASSED [100%]
===================================================================================================== FAILURES =====================================================================================================
_____________________________________________________________________________________________ test_doctests_reordered ______________________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_doctests_reordered0')>
def test_doctests_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def foo():
'''
>>> foo()
9001
'''
return 9001
def bar():
'''
>>> bar()
9002
'''
return 9002
"""
)
args = ["-v", "--doctest-modules", "--randomly-seed=1"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=2)
> assert out.outlines[8:10] == [
"test_one.py::test_one.bar PASSED",
"test_one.py::test_one.foo PASSED",
]
E AssertionError: assert ['', 'test_one.py::test_one.bar PASSED'] == ['test_one.py::test_one.bar PASSED', 'test_one.py::test_one.foo PASSED']
E At index 0 diff: '' != 'test_one.py::test_one.bar PASSED'
E Full diff:
E - ['test_one.py::test_one.bar PASSED', 'test_one.py::test_one.foo PASSED']
E + ['', 'test_one.py::test_one.bar PASSED']
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:432: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=1
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_doctests_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 2 items
test_one.py::test_one.bar PASSED
test_one.py::test_one.foo PASSED
================================================================================================ 2 passed in 0.07s =================================================================================================
_____________________________________________________________________________ test_test_functions_reordered_when_randomness_in_module ______________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_test_functions_reordered_when_randomness_in_module0')>
def test_test_functions_reordered_when_randomness_in_module(ourtestdir):
ourtestdir.makepyfile(
test_one="""
import random
import time
random.seed(time.time() * 100)
def test_a():
pass
def test_b():
pass
def test_c():
pass
def test_d():
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::test_c PASSED",
"test_one.py::test_a PASSED",
"test_one.py::test_b PASSED",
"test_one.py::test_d PASSED",
]
E AssertionError: assert ['',\n 'test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_b PASSED'] == ['test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_b PASSED',\n 'test_one.py::test_d PASSED']
E At index 0 diff: '' != 'test_one.py::test_c PASSED'
E Full diff:
E [
E + '',
E 'test_one.py::test_c PASSED',
E 'test_one.py::test_a PASSED',
E 'test_one.py::test_b PASSED',
E - 'test_one.py::test_d PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:402: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_test_functions_reordered_when_randomness_in_module0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_one.py::test_c PASSED
test_one.py::test_a PASSED
test_one.py::test_b PASSED
test_one.py::test_d PASSED
================================================================================================ 4 passed in 0.07s =================================================================================================
__________________________________________________________________________________________ test_test_functions_reordered ___________________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_test_functions_reordered0')>
def test_test_functions_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def test_a():
pass
def test_b():
pass
def test_c():
pass
def test_d():
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::test_c PASSED",
"test_one.py::test_a PASSED",
"test_one.py::test_b PASSED",
"test_one.py::test_d PASSED",
]
E AssertionError: assert ['',\n 'test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_b PASSED'] == ['test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_b PASSED',\n 'test_one.py::test_d PASSED']
E At index 0 diff: '' != 'test_one.py::test_c PASSED'
E Full diff:
E [
E + '',
E 'test_one.py::test_c PASSED',
E 'test_one.py::test_a PASSED',
E 'test_one.py::test_b PASSED',
E - 'test_one.py::test_d PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:368: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_test_functions_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_one.py::test_c PASSED
test_one.py::test_a PASSED
test_one.py::test_b PASSED
test_one.py::test_d PASSED
================================================================================================ 4 passed in 0.07s =================================================================================================
_______________________________________________________________________________________________ test_files_reordered _______________________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_files_reordered0')>
def test_files_reordered(ourtestdir):
code = """
def test_it():
pass
"""
ourtestdir.makepyfile(test_a=code, test_b=code, test_c=code, test_d=code)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_b.py::test_it PASSED",
"test_a.py::test_it PASSED",
"test_d.py::test_it PASSED",
"test_c.py::test_it PASSED",
]
E AssertionError: assert ['',\n 'test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED'] == ['test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED',\n 'test_c.py::test_it PASSED']
E At index 0 diff: '' != 'test_b.py::test_it PASSED'
E Full diff:
E [
E + '',
E 'test_b.py::test_it PASSED',
E 'test_a.py::test_it PASSED',
E 'test_d.py::test_it PASSED',
E - 'test_c.py::test_it PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:248: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_files_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_b.py::test_it PASSED
test_a.py::test_it PASSED
test_d.py::test_it PASSED
test_c.py::test_it PASSED
================================================================================================ 4 passed in 0.07s =================================================================================================
_______________________________________________________________________________________ test_doctests_in_txt_files_reordered _______________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_doctests_in_txt_files_reordered0')>
def test_doctests_in_txt_files_reordered(ourtestdir):
ourtestdir.tmpdir.join("test.txt").write(
"""\
>>> 2 + 2
4
"""
)
ourtestdir.tmpdir.join("test2.txt").write(
"""\
>>> 2 - 2
0
"""
)
args = ["-v", "--randomly-seed=2"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=2)
> assert out.outlines[8:10] == [
"test2.txt::test2.txt PASSED",
"test.txt::test.txt PASSED",
]
E AssertionError: assert ['', 'test2.txt::test2.txt PASSED'] == ['test2.txt::test2.txt PASSED', 'test.txt::test.txt PASSED']
E At index 0 diff: '' != 'test2.txt::test2.txt PASSED'
E Full diff:
E - ['test2.txt::test2.txt PASSED', 'test.txt::test.txt PASSED']
E + ['', 'test2.txt::test2.txt PASSED']
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:513: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=2
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_doctests_in_txt_files_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 2 items
test2.txt::test2.txt PASSED
test.txt::test.txt PASSED
================================================================================================ 2 passed in 0.06s =================================================================================================
_____________________________________________________________________________________ test_files_reordered_when_seed_not_reset _____________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_files_reordered_when_seed_not_reset0')>
def test_files_reordered_when_seed_not_reset(ourtestdir):
code = """
def test_it():
pass
"""
ourtestdir.makepyfile(test_a=code, test_b=code, test_c=code, test_d=code)
args = ["-v", "--randomly-seed=15"]
args.append("--randomly-dont-reset-seed")
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_b.py::test_it PASSED",
"test_a.py::test_it PASSED",
"test_d.py::test_it PASSED",
"test_c.py::test_it PASSED",
]
E AssertionError: assert ['',\n 'test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED'] == ['test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED',\n 'test_c.py::test_it PASSED']
E At index 0 diff: '' != 'test_b.py::test_it PASSED'
E Full diff:
E [
E + '',
E 'test_b.py::test_it PASSED',
E 'test_a.py::test_it PASSED',
E 'test_d.py::test_it PASSED',
E - 'test_c.py::test_it PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:268: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_files_reordered_when_seed_not_reset0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_b.py::test_it PASSED
test_a.py::test_it PASSED
test_d.py::test_it PASSED
test_c.py::test_it PASSED
================================================================================================ 4 passed in 0.07s =================================================================================================
___________________________________________________________________________________________ test_it_runs_before_stepwise ___________________________________________________________________________________________
self = <module 'py.error'>, func = <built-in function remove>, args = ('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0/__pycache__',), kwargs = {}
__tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory'), tb = <traceback object at 0x7fdb4e834f00>, errno = 2
def checked_call(self, func, *args, **kwargs):
""" call a function and raise an errno-exception if applicable. """
__tracebackhide__ = True
try:
> return func(*args, **kwargs)
E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0/__pycache__'
/gnu/store/k49pbvmdlv7cjhfgsgkdjqv4rd94gx7x-python-py-1.10.0/lib/python3.9/site-packages/py/_error.py:66: FileNotFoundError
During handling of the above exception, another exception occurred:
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0')>
def test_it_runs_before_stepwise(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def test_a():
assert 0
def test_b():
assert 0
"""
)
out = ourtestdir.runpytest("-v", "--randomly-seed=1", "--stepwise")
out.assert_outcomes(failed=1)
# Now make test_b pass
ourtestdir.makepyfile(
test_one="""
def test_a():
assert 0
def test_b():
assert 1
"""
)
> ourtestdir.tmpdir.join("__pycache__").remove()
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:544:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/gnu/store/k49pbvmdlv7cjhfgsgkdjqv4rd94gx7x-python-py-1.10.0/lib/python3.9/site-packages/py/_path/local.py:226: in remove
py.error.checked_call(os.remove, self.strpath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <module 'py.error'>, func = <built-in function remove>, args = ('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0/__pycache__',), kwargs = {}
__tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory'), tb = <traceback object at 0x7fdb4e834f00>, errno = 2
def checked_call(self, func, *args, **kwargs):
""" call a function and raise an errno-exception if applicable. """
__tracebackhide__ = True
try:
return func(*args, **kwargs)
except self.Error:
raise
except (OSError, EnvironmentError):
cls, value, tb = sys.exc_info()
if not hasattr(value, 'errno'):
raise
__tracebackhide__ = False
errno = value.errno
try:
if not isinstance(value, WindowsError):
raise NameError
except NameError:
# we are not on Windows, or we got a proper OSError
cls = self._geterrnoclass(errno)
else:
try:
cls = self._geterrnoclass(_winerrnomap[errno])
except KeyError:
raise value
> raise cls("%s%r" % (func.__name__, args))
E py.error.ENOENT: [No such file or directory]: remove('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0/__pycache__',)
/gnu/store/k49pbvmdlv7cjhfgsgkdjqv4rd94gx7x-python-py-1.10.0/lib/python3.9/site-packages/py/_error.py:86: ENOENT
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=1
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_runs_before_stepwise0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 2 items
stepwise: no previously failed tests, not skipping.
test_one.py::test_b FAILED
===================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________________________________ test_b ______________________________________________________________________________________________________
def test_b():
> assert 0
E assert 0
test_one.py:6: AssertionError
============================================================================================= short test summary info ==============================================================================================
FAILED test_one.py::test_b - assert 0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================ 1 failed in 0.16s =================================================================================================
____________________________________________________________________________________________ test_entrypoint_injection _____________________________________________________________________________________________
testdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_entrypoint_injection0')>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fdb4e3fa490>
def test_entrypoint_injection(testdir, monkeypatch):
"""Test that registered entry points are seeded"""
class _FakeEntryPoint:
"""Minimal surface of Entry point API to allow testing"""
def __init__(self, name, obj):
self.name = name
self._obj = obj
def load(self):
return self._obj
entry_points: List[_FakeEntryPoint] = []
def fake_entry_points(*, group):
return entry_points
monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points)
reseed = Mock()
entry_points.append(_FakeEntryPoint("test_seeder", reseed))
# Need to run in-process so that monkeypatching works
testdir.runpytest("--randomly-seed=1")
> assert reseed.call_args == ((1,),)
E assert None == ((1,),)
E +None
E -((1,),)
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:738: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=1
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_entrypoint_injection0
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collected 0 items
============================================================================================== no tests ran in 0.01s ===============================================================================================
________________________________________________________________________________________ test_class_test_methods_reordered _________________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_class_test_methods_reordered0')>
def test_class_test_methods_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
from unittest import TestCase
class T(TestCase):
def test_a(self):
pass
def test_b(self):
pass
def test_c(self):
pass
def test_d(self):
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::T::test_c PASSED",
"test_one.py::T::test_b PASSED",
"test_one.py::T::test_a PASSED",
"test_one.py::T::test_d PASSED",
]
E AssertionError: assert ['',\n 'test_one.py::T::test_c PASSED',\n 'test_one.py::T::test_b PASSED',\n 'test_one.py::T::test_a PASSED'] == ['test_one.py::T::test_c PASSED',\n 'test_one.py::T::test_b PASSED',\n 'test_one.py::T::test_a PASSED',\n 'test_one.py::T::test_d PASSED']
E At index 0 diff: '' != 'test_one.py::T::test_c PASSED'
E Full diff:
E [
E + '',
E 'test_one.py::T::test_c PASSED',
E 'test_one.py::T::test_b PASSED',
E 'test_one.py::T::test_a PASSED',
E - 'test_one.py::T::test_d PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:339: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_class_test_methods_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_one.py::T::test_c PASSED
test_one.py::T::test_b PASSED
test_one.py::T::test_a PASSED
test_one.py::T::test_d PASSED
================================================================================================ 4 passed in 0.15s =================================================================================================
______________________________________________________________________________________________ test_classes_reordered ______________________________________________________________________________________________
ourtestdir = <Testdir local('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_classes_reordered0')>
def test_classes_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
from unittest import TestCase
class A(TestCase):
def test_a(self):
pass
class B(TestCase):
def test_b(self):
pass
class C(TestCase):
def test_c(self):
pass
class D(TestCase):
def test_d(self):
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::D::test_d PASSED",
"test_one.py::B::test_b PASSED",
"test_one.py::C::test_c PASSED",
"test_one.py::A::test_a PASSED",
]
E AssertionError: assert ['',\n 'test_one.py::D::test_d PASSED',\n 'test_one.py::B::test_b PASSED',\n 'test_one.py::C::test_c PASSED'] == ['test_one.py::D::test_d PASSED',\n 'test_one.py::B::test_b PASSED',\n 'test_one.py::C::test_c PASSED',\n 'test_one.py::A::test_a PASSED']
E At index 0 diff: '' != 'test_one.py::D::test_d PASSED'
E Full diff:
E [
E + '',
E 'test_one.py::D::test_d PASSED',
E 'test_one.py::B::test_b PASSED',
E 'test_one.py::C::test_c PASSED',
E - 'test_one.py::A::test_a PASSED',
E ]
/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/tests/test_pytest_randomly.py:307: AssertionError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.9.9, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /gnu/store/j3cx0yaqdpw0mxizp5bayx93pya44dhn-python-wrapper-3.9.9/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-2/source/.hypothesis/examples')
Using --randomly-seed=15
rootdir: /tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_classes_reordered0, configfile: pytest.ini
plugins: hypothesis-6.0.2, randomly-3.11.0, Faker-13.3.4, xdist-2.1.0, forked-1.3.0
collecting ... collected 4 items
test_one.py::D::test_d PASSED
test_one.py::B::test_b PASSED
test_one.py::C::test_c PASSED
test_one.py::A::test_a PASSED
================================================================================================ 4 passed in 0.26s =================================================================================================
============================================================================================= short test summary info ==============================================================================================
FAILED tests/test_pytest_randomly.py::test_doctests_reordered - AssertionError: assert ['', 'test_one.py::test_one.bar PASSED'] == ['test_one.py::test_one.bar PASSED', 'test_one.py::test_one.foo PASSED']
FAILED tests/test_pytest_randomly.py::test_test_functions_reordered_when_randomness_in_module - AssertionError: assert ['',\n 'test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_...
FAILED tests/test_pytest_randomly.py::test_test_functions_reordered - AssertionError: assert ['',\n 'test_one.py::test_c PASSED',\n 'test_one.py::test_a PASSED',\n 'test_one.py::test_b PASSED'] == ['test_one.p...
FAILED tests/test_pytest_randomly.py::test_files_reordered - AssertionError: assert ['',\n 'test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED'] == ['test_b.py::test_it PAS...
FAILED tests/test_pytest_randomly.py::test_doctests_in_txt_files_reordered - AssertionError: assert ['', 'test2.txt::test2.txt PASSED'] == ['test2.txt::test2.txt PASSED', 'test.txt::test.txt PASSED']
FAILED tests/test_pytest_randomly.py::test_files_reordered_when_seed_not_reset - AssertionError: assert ['',\n 'test_b.py::test_it PASSED',\n 'test_a.py::test_it PASSED',\n 'test_d.py::test_it PASSED'] == ['te...
FAILED tests/test_pytest_randomly.py::test_it_runs_before_stepwise - py.error.ENOENT: [No such file or directory]: remove('/tmp/guix-build-python-pytest-randomly-3.11.0.drv-0/pytest-of-maxim/pytest-2/test_it_r...
FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - assert None == ((1,),)
FAILED tests/test_pytest_randomly.py::test_class_test_methods_reordered - AssertionError: assert ['',\n 'test_one.py::T::test_c PASSED',\n 'test_one.py::T::test_b PASSED',\n 'test_one.py::T::test_a PASSED'] ==...
FAILED tests/test_pytest_randomly.py::test_classes_reordered - AssertionError: assert ['',\n 'test_one.py::D::test_d PASSED',\n 'test_one.py::B::test_b PASSED',\n 'test_one.py::C::test_c PASSED'] == ['test_one...
========================================================================================== 10 failed, 26 passed in 35.53s ==========================================================================================
Any ideas what could be causing these?
The direct dependencies used are:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Thank you!
I couldn't find it referenced how to use this plugin to reorder tests randomly (vs. affecting the random seed). Does this happen implicitly if the plugin is installed? I think making more explicit that this happens would be a nice addition to the docs.
cpython defaults to using (os.urandom or time.time() + pid + time.monotonic()) for a random.seed: https://github.com/python/cpython/blob/d0981e61a5869c48e0a70a512967558391272a93/Modules/_randommodule.c#L233-L262
these functions aren't actually available to python code, but random.Random().getrandbits(64)
seems to be the next best thing.
3.8.11
3.10
I am trying to update pytest-randomly
from 3.8
to 3.10
. But my test suite now fails:
============================= test session starts ==============================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=2912292964
rootdir: /home/runner/work/returns/returns, configfile: setup.cfg
plugins: randomly-3.10.0, subtests-0.5.0, xdist-2.3.0, hypothesis-6.14.6, returns-0.16.0, mypy-plugins-1.7.0, anyio-3.3.0, forked-1.3.0
collected 830 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 637, in perform_collect
INTERNALERROR> hook.pytest_collection_modifyitems(
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR> self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 212, in pytest_collection_modifyitems
INTERNALERROR> _shuffle_by_class(list(group), seed),
INTERNALERROR> File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 239, in _shuffle_by_class
INTERNALERROR> klass_items.sort()
INTERNALERROR> TypeError: '<' not supported between instances of 'YamlTestItem' and 'YamlTestItem'
Link: https://github.com/dry-python/returns/pull/1021/checks?check_run_id=3322840792
I guess this happens because we use pytest-mypy-plugins
where we define our tests in yml
files, example: https://github.com/dry-python/returns/blob/master/typesafety/test_functions/test_tap.yml
Steps to reproduce:
poetry install
poetry run pip install 'pytest-randomly==3.10.0'
poetry run pytest typesafety -p no:cov -o addopts="" --mypy-ini-file=setup.cfg
: https://github.com/dry-python/returns/blob/master/.github/workflows/test.yml#L67See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent
While doing a mass rebuild for python 3.9, https://www.archlinux.org/packages/community/any/python-pytest-randomly/ failed to pass the testsuite with the following error log:
============================= test session starts ==============================
platform linux -- Python 3.9.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
Using --randomly-seed=610203316
rootdir: /build/python-pytest-randomly/src/pytest-randomly-3.4.1
plugins: randomly-3.4.1, Faker-4.14.2, forked-1.3.0, xdist-2.1.0
collected 34 items / 1 deselected / 33 selected
tests/test_pytest_randomly.py .....................F........... [100%]
=================================== FAILURES ===================================
__________________ test_it_works_with_the_simplest_test_items __________________
ourtestdir = <Testdir local('/tmp/pytest-of-builduser/pytest-0/test_it_works_with_the_simplest_test_items0')>
def test_it_works_with_the_simplest_test_items(ourtestdir):
ourtestdir.makepyfile(
conftest="""
import pytest
class MyCollector(pytest.Collector):
def __init__(self, fspath, items, **kwargs):
super(MyCollector, self).__init__(fspath, **kwargs)
self.items = items
def collect(self):
return self.items
class NoOpItem(pytest.Item):
def __init__(self, path, parent, module=None):
super(NoOpItem, self).__init__(path, parent)
if module is not None:
self.module = module
def runtest(self):
pass
def pytest_collect_file(path, parent):
if not str(path).endswith('.py'):
return
return MyCollector(
fspath=str(path),
items=[
NoOpItem(str(path), parent, 'foo'),
NoOpItem(str(path), parent),
],
parent=parent,
)
"""
)
args = ["-v"]
out = ourtestdir.runpytest(*args)
> out.assert_outcomes(passed=2)
E AssertionError: assert {'errors': 1,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E Omitting 4 identical items, use -vv to show
E Differing items:
E {'errors': 1} != {'errors': 0}
E {'passed': 0} != {'passed': 2}
E Use -v to get the full diff
/build/python-pytest-randomly/src/pytest-randomly-3.4.1/tests/test_pytest_randomly.py:467: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.0, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
Using --randomly-seed=610203316
rootdir: /tmp/pytest-of-builduser/pytest-0/test_it_works_with_the_simplest_test_items0, configfile: pytest.ini
plugins: randomly-3.4.1, Faker-4.14.2, forked-1.3.0, xdist-2.1.0
collecting ... collected 0 items / 1 error
==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
Direct construction of NoOpItem has been deprecated, please use NoOpItem.from_parent.
See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent for more details.
=========================== short test summary info ============================
ERROR
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.08s ===============================
=========================== short test summary info ============================
FAILED tests/test_pytest_randomly.py::test_it_works_with_the_simplest_test_items
================== 1 failed, 32 passed, 1 deselected in 9.67s ==================
FYI I think there's a compatibility issue with pytest-xdist==1.30.0, see pytest-dev/pytest-xdist#472
I wish to run as many tests as possible using a random order and have this as the default. But for the ones which cannot run in random order, I wish to mark them somehow rather than executing several different commands in the terminal.
Will pytest-ordering work for this if I have pytest-randomly installed as well?
Hi all, I was wondering if there is capacity to release pytest-randomly on conda too?
importlib_metadata is used in the source code but it's specified to be installed only for Python versions less than 3.10 . This will cause issue in Python 3.10 since the dependency is not available causing ImportError. importlib.metadata
is part of stdlib since Python 3.8 so I guess this import can be made as per python version under which the code is run to use importlib_metadata
or to use importlib_metadata
as a hard dependency removing the required python version limit in requirements.
src/pytest_randomly.py
6:from importlib_metadata import entry_points
setup.cfg
1:[metadata]
35: importlib-metadata >= 3.6.0 ; python_version < "3.10"
requirements/requirements.in
3:importlib-metadata >= 3.6.0 ; python_version < "3.10"
requirements/py37.txt
29:importlib-metadata==3.10.0 ; python_version < "3.10" \
119: # via importlib-metadata
123: # via importlib-metadata
requirements/py36.txt
29:importlib-metadata==3.10.0 ; python_version < "3.10" \
129: # via importlib-metadata
133: # via importlib-metadata
requirements/py39.txt
29:importlib-metadata==3.10.0 ; python_version < "3.10" \
115: # via importlib-metadata
requirements/py38.txt
29:importlib-metadata==3.10.0 ; python_version < "3.10" \
115: # via importlib-metadata
tox -e py
.package create: /root/checked_repos_clone/pytest-randomly/.tox/.package
.package installdeps: setuptools >= 40.6.0, wheel
py create: /root/checked_repos_clone/pytest-randomly/.tox/py
py inst: /root/checked_repos_clone/pytest-randomly/.tox/.tmp/package/1/pytest-randomly-3.7.0.tar.gz
py installed: attrs==20.3.0,iniconfig==1.1.1,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.3,pytest-randomly @ file:///root/checked_repos_clone/pytest-randomly/.tox/.tmp/package/1/pytest-randomly-3.7.0.tar.gz,toml==0.10.2
py run-test-pre: PYTHONHASHSEED='1324704500'
py run-test: commands[0] | python -W error::DeprecationWarning -W error::PendingDeprecationWarning -m pytest -p no:randomly
========================================================================= test session starts ==========================================================================
platform linux -- Python 3.10.0a0, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py/.pytest_cache
rootdir: /root/checked_repos_clone/pytest-randomly
collected 0 items / 1 error
================================================================================ ERRORS ================================================================================
____________________________________________________________ ERROR collecting tests/test_pytest_randomly.py ____________________________________________________________
ImportError while importing test module '/root/checked_repos_clone/pytest-randomly/tests/test_pytest_randomly.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_pytest_randomly.py:6: in <module>
import pytest_randomly
.tox/py/lib/python3.10/site-packages/pytest_randomly.py:6: in <module>
from importlib_metadata import entry_points
E ModuleNotFoundError: No module named 'importlib_metadata'
======================================================================= short test summary info ========================================================================
ERROR tests/test_pytest_randomly.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================================== 1 error in 0.25s ===========================================================================
ERROR: InvocationError for command /root/checked_repos_clone/pytest-randomly/.tox/py/bin/python -W error::DeprecationWarning -W error::PendingDeprecationWarning -m pytest -p no:randomly (exited with code 2)
_______________________________________________________________________________ summary ________________________________________________________________________________
ERROR: py: commands failed
In the recent re-organisation of the tests, the new module isnt being included into the sdist.
When running with deprecation warnings on I encountered this
pytest_randomly.py:67: DeprecationWarning: factory.fuzzy.set_random_state
is deprecated. You should use factory.random.set_random_state
instead
factory_set_random_state(random_states[seed])
Hi everybody,
thanks for developing pytest-randomly
! I have a big test suite full of tests working with random data and with changing seeds I feel more confident that I will eventually discover all weird edge cases.
What I would like to propose is to repeat tests with different random seeds derived from the base seed. E.g., I have a test I would like to repeat five times and -randomly-seed=5
. Then, the test should be run with seeds 5, 6, 7, 8, 9.
I imagine a solution like
import numpy as np
@pytest.mark.randomly(repetitions=5)
def test():
arr = np.random.randint(low=0, high=10, size=2)
assert arr[0] + arr[1] == np.sum(arr)
Currently, I interfere in the test generation routine and every test with an argument named seed
is parametrized with increments of --randomly-seed
.
Thanks for providing this plugin, looks great!
I am running tests in a project relying in PyTorch
and I want to use the corresponding rng
. I tried to follow your documentation and added
[options.entry_points]
pytest_randomly.random_seeder =
torch = torch.manual_seed
to my setup.cfg
.
However, torch
is not in the name space when reseed function is called in
pytest-randomly/src/pytest_randomly.py
Line 152 in f7e377b
What am I missing?
Thanks for your help!
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with --no-isolation
I'm using during all processes only locally installed modulesIn build env are installed only modules required pytest-randomly
test suite.
Here is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-randomly-3.12.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-randomly-3.12.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
Using --randomly-seed=542622231
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.12.0, configfile: pyproject.toml
plugins: randomly-3.12.0, forked-1.4.0, Faker-13.11.1, xdist-2.5.0
collected 36 items
tests/test_pytest_randomly.py ........F..........F................ [100%]
================================================================================= FAILURES =================================================================================
________________________________________________________________________ test_entrypoint_injection _________________________________________________________________________
pytester = <Pytester PosixPath('/tmp/pytest-of-tkloczko/pytest-73/test_entrypoint_injection0')>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5a9189c0a0>
def test_entrypoint_injection(pytester, monkeypatch):
"""Test that registered entry points are seeded"""
(pytester.path / "test_one.py").write_text("def test_one(): pass\n")
class _FakeEntryPoint:
"""Minimal surface of Entry point API to allow testing"""
def __init__(self, name, obj):
self.name = name
self._obj = obj
def load(self):
return self._obj
entry_points: list[_FakeEntryPoint] = []
def fake_entry_points(*, group):
return entry_points
monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points)
reseed = mock.Mock()
entry_points.append(_FakeEntryPoint("test_seeder", reseed))
# Need to run in-process so that monkeypatching works
pytester.runpytest_inprocess("--randomly-seed=1")
> assert reseed.mock_calls == [
mock.call(1),
mock.call(1),
mock.call(0),
mock.call(1),
mock.call(2),
]
E assert [] == [call(1), cal...l(1), call(2)]
E Right contains 5 more items, first extra item: call(1)
E Use -v to get more diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.12.0/tests/test_pytest_randomly.py:749: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
Using --randomly-seed=1
rootdir: /tmp/pytest-of-tkloczko/pytest-73/test_entrypoint_injection0
plugins: randomly-3.12.0, forked-1.4.0, Faker-13.11.1, xdist-2.5.0
collected 1 item
test_one.py . [100%]
============================== 1 passed in 0.06s ===============================
_______________________________________________________________________ test_it_runs_before_stepwise _______________________________________________________________________
self = <module 'py.error'>, func = <built-in function remove>, args = ('/tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0/__pycache__',), kwargs = {}
__tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory'), tb = <traceback object at 0x7f5a916a5240>, errno = 2
def checked_call(self, func, *args, **kwargs):
""" call a function and raise an errno-exception if applicable. """
__tracebackhide__ = True
try:
> return func(*args, **kwargs)
E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0/__pycache__'
/usr/lib/python3.8/site-packages/py/_error.py:66: FileNotFoundError
During handling of the above exception, another exception occurred:
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0')>
def test_it_runs_before_stepwise(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def test_a():
assert 0
def test_b():
assert 0
"""
)
out = ourtestdir.runpytest("-v", "--randomly-seed=1", "--stepwise")
out.assert_outcomes(failed=1)
# Now make test_b pass
ourtestdir.makepyfile(
test_one="""
def test_a():
assert 0
def test_b():
assert 1
"""
)
> ourtestdir.tmpdir.join("__pycache__").remove()
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.12.0/tests/test_pytest_randomly.py:552:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.8/site-packages/py/_path/local.py:226: in remove
py.error.checked_call(os.remove, self.strpath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <module 'py.error'>, func = <built-in function remove>, args = ('/tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0/__pycache__',), kwargs = {}
__tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory'), tb = <traceback object at 0x7f5a916a5240>, errno = 2
def checked_call(self, func, *args, **kwargs):
""" call a function and raise an errno-exception if applicable. """
__tracebackhide__ = True
try:
return func(*args, **kwargs)
except self.Error:
raise
except (OSError, EnvironmentError):
cls, value, tb = sys.exc_info()
if not hasattr(value, 'errno'):
raise
__tracebackhide__ = False
errno = value.errno
try:
if not isinstance(value, WindowsError):
raise NameError
except NameError:
# we are not on Windows, or we got a proper OSError
cls = self._geterrnoclass(errno)
else:
try:
cls = self._geterrnoclass(_winerrnomap[errno])
except KeyError:
raise value
> raise cls("%s%r" % (func.__name__, args))
E py.error.ENOENT: [No such file or directory]: remove('/tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0/__pycache__',)
/usr/lib/python3.8/site-packages/py/_error.py:86: ENOENT
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=1
rootdir: /tmp/pytest-of-tkloczko/pytest-73/test_it_runs_before_stepwise0, configfile: pytest.ini
plugins: randomly-3.12.0, forked-1.4.0, Faker-13.11.1, xdist-2.5.0
collecting ... collected 2 items
stepwise: no previously failed tests, not skipping.
test_one.py::test_b FAILED
=================================== FAILURES ===================================
____________________________________ test_b ____________________________________
def test_b():
> assert 0
E assert 0
test_one.py:6: AssertionError
=========================== short test summary info ============================
FAILED test_one.py::test_b - assert 0
!!!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!!!
============================== 1 failed in 0.19s ===============================
========================================================================= short test summary info ==========================================================================
FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - assert [] == [call(1), cal...l(1), call(2)]
FAILED tests/test_pytest_randomly.py::test_it_runs_before_stepwise - py.error.ENOENT: [No such file or directory]: remove('/tmp/pytest-of-tkloczko/pytest-73/test_it_runs...
====================================================================== 2 failed, 34 passed in 27.40s =======================================================================
nose-randomly
has the feature that nosetests path.One path.Two
still runs path.One
and path.Two
in order, allowing reproduction with --randomly-seed=X
much easier - once you've found a bad test order, just copy/paste it and rerun deleting paths one at a time.
pytest-randomly
does not have this property because the pytest collection hooks are more generic. If possible adding this property would make it easier to debug such failures.
As noted by @tolomea
I run pytest-randomly
's test suite during RPM build.
With recent Pytest (5.3+) there is an error:
test_pytest_randomly.py .....F..............ss... [100%]
=================================== FAILURES ===================================
___________________ test_passing_nonsense_for_randomly_seed ____________________
ourtestdir = <Testdir local('/usr/src/tmp/pytest-of-builder/pytest-0/test_passing_nonsense_for_randomly_seed0')>
def test_passing_nonsense_for_randomly_seed(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def test_a():
pass
"""
)
out = ourtestdir.runpytest("--randomly-seed=invalidvalue")
assert out.ret != 0
out.stderr.fnmatch_lines(
[
(
> "pytest.py: error: argument --randomly-seed: 'invalidvalue' "
+ "is not an integer or the string 'last'"
)
]
)
E Failed: nomatch: "pytest.py: error: argument --randomly-seed: 'invalidvalue' is not an integer or the string 'last'"
E and: 'ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]'
E and: "pytest: error: argument --randomly-seed: 'invalidvalue' is not an integer or the string 'last'"
E and: ''
E remains unmatched: "pytest.py: error: argument --randomly-seed: 'invalidvalue' is not an integer or the string 'last'"
Sure, adjusting it like as follows helps:
(
- "pytest.py: error: argument --randomly-seed: 'invalidvalue' "
+ "pytest: error: argument --randomly-seed: 'invalidvalue' "
+ "is not an integer or the string 'last'"
)
Hi!
Just stumbled on this plugin! It looks like a much better version (and maintained!) of the honorable https://github.com/klrmn/pytest-random.
Would you like to consider moving it under the pytest-dev
organization for more visibility? You can read more about this here. ๐
Maybe related to installed dependencies I could imagine, but I find it hard to debug the issue as I have no clue what is going on.
============================= test session starts ==============================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.9.0, pluggy-0.13.1
Using --randomly-seed=3805774731
rootdir: /build/source
plugins: randomly-3.6.0, xdist-2.2.1, forked-1.3.0, Faker-5.5.1
collected 34 items
tests/test_pytest_randomly.py .......................F.......... [100%]
=================================== FAILURES ===================================
__________________________ test_entrypoint_injection ___________________________
testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_entrypoint_injection0')>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fffcc272d00>
def test_entrypoint_injection(testdir, monkeypatch):
"""Test that registered entry points are seeded"""
class _FakeEntryPoint:
"""Minimal surface of Entry point API to allow testing"""
def __init__(self, name, obj):
self.name = name
self._obj = obj
def load(self):
return self._obj
entry_points = []
def fake_entry_points():
return SimpleNamespace(select=fake_select)
def fake_select(*, group):
return entry_points
monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points)
reseed = Mock()
entry_points.append(_FakeEntryPoint("test_seeder", reseed))
# Need to run in-process so that monkeypatching works
testdir.runpytest("--randomly-seed=1")
> assert reseed.call_args == ((1,),)
E AssertionError: assert None == ((1,),)
E + where None = <Mock id='140736618505888'>.call_args
/build/source/tests/test_pytest_randomly.py:690: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.9.0, pluggy-0.13.1
Using --randomly-seed=1
rootdir: /build/pytest-of-nixbld/pytest-0/test_entrypoint_injection0
plugins: randomly-3.6.0, xdist-2.2.1, forked-1.3.0, Faker-5.5.1
collected 0 items
============================ no tests ran in 0.01s =============================
=========================== short test summary info ============================
FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - AssertionEr...
======================== 1 failed, 33 passed in 27.88s =========================
If doctests are enabled the following exception is raised:
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/main.py", line 96, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/main.py", line 130, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/main.py", line 139, in pytest_collection
INTERNALERROR> return session.perform_collect()
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/main.py", line 592, in perform_collect
INTERNALERROR> config=self.config, items=items)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/marscher/miniconda3/lib/python3.5/site-packages/pytest_randomly.py", line 125, in pytest_collection_modifyitems
INTERNALERROR> current_module = item.module
INTERNALERROR> AttributeError: 'DoctestItem' object has no attribute 'module'
Using pytest-randomly 3.3.0 with a recent pytest fails with:
INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_configure_node' in plugin <module 'pytest_randomly' from '/home/travis/virtualenv/python3.6.10/lib/python3.6/site-packages/pytest_randomly.py'>
Versions:
platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.0, pluggy-0.13.1 -- /home/travis/virtualenv/python3.6.10/bin/python
plugins: randomly-3.3.0, cov-2.8.1, requests-mock-1.7.0
Full log: https://travis-ci.org/github/lemon24/reader/jobs/675355855
Running it on my laptop with the latest pytest version gives the same result:
% pip install pytest-randomly==3.3.0
...
% pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
Using --randomly-seed=1586965045
rootdir: /Users/lemon/code/reader, inifile: setup.cfg
plugins: requests-mock-1.7.0, randomly-3.3.0, cov-2.8.1
collected 534 items
...
INTERNALERROR> pluggy.manager.PluginValidationError: unknown hook 'pytest_configure_node' in plugin <module 'pytest_randomly' from '/Users/lemon/code/reader/_venv/lib/python3.7/site-packages/pytest_randomly.py'>
============================ no tests ran in 0.66s =============================
% pip install 'pytest-randomly<3.3.0'
...
% pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
Using --randomly-seed=1586965126
rootdir: /Users/lemon/code/reader, inifile: setup.cfg
plugins: requests-mock-1.7.0, randomly-3.2.1, cov-2.8.1
collected 534 items
...
================== 460 passed, 72 skipped, 2 xfailed in 3.24s ==================
I will try adding a minimal repro later.
would it be possible to get pytest-randomly to shuffle based on overlapping fixture scopes rather than just module, class, fn
+ /usr/bin/python3 -Bm pytest -ra -p no:randomly
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collected 37 items
. . [ 2%]
tests/test_pytest_randomly.py .........FFFFFFF.F................ [100%]
================================================================================= FAILURES =================================================================================
___________________________________________________________________________ test_files_reordered ___________________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_files_reordered0')>
def test_files_reordered(ourtestdir):
code = """
def test_it():
pass
"""
ourtestdir.makepyfile(test_a=code, test_b=code, test_c=code, test_d=code)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_d.py::test_it PASSED",
"test_c.py::test_it PASSED",
"test_a.py::test_it PASSED",
"test_b.py::test_it PASSED",
]
E AssertionError: assert ['', 'test_d....st_it PASSED'] == ['test_d.py::...st_it PASSED']
E At index 0 diff: '' != 'test_d.py::test_it PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:241: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_files_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_d.py::test_it PASSED
test_c.py::test_it PASSED
test_a.py::test_it PASSED
test_b.py::test_it PASSED
============================== 4 passed in 0.11s ===============================
_________________________________________________________________ test_files_reordered_when_seed_not_reset _________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_files_reordered_when_seed_not_reset0')>
def test_files_reordered_when_seed_not_reset(ourtestdir):
code = """
def test_it():
pass
"""
ourtestdir.makepyfile(test_a=code, test_b=code, test_c=code, test_d=code)
args = ["-v", "--randomly-seed=15"]
args.append("--randomly-dont-reset-seed")
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_d.py::test_it PASSED",
"test_c.py::test_it PASSED",
"test_a.py::test_it PASSED",
"test_b.py::test_it PASSED",
]
E AssertionError: assert ['', 'test_d....st_it PASSED'] == ['test_d.py::...st_it PASSED']
E At index 0 diff: '' != 'test_d.py::test_it PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:261: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_files_reordered_when_seed_not_reset0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_d.py::test_it PASSED
test_c.py::test_it PASSED
test_a.py::test_it PASSED
test_b.py::test_it PASSED
============================== 4 passed in 0.11s ===============================
__________________________________________________________________________ test_classes_reordered __________________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_classes_reordered0')>
def test_classes_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
from unittest import TestCase
class A(TestCase):
def test_a(self):
pass
class B(TestCase):
def test_b(self):
pass
class C(TestCase):
def test_c(self):
pass
class D(TestCase):
def test_d(self):
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::D::test_d PASSED",
"test_one.py::C::test_c PASSED",
"test_one.py::A::test_a PASSED",
"test_one.py::B::test_b PASSED",
]
E AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
E At index 0 diff: '' != 'test_one.py::D::test_d PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:300: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_classes_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_one.py::D::test_d PASSED
test_one.py::C::test_c PASSED
test_one.py::A::test_a PASSED
test_one.py::B::test_b PASSED
============================== 4 passed in 0.11s ===============================
____________________________________________________________________ test_class_test_methods_reordered _____________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_class_test_methods_reordered0')>
def test_class_test_methods_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
from unittest import TestCase
class T(TestCase):
def test_a(self):
pass
def test_b(self):
pass
def test_c(self):
pass
def test_d(self):
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::T::test_d PASSED",
"test_one.py::T::test_c PASSED",
"test_one.py::T::test_a PASSED",
"test_one.py::T::test_b PASSED",
]
E AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
E At index 0 diff: '' != 'test_one.py::T::test_d PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:332: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_class_test_methods_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_one.py::T::test_d PASSED
test_one.py::T::test_c PASSED
test_one.py::T::test_a PASSED
test_one.py::T::test_b PASSED
============================== 4 passed in 0.11s ===============================
______________________________________________________________________ test_test_functions_reordered _______________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_test_functions_reordered0')>
def test_test_functions_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def test_a():
pass
def test_b():
pass
def test_c():
pass
def test_d():
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::test_d PASSED",
"test_one.py::test_c PASSED",
"test_one.py::test_a PASSED",
"test_one.py::test_b PASSED",
]
E AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
E At index 0 diff: '' != 'test_one.py::test_d PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:361: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_test_functions_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_one.py::test_d PASSED
test_one.py::test_c PASSED
test_one.py::test_a PASSED
test_one.py::test_b PASSED
============================== 4 passed in 0.11s ===============================
_________________________________________________________ test_test_functions_reordered_when_randomness_in_module __________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_test_functions_reordered_when_randomness_in_module0')>
def test_test_functions_reordered_when_randomness_in_module(ourtestdir):
ourtestdir.makepyfile(
test_one="""
import random
import time
random.seed(time.time() * 100)
def test_a():
pass
def test_b():
pass
def test_c():
pass
def test_d():
pass
"""
)
args = ["-v", "--randomly-seed=15"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=4, failed=0)
> assert out.outlines[8:12] == [
"test_one.py::test_d PASSED",
"test_one.py::test_c PASSED",
"test_one.py::test_a PASSED",
"test_one.py::test_b PASSED",
]
E AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
E At index 0 diff: '' != 'test_one.py::test_d PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:395: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=15
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_test_functions_reordered_when_randomness_in_module0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 4 items
test_one.py::test_d PASSED
test_one.py::test_c PASSED
test_one.py::test_a PASSED
test_one.py::test_b PASSED
============================== 4 passed in 0.11s ===============================
_________________________________________________________________________ test_doctests_reordered __________________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_doctests_reordered0')>
def test_doctests_reordered(ourtestdir):
ourtestdir.makepyfile(
test_one="""
def foo():
'''
>>> foo()
9001
'''
return 9001
def bar():
'''
>>> bar()
9002
'''
return 9002
"""
)
args = ["-v", "--doctest-modules", "--randomly-seed=5"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=2)
> assert out.outlines[8:10] == [
"test_one.py::test_one.bar PASSED",
"test_one.py::test_one.foo PASSED",
]
E AssertionError: assert ['', 'test_on...e.bar PASSED'] == ['test_one.py...e.foo PASSED']
E At index 0 diff: '' != 'test_one.py::test_one.bar PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:425: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=5
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_doctests_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 2 items
test_one.py::test_one.bar PASSED
test_one.py::test_one.foo PASSED
============================== 2 passed in 0.11s ===============================
___________________________________________________________________ test_doctests_in_txt_files_reordered ___________________________________________________________________
ourtestdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-18/test_doctests_in_txt_files_reordered0')>
def test_doctests_in_txt_files_reordered(ourtestdir):
ourtestdir.tmpdir.join("test.txt").write(
"""\
>>> 2 + 2
4
"""
)
ourtestdir.tmpdir.join("test2.txt").write(
"""\
>>> 2 - 2
0
"""
)
args = ["-v", "--randomly-seed=1"]
out = ourtestdir.runpytest(*args)
out.assert_outcomes(passed=2)
> assert out.outlines[8:10] == [
"test2.txt::test2.txt PASSED",
"test.txt::test.txt PASSED",
]
E AssertionError: assert ['', 'test2.t...2.txt PASSED'] == ['test2.txt::...t.txt PASSED']
E At index 0 diff: '' != 'test2.txt::test2.txt PASSED'
E Use -v to get the full diff
/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/tests/test_pytest_randomly.py:498: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
Using --randomly-seed=1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.8.0/.hypothesis/examples')
rootdir: /tmp/pytest-of-tkloczko/pytest-18/test_doctests_in_txt_files_reordered0, configfile: pytest.ini
plugins: randomly-3.8.0, forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, cases-3.4.6, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, flaky-3.7.0, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, hypothesis-6.13.7, Faker-8.4.0, cov-2.12.1
collecting ... collected 2 items
test2.txt::test2.txt PASSED
test.txt::test.txt PASSED
============================== 2 passed in 0.10s ===============================
========================================================================= short test summary info ==========================================================================
FAILED tests/test_pytest_randomly.py::test_files_reordered - AssertionError: assert ['', 'test_d....st_it PASSED'] == ['test_d.py::...st_it PASSED']
FAILED tests/test_pytest_randomly.py::test_files_reordered_when_seed_not_reset - AssertionError: assert ['', 'test_d....st_it PASSED'] == ['test_d.py::...st_it PASSED']
FAILED tests/test_pytest_randomly.py::test_classes_reordered - AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
FAILED tests/test_pytest_randomly.py::test_class_test_methods_reordered - AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
FAILED tests/test_pytest_randomly.py::test_test_functions_reordered - AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py...est_b PASSED']
FAILED tests/test_pytest_randomly.py::test_test_functions_reordered_when_randomness_in_module - AssertionError: assert ['', 'test_on...est_a PASSED'] == ['test_one.py......
FAILED tests/test_pytest_randomly.py::test_doctests_reordered - AssertionError: assert ['', 'test_on...e.bar PASSED'] == ['test_one.py...e.foo PASSED']
FAILED tests/test_pytest_randomly.py::test_doctests_in_txt_files_reordered - AssertionError: assert ['', 'test2.t...2.txt PASSED'] == ['test2.txt::...t.txt PASSED']
====================================================================== 8 failed, 27 passed in 38.43s =======================================================================
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.