Provide reverse order about pytest-randomly HOT 6 CLOSED

Hi @thbde

Thanks for pointing outt the research!

However, in this case, the commit when a flawed test is introduced may be different to the commit for which the test fails. This leads to higher effort for the root cause detection.

Yes it does.

Running the tests in reverse order is a simple and effective approach to detect simple order issues.

Yes it is. Django's test framework includes a --reverse flag for this reason.

You know I would've thought this would exist in pytest core, but it doesn't.

You can make a plugin to allow --reverse to reverse the tests with just a few lines:

def pytest_addoption(parser):
    parser._addoption(
        "--reverse",
        action="store",
        ... # todo: fill in remaining arguments
    )


def pytest_collection_modifyitems(session, config, items):
    if config.getoption("reverse"):
        items[:] = items[::-1]

Given this is part of Django's test runner, and it's such a small amount of code, it might be suitable for pytest core. Since it's not randomness-based, I don't think it's appropriate here.

I'm interested to see what the pytest core devs say, if you open an issue on pytest core (please tag me, and refer here).

from pytest-randomly.

I just read the paper and found it so interesting I decided it's really worth having this ability in pytest. I've create a plugin pytest-reverse to release a full version of that code snippet. Please test it!

And please do open an issue on pytest core, I think this would be a small and valuable addition.

from pytest-randomly.

I was a bit busy, I will open the issue as suggested. Thank you for your fast feedback.

Regarding your positive reaction to the paper, maybe a short summary and a link to the paper could be a nice addition to the README. This would provide additional arguments and resources to for the interested user of randomly. I can open a PR with some text if you want.

Regarding the place of the option (here or core): reverse might be a special version of randomness :) (at least in contrast to in-order). But I am also happy with a core option.

from pytest-randomly.

Yes please, and on pytest-reverse ! It's also worth mentioning pytest-reverse on https://docs.pytest.org/en/latest/flaky.html

from pytest-randomly.

I was busy with some other tasks, but here we go. I also added a link to pytest-reverse.

I tried to keep the amount of text low. Please share your thoughts if you prefer more explanation and reasoning like in my initial post here.

Feel free to close this issue after accepting the PR.

from pytest-randomly.

Thank you! 🙏

Good work with the references section.

from pytest-randomly.