pytest session timeout support. about pytest-timeout HOT 5 CLOSED

We can use this statement to stop the whole session.
request.session.shouldstop = 'time limit reached: %0.2f seconds' % elapsed

Could you describe a bit more the entire process you mean here? Not sure I follow.

Using CI to timeout pytest cannot do a graceful clean and test result report. Thank you.

Have you considered looking if it's possible to get the test reports written out on external termination instead?

from pytest-timeout.

Hi @flub , thanks for your reply and help.

I tried to replace pytest.fail with request.session.shouldstop in the following function to do the concept proof that pytest-timeout could work for pytest session timeout:

def timeout_sigalrm(item, timeout):
    """Dump stack of threads and raise an exception
    This will output the stacks of any threads other then the
    current to stderr and then raise an AssertionError, thus
    terminating the test.
    """
    if SUPPRESS_TIMEOUT:
        return
    __tracebackhide__ = True
    nthreads = len(threading.enumerate())
    if nthreads > 1:
        write_title("Timeout", sep="+")
    dump_stacks()
    if nthreads > 1:
        write_title("Timeout", sep="+")
    pytest.fail("Timeout >%ss" % timeout)

The timeout mechanism can stop the whole pytest session.
But I was not sure the original design why pytest-timeout only timeout a test but not providing another option to timeout the whole session.

The use case here:
CI(Jenkins) set a hard timeout, say 5 hrs for pytest jobs. pytest with --junit-xml reports the whole results only after test suite. If the CI timeout was hit and the whole pytest was killed by an external signal, junit test reports were lost.

Two fixes to consider:

1. Have a pytest session timeout mechanism and gracefully cleanup for a test suite. Avoid using CI to set external timeout and kill pytest process.
2. junitxml plugin report test result after each test, not the whole test suite.

The first fix could fit pytest-timeout domain. That's why I posted an issue here.

The second fix could be in junitxml.py in pytest.

Thank you very much for your reading and time.

from pytest-timeout.

Some context on why pytest-timeout doesn't have this feature: It is my personal opinion that timeouts of this nature (whole test run is taking too long) are the CI system's problem. And I also believe that this should not happen normally and is a serious bug in the test suite if it does. This is the reason that I've never attempted to solve this problem of lost test reporting in case the timeout triggers (the fact that the signal method doesn't have this problem is entirely accidental).

Having said that, I do not oppose if someone would like to contribute such a feature. I'm happy to give feedback on a design and review PRs.

To answer you question about the original design of pytest. It is solving the problem of a test being stuck (the first paragraph of the readme tries to say this: "if you don't know why the test suite hangs"). This kind of problem is a serious bug and that reporting is broken at this point is not much of a concern, once the bug is fixed CI and reporting will be fine again. The concern is showing stack traces so you can debug where the code deadlocked.

I tried to replace pytest.fail with request.session.shouldstop in the following function to do the concept proof that pytest-timeout could work for pytest session timeout

When a signal handler runs, it pushes a new frame on the stack in which it executes. Once that frame returns the previous frame will continue where it left off. If the previous frame was stuck, it will just keep being stuck. pytest.fail() however raises an exception, one which is usually not caught by any of the non-pytest code. So when the frame of the signal handler returns and the exception is set, the exception will usually propagate up the stack until pytest catches it. This is why your modification no longer works: you stopped raising an exception. I suspect this will work better if you insert your shouldstop line before the pytest.fail() call instead of replacing it.

from pytest-timeout.

Hi @flub , thank you very much for your help. You were right. I should insert shouldstop line before the pytest.fail() call.

I will try to make a PR later.

from pytest-timeout.

Going to close this, this essentially asks for graceful termination and graceful termination is explicitly not the domain of pytest-timeout. It's purpose is last-resort termination.

from pytest-timeout.