How can I blacklist tests which will SKIP or CANCEL on my machine? about avocado-misc-tests HOT 4 CLOSED

The existence of GenWqe is checked during the test suites class level setUp(). A problem could be that it is hard to distinguish these two cases: (1) the device has no GenWqe and should SKIP the tests and (2) the device has GenWqe but it is not working correctly and CANCEL the tests (as the check is executed in the setUp() FAIL is no option here I guess).

from avocado-misc-tests.

How can I identify tests which SKIPed and CANCELed due to an unsuitable configuration of the system under test? Is there a way to "blacklist" these tests prior to test execution?

No. In Avocado, the test discovery (used as the only step by avocado list and as the first step by avocado run) follows a design (security) decision of not executing code from the test. We have to discover all tests by static syntax analysis. Checking whether a test will be skipped/canceled or not before actually running any test code would probably be impossible.

The existence of GenWqe is checked during the test suites class level setUp(). A problem could be that it is hard to distinguish these two cases: (1) the device has no GenWqe and should SKIP the tests and (2) the device has GenWqe but it is not working correctly and CANCEL the tests (as the check is executed in the setUp() FAIL is no option here I guess).

Once you understand the statuses meaning, you can use them as you wish. It's acceptable to call self.fail() from the setUp(). The main difference between SKIP/CANCEL and FAIL is that FAIL will make the Job to exit with a non-zero exit status:

~$ avocado run canceltest.py
JOB ID     : 188d3bd44b28913ed1b5f26d71e51d3c6c6daa87
JOB LOG    : /home/apahim/avocado/job-results/job-2017-11-09T13.29-188d3bd/job.log
 (1/1) canceltest.py:CancelTest.test: CANCEL (0.00 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1
JOB TIME   : 0.83 s
JOB HTML   : /home/apahim/avocado/job-results/job-2017-11-09T13.29-188d3bd/results.html
~$ echo $?
0

~$ avocado run failtest.py
JOB ID     : c373f5c43d78a980ee68319ffc82b525feece90b
JOB LOG    : /home/apahim/avocado/job-results/job-2017-11-09T13.29-c373f5c/job.log
 (1/1) failtest.py:FailTest.test: FAIL (0.04 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1.00 s
JOB HTML   : /home/apahim/avocado/job-results/job-2017-11-09T13.29-c373f5c/results.html
~$ echo $?
1

Now, between SKIP and CANCEL, the difference is that a skipped test cannot have any part of it executed (not even setUp()). A canceled test, can. Meaning that, to SKIP a test, the condition check code has to live outside the test class. To CANCEL a test, the condition can live inside the test code and it also means that setUp() and tearDown() will both be executed.

from avocado-misc-tests.

Btw, not really related, but maybe a way out: you can tag the tests and then run only the tests tagged with a given tag: http://avocado-framework.readthedocs.io/en/55.0/WritingTests.html#categorizing-tests

from avocado-misc-tests.

No. In Avocado, the test discovery (used as the only step by avocado list and as the first step by avocado run) follows a design (security) decision of not executing code from the test. We have to discover all tests by static syntax analysis. Checking whether a test will be skipped/canceled or not before actually running any test code would probably be impossible.

Ok, that's reasonable.

Once you understand the statuses meaning, you can use them as you wish. It's acceptable to call self.fail() from the setUp(). The main difference between SKIP/CANCEL and FAIL is that FAIL will make the Job to exit with a non-zero exit status:

I didn't think about that ...

Now, between SKIP and CANCEL, the difference is that a skipped test cannot have any part of it executed (not even setUp()). A canceled test, can. Meaning that, to SKIP a test, the condition check code has to live outside the test class. To CANCEL a test, the condition can live inside the test code and it also means that setUp() and tearDown() will both be executed.

... then there is no doubt about how to interpret the test status any more. Thanks for the explanation.

Btw, not really related, but maybe a way out: you can tag the tests and then run only the tests tagged with a given tag: http://avocado-framework.readthedocs.io/en/55.0/WritingTests.html#categorizing-tests

I will consider that for potential tests I add by myself or for possible PR for existing tests.

from avocado-misc-tests.