Adding exclude list capabilities at config level about go-ftw HOT 5 CLOSED

So it is more an ignorelist maybe? We can do both.

from go-ftw.

I think it's important that excludelist would contains tests identifiers which could be ignored, if the test failed - not skipped the test. :)

from go-ftw.

After doing a first pass, I come with this. Having three additional maps for ignore, pass or fail.

An example is:

testmodify:
  ignore:
    '920400-1': 'This test result must be ignored'
    ... (this is a list)
  forcepass:
    '123456-1': 'We expect this test to pass once issue github.com/.... is resolved'
    ... (this is a list)
  forcefail:
    '200200-4': 'Force fail this test until we find a solution to bug #1234'
    ... (this is a list)

The id of the test should match with the proper test you want to modify its results, and the text should be handy for printing in the results.

This only modifies the results, but the test is still executed.

What do you think?

from go-ftw.

I don’t think we need that many kinds of lists - if case of ftwrunner I made only one list, but that list can modify the result as this:

if TEST == PASSED:
    if TEST in WHITELIST:
        result = PASSED_BUT_LISTED  // this is important for user, because the test can be removed from the list 
    else:
        result = PASSED  // this is the expected
else:
    if TEST in WHITELIST:
        result = FAILED_BUT_LISTED // this is the expected
    else:
        result = FAILED  // ops...

This only modifies the results, but the test is still executed.

Agree. But not sure this implies the number of lists.

from go-ftw.

This was implemented.

from go-ftw.