kanecheshire / retryable Goto Github PK

Non-parallel test runs incorrectly report captured flakes as failures, probably because Fastlane itself fails the run rather than Xcode.

Current workaround is to run tests in parallel since Fastlane doesn't see flakes as failures that way;

If I might suggest something to this incredible approach to tests (can't thank you enough) is that tests that are marked as flaky are expected to fail.

Something like this:

    open func flaky(_ flakiness: Reliability.Flakiness, _ tests: () -> Void) {
        reliability = .flaky(flakiness)
        XCTExpectFailure("This is a flaky test", options: .nonStrict(), failingBlock: tests)
        reliability = .reliable
    }

This will improve test reporting, since if it's an expected failure, it won't show up as a failure in the navigator.

This would allow people to use Retryable with other libs that require subclassing a test case to work.

Super easy to fix but worth mentioning that Xcode 11 will try to be a bit smarter, and any suite reports that have the same name will be combined into one section in the Xcode Report navigator.

Example, if you have two test cases that both have one retried test each, Retryable would create two suites called "Retrying 1 failed test". Xcode 10 would display these as two separate sections in the report, Xcode 11 shows them as one section, with two failures, but the name says just "1 failed test".

Could be fixed by adding a UUID to each suite name or somehow creating a name derived from the failed test case's names.