The Buildkite <-> Phabricator bridge can ignore failed tests in the "unit tests" view of Harbormaster about llvm-premerge-checks HOT 9 CLOSED

@ldionne I have updated the script that generates the report - now it should fail on anything that is not "passing" (previously we were only considered "pass" or "fail" as interesting), it also should correctly process recursive builds and report all failing tests. Not sure if reporting all failed jobs will be too verbose, please ping me if it is going out of hand.

from llvm-premerge-checks.

I will take a look on Monday.

from llvm-premerge-checks.

Hi @ldionne , unfortunately I didn't have a chance to look at it this week (on a bright side we solved one issue with win machines with @mstorsjo in #440) . I am also leaving for 2 week vacation.

@movsic maybe you want to exercise your buildkite knowledge? 😉

from llvm-premerge-checks.

This just happened again! https://buildkite.com/llvm-project/libcxx-ci/builds/22675 is skipped, but https://llvm.org/D143914 shows the pre-merge checks as passed. I think there's a lot of value in fixing this TBH, otherwise we don't know whether we can rely on a green checkmark from the Phabricator UI.

from llvm-premerge-checks.

On the surface it seems that the issue is with the part that parses the results and passes back to phabricator.
@ldionne will it be better to not report any of the results back to phabricator and just report that "there are failures - go figure"? (that will be the quickest option).
Otherwise I am looking into properly parsing the results.

from llvm-premerge-checks.

@ldionne I am also confused by

In particular, we had some tests in the no-exceptions CI job that were never reported in the Harbormaster view
while in...

in https://reviews.llvm.org/harbormaster/unit/220338/ you posted there are 4 failures reported by "no exceptions" - exactly ones in https://buildkite.com/llvm-project/libcxx-ci/builds/20939#0186fc1a-7a90-43ca-b8bb-53b32fe23a28

from llvm-premerge-checks.

On the surface it seems that the issue is with the part that parses the results and passes back to phabricator.
@ldionne will it be better to not report any of the results back to phabricator and just report that "there are failures - go figure"? (that will be the quickest option).

Yes, honestly I think that's what I would do.

@ldionne I am also confused by

In particular, we had some tests in the no-exceptions CI job that were never reported in the Harbormaster view
while in...

in https://reviews.llvm.org/harbormaster/unit/220338/ you posted there are 4 failures reported by "no exceptions" - exactly ones in https://buildkite.com/llvm-project/libcxx-ci/builds/20939#0186fc1a-7a90-43ca-b8bb-53b32fe23a28

Hmm, you are right, I'm not sure what I was referring to anymore. The noexceptions Lit tests failing do seem to be reported faithfully. However, some other jobs are failing before we even get to the Lit tests and those results are not reported -- so IMO the right path here would be to not attempt to report anything and just provide a quick link to the Buildkite job itself, where people can see the problems directly.

from llvm-premerge-checks.

@ldionne is it better now?

from llvm-premerge-checks.

I think it is, thanks!

from llvm-premerge-checks.