Create Github action to run regression test for newly added collections about l2ss-py-autotest HOT 8 CLOSED

After initial review, it looks like the cron timer is all set to go (just commented out) and the steps to calculate the diff are there. The two jobs for creating the pull requests are also there. What's missing are the pull request updates after testing:

If all tests pass: The pr will be labeled verified and automatically merged
If any test fails or has an unknown error: The pr will be labeled bug and failed verification and will remain open
If any tests are skipped: The pr will be labeled unverified and will remain open

So what I need to determine next (maybe @sliu008 or @frankinspace can help with this) is how to tell if collections are skipped, how to tell if they are failed, and how to tell if everything worked properly.

If I'm understanding the rest properly, it looks like the diff list is the new file for the PR, so most likely there would be some indication in that list as to whether or not a collection was skipped or failed. I'll look into that file a little more and see what I can find.

Work to be done includes:

1. Determine failed items
2. Determine skipped items
3. Add a step for updating PR labels on failed items
4. Add a step for updating PR labels on skipped items
5. Turn on cron timer

from l2ss-py-autotest.

@jonathansmolenski Did you check the https://github.com/podaac/l2ss-py-autotest/blob/main/.github/workflows/verify.yml workflow, most of that logic is in there.

from l2ss-py-autotest.

Thanks @frankinspace I hadn't gone through that yet. Should be everything I need

from l2ss-py-autotest.

@frankinspace @sliu008 I'm pretty happy witht he workflow right now, but I want to verify one thing. When the results are marked as "skipped" it looks like the pull requests is merged. Is that the proper action to take in that scenario?

from l2ss-py-autotest.

No, I think the intent for skipped is that someone needs to acknowledge the situation and manually deal with the PR.

from l2ss-py-autotest.

A few updates.

I've added a step to the verify.yml file that adds a check (fails the check) if any of the tests are failed, skipped, or errored out.

Next, I've updated the branch main so that it requires my check to be passed in order to auto-merge.

I'll probably have to add a new step to pass the check if nothing has gone wrong, and I'll need to fix the issue where if the UAT commit completes, the OPS check can't because it needs to pull the main branch.

from l2ss-py-autotest.

@jonathansmolenski I think you closed this unintentionally?

from l2ss-py-autotest.

The github action in question can be found here: https://github.com/podaac/l2ss-py-autotest/blob/main/.github/workflows/diff.yml

from l2ss-py-autotest.