Request for Advice on Using Policy Bot in Open Source Projects for Testing, Approving, Merging of PRs about policy-bot HOT 3 OPEN

As you discovered, Policy Bot was designed for overall/merge approval workflows and doesn't support the phased approval (approved for testing vs approved for merge) that your workflow requires. Given the limitations of GitHub Actions and Policy Bot, I don't think you can achieve what you want using only features that are available today.

That said, here's an idea that might work with one new Policy Bot feature:

1. Implement the feature requested in #387, which is a way for Policy Bot to leave GitHub reviews instead of only posting status checks. Since this is a feature we don't have a use for internally, I'm not sure my team will have time to implement it, but I think it should be relatively straightforward to add and I'm happy to review a contribution or discuss the implementation.
2. For your organization, run two instances of Policy Bot, the "Test" instance and the "Merge" instance
3. The "Test" instance looks at a test-specific policy file and leaves an approving review when it is satisfied. You configure the GitHub branch protection and the "Merge" policy to ignore approvals from this app instance. Instead, this approval is the trigger condition for the GitHub Actions workflow.
4. The "Merge" instance looks at a merge-specific policy file and approves the PR (either by leaving a review or via status check) when code review is complete and the PR is ready to merge

Running two instances of Policy Bot is awkward, but they can be configured to not conflict and it avoids adding more complicated features.

To achieve what you want with only a single instance, I think we'd have to add significant new features. We don't really have a concept of sub-policies or policies for different actions (the disapproval policy is an explicit special case) and I don't immediately see a place to hook in logic to leave comments or add labels when certain conditions are met.

from policy-bot.

Another alternative we considered was Mergify, which is free for open-source, but limited on features. I believe the features we needed are not available in the open-source offering.

https://docs.mergify.com/

from policy-bot.

Thanks @bluekeyes for the phenomenally prompt response. That's very helpful! Will discuss with the team what we should do, but the suggestion makes sense. Also, thanks for confirming our suspicion that we would need to deploy two bots for the reasons you mentioned.

from policy-bot.