Giter Club home page Giter Club logo

ok-to-test's Introduction

Ok To Test

Example workflow configuration showing how to use GitHub Actions secrets in pull requests from forks πŸ΄πŸ”‘

Summary

An Ok To Test workflow is configured so that when someone with write access to this repository comments ok-to-test sha=<head-sha> on a pull request from a fork, a "privileged" Integration tests workflow needing secrets is triggered. In parallel, a "non-privileged" Unit tests workflow not needing secrets is triggered on any pull request.

About

GitHub Actions purposely limits the secrets available to pull requests from forks for security reasons:

Though this provides peace of mind, many projects depend on the fork pull request model. If you've configured a GitHub Actions test workflow to trigger on pull requests, and those tests require secrets, the secrets aren't available and the workflow fails.

No longer with this workaround, which shows an example Prow-like /ok-to-test sha=<head-sha> slash command configuration! πŸ₯³

This project is not affiliated with GitHub.

Setup

This is a template repository with three example workflows. Start by creating a new repository ("Use this template"). Then, consider for your use case:

  1. Which type of token you'll use to emit the repository_dispatch event in Ok To Test. Set the secrets in your repository accordingly, e.g. I used a GitHub App and had to save secrets called APP_ID and PRIVATE_KEY. Remember: if you also choose GitHub App authentication (preferred), you must create and install it on the repo(s) in which this configuration will run. See Creating A GitHub App for a basic overview of how to do this.
  2. Which workflow(s) need secrets. In this example, it's Integration tests, and I would need to fill in my tests here.
  3. Which workflow(s) do not need secrets. In this example, it's Unit tests. These types of workflows can simply trigger on pull request.
  4. The Permissions required for your GITHUB_TOKEN. The workflows used to implement ok-to-test require the ability to: add reactions to your pull request comments, and update the status of your pull request checks. Currently GitHub Actions' built-in GITHUB_TOKEN is read-only by default. The example workflows in this repo explicitly grant the necessary write permissions to the jobs that require them. You can read more about this in the GitHub Docs, which also describe how to update the defaults.

Usage

As someone with write access, comment /ok-to-test sha=<head-sha> on an incoming pull request to set off this Rube Goldberg machine πŸ˜„. The head sha is the first seven characters of the most recent commit of the incoming pull request. For example, /ok-to-test sha=742c71a.

Example

  1. A fork pull request is opened.
  2. A unit test workflow runs. Secrets are not available to this workflow.
  3. Someone with write access looks over the pull request code. ⚠️ Before proceeding, they should be sure the code isn't doing anything malicious like secret logging. ⚠️
  4. They comment /ok-to-test sha=<head-sha> on the pull request.
  5. A repository_dispatch API request is sent to this repository. See guidance below on how to authenticate.
  6. An integration test workflow runs, checking out the merge commit if the head sha hasn't changed since the comment was made. Secrets are available to this workflow! πŸ’«
  7. The pull request status check is updated to reflect the success or failure of the integration test workflow.

Note that this sequence also works for branch based pull requests, as you'd expect!

Authentication

Choose one of these authentication methods for the repository_dispatch helper action, peter-evans/slash-command-dispatch, in ok-to-test.yml:

GitHub Apps have distinct identities on GitHub – no seat taken up by a machine account, no potential for leaking your personal credentials, and no rate limit sharing!

Creating a GitHub App

Here we are using a GitHub App as an authentication entity. Below are some brief instructions on how to setup a GitHub App for this purpose, note that there are other methods of creating a GitHub App such as with a manifest file (e.g. one similar to app.yml). (The instructions below are for setting up an app within your user, but you can also do it for your organization.)

  1. Go to Settings > Developer Settings > GitHub Apps, and select New GitHub App.
  2. Enter a name for your app (this needs to be unique across GitHub), and fill in the required URL fields. You can fill in these URLs with fake values - they do not need to resolve, so you can use:
    • Homepage URL = http://example.com
  3. You can ignore the 'Callback URL' field, and untick 'Webook' > Active
  4. Under Repository Permissions, set:
    • Contents = 'Read and Write'
    • Metadata = 'Read-only'
  5. Click 'Create GitHub app'
  6. Click 'Generate Private key' (this will be downloaded to your computer), and take a note of the App ID field
  7. Install the GitHub App into your user or organization, by clicking 'Install' under the 'Install App' tab, and choose whether you want to give the app access to all of your user's / org's repositories, or just specific ones
  8. Go to the repository that you want to use ok-to-test with, and then Settings > Secrets and variables > Actions and create two new secrets:
    • APP_ID with the value for the App ID field that you noted earlier
    • PRIVATE_KEY, copying and pasting in the full contents of the Private Key file that you generated and downloaded earlier

Credits

Contributing

Pull requests are welcome!

License

MIT

ok-to-test's People

Contributors

alzafacon avatar imjohnbo avatar imnotjohnbo avatar jagoodhand avatar kjaksik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

ok-to-test's Issues

Using Oauth App Token

I want to use the OAuth App method to authenticate, but how do you create the token in the first place? I see that you're intending to use secrets in the Slash Command Dispatch, but how does the token get created and put into Github secrets?

My interpretation was for this code:

- name: Slash Command Dispatch
      uses: peter-evans/slash-command-dispatch@v1
      env:
        TOKEN: ${{ steps.generate_token.outputs.token }}
      with:
        token: ${{ env.TOKEN }} # GitHub App installation access token
        # token: ${{ secrets.PERSONAL_ACCESS_TOKEN }} # PAT or OAuth token will also work
        reaction-token: ${{ secrets.GITHUB_TOKEN }}
        issue-type: pull-request
        commands: ok-to-test
        named-args: true
        permission: write

to become

- name: Slash Command Dispatch
      uses: peter-evans/slash-command-dispatch@v1
      with:
        token: ${{ secrets.OAUTH-TOKEN}} # PAT or OAuth token will also work
        reaction-token: ${{ secrets.GITHUB_TOKEN }}
        issue-type: pull-request
        commands: ok-to-test
        named-args: true
        permission: write

but that implies that there is a OAUTH-TOKEN in Github secrets. I know that the PAT is easy to create and put into secrets, but when creating an OAuth App in the UI, there's no where suggesting how to create a token. I'm reading steps #2 and #3 from these docs, but there's nothing about putting that token into Github secrets... Any guidance would help! I see that the flow using the Github app installation is well-documented, but using the Oauth token doesn't have an example, one would be appreciated!

Unhandled error: TypeError: Cannot read property 'get' of undefined

Hey! Can you help me debug what is happening with the actions/github-script@v5 step on my CI?

https://github.com/external-secrets/external-secrets/runs/4166526723?check_suite_focus=true

We are basically following the same pattern suggested here in this repo. But out of nowhere this started to break. I just tried changing await github.pulls.get to await github.pull_requests.get but of course it wasn't it πŸ˜…

How did you figure out what to include in this script?

Having difficulty knowing how to set up the GitHub App

Hi there,

Thanks a lot for creating this, this looks really nice and exactly what I need. I understand pretty much everything regarding the workflow part of things but I'm struggling to know what kind of parameters to put in when creating the github app even when following URL: https://docs.github.com/en/free-pro-team@latest/developers/apps/creating-a-github-app

Especially not sure what to put in the URL sections as well as what kind of permissions it will need.

Do you have more info on how to do this best?

Using personal access token

I am using the personal access token authentication method. However, I can't seem to get it to work. Here is my ok-to-test.yml. What lines should be commented out in order to make it work for the personal access token? Thanks.

Bug: grab sha sooner or as argument

The sha should be referenced sooner that this so that this scenario doesn't happen:

  • Time T: admin comments /ok-to-test
  • Time T + 1: malicious code introduced into PR branch
  • Time T + 2: PR branch checked out

Solution 1: accept sha argument like /ok-to-test 827asa
Solution 2: shorten time window by calculating the merge commit in ok-to-test.yml instead of integration.yml

See explicit logs in original workflow run

@imjohnbo hello, we have applied your template in our repo, yet there is some functionality we want cannot be achieved. So we would like to hear some insights from you. So basically we would like to see the logs in the original workflow instead of the /ok-to-test command workflow for a forked-based pull request. For example, if you look here It says the check is skipped. Let's say the owner triggered the /ok-to-test workflow and it fails, here it will only display the check fails. The real logs are displayed in the newly triggered /ok-to-test workflow. So what we want is that right here in the original workflow we can see the error log, or at least can display an url pointing to the actual running workflow. Do you think this is achievable? Thanks

Explain actions/checkout for forked PRs vs normal PRs

Hi! I'm having trouble understanding how to get forked PRs to checkout the code of the head SHA of the PR because when I use your example code of:

ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge'

The Integration Tests come back as having succeeded, but the individual tests say they are still pending. In the Github Actions UI, it says that the tests for the forked branches are skipped even though all the criteria pass. Here is my integration file for my open source repo: https://github.com/chanzuckerberg/terraform-provider-snowflake/blob/main/.github/workflows/integration.yml. I took out the above line because it didn't seem to be running the most recent commit on my PR. Any advice?

Example integration

Came across your solution in trying to find a way to share secrets in a safe way to fork contributors (for CI purposes). Would you mind sharing some example how I could incorporate this into an existing GitHub Actions CI workflow? Still relatively new to actions but solutions for secret access are few and far between. This looks promising!

Question

Why invoke a repo dispatch event? Can’t we instead open a check run from the issue_comment event directly? Would something like this work:

If owner OR comment = β€œ/ok-to-test”
...tests?

Thanks just learning I’m obviously missing something

Script error - Update Check Runs

Thanks for putting this together, it's really useful. I've been experiencing a couple of (what I imagine are) permission errors related to the GitHub App configuration. Are there some additional permissions that need to be granted beyond those stated below?

default_permissions:
  contents: :write
  metadata: :read

This script:

script: |
const { data: pull } = await github.rest.pulls.get({
...context.repo,
pull_number: process.env.number
});
const ref = pull.head.sha;
const { data: checks } = await github.rest.checks.listForRef({
...context.repo,
ref
});
const check = checks.check_runs.filter(c => c.name === process.env.job);
const { data: result } = await github.rest.checks.update({
...context.repo,
check_run_id: check[0].id,
status: 'completed',
conclusion: process.env.conclusion
});
return result;

Fails with this error for me:

2023-03-05T18:53:26.8384989Z RequestError [HttpError]: Resource not accessible by integration
2023-03-05T18:53:26.8387986Z     at /home/runner/work/_actions/actions/github-script/v6/dist/index.js:6803:21
2023-03-05T18:53:26.8388365Z     at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-03-05T18:53:26.8388932Z     at async eval (eval at callAsyncFunction (/home/runner/work/_actions/actions/github-script/v6/dist/index.js:15099:16), <anonymous>:13:26)
2023-03-05T18:53:26.8389463Z     at async main (/home/runner/work/_actions/actions/github-script/v6/dist/index.js:15198:20) {
2023-03-05T18:53:26.8389745Z   status: 403,
2023-03-05T18:53:26.8389922Z   response: {
2023-03-05T18:53:26.8390378Z     url: 'https://api.github.com/repos/GoodhandSolutions/terraform-snowflake-playground/check-runs/11775549491',
2023-03-05T18:53:26.8391081Z     status: 403,
2023-03-05T18:53:26.8391282Z     headers: {
2023-03-05T18:53:26.8391572Z       'access-control-allow-origin': '*',
2023-03-05T18:53:26.8392544Z       'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
2023-03-05T18:53:26.8393279Z       connection: 'close',
2023-03-05T18:53:26.8393538Z       'content-encoding': 'gzip',
2023-03-05T18:53:26.8393865Z       'content-security-policy': "default-src 'none'",
2023-03-05T18:53:26.8394227Z       'content-type': 'application/json; charset=utf-8',
2023-03-05T18:53:26.8394548Z       date: 'Sun, 05 Mar 2023 18:53:26 GMT',
2023-03-05T18:53:26.8394962Z       'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
2023-03-05T18:53:26.8395324Z       server: 'GitHub.com',
2023-03-05T18:53:26.8395714Z       'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
2023-03-05T18:53:26.8396063Z       'transfer-encoding': 'chunked',
2023-03-05T18:53:26.8396408Z       vary: 'Accept-Encoding, Accept, X-Requested-With',
2023-03-05T18:53:26.8396746Z       'x-content-type-options': 'nosniff',
2023-03-05T18:53:26.8397024Z       'x-frame-options': 'deny',
2023-03-05T18:53:26.8397350Z       'x-github-api-version-selected': '2022-11-28',
2023-03-05T18:53:26.8397678Z       'x-github-media-type': 'github.v3',
2023-03-05T18:53:26.8398039Z       'x-github-request-id': '86C4:60B2:2DCE608:5F0BA85:6404E526',
2023-03-05T18:53:26.8398348Z       'x-ratelimit-limit': '1000',
2023-03-05T18:53:26.8398628Z       'x-ratelimit-remaining': '980',
2023-03-05T18:53:26.8398911Z       'x-ratelimit-reset': '1678044246',
2023-03-05T18:53:26.8399189Z       'x-ratelimit-resource': 'core',
2023-03-05T18:53:26.8399470Z       'x-ratelimit-used': '20',
2023-03-05T18:53:26.8400037Z       'x-xss-protection': '0'
2023-03-05T18:53:26.8400235Z     },
2023-03-05T18:53:26.8400407Z     data: {
2023-03-05T18:53:26.8400810Z       message: 'Resource not accessible by integration',
2023-03-05T18:53:26.8403050Z       documentation_url: 'https://docs.github.com/rest/reference/checks#update-a-check-run'
2023-03-05T18:53:26.8403371Z     }
2023-03-05T18:53:26.8403527Z   },
2023-03-05T18:53:26.8403684Z   request: {
2023-03-05T18:53:26.8403938Z     method: 'PATCH',
2023-03-05T18:53:26.8404396Z     url: 'https://api.github.com/repos/GoodhandSolutions/terraform-snowflake-playground/check-runs/11775549491',
2023-03-05T18:53:26.8404722Z     headers: {
2023-03-05T18:53:26.8405028Z       accept: 'application/vnd.github.-preview+json',
2023-03-05T18:53:26.8405475Z       'user-agent': 'actions/github-script octokit-core.js/3.6.0 Node.js/16.16.0 (linux; x64)',
2023-03-05T18:53:26.8405836Z       authorization: 'token [REDACTED]',
2023-03-05T18:53:26.8406169Z       'content-type': 'application/json; charset=utf-8'
2023-03-05T18:53:26.8406411Z     },
2023-03-05T18:53:26.8406688Z     body: '{"status":"completed","conclusion":"success"}',
2023-03-05T18:53:26.8407009Z     request: { agent: [Agent], hook: [Function: bound bound register] }
2023-03-05T18:53:26.8407248Z   }
2023-03-05T18:53:26.8407404Z }
2023-03-05T18:53:26.8437830Z ##[error]Unhandled error: HttpError: Resource not accessible by integration

Similarly the dispatching action, raises warnings for: Failed to set reaction on comment ID 1455172514.

Are these just user errors on my part or is it something else?

PS. I think the action versions need to be updated to prevent warnings being raised about the Node version. I can raise a PR for that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.