pulumi / ci-mgmt Goto Github PK

Configuration for all things CI

License: Apache License 2.0

Makefile 25.30% JavaScript 0.24% TypeScript 50.72% Python 2.91% Shell 14.91% Go 5.22% Dockerfile 0.69%

ci-mgmt's Introduction

Pulumi CI Management

Purpose

This repository contains code to manage CI/CD for the many Pulumi providers in a consistent and (mostly) automated manner. The repo's intended audience are Pulumi Corp engineers, but its contents may also serve as a helpful example for Pulumi community members looking to maintain their own providers with a similar CI/CD process to Pulumi Corp.

Pulumi providers use GitHub Actions for CI/CD. Because we maintain a long list of providers, we use this repository to:

Generate GitHub Actions Workflow files for any provider. These can be deployed to all providers or a single provider respectively by the GitHub Actions workflows in this repository.
Keep an inventory of existing Pulumi providers.
Maintain logic for branch protection across provider repositories.

Usage

This repository has the following components:

The provider-ci and native-provider-ci directories contain code to generate GitHub Actions workflow files for Pulumi providers, as well as the generated output for each provider (retained for the purpose of convenient output diffing).
The infra/providers/ directory contains a Pulumi program which uses the Pulumi GitHub provider to ensure consistent branch protections across our provider repositories.

For an overview of how Pulumi programs work, see the Pulumi docs.
GitHub Actions workflows to automate common operations across all providers or a single provider.

Prerequisites

The following tools are required for generating and deploying GitHub Actions workflows:

Building

After checking out the code, run the following command:

cd provider-ci && make clean && make -j

Common commands:

make: Generate all code and check the output.
make provider NAME=aws: Generate code for single provider with debug information
make lint-providers: Check the generated code for all providers.
make lint-providers/aws/repo: Check the generated code for a specific provider.

Adding generated workflows to your local provider repository

Sometimes, you want those changes NOW rather than having to wait for a GitHub PR. This example command will generate workflows for pulumi-datadog, and place them in the specified --out directory. Adjust for your provider and filesystem.

./bin/provider-ci generate --name pulumi/pulumi-datadog --template bridged-provider --config ./providers/datadog/config.yaml --out ../../pulumi-dtadog

Adding a New Bridged Provider

To add a new provider:

A new provider needs a top-level .ci-mgmt.yaml in its own repository with the following basic configuration:

# Required values:
provider: foo # substitute the name of your provider, without the pulumi- prefix
env: # A map of required configuration for any integration tests, etc.
  AN_OPTION: value
  ANOTHER_OPTION: true
  # etc.
lint: true # Linting should be true in most cases, unless failing rules in the upstream provider makes this impractical.

# Optional values:
docker: true # Whether the provider's tests use Docker to run. If set to true, a file `testing/docker-compose.yml` must be present in the provider repository.
setup-script: testing/setup.sh # Path to a script that's used for testing bootstraps

ci-mgmt will read your provider's .ci-mgmt.yaml and generate the standard set of CI files from templates.

Add your provider to provider-ci/providers.json in alphabetical order. This ensures your provider receives regular updates and maintenance.
Commit the changes to provider-ci/providers.json and open a pull request.
To receive a pull request with your new config files, you can run the Update Workflows, Single Bridged Provider workflow run, using your provider name as the input. Another option is to wait for the nightly cronjob to send this pull request automatically.
If you would like to manually generate the configuration to get started right away, you can do so in your provider repository root:
```
go run github.com/pulumi/ci-mgmt/provider-ci@master generate \
 --name pulumi/pulumi-$(PROVIDER_NAME) \
 --out . \
 --template bridged-provider \
 --config .ci-mgmt.yaml
```
The generated files will be writen to your current directory.

Updating All Bridged Providers

The Update GH Workflows, ecosystem providers Workflow runs on a nightly schedule. You may trigger this Workflow manually; however be aware that this causes a lot of GitHub Actions to run at the same time, which may cause rate limiting across the org. Plan ahead and do this at a low-traffic time.

Updating GitHub workflow schema

Fetch the latest JSON Schema then re-generate type definitions:

make discovery

Automatically editing source code across provider repositories

You can apply ad-hoc source edits across provider repositories even on files that are not managed or generated by ci-mgmt. For example, you might need to automatically update example code to use a newer Pulumi SDK major version dependency or a newer version of the underlying infrastructure such as .NET Framework. This can be done with sourcemigrator:

describe your desired edits as a SourceMigration in tools/sourcemigrator

test your changes locally:

cd tools/sourcemigrator
npx ts-node index.ts ~/code/my-provider

stand up a PR to ci-mgmt
trigger an action such as update-workflows-bridged-providers.yml from the PR; this will create PRs that synchronize the selected repositories with ci-mgmt and apply the source migration as part of the change
merge the PR to ci-mgmt
merge the PR to the desired provider repositories (not needed if these PRs are set to automerge)

ci-mgmt's People

Contributors

Stargazers

Watchers

Forkers

kevinschoonover isabella232 friel-test kmosher-alt-test

ci-mgmt's Issues

create a deployment script

the pulumi project for managing these providers is not fit for purpose. We should create a deployment script that manages this for us

Deploy update-upstream-provider workflow to all non-forked providers

Template and deploy the workflow in https://github.com/pulumi/pulumi-yandex/blob/master/.github/workflows/update-upstream-provider.yml to all non-forked providers.

Depends on https://github.com/pulumi/platform-integrations-team/issues/7

Implement GHA Artifact Cleanup

https://github.com/c-hive/gha-remove-artifacts

Allow update-upstream-provider workflow to handle providers with shims

If a provider has a directory called provider/shim, the update-upstream-provider workflow should:

Update the reference to the upstream provider in provider/shim/go.mod
cd provider/shim && go mod tidy
cd ../ && go mod tidy

This will become increasingly important as upstream TF providers adapt to current best practices for TF providers.

(build) Add organization secrets to pulumi org

Instead of copying all the config values to each provider repo, we need to add organization secrets

Remove custom action from Docker image builds?

The custom GH Action we use to build our Docker images was created at a time when the official Docker GH actions were not adequate for our needs.

Investigate whether we can move away from maintaining a custom GH Action in favor of one of the following, in order of preference:

Official Docker actions
Straight run (shell) commands.

add moolumi org

We're currently using the pulumi org in the SaaS for tests, we need to use the moolumi org

actions generator should also generate goreleaser & golangci-lint config

We currently generate the github actions configuration programmatically, but we could also generate the goreleaser config and golangci-lint config because it's also a YAML configuration file

(build) fail the build if the schema changes in CI

During the initial prerequisites step of our build, we need to ensure there's no changes to the schema and fail so we don't generate an artifact that isn't valid

(build) add a comment to the top of generated files

PoC: Automate updating forks of Terraform providers

Create a PoC to automate the updating of Terraform provider forks. The process is documented in https://github.com/pulumi/platform-providers-team/blob/main/playbooks/tf-provider-updating-forks.md

migrate providers to run GHA test

(build) publish scripts assume travis

There are various parts of the publish script that use travis environment variables which we need to remove

(build) make sure version calculated is a valid semver before building

We have had several builds fail/error in our existing build process because they didn't generate a valid semver tag (for example, v.3.3.3 is not valid)

If we're performing a build, we should verify that the version is valid semver, and fail if it isnt.

/v2 upstream references do not work (pulumi-artifactory)

update-upstream-provider assumes that there's no /v2 suffix to a reference when it's updated. We need to add support for this, as terraform-provider-artifactory is one of the few providers that do add the suffix.

Fixup LDFlags - most are being passed without a version

We should maybe make this configurable in the config.yaml file of each provider?

@jaxxstorm

Update bridge workflow: Programmatically determine the SDK dependency

A provider can't support an SDK dependency that is a more recent version than the version referenced in pulumi-terraform-bridge. Therefore, we should look up the SDK version programmatically instead of allowing it as an arbitrary parameter.

Strip binaries for plugins

Our current binaries can get quite large, especially for some of the larger provider plugins:

du -sh /Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/
329M	/Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/

If we add the -w and -s LDFLAGS, we get:

du -sh /Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/
287M	/Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/

If we strip the binary further using upx

du -sh /Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/
 56M	/Users/lbriggs/src/git/pulumi/pulumi-aws/dist/pulumi-aws_darwin_amd64/

We should make use of this if possible

(build) Add parallel github action build

Each provider needs to have the github actions build configuration added to it

note this issue is to track COMPLETED migrations, including publishing

(build) add failure notifications to slack channel

fill in

Improve error messaging when upstream revision cannot be found

In update-upstream-provider, we should provide a better error message and immediately fail the build if we cannot find the supplied upstream revision, rather than failing go mod edit .....null.

This can be as simple as writing a message to stderr and executing false, since the step is of type run (i.e. a Bash script).

Refactor Native Providers to be composable builds

Workflow dispatch for upstream providers and bridge updates should take a ref, not a version number

We currently expect a semver to be passed to the update-upstream-provider and update-bridge workflows - the workflow itself will tack on the "v" prefix that go module tags expect. This constraint is unnecessarily rigid in that go mod edit will accept any valid git ref - it does not need to be, e.g. v1.2.3.

Accepting a ref instead of a semver will enable us to do a few things:

Semi-automate provider upgrades for forked providers, e.g. because we can pass upstream-v1.2.3 as an argument.
Update the bridge to an arbitrary tag for a provider, e.g. to test some beta functionality.

Migrate remaining GHA back to this org to be centrally managed

This includes:

and start to build common workflows between these systems

(experiment) consider removing Makefiles

One of the more frustrating elements of the build process is dealing with the Makefile syntax we're currently using:

It's difficult to generate Makefiles programmatically
The syntax doesn't lend itself well to multi stage builds
It's quickly becoming complex

Ideally, we could try and replace this with a task runner which will allow us to generate these files programmatically with jk.

I've had some success before with https://github.com/go-task/task which uses a YAML config file, so we should try it out

PR comment is null

In the following PR, we triggered a run of the integration tests, but the link back to the actions run was apparently null: pulumi/pulumi-aws#1792

Looks like the issue is that the step is called var, but is referenced as vars:

    - id: var
      name: Create URL to the run output
      run: echo ::set-output name=run-url::https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID
    - name: Update with Result
      uses: peter-evans/create-or-update-comment@v1
      with:
        body: Please view the PR build - ${{ steps.vars.outputs.run-url }}

Slack notifications do not fire for release builds

The release build is missing the Slack notification step, which means we won't know if a release fails. We should add it.

Use tar.gz for provider binaries

As part of the pre-requisite steps, we should tar.gz the binaries to be faster and more efficient for storage

Assign platform integrations team as reviewer on PRs generated by update-all-providers

Add a nightly cron for the providers

This cron should do everything that the default run does EXCEPT the publish step

Add a workflow to update a single provider

We only have the ability to deploy updates in workflows to all providers. We also would find it useful to have the ability to deploy a single provider.

Roll out GO_DEP_ROOT envvar, and fix go test dependency paths

This PR added support for a new env var to fix our go integration testing setup.

We need to roll this env var out to all of the providers and do a sweep of the go integration tests as some of them have dependencies that need to be updated. They should be in the form:

github.com/pulumi/pulumi-REPO/vN

See an example of a typo here:

https://github.com/pulumi/pulumi-aws/blob/master/examples/examples_go_test.go#L19

ci changes should automerge

Add notification if we fail to create a PR for a provider upgrade

A failed update-upstream-provider workflow attempt (e.g. because there are new resources) currently alerts to #builds in Slack. The failed attempt is not actually an alert-type occurrence because while it requires human intervention to add the mappings, it's not urgent.

The failed attempt should instead add a comment to the linked issue (if provided) indicating the automation attempt failed.

~~In addition, will need to change the code here to ignore the workflow in the #builds channel.~~ <-- @stack72 took care of this

Python tests in GHA failing

Python tests running in GHA are failing in pulumi/examples, pulumi/pulumi-azure, etc.

The integration tests run pipenv run pip install -r requirements.txt to install dependencies in the virtual environment created by Pipenv. That's how it's supposed to work, but it looks like in the GHA runner it's installing the dependencies globally, so when pipenv pulumi up is run, it can't find any of the dependencies because they weren't installed in Pipenv's virtual environment (they were installed globally, which isn't visible to the python in the virtual environment):

 Diagnostics:
   pulumi:pulumi:Stack (aws-py-s3-folder-p-it-fv-az60-aws-py-s3--92dffaeb):
     Traceback (most recent call last):
       File "/home/runner/.pulumi/bin/pulumi-language-python-exec", line 14, in <module>
         import pulumi
     ModuleNotFoundError: No module named 'pulumi'
     It looks like the Pulumi SDK has not been installed. Have you run pip install?
     If you are running in a virtualenv, you must run pip install -r requirements.txt from inside the virtualenv.

     error: an unhandled error occurred: Program exited with non-zero exit code: 1

provider build migration phase 2

phase 2 of the provider build migration happens after #7 but before #2

add pypi publish key to org secrets
add nodejs publish key to org secrets
add nuget publish key to org secrets
add SDK publish task to each release action
disable publishing in travis config

Add extra/missing mapping configuration to providers

An extra mapping should fail the build for all providers.
A missing mapping should fail the build for all providers except yandex and alicloud.

Insert a comment in generated workflow files indicating they are generated

We don't have an indicator in the workflow files that they are automatically generated and will be overwritten when this repo is pushed. We should insert a into each generated comment like:

# This file is automatically generated.  Any content added to this file risks being overwritten.
# To make changes to the content of the file, see: https://github.com/pulumi/ci-mgmt/

Because our current templating tool jk doesn't support comments, we'd likely have to use bash to implement, probably using one of the answers here: https://superuser.com/questions/246837/how-do-i-add-text-to-the-beginning-of-a-file-in-bash

Improve README

Spruce up the README:

Ensure MD lint passes.
Add additional instructional content, explanation.

Ensure that latest build + deploy scripts are available in all providers

Ensure that the major-version of each provider is reflected in this repo (@stack72 TO-DO)
Swap out the use of MP to be a GHA
- We build a dynamic matrix (https://tomasvotruba.com/blog/2020/11/16/how-to-make-dynamic-matrix-in-github-actions/) to fuel out GHA
- generate scripts for ALL providers
- foreach provider
  - checkout provider repo
  - copy the files across
  - open a PR

Support default branches not named "master"

update-upstream-provider assumes master for the default branch, which breaks on pulumi-artifactory, which has a default branch named main. We need to allow for both cases easily.

Add ability to trigger the update-bridge workflow for all providers

Ensure that the update-bridge workflow is complete and accurate. (make tfgen build_sdks, go mod edit command is correct)
Extend the workflow to accept a Pulumi version which will update the Pulumi SDK dependency in both the provider/go.mod and sdk/go.mod.
~~Ensure the workflow works for shim-ed providers.~~ <-- n/a - the bridge is not referenced in the shim
Create a workflow_dispatch-triggered workflow to which invokes the update-bridge workflow for all bridged providers, similar to the work we did that submits a PR for all providers for changes to workflows defined in this repo.
Ensure that the platform integrations team is added as a reviewer for any generated PRs