stairs:
  - name: foo
    scope: project
    buildkite:
      branches: "master develop feature"
      command: make foo

Above will render the branches part as:

steps:
  - branches: master develop feature  # should be wrapped in quotes
    command: make foo

Inspiration: https://github.com/BastiPaeltz/gitlab-ci-monorepo

Would be interesting to generalize the repo to other CIs that support dynamic pipelines. There's a way to do it in Jenkins using a Jenkinsfile.

Some resources that might be useful:

• https://jenkins.io/doc/book/pipeline/
• https://stackoverflow.com/questions/45711473/how-to-load-dynamic-jenkinsfile-during-a-build
• https://jenkins.io/blog/2017/09/25/declarative-1/
• https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md
• https://support.cloudbees.com/hc/en-us/articles/226122247-How-to-Customize-Checkout-for-Pipeline-Multibranch-

For projects, it's possible to blacklist steps by putting them into the skip: block.
It would be great to have the same option with whitelisting, for example with an include: block.

Using the details from README I tried to use the plugin v0.9.2. This doesn't work as this command fails because v0.9.2 is only tagged and not yet released.

The binary asset being downloaded by the plugin does not exist and gives 404. 😅

https://github.com/jwplayer/buildpipe-buildkite-plugin/releases/download/v0.9.2/buildpipe-linux

It would be great to not only have the project path (BUILDPIPE_PROJECT_PATH) but also the project name/label available as environment variables in pipeline steps.

a blank env key gets appended to the group keys, causing the YAML to be invalid

Here is our output generated by the plugin:

Which generates the following error:

2022-09-22 19:56:58 WARN   POST https://agent.buildkite.com/v3/jobs/018366c6-14b4-4e2e-95e5-08817c16d4ee/pipelines: 422 `env` is not a valid property on a step group. Please check the documentation to get a full list of supported properties. (Attempt 1/60 Retrying in 5s)
--
  | 2022-09-22 19:56:58 ERROR  Unrecoverable error, skipping retries
  | 2022-09-22 19:56:58 FATAL  Failed to upload and process pipeline: POST https://agent.buildkite.com/v3/jobs/018366c6-14b4-4e2e-95e5-08817c16d4ee/pipelines: 422 `env` is not a valid property on a step group. Please check the documentation to get a full list of supported properties.
  | 🚨 Error: The command exited with status 1

buildpipe does not have the --version flag in place.
Since users call buildpipe directly from the command line I think this flag should be supported.

Dependencies currently do no work when multiple projects are affected. This is due to Buildpipe no making the key unique causing a collision.

https://github.com/23andMe/Yamale

Say I have two projects:

toy-project
toy-project-api

I would like to have one deploy without having both deploy.

When only documentation is changed (E.g. */README.md) don't trigger project to build

It would be nice to rewrite to Go so that we can ship a binary instead of having to install with pip

• Remove stairs deploy option
• deploy in spec (maybe rename) should apply to tags

Unless I'm missing something (and that's really possible), it's impossible to trigger a build without a code change that flags which paths/labels to build. (There was a pull request - #13 - when this project was still in Python to that regard, and it looks like it was merged, so this may be a regression.)

Use case: Our Ansible roles are in a monorepo. We would like to trigger a Packer run for two of those roles weekly so that we pick up changes to packages upstream from us. To do that, we'd need to be able to add to the project paths that will be built by getAffectedProjects.

This feature is similar to #74 .

Hello,

I'm currently working out an issue where BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS="*" is ignored in the environmental variables section of the Buildkite UI's "manual" build feature and when injected into a trigger from a different pipeline. In both cases only the last changed project is built. This behavior does not change when manually specifying comma separated project names vs. the wildcard. I did confirm the variable is being passed into the pipeline build. Please let me know what relevant debug or logging information I can provide.

Thanks,

-- Jim

Hi there 👋

First of all, thank you for this plugin, I have been looking for something similar for some time and it seems I just found it!

There is an issue I'd like to check with you is that I'm currently unable to use Buildkite trigger as it errors out due to the following:

`env` is not a valid property on the `trigger` step. Please check the documentation to get a full list of supported properties.

This seems similar to the issue #39 but what it seems to be the case with me is that even though I don't specify global any env, it tries to add to buildpipe generated yaml anyway.

Here's an example:

Given the dynamic pipeline below:

env:
  FOO: bar

projects:
  - label: "my-project"
    path:
      - internal/foo
    key: deploy
    trigger: "another-pipeline"
    build:
      env:
        BUILDPIPE_SCOPE: project

The result I get is the following:

  | [2022-12-20T04:44:49Z]   build:
  | [2022-12-20T04:44:49Z]     env:
  | [2022-12-20T04:44:49Z]       BUILDPIPE_SCOPE: project
  | [2022-12-20T04:44:49Z]   env:
  | [2022-12-20T04:44:49Z]     FOO: bar
  | [2022-12-20T04:44:49Z]   key: deploy
  | [2022-12-20T04:44:49Z]   label: 'my-project'
  | [2022-12-20T04:44:49Z]   trigger: another-pipeline

As you can see above, buildpipe is adding the key env to the same level as build, which is not supported.

Even if I remove the global environment variables, I get the same error as the env is still being added:

# no global env vars this time
projects:
  - label: "my-project"
    path:
      - internal/foo
    key: deploy
    trigger: "another-pipeline"
    build:
      env:
        BUILDPIPE_SCOPE: project

Output:

  | [2022-12-20T04:44:49Z]   build:
  | [2022-12-20T04:44:49Z]     env:
  | [2022-12-20T04:44:49Z]       BUILDPIPE_SCOPE: project
  | [2022-12-20T04:44:49Z]   env: {}
  | [2022-12-20T04:44:49Z]   key: deploy
  | [2022-12-20T04:44:49Z]   label: 'my-project'
  | [2022-12-20T04:44:49Z]   trigger: another-pipeline

As you can see above, the key env is being generated but it is empty (i.e. env: {}).

Let me know if this seems to be a legit issue or if I'm using the plugin wrong. Nevertheless, thank you for your work!

Cheers 🙏

Trying to nest plugins causes a representation error:

    buildkite:
      command:
        - docker-compose -f docker-compose-ci.yml build
      plugins:
        - docker-login#v2.0.1:
          - server: containers.amphoratech.net
            username: myUser
            password-env: MY_PASS

This will throw

Traceback (most recent call last):
--
  | File "/usr/bin/buildpipe", line 10, in <module>
  | sys.exit(main())
  | File "/usr/lib/python3.6/site-packages/buildpipe/__main__.py", line 17, in main
  | pipeline.create_pipeline(**vars(args))
  | File "/usr/lib/python3.6/site-packages/buildpipe/pipeline.py", line 206, in create_pipeline
  | steps.to_yaml(filename=outfile, Dumper=yaml.dumper.SafeDumper)
  | File "/usr/bin/box.py", line 733, in to_yaml
  | encoding=encoding, errors=errors, **yaml_kwargs)
  | File "/usr/bin/box.py", line 106, in _to_yaml
  | **yaml_kwargs)
  | File "/usr/lib/python3.6/site-packages/yaml/__init__.py", line 290, in dump
  | return dump_all([data], stream, Dumper=Dumper, **kwds)
  | File "/usr/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all
  | dumper.represent(data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent
  | node = self.represent_data(data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data
  | node = self.yaml_representers[data_types[0]](self, data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict
  | return self.represent_mapping('tag:yaml.org,2002:map', data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping
  | node_value = self.represent_data(item_value)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data
  | node = self.yaml_representers[data_types[0]](self, data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list
  | return self.represent_sequence('tag:yaml.org,2002:seq', data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence
  | node_item = self.represent_data(item)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data
  | node = self.yaml_representers[data_types[0]](self, data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict
  | return self.represent_mapping('tag:yaml.org,2002:map', data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping
  | node_value = self.represent_data(item_value)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data
  | node = self.yaml_representers[data_types[0]](self, data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list
  | return self.represent_sequence('tag:yaml.org,2002:seq', data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence
  | node_item = self.represent_data(item)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data
  | node = self.yaml_representers[None](self, data)
  | File "/usr/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined
  | raise RepresenterError("cannot represent an object", data)
  | yaml.representer.RepresenterError: ('cannot represent an object', <Box: {'docker-login#v2.0.1': None, 'server': 'containers.amphoratech.net', 'username': 'myUser', 'password-env': 'MY_PASS'}>)

Probably related to this: https://stackoverflow.com/questions/18374276/pyaml-throwing-a-representation-error

Tag rules should work similar to how Buildkite does it

This is not really an issue but more of a request for help.

I am not sure if this can be easily done with this repo, but I want to detect changes for a project only if the latest commit had changes in the respective project's files.

Is this somehow possible with buildpipe?

I only found this block where the git log command actually does what I need, but only on the deploy branch or master.

if branch == deploy_branch:
    command = ['git', 'log', '-m', '-1', '--name-only', '--pretty=format:', commit]
else:
    command = ['git', 'whatchanged', '--name-only', '--pretty=format:', 'origin..HEAD']

Hey there 👋

This looks like an awesome project but I was hoping to get some clarification.

You mention "Especially useful for monorepos where you want to create dependencies between projects." Is there an example that shows how to define dependencies between projects? I can't find docs around this anywhere.

If it helps, here's the use case I have in mind. I'm looking to do Packer builds, where if I change the base image, I want to trigger all subsequent builds:

Ex:

└── packer
    ├── golang
    ├── node
    ├── python
    ├── ruby
    └── ubuntu

I would like the language-specific images to have a dependency on the ubuntu build, so if I update the base image, all the others will be updated.

Similarly, I might want to do something like run tests on all projects in the monorepo if a shared library changes.

Hi, I couldn't figure out a way to define dependencies between steps within one project. The Buildkite native way would be to use a key: foo and a depends_on: foo property on two separate steps. But when I do that using Buildpipe and have more than one project, the key: foo would be used over and over again and I get the error

The key "foo" has already been used by another step in this build 

I've tried key: foo-${BUILDPIPE_PROJECT_LABEL} but that didn't work either.

How can I use keys on steps when using Buildpipe?

Hi folks.

We recently swapped our repo over to use GitHub's rebase strategy for PR merges and noted that the default diff command for buildpipe doesn't work anymore (as noted in the docs).

Before re-inventing the wheel - wondering if the Buildpipe community has solved for a diff command for this scenario yet and whether we might document different suggested diff commands for the different strategies.

First thought is to rely on git tags for the diffing the default branch HEAD to a prior build but the risk of drift is a little concerning.

Anyone encountered this and come up with an elegant solution?

Thanks

P.S I'd have preferred to submit this as a discussion rather than an issue - happy to be redirected to a different forum. Thanks :)

We need better handling of pipeline upload errors. Uploading an incorrectly formatted pipeline will just result in an unhelpful error that's hard to debug:

FATA[0000] Error running command: exit status 1

It would be very cool to have an option to define environment variables on a global level and project level in the pipeline file.
Buildkite had that option in version 2 to have global env vars but they removed it in v3 for some reason.

I have something in mind that could look like this:

projects:
  - label: project-xyz
    path: path/to/my/project
    env:
      THIS_IS_A_PROJECT_SCOPED_ENV_VAR: def456

env:
  - THIS_IS_A_PIPELINE_GLOBAL_ENV_VAR: abc123

steps:
  - label: ":docker: build - "
    command: echo "this is a build step"
    env:
      THIS_IS_A_STEP_SCOPED_ENV_VAR: xyz999

Buildpipe could already take care of the double dollar signs needed for some of those variables depending on where they are defined in the pipeline.

I want to use buildpipe to detect changes and to trigger different buildkite pipelines through buildkite's built-in trigger step

But the syntax that buildpipe produces seems to be wrong for this step.
The trigger step does not have the env attribute on its top level but only under the build attribute.

Step produced by buildpipe on my pipeline:

- async: true
  build:
    branch: $BUILDKITE_BRANCH
    commit: $BUILDKITE_COMMIT
    message: $BUILDKITE_MESSAGE
  env:
    BUILDPIPE_STAIR_NAME: trigger-<mypipeline>
    BUILDPIPE_STAIR_SCOPE: stair
  label: <mylabel>
  trigger: <slug of a different pipeline>

How it should look like:

- async: true
  build:
    branch: $BUILDKITE_BRANCH
    commit: $BUILDKITE_COMMIT
    message: $BUILDKITE_MESSAGE
    env:
      BUILDPIPE_STAIR_NAME: trigger-<mypipeline>
      BUILDPIPE_STAIR_SCOPE: stair
  label: <mylabel>
  trigger: <slug of a different pipeline>