vital-software / monofo-buildkite-plugin Goto Github PK
View Code? Open in Web Editor NEWBuildkite dynamic pipeline generator for mono-repos
License: GNU General Public License v3.0
Buildkite dynamic pipeline generator for mono-repos
License: GNU General Public License v3.0
monorepo:
match: true
should give Warning: unknown option 'match' in monorepo config
(it should be matches
not match
)
Example setup:
# pipeline.a.yml
steps:
- key: a
command: # does something with side effects
# pipeline.b.yml
monorepo:
produces: b.tar.gz
steps:
- key: b
depends_on: a
command: # creates b.tar.gz
# pipeline.c.yml
monorepo:
expects: b.tar.gz
steps:
- key: c
depends_on: b
command: # uses b.tar.gz, and relies on side-effects of `a` happening first
If b has no matches and is skipped, currently we:
c
), for the artifact step (so that b.tar.gz
is guaranteed to exist before step c
runs).This is fine logic, however depending on the nature of the different steps, some builds may have ordering concerns between a
and c
. Such builds might be relying on the transitive nature of depends_on
(as they should be able to) so that their pipeline is simple. In our specific example, a = node_modules, b = typescript, and c = some tests.
Just running c
after the replacement of b
isn't enough. Instead, it has to run after all of b
's depends_on
s, if they're still in the build.
Result: if b is skipped, c's depends_on should be [artifact step + (depends on of b) + (depends on of c)].filter(still in build), for a result of: [artifact step, a]
When using MONOFO_INTEGRATION_BRANCH, if the diff strategy fails, the fallback is used. This implies if you want to e.g. have an integration branch called 'dev', you still need to set MONOFO_DEFAULT_BRANCH=dev
. But the ordering here prevents the integration diff from ever triggering in that case:
I think the solution is to reverse this ordering, so that we can set both MONOFO_INTEGRATION_BRANCH
and MONOFO_DEFAULT_BRANCH
, and have the integration diff happen first (and the default branch diff only happen if that fails)
https://developer.github.com/v3/repos/statuses/
maybe configured like
monorepo:
excluded:
statuses: ['foo/bar']
Because there's no easy way to map a build ID back to e.g. a build URL to use
Summary: the integration branch feature MONOFO_DEFAULT_BRANCH
, as implied by the name, overwrites the default branch (i.e. in the environment) so that we can treat dev
as an integration branch. However, the diffing logic is suboptimal in cases where an integration branch is reset to the default branch regularly, because monofo does no assume any relationship between an integration branch and the default branch.
dev
on Buildkite, none of them match up to the ancestors of the current commit (because they all, as far back as the start of the week, happened on a parallel branch of development that is now thrown away by the hard reset)For simplicity, we want to take on the concept of an integration branch as distinct from the default branch, deprecate MONOFO_DEFAULT_BRANCH, add MONOFO_INTEGRATION_BRANCH, and provide better semantics. Those semantics are:
We still use the integration branch when looking up base build commits in Git, and we prefer builds that happened on the branch when matching builds on the Buildkite API, but we also consider default-branch commits that are ancestors of the integration branch.
We can use
Example: ?branch[]=master&branch[]=testing returns all builds on master and testing branches
to do so, and retain ordering
If a pipeline.yml file contains no contents at all (in this case, it only contained a lengthy comment), then the following error is given and monofo crashes!
TypeError: Cannot destructure property 'monorepo' of 'js_yaml_1.load(...)' as it is null.
at Function.read (/var/lib/buildkite-agent/.npm/_npx/7612/lib/node_modules/monofo/build/src/config.js:167:17)
at async Promise.all (index 4)
at async Function.readAll (/var/lib/buildkite-agent/.npm/_npx/7612/lib/node_modules/monofo/build/src/config.js:229:17)
at async Function.getAll (/var/lib/buildkite-agent/.npm/_npx/7612/lib/node_modules/monofo/build/src/config.js:237:25)
https://buildkite.com/docs/pipelines/group-step
Group steps should be supported
Initial support only requires that we can analyze the depends_on which now might be within the group steps themselves - the ones on the group should be ok (or maybe we need to merge them down onto the steps) - this is bug level, because we should support all Buildkite steps
Later support for grouping (they have to be grouped together in the upload dynamically, not the same group given multiple times) would be a nice to have
Somewhat similar to #195, for task-like pipelines, it's nice to be able to opt out of the self-matching, otherwise modifying the tasks always causes them to run.
Make a generator mode where we put an input step first, asking which subpipelines to run
pure: true
and matches: true
matches: **
but don't show huge list of file matchesSimilar to #338
The list of files given to tar must contain (empty, inferred) intermediate directories - this is necessary for it to be processed into a catar
See https://github.com/dominics/desync-tar-requirements#example-output for a larger run-down of the requirements
main
branch failed. π¨I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.
You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. Iβm sure you can resolve this πͺ.
Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.
Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the main
branch. You can also manually restart the failed CI job that runs semantic-release.
If you are not sure how to resolve this, here is some links that can help you:
If those donβt help, or if this issue is reporting something you think isnβt right, you can always ask the humans behind semantic-release.
An npm token must be created and set in the NPM_TOKEN
environment variable on your CI environment.
Please make sure to create an npm token and to set it in the NPM_TOKEN
environment variable on your CI environment. The token must allow to publish to the registry https://registry.npmjs.org/
.
Good luck with your project β¨
Your semantic-release bot π¦π
The inject-artifact step can become quite unwieldy with large, or large numbers of, artifacts. It's necessary because of a few reasons:
So, for these reasons, monofo tries to ensure that every build contains every artifact (that's used by another pipeline), so that if a pipeline is skipped, that artifact can be injected and the cached version used.
But inject-artifacts is not a good mechanism. Ideally, it should be unnecessary, if we can work around the above reasons. This requires a few things:
The hardest bit:
BUILDKITE_PLUGIN_ARTIFACTS_BUILD
that tell it to use a specific build IDpre-command
hook that does downloading of artifacts in parallel)
MONOFO_<COMPONENT>_BASE_BUILD_ID
?)π That should be it: no more inject-artifacts step except for things that really need to be in the Buildkite artifacts system for the current build
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Warning
These dependencies are deprecated:
Datasource | Name | Replacement PR? |
---|---|---|
npm | @google/semantic-release-replace-plugin |
|
npm | @types/log-update |
|
npm | @types/mkdirp |
|
npm | @types/rimraf |
|
npm | @types/tempy |
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
split2
, @types/split2
)tiny-async-pool
, @types/tiny-async-pool
)@oclif/core
, @oclif/plugin-autocomplete
, @oclif/plugin-commands
, @oclif/plugin-help
, @oclif/plugin-not-found
, @oclif/plugin-version
, oclif
)@types/jest
, jest
, jest-dynalite
, nock
, ts-jest
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
, eslint
, eslint-config-airbnb-typescript
, eslint-config-prettier
, eslint-plugin-import
, eslint-plugin-jest
, eslint-plugin-prettier
, prettier
)@commitlint/cli
, @commitlint/config-conventional
)@types/jest
, jest
, ts-jest
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
, eslint
, eslint-config-airbnb-typescript
, eslint-config-prettier
, eslint-plugin-jest
, eslint-plugin-prettier
, prettier
)These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.
@aws-sdk/client-dynamodb
, @aws-sdk/credential-provider-node
, @aws-sdk/lib-dynamodb
, @aws-sdk/types
)@oclif/core
, @oclif/plugin-autocomplete
, @oclif/plugin-commands
, @oclif/plugin-help
, @oclif/plugin-not-found
, @oclif/plugin-version
, oclif
)glob
, @types/glob
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
.buildkite/buildkite/pipeline.yml
docker-compose v3.9.0
improbable-eng/metahook v0.4.1
artifacts v1.5.0
improbable-eng/metahook v0.4.1
artifacts v1.5.0
docker-compose v3.9.0
docker-compose v3.9.0
improbable-eng/metahook v0.4.1
seek-oss/aws-sm v2.3.1
.buildkite/monofo/pipeline.node-modules.yml
docker-compose v3.9.0
vital-software/monofo v5.0.11
.buildkite/monofo/pipeline.plugin-lint.yml
docker-compose v3.9.0
.buildkite/monofo/pipeline.plugin-test.yml
docker-compose v3.9.0
.buildkite/monofo/pipeline.release.yml
improbable-eng/metahook v0.4.1
vital-software/monofo v5.0.11
docker-compose v3.9.0
seek-oss/aws-sm v2.3.1
.buildkite/monofo/pipeline.test.yml
docker-compose v3.9.0
vital-software/monofo v5.0.11
vital-software/monofo v5.0.11
.buildkite/monofo/pipeline.typescript.yml
docker-compose v3.9.0
vital-software/monofo v5.0.11
test/projects/kitchen-sink/.buildkite/pipeline.bar.yml
artifacts v1.3.0
artifacts v1.3.0
test/projects/kitchen-sink/.buildkite/pipeline.baz.yml
artifacts v1.3.0
test/projects/kitchen-sink/.buildkite/pipeline.foo.yml
artifacts v1.3.0
test/projects/kitchen-sink/.buildkite/pipeline.qux.yml
artifacts v1.3.0
test/projects/skipped/.buildkite/pipeline.bar.yml
artifacts v1.3.0
artifacts v1.3.0
test/projects/skipped/.buildkite/pipeline.foo.yml
artifacts v1.3.0
docker-compose.buildkite.yml
docker-compose.local.yml
docker-compose.yml
Dockerfile
package.json
@aws-sdk/client-dynamodb 3.80.0
@aws-sdk/credential-provider-node 3.80.0
@aws-sdk/lib-dynamodb 3.80.0
@aws-sdk/types 3.78.0
@oclif/core 1.7.0
@oclif/plugin-autocomplete 1.2.0
@oclif/plugin-help 5.1.12
@oclif/plugin-not-found 2.3.1
@oclif/plugin-commands 2.1.0
@oclif/plugin-version 1.0.4
bluebird 3.7.2
chalk 4.1.2
command-exists 1.2.9
compare-versions 4.1.3
debug 4.3.4
execa 5.1.1
glob 8.0.1
got 11.8.3
js-yaml 4.1.0
lodash 4.17.21
log-update 4.0.0
minimatch 5.0.1
mkdirp 1.0.4
pretty-bytes 5.6.0
rimraf 3.0.2
split2 4.1.0
tempy 1.0.1
tiny-async-pool 1.3.0
toposort 2.0.2
@commitlint/cli 16.2.4
@commitlint/config-conventional 16.2.4
@google/semantic-release-replace-plugin 1.1.0
@semantic-release/changelog 6.0.1
@semantic-release/git 10.0.1
@tsconfig/node14 1.0.1
@types/bluebird 3.5.36
@types/command-exists 1.2.0
@types/debug 4.1.7
@types/glob 7.2.0
@types/jest 27.5.0
@types/js-yaml 4.0.5
@types/lodash 4.14.182
@types/log-update 3.1.0
@types/minimatch 3.0.5
@types/mkdirp 1.0.2
@types/rimraf 3.0.2
@types/split2 3.2.1
@types/tempy 0.3.0
@types/tiny-async-pool 1.0.1
@types/toposort 2.0.3
@typescript-eslint/eslint-plugin 5.22.0
@typescript-eslint/parser 5.22.0
cz-conventional-changelog 3.3.0
eslint 8.14.0
eslint-config-airbnb-typescript 17.0.0
eslint-config-prettier 8.5.0
eslint-plugin-import 2.26.0
eslint-plugin-jest 26.1.5
eslint-plugin-prettier 4.0.0
husky 7.0.4
jest 28.0.3
jest-dynalite 3.5.1
nock 13.2.4
npm-run-all 4.1.5
oclif 3.0.1
pkg 5.6.0
prettier 2.6.2
semantic-release 19.0.2
stdout-stderr 0.1.13
ts-jest 28.0.1
typescript 4.6.4
node >=14.16
.nvmrc
main
branch failed. π¨I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.
You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. Iβm sure you can fix this πͺ.
Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.
Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the main
branch. You can also manually restart the failed CI job that runs semantic-release.
If you are not sure how to resolve this, here are some links that can help you:
If those donβt help, or if this issue is reporting something you think isnβt right, you can always ask the humans behind semantic-release.
The npm token configured in the NPM_TOKEN
environment variable must be a valid token allowing to publish to the registry https://registry.npmjs.org/
.
If you are using Two Factor Authentication for your account, set its level to "Authorization only" in your account settings. semantic-release cannot publish with the default "
Authorization and writes" level.
Please make sure to set the NPM_TOKEN
environment variable in your CI with the exact value of the npm token.
Good luck with your project β¨
Your semantic-release bot π¦π
i.e. act as if monorepo: { matches: ".buildkite/pipeline.foo.yml" }
is present in .buildkite/pipeline.foo.yml
, automatically
_SKIP
Say you have an artifact named example.catar.caibx
and you're trying to do the classic "incremental build" pattern:
monorepo:
pure: true
expects: dependency.catar.caibx
produces: example.catar.caibx
matches:
- "some-path/**"
steps:
- name: "Compile example archive"
depends_on:
- dependency
commands:
- do-the-compile
- produce-example-file-list-to-cache # produces example.list
env:
MONOFO_ARTIFACT_EXAMPLE_BUILD_ID: ${MONOFO_ARTIFACT_EXAMPLE_BUILD_ID:-$MONOFO_BASE_BUILD_ID} # provides the place to get the base artifact for the incremental build from
MONOFO_ARTIFACT_EXAMPLE_SOFT_FAIL: 1 # says: don't worry if I can't get a base artifact for the incremental build
plugins:
- docker-compose#v3.9.0:
run: node
- vital-software/monofo#v4.0.0:
download:
- dependency.catar.caibx
- example.catar.caibx # pre-cache incremental build
upload:
example.catar.caibx:
filesFrom: example.list
null: true
The plan was to have MONOFO_ARTIFACT_EXAMPLE_SKIP=1
cause just the incremental build part of this step to be skipped, so that the produced example was run on a "clean" checkout, rather than checkout + previous result.
MONOFO_ARTIFACT_EXAMPLE_SKIP=1
, any steps in other partial pipelines that want to actually depend on the latest copy of the artifact will also be skippedFor the incremental use-case, we can detect that the same dependency is being uploaded later in the current step. And the skip the artifact iff we are uploading it.
We should use an additional environment variable for this new behavior, to retain BC: suggest MONOFO_ARTIFACT_EXAMPLE_SKIP_PRECACHE=1
The base build commit and Buildkite build ID (if there is one) should be transmitted to the final pipeline as an env var.
As a simple way to support alternate/multiple default branches, add support for a MONOFO_DEFAULT_BRANCH
environment variable that can be used with the same meaning as (but a higher priority than) BUILDKITE_PIPELINE_DEFAULT_BRANCH
Some sub-pipelines might only be triggered in certain circumstances, e.g. when PIPELINE_RUN_ or PIPELINE_RUN_ONLY are given. One use case I've seen for this is 'task' pipelines, run by schedules, where the task wants to reused some of the monofo build, but doesn't want to ever be pulled in by PIPELINE_RUN_ALL
This might be task: true
or runs_on: { all: false }
or similar:
monorepo:
name: some-task
task: true
expects:
- node-modules.tar.lz4
- typescript.tar.lz4
Make sure the npx monofo@latest download
command can also be used to pre-cache artifacts from other pipelines; that would mean we can allow users to precache e.g. node-modules.caidx in their agent bootstrap
Error: Command failed with exit code 1: desync untar --config /tmp/286afb41f536a05b85840a47535e765e/config --verbose --no-same-owner --index --store s3+https://s3-us-west-2.amazonaws.com/vital-buildkite-cache-us-west-2/desync/store --cache /tmp/desync/monofo - .
reading /tmp/286afb41f536a05b85840a47535e765e/config: EOF
However, upon checking (later, after the build failed), that config file was present and filled in
Found 0 changed files
leads to every component not matching, even those with matches: true
When using this plugin in our mono repo with git lfs enabled, it always considers the lfs files to have changed.
The agent which performs the upload doesn't actually perform the git lfs checkout.
When run locally on a git lfs checkout it appears to work correctly (with the lfs checked out or not), so I can only assume it is something about the way buildkite is performing the checkout.
env:
FOO: "bar"
monorepo:
matches: true
This should be treated as if it had a steps: []
entry present
Buildkite's branch pattern matching (https://buildkite.com/docs/pipelines/branch-configuration#pipeline-level-branch-filtering) doesn't seem to respect folder paths (e.g. task/KATOA-1234/branch-name
) while minimatch does. Buildkite's behaviour is also more expected (probably), so it would be good to match it closer
So that you can use
PIPELINE_RUN_ONLY=component-a,component-b
I have a pipeline that's roughly like this:
monorepo:
pure: true
matches:
- "**/package.json"
- yarn.lock
- scripts/yarn-install
produces: node-modules.tar.lz4
steps:
- name: ":nodejs: Install"
key: node-modules
depends_on: node-image
command: scripts/yarn-install
Even though this pipeline doesn't declare any expects
, when we skip the node-image
containing pipeline, we also make this step wait for the inject artifacts step. That must be unnecessary: this pipeline declares no expects
so it can't need to wait for artifacts to be pulled from previous builds (and if it does, it should declare that fact).
Removing this unnecessary depends_on replacement would allow this step to run immediately, rather than waiting for inject-artifacts.
Should be e.g.
./node_modules/
./data-platform/node_modules
./data-platform/checks/node_modules
ends up being just based on name, so:
./data-platform/checks/node_modules
./data-platform/node_modules
./node_modules
which is the exact opposite of what we need!
Maybe GNU tar
's --sort=inode
is doing this internally, but we need a way to do it for our file list
Background context: In repositories where I'm using Monofo, I often have to pull in a "decompress artifact" and "compress artifact" helper, for dealing with large directories with many files (e.g. for caching node_modules between builds). The artifacts themselves are .tar.lz4 archives, for example.
Monofo itself also uses buildkite-agent artifact download
and buildkite-agent artifact upload
in the inject-artifacts step. If the artifact download there had a caching layer wrapping it, it wouldn't have to do as much downloading work.
buildkite-agent artifact search 'output/coverage-data-platform/cobertura.part.3.xml' . --build 'd31b7e49-95cf-4cea-b091-9756a59e3605'
2021-06-03 14:19:22 INFO Searching for artifacts: "output/coverage-data-platform/cobertura.part.3.xml"
a408d0f8-807d-467d-96bc-5cdef1238c0b output/coverage-data-platform/cobertura.part.3.xml 2021-06-03T00:14:23Z
$ buildkite-agent artifact shasum 'output/coverage-data-platform/cobertura.part.3.xml' . --build 'd31b7e49-95cf-4cea-b091-9756a59e3605'
2021-06-03 14:21:44 INFO Searching for artifacts: "output/coverage-data-platform/cobertura.part.3.xml"
f62be7358e7ec7016cd7b53ffee4afac3c4942db
More important for big artifacts obviously. Instead of downloading, get the above info, and check a cache dir for the artifact ID as the file key. Double-check integrity by sha1sum, and fall back to re-downloading if needed. Make sure moves into the file cache are atomic to prevent most integrity errors.
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.