Release 6.4.0 about ci HOT 24 CLOSED

An addition to step 1 in the process:

• Once all the commits were on master, we created a release note file for 6.4.0 based on latest.md and made sure to clear the latest.md file. Once the changes are merged, then we can proceed and create the release branch. Pull request link.

This should also result in amending the last step as that should just edit the 6.4.0 release notes file rather than create it.

from ci.

• An addition after the steps in the process:
  * the pipeline filters out commits to "release-notes" directory , hence we need to make sure that they are picked up somehow prior to kicking off shipit as otherwise release note updates wont end up back on master.

from ci.

TODO - we created the release/6.4.x branch off of master for concourse-chart - CONFIRM that is indeed correct

from ci.

• TODO PRE RELEASE - do a dummy commit to ensure all the PR docs get merged back into master

from ci.

• TODO PRE RELEASE - pin gdn to the older version 1.19.12 as the newer patch might have issues that we haven't had the opportunity to investigate yet - https://pivotal.slack.com/archives/CBHN4GXTR/p1594236571255600

from ci.

TODO - we created the release/6.4.x branch off of master for concourse-chart - CONFIRM that is indeed correct

Chatted with Taylor and new features are always merged into the dev branch of concourse-chart repo, since, 6.4.0 is a minor release, it made more sense to create the release branch off the dev branch.

• TODO - we should start highlighting this in the docs

from ci.

TODO - take a look at the list of resource types below and bump those with new PRs merged in:

• bosh-io-release-resource - bumped to v1.0.3
• bosh-io-stemcell-resource - v1.0.3
• cf-resource - v1.1.0
• docker-image-resource - bumped to v1.3.4
• git-resource - bumped to v1.8.0
• github-release-resource - bumped to v1.3.1
• hg-resource
• pool-resource
• registry-image-resource bumped to 0.11.1
• s3-resource bumped to 1.1.0
• semver-resource
• time-resource bumped to 1.2.2
• tracker-resource bumped to 1.0.3
• mock-resource bumped to 0.8.0

from ci.

TODO PRE RELEASE - pin gdn to the older version 1.19.12 as the newer patch might have issues that we haven't had the opportunity to investigate yet - https://pivotal.slack.com/archives/CBHN4GXTR/p1594236571255600

Pinned the gdn resource

from ci.

• TODO PRE RELEASE - Should we be deploying CI with Helm or Kapp ?

from ci.

• Go to the https://project.concourse-ci.org/releases page and take a look at what resource types have had new PRs merged in. If the PRs that got merged in were only small fixes, then bump a patch release of that resource type in ci.concourse-ci.org. If they add features then do a minor version bump. If it adds breaking changes then do a major version bump.

• Add your release pipeline to the reconfigure-pipeline

• Bump the appropriate versions for resource types. Go to the releases page https://project.concourse-ci.org/releases and take a look at which resource type repositories have had new commits or PRs. Take a look at what those changes entail and bump the version in their respective pipeline in ci.concourse-ci.org.

  • If the changes were only README or repo restructuring changes with no user impact, you don't need to bump the version
  • If the changes were small bug fixes or changes, you can do a patch version bump
  • If the changes were adding of features, you can do a minor version bump
  • If the changes involve a breaking changes, that should be a major version bump

• TODO - cleanup duplicate steps for resource types

from ci.

TODO - Add section for DOCs PRs to approve just after releasing the binaries and prior to the promote pipeline that updates docs

• concourse/docs#344

from ci.

PR for release notes changes concourse/concourse#5873

from ci.

This issue is going to be a blocker for the release concourse/concourse#5875

from ci.

TODO - we should add upgrading CI as part of the release process

from ci.

TODO - upgrade the remaining deployments for CI as well (eg. PR workers )

from ci.

TODO - Noticed a snag with the concourse-chart, running publish-chart-minor only updated the chat version but not the app version - https://ci.concourse-ci.org/teams/main/pipelines/helm-chart/jobs/publish-chart-minor/builds/4

Need to document the proper steps to doing this. The bump concourse job appears to pull form dev to master and not from the release branch.

from ci.

TODO - Noticed a snag with the concourse-chart, running publish-chart-minor only updated the chat version but not the app version - https://ci.concourse-ci.org/teams/main/pipelines/helm-chart/jobs/publish-chart-minor/builds/4

Need to document the proper steps to doing this. The bump concourse job appears to pull form dev to master and not from the release branch.

Had to manually bump the chart as the job kept failing due to the merge conflict, which was introduced by me running the bump chart job prematurely. The bump concourse job needs some TLC so it works for the release process.

from ci.

TODO
Updating CI, CI-PR and CI-Topgun-worker is a bit wonky and documentation and process needs some TLC so its simpler

• ci uses kapp, the others use helm
• ci-topgun-worker is in the topgun cluster and not part of hush-house
• ci-topgun-worker's state in the ci repo didn't match its state in the deployed cluster - release name is ci-topgun in the former and ci-topgun-worker in the latter

Windows worker is on BOSH ( deployment - concourse-prod-windows-worker & state is in concourse/prod/deployments repo )

from ci.

TODO

To deploy the ci environment, run `make deploy-ci`.

If you want to force a rolling update (recreate all pods), say after updating secrets, increment the `rollingUpdate` annotation declared in [`values.yaml`](./values.yaml).

Is this only for the cases when we update the secrets but not change anything else (eg. image digest/tag) or for every case ?

from ci.

Deployed HH & Workers

TODO

• roll windows and any other remote workers of HH
• understand the CONCOURSE_LOG_LEVEL & db query log issue - ( Rui & I observed that the flags are not respected from the template and we were unsure where they were being set from )
• why does CI have datadog as a dep and HH doesn't in the Charts.yaml . Furthermore there are many datadog pods in the CI namespace - not clear whether they belong there are not

from ci.

Cleaned up hush house windows worker on bosh

 bosh -e prod -d hush-house-windows-worker delete-deployment

as the worker wasn't connected to hush house and had been failing to do so from at least July 14 ( logs were rolled so we don't know its earlier state ). Also, its worker public key wasn't on hush house either.

It can easily be re-created by using the ci windows worker BOSH manifest

from ci.

Updated checklist

The documentation we have been using as reference to this release is listed below:

• Concourse Release Process
• Proposed changes to release process
• 6.3.0 release steps

Steps for a new major/minor release:

• [] Once all the commits are into master for your release, create the release notes. Move the contents of release-notes/latest.md into a new release note file of release-notes/v<M.m.p>. Then create your release branch on the concourse/concourse github repo with the following format release/M.m.x (M being the major version and m being the minor version)

• [] Create the release branch on the concourse/concourse-bosh-release repository. Make any missing changes to the spec of web or worker depending on if the release contains any changes that adds or modifies any flags.

  • Any changes you make on the branch will not get automatically merged back to master so try to make the changes on master and then create the branch from there.
  • We should really have something that will merge the branch back to master. (like we do for the concourse/concourse branches)

• [] Create the release branch on concourse/concourse-bosh-deployment repository.

• [] Create the release branch on concourse/concourse-chart repository. Same with the concourse-bosh-release repo, make any missing changes to values.yaml or templates/web-deployment.yaml for changes to flags on web or templates/worker-deployment.yaml for changes to flags on the worker. New features to the chart are added to the dev branch vs master gets bug-fixes. This distinction should guide whether the release branch should be created from dev (major/minor release) or master (patch release).

• [] Bump the appropriate versions for resource types. Go to the releases page https://project.concourse-ci.org/releases and take a look at which resource type repositories have had new commits or PRs. Take a look at what those changes entail and bump the version in their respective pipeline in ci.concourse-ci.org.

  • If the changes were only README or repo restructuring changes with no user impact, you don't need to bump the version
  • If the changes were small bug fixes or changes, you can do a patch version bump
  • If the changes were adding of features, you can do a minor version bump
  • If the changes involve a breaking changes, that should be a major version bump

• [] Add your release pipeline to the reconfigure-pipeline

• [] Go through all the undocumented PRs in the release page for your milestone https://project.concourse-ci.org/releases/concourse?milestone=v<M.m.p> and make sure that everything has release notes. You can organize which PRs by clicking on the button to add whichever label best fits that PR.

  • If there is an existing release note, add a release/documented label
  • If there is no release note and the changes have no user impact, add a release/no-impact label
  • If there is no release note and the changes have user impact, document it (or delegate) and add a release/documented after finished.

  Note: The pipeline ignores the release-notes path so a build would have to be fired manually to get updates through the pipeline.

• [] Once the all source code changes are finalize, Concourse RC version should be deployed to CI

  • including all the external workers (pr-worker, ci-topgun-worker, darwin-worker & BOSH deployed windows worker)

• [] Once everything is ready, the shipit job can be triggered. Subsequently, the promote concourse job will run automatically. The publish-docs job needs to be triggered manually.

• [] The helm-chart pipeline is used to bump & then publish the chart.

• [] Once the Concourse release is shipped, Concourse should be deployed to Hush-House

from ci.

Theme of issues

• updating our envs is fragmented due to external workers and different tooling (kapp vs helm & BOSH)
• our pipelines are fragmented (release, promote, helm-chart, helm-chart enterprise) and the relationships are not obvious

from ci.

Updated checklist

The documentation we have been using as reference to this release is listed below:

• Concourse Release Process
• Proposed changes to release process
• 6.3.0 release steps

Steps for a new major/minor release:

• [] Once all the commits are into master for your release, create the release notes. Move the contents of release-notes/latest.md into a new release note file of release-notes/v<M.m.p>. Then create your release branch on the concourse/concourse github repo with the following format release/M.m.x (M being the major version and m being the minor version)

• [] Create the release branch on the concourse/concourse-bosh-release repository. Make any missing changes to the spec of web or worker depending on if the release contains any changes that adds or modifies any flags.

  • Any changes you make on the branch will not get automatically merged back to master so try to make the changes on master and then create the branch from there.
  • We should really have something that will merge the branch back to master. (like we do for the concourse/concourse branches)

• [] Create the release branch on concourse/concourse-bosh-deployment repository.

• [] Create the release branch on concourse/concourse-chart repository. Same with the concourse-bosh-release repo, make any missing changes to values.yaml or templates/web-deployment.yaml for changes to flags on web or templates/worker-deployment.yaml for changes to flags on the worker. New features to the chart are added to the dev branch vs master gets bug-fixes. This distinction should guide whether the release branch should be created from dev (major/minor release) or master (patch release).

• [] Bump the appropriate versions for resource types. Go to the releases page https://project.concourse-ci.org/releases and take a look at which resource type repositories have had new commits or PRs. Take a look at what those changes entail and bump the version in their respective pipeline in ci.concourse-ci.org.

  • If the changes were only README or repo restructuring changes with no user impact, you don't need to bump the version
  • If the changes were small bug fixes or changes, you can do a patch version bump
  • If the changes were adding of features, you can do a minor version bump
  • If the changes involve a breaking changes, that should be a major version bump

• [] Add your release pipeline to the reconfigure-pipeline

• [] Go through all the undocumented PRs in the release page for your milestone https://project.concourse-ci.org/releases/concourse?milestone=v<M.m.p> and make sure that everything has release notes. You can organize which PRs by clicking on the button to add whichever label best fits that PR.

  • If there is an existing release note, add a release/documented label
  • If there is no release note and the changes have no user impact, add a release/no-impact label
  • If there is no release note and the changes have user impact, document it (or delegate) and add a release/documented after finished.

  Note: The pipeline ignores the release-notes path so a build would have to be fired manually to get updates through the pipeline.

• [] Once the all source code changes are finalize, Concourse RC version should be deployed to CI

  • including all the external workers (pr-worker, ci-topgun-worker, darwin-worker & BOSH deployed windows worker)

• [] Once everything is ready, the shipit job can be triggered. Subsequently, the promote concourse job will run automatically. The publish-docs job needs to be triggered manually.

• [] The helm-chart pipeline is used to bump & then publish the chart.

• [] Once the Concourse release is shipped, Concourse should be deployed to Hush-House

Hey @clarafu
Updated your checklist based on my experience shipping 6.4.x

from ci.