trecnoc / concourse-pipelines Goto Github PK

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Bionic stemcells will soon come out of 'Beta' status and we should start mirroring these.

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Is your feature request related to a problem? Please describe.
Implement Concourse CI to automaticly build a new release when the cached buildpack is uploaded to Nexus

Describe the solution you'd like

• Check for new cached buildpack in Nexus
• Delete current blob
• Add new blob
• Create release with the cached buildpack version

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

We're going to jump from CF version 2.7.0 to 3.6.0, so we need all the artifacts for 3.6.0

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Currently triggers on both RabbitMq and Broker release. Should only trigger on rabbit.

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

The following RabbitMQ release used to be published to bosh.io but have stopped for a while (see related ticket):

• pivotal-cf/cf-rabbitmq-release
• pivotal-cf/cf-rabbitmq-multitenant-broker-release
• pivotal-cf/cf-rabbitmq-smoke-tests-release

Instead of pulling the releases from bosh.io we should in the mean time build them ourselves

Related pivotal-cf/cf-rabbitmq-release#408

Currently when a new release of RabbitMQ is published the pipeline is triggerred and will pull the following releases also:

• Cf-CLI
• RabbitMQ Smoke Tests
• Broker

The problem is that sometimes these releases are published after RabbitMQ and we miss these. Also if they are not updated we re-download the same artifacts.

We should make all 4 release trigger the pipeline to download only the updated releases.

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Pipelines fails to mirror artifacts, in the put: mirrror step we get:

Creating remote destination directory '/data/repo/buildpacks'
Transfer content from sub-directory 'buildpacks' only

Executing transfer
sending incremental file list
rsync: change_dir "/tmp/build/put/buildpacks" failed: No such file or directory (2)

sent 20 bytes  received 12 bytes  21.33 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3]

Similar to are other pipelines, whenever we get a new MySQL release also fetch BPM and Routing

Releases required:

• logsearch-boshrelease (triggers for all others)
• bpm
• routing
• stemcell

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Related to #47

Our current pipeline pull the latest Bosh-DNS release but should actually use the version specified in the runtime-config provided by the bosh-deployment repo.

When bosh-lite downloads were introduced we started downloading with curl the same files twice. An ops file should be written and added to the bosh-list manifest to remove these duplicates.

Here is an output of the process director script:

Generate a manifest with all the required ops files for a vSphere director
Fetching releases
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-270.5.0-ubuntu-xenial-456.12-20190816-213325-836676604-20190816213335.tgz
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bpm-1.1.3-ubuntu-xenial-456.12-20190816-213112-769707942-20190816213128.tgz
Downloading artifact: https://bosh.io/d/github.com/cloudfoundry/bosh-vsphere-cpi-release?v=53.0.1
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/uaa-73.7.0-ubuntu-xenial-456.12-20190816-215108-971707017-20190816215121.tgz
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/credhub-2.5.0-ubuntu-xenial-456.12-20190816-212632-076204669-20190816212639.tgz
Downloading artifact: https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=21.0.0
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/backup-and-restore-sdk-1.17.0-ubuntu-xenial-456.12-20190816-213639-824620335-20190816213702.tgz
Fetching the stemcell https://s3.amazonaws.com/bosh-core-stemcells/456.12/bosh-stemcell-456.12-vsphere-esxi-ubuntu-xenial-go_agent.tgz
Generate a manifest with all the required ops files for a Bosh-Lite director
Fetching releases
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-270.5.0-ubuntu-xenial-456.12-20190816-213325-836676604-20190816213335.tgz
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bpm-1.1.3-ubuntu-xenial-456.12-20190816-213112-769707942-20190816213128.tgz
Downloading artifact: https://bosh.io/d/github.com/cppforlife/bosh-virtualbox-cpi-release?v=0.2.0
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/garden-runc-1.19.5-ubuntu-xenial-456.12-20190816-215444-084934166-20190816215455.tgz
Downloading artifact: https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-warden-cpi-40-ubuntu-xenial-456.12-20190816-213008-895995834-20190816213019.tgz
Downloading artifact: https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=18
Fetching the stemcell https://s3.amazonaws.com/bosh-core-stemcells/456.12/bosh-stemcell-456.12-vsphere-esxi-ubuntu-xenial-go_agent.tgz
Processing bosh-dns-release-1.12.0.tgz

Duplicates have been strikenthrough

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

CF

The CF pipeline is pulling some CF CLI which have been archived or are not required anymore and is missing some:

Plugins:

• antifreeze: not required anymore
• autopilot: not required anymore
• cf-buildpack-management: not required anymore
• stack-auditor: missing, we should pull this

We can also now remove the CF additional pull since we caught up with what we currently have mirrored.

Github

This pipeline is not required at all anymore

Memcache

This pipeline is not required at all anymore

Looks like the buildpack removed support for CFLinuxFS2

From Concourse it fails to find the correct version of Bundler provided by the Docker image

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier

Support for cflinuxfs2 has been removed

Missing releases:

• routing
• bpm

• Should use the common tasks from the concourse-pipelines-tasks repo
• Should use the Rsync resource instead of the task
• Should use the newer Slack Notifier