concourse / prod Goto Github PK

similar to pks worker, tag everything to do with topgun or that worker appropriately.... you should be able to still use the bosh-deployment resource. the manifest for that worker is in concourse/prod, called something like topgun-worker.yml

first attempt, we increased the workers-3 node pool by 5 and allocated a new Strabo-workers deployment on to that node pool. It didn't work. @cirocosta mentioned that this doesn't actually fix things since the resources won't be isolated for Strabo.

second attempt, create a separate node pool Strabo-workers with 5 k8s workers. Move the Strabo-workers helm deployment onto it and see if the pipeline passes.

add a wiki entry for how to search for logs in stackdriver - can it be as easy as papertrail?

https://github.com/concourse/docs/blob/master/lit/docs/observation.lit#L47-L56

figure out where those sample pipelines come from, or invent/choose a place for them to live and tell osis. put their existence in code (like the reconfigure pipeline)

in a perfect world, creds live only in vault (/its appropriately supported storage backend) and manual deploys use the same technology
there are some helm deployment resource types that might help

• auto-unseal
  • Understand auto-unseal with Google Cloud Key Management Service (extraEnvVars in values.yaml from the chart vault-helm and auto-unseal settings in vault config file,
  • sample terraform config for gcpkms and terraform docs for gcpkms
  • terraform update with adding gkms crypto and key ring.
  • add gcp credential to hush-house-values-nci in lastpass.
  • run command vault operator init to initialize vault.

We should take a look at whether cert rotation can be more automated (or documented better) than our current process.

Currently we need to:

• update the certs in credhub for metrics and re-deploy
• add the certs to the lastpass entry for the directory vars and run the create script
• ????
• profit

These deploys and credhub interactions could at least be assisted by a pipeline

please put your configuration in code with terraform

We used to use an old version of Vault in BOSH prod, which may have used the filesystem backend as a default option. While it technically works, we may want to shift to a less error-prone storage backend because it is all of our production secrets after all, and it would be quite annoying to have to recover from someone accidentally running rm rf /vault for example while SSH'd into the VM/container.

All of the possible backends are documented here: https://www.vaultproject.io/docs/configuration/storage/index.html

We don't really need highly consistent/replicated/sharded/etc persistence so a lot of these strategies are overkill, but I'd say that the one feature we could benefit from is being able to easily make snapshots for backing up and restoring.

We already use GCS so my instinct would be to just pick that one, but this story includes room for doing some investigation.

An annoying thing we will have to do once is perform a data migration into the new schema. As far as I can tell there is no officially-supported way to migrate data between different backends.

• Research and choose a new backend
• Backup current prod Vault data
• Set up the new backend
• Update operating docs github.com/pivotal/concourse-ops wiki
• Restore old secrets into new backend
• Decommission old backend

Hey,

We've been recently receiving complaints that resources like docker-image and
registry-image have been failing with "429 Too Many Requests".

While we did introduce retries at the resource-type level for registry-image, (see
concourse/registry-image-resource#69) those using
docker-image (or trying to reach dockerhub directly) would still suffer from the
limit being place on our IP.

My hypothesis is that by removing the NAT machine that we have in the bosh
network (which ends up making every request from any of the 40+ machines we
have going out from that single IP), we can then get rid of the problems we're
currently facing w/ regards to limits on the number of requests (aside from
reducing one hop and a single point of failure).

Last week, I naively tried just removing the routes that we have set at the
network level

but that didn't really work as expected as the machines that we create in the
bosh network do not assign ephemeral external IPs:

"The instance must have an external IP address. An external IP can be assigned
to an instance when it is created or after it has been created."

(from https://cloud.google.com/vpc/docs/vpc#internet_access_reqs)

Given that we're on GCP, we can overcome that by using the
ephemeral_external_ip property - see https://bosh.io/docs/google-cpi/#networks.

Should we do that? I think so - if we don't have the requirement of having those
machines completely unreachable at all (not really true in our case), I think we
should just drop it.

Thanks!

We would like to migrate the CI (prod) deployment to run on K8s

TODOs v1

• Create address & DNS entry for nci.concourse-ci.org
• Create a worker node pool
• Create a CloudSQL instance for CI
• Helm deploy workers
• Helm deploy web nodes
• Helm deploy vault
• Move secrets over
• Setup auth for the vault server
• Ensure Vault is secure Prod Deployment checklist
• Migrate the pipelines ( choose which ones need to be migrated )
• Migrate example pipelines used in https://concourse-ci.org/examples.html #38
• Audit & reconcile Bosh manifest values vs. K8s Chart values
• Move over DNS entry from nci to ci making it official #39
• Logging solution #40
• add X-Frame-Option #41
  • concourse/concourse-chart#22
• Unmute datadog Hush-House SLI alert #42
• have baggageclaim pipeline setup
  • concourse/ci#221

TODOS v2

• Point workers to the new CI env

  • darwin
  • windows
  • k8s-topgun
    • concourse/ci#192
  • bosh-topgun needs to be continuously deployed - #43

• Move untrusted workers into K8s: concourse/ci#211

• Move metrics for CI over to K8s

• @clarafu @vito check that there's a backup before upgrading to v6

• Scale down BOSH deployed prod/ci env to 1 or 0 ?? #46

• switch deployment to use the concourse-rc, Setup CI to auto deploy CI #45

• #44 Pick a scalable storage backend for vault ( such that backups, snapshots are much easier to manage ), auto-unseal with Google Cloud Key Management Service (extraEnvVars in values.yaml, auto-unseal settings in vault config file, sample terraform config for gcpkms, terraform docs for gcpkms)

• switch to datadawg if grafana gets hard to manage

use the 'extra environment variables' thing in the helm chart to avoid the empty string issue.

concourse/oxygen-mask is where they live. think about fitting this into the reconfigure-pipelines pipeline. think about creating a separate worker just for running these pipelines? @kcmannem has an opinion

Tentative plans:

Pre-work:

• merge concourse/ci#231
• merge concourse/hush-house#90
• Figure out prod deploys: #45
  • windows worker on prod should move its config to ci

PR to concourse/prod:

• remove ci-concourse-ci-org-dns from https://github.com/concourse/prod/blob/master/iaas/dns.tf

PR to concourse/ci:

• topgun worker - update configuration in concourse/ci with new DNS
• k8s-topgun worker - update configuration in ci with new DNS
• windows worker - update configuration in ci with new DNS
• concourse-pipeline resource inside the reconfigure pipeline
• slack notification resource inside the main pipeline

PR to concourse/hush-house:

• change concourse-nci-address in https://github.com/concourse/hush-house/blob/master/terraform/main.tf
• external url on the new environment needs to change too
• change cluster name from ci-house to ci

Secrets:

• TLS cert on API/web

manual followup steps:

• terraform apply in prod
• terraform apply in hush-house
• deploy CI and all its worker pools, using the results of #45
• github auth redirect url

makes sense to do at the same time as #39