alphagov / paas-cf Goto Github PK

https://github.com/alphagov/paas-cf/blob/master/concourse/scripts/pipecleaner.py

@jmcarp just pointed me to this script—seems super useful! Would you consider breaking it out to its own repository, so it can be installed/run more easily?

This is only after a cursory skim of your pipelines, so I may have missed some special requirement.

There are three Concourse resources in the main org that can abstract away deploying with BOSH and updating releases and stemcells.

See as an example the work the Buildpacks team have done to deploy rolling versions of CF for testing.

The pre-requisites section of the documentation does not include details about the creation of the shared-cf-blosh-blobstore- bucket(s). These buckets are not created by the paas-cf pipeline, so should be listed in the pre-requisites section.

Further to this, because S3 bucket names are globally unique, it is not possible for any other organisation outside of GDS to use the same bucket-name format, so it would be good to be able to parametrise the shared-cf-bosh-blobstore- part of the bucket name for different organisations to be able to use paas-cf without having to fork and change code.

I want to suggest an improvement for your pipelines to pin the version of the containers.

I suggest this here because there is no other official channel to communicate.

In concourse, you can use a resource as image for a task, by using the get instead of specify task.config.docker_resource.

This way you are pining the version of the container to use:

diff --git a/concourse/pipelines/create-cloudfoundry.yml b/concourse/pipelines/create-cloudfoundry.yml
index 810c3d7..21af57d 100644
--- a/concourse/pipelines/create-cloudfoundry.yml
+++ b/concourse/pipelines/create-cloudfoundry.yml
@@ -55,6 +55,7 @@ groups:
   - name: credentials
     jobs:
       - clear-cloudfoundry-credentials
+
 resource_types:
 - name: s3-iam
   type: docker-image
@@ -67,6 +68,11 @@ resource_types:
     repository: governmentpaas/semver-resource
 
 resources:
+  - name: cf-acceptance-tests-container
+    type: docker-image
+    source:
+      repository: governmentpaas/semver-resource
+
   - name: pipeline-trigger
     type: semver-iam
     source:
@@ -1129,6 +1135,8 @@ jobs:
           - get: graphite-nozzle
           - get: datadog-tfstate
           - get: paas-rubbernecker
+          - get: cf-acceptance-tests-container
+            passed: ['cf-deploy']
 
       - aggregate:
         - task: extract-cf-terraform-outputs
@@ -1931,7 +1939,8 @@ jobs:
           - get: bosh-CA
           - get: cf-secrets
             passed: ['cf-deploy']
-
+          - get: cf-acceptance-tests-container
+            passed: ['cf-deploy']
       - do:
         - task: create-temp-user
           file: paas-cf/concourse/tasks/create_admin.yml
@@ -1972,7 +1981,7 @@ jobs:
                     ./cf-release/jobs/acceptance-tests/spec \
                     acceptance_test_properties.yml \
                       > ./test-config/run
-
+                                                                        t
                   chmod +x ./test-config/run
 
                   ./paas-cf/platform-tests/bosh-template-renderer/render.rb \
@@ -1983,11 +1992,9 @@ jobs:
 
         - task: run-tests
           config:
+            image: cf-acceptance-tests-container
             platform: linux
-            image_resource:
-              type: docker-image
-              source:
-                repository: governmentpaas/cf-acceptance-tests
+g
             params:
               DISABLE_CF_ACCEPTANCE_TESTS: {{disable_cf_acceptance_tests}}
             inputs:

CIDR access IPs for admin endpoints is configured in globals.tf and other .tf variable files and deployed by Concourse pipeline.

In order to change the cidr ranges, it is necessary to fork paas-cf

It would be great to be able to deploy paas-cf for non-GDS organisations without forking.

Hi there,

I stumbled on this FIXME while doing a GH code search for usage of a library I work on:

As zorkian/go-datadog-api#56 has been closed off for a while now, you should be able to remove your work around when and if you see fit.

The Vagrantfile contains hard-coded subnet and security group IDS which correspond to GDS's AWS account only.

It would be really good to parametrise these settings (or at least document that these will need to change for deployment outside GDS)

The README.md states that IAM instance profiles are configured through the account-wide-terraform repository and that it is not publically accessible due to the state file being stored within that repository. No further details of the required instance profiles are provided.

The following IAM instance profiles are required as pre-requisites before deployment can complete successfully.

• bootstrap-concourse
• deployer-concourse
• bosh-director
• bosh-managed
• cf-cloudcontroller
• rds-broker

It would be really great if a sanitised version of the account-wide-terraform repository could be provided, but failing that, details of the required IAM roles and policies for each instance profile should be documented fully.

Because the terraform runs happen within the Concourse deployment pipeline(s), the use of hardcoded DNS names and Zone IDs within the TFVars makes it necessary to fork paas-cf and update pipelines to pull from the forked repo instead.

It would be great for other organisations outside GDS to be able to deploy paas-cf without having to fork the codebase.

Background

The lego has already renamed it’s import path from "github.com/xenolf/lego" to "github.com/go-acme/lego".

But gomodules/dns still used the old path:
https://github.com/alphagov/paas-cf/blob/master/tools/user_emails/go.mod#L17

github.com/xenolf/lego v2.5.0+incompatible

When you use the old path "github.com/xenolf/lego" to import the lego, it will be very easy to reintroduce lego through the import statements "import github.com/go-acme/lego" in the go source file of lego.
https://github.com/go-acme/lego/blob/v2.5.0/acme/api/authorization.go#L6

package main
import (
	"fmt"
	"github.com/go-acme/lego"
	"net/http"
)
…

The "github.com/go-acme/lego" and "github.com/xenolf/lego" are the same repos. This will work in isolation, bring about potential risks and problems.

Solution

Replace all the old import paths, change "github.com/xenolf/lego" to "github.com/go-acme/lego".
Where did you import it: https://github.com/alphagov/paas-cf/search?q=github.com%2Fxenolf%2Flego&unscoped_q=github.com%2Fxenolf%2Flego