We currently only allow PutObject access to our s3 bucket (here: https://github.com/samhstn/samhstn/blob/master/infra/codebuild.yaml#L90)

but the command:

aws s3 sync static s3://samhstn.com --delete --exclude badge

also needs DeleteObject access for removing no longer needed files.

Currently we pass parameters into our cloudformation template.

This is not a nice way to deal with our keys, and not how parameters should be used with cloudformation templates.

We should instead use ssm and kms to manage our keys following this article.

We currently reference a hardcoded codebuild badge in our README.md.

This will change every time we re-deploy our ./infra/codebuild.yaml template.

We should instead reference a url, something like:

[![Build Status](https://samhstn.com/repo/samhstn/badge)](https://console.aws.amazon.com/codesuite/codebuild/projects/CodeBuild)

We will create an S3 bucket which will redirect to our CodeBuild badge (following this article).

This bucket redirect url will update with every deployment and always point to the correct place.

Currently the aws console is being cluttered by different resources from different projects.

It would be really nice if we created an Admin type user who can view, create, update or delete resources which are related to the samhstn project - and only the samhstn project.

That way we can have a cleaner view of exactly what we are working on at any time and not accidentally change anything not related to what we are working on.

These permissions should be set referencing aws tagged resources - where these users should only see resources tagged with samhstn and should always include the samhstn tag when interacting with any resources.

This should affect both the Management console access user as well as the programmatic access user.

Look to use AWS::SecretsManager::Secret for our application secret key base and pass this to our ec2 instance on launch.

We currently use a hardcoded, example one (see here).

This is overkill.

There should be no css pre-processing.

Maintaining this is going to be hard enough without the adding another package - adding dependencies should be avoided at all costs.

Currently we manually add the webhook through the Github interface in the Configure Codepipelines section.

It would be super smooth to add a curl command which lists our repo webhooks and adds the one we need (documentation here) referencing our CodePipeline webhook url.

This would involve updating the Authorize Github section to include the admin:repo_hook scope.

Desktop

Mobile

When our master branch is updated,

we should trigger a codebuild job,

which will compile our code,

then write these static files to our s3 bucket,

which will update samhstn.com

We currently work off the aws provided nodejs image.

We are going to be installing some of the same dependencies on every CodeBuild run (Cfn linter for instance).

We can save this installation time and reduce CodeBuild logic if we instead reference a samhstn image which we have control of.

We should deploy this image to ECR and reference it in each of our CodeBuild builds.

Our CodeBuild should update our Dockerfile on ecs if there are changes to be made.

The nicest way I can think of doing this is:

• Create an ssm parameter called /Samhstn/DockerfileDigest
• On every deploy, we should run a check if md5sum is the same
• If it matches, don't update our ecr image, otherwise do.

The command to check if there is a match is:

echo "$DOCKERFILE_DIGEST  infra/Dockerfile" | md5sum -c -

Our S3 buckets are currently publicly accessible.

We should make sure that S3 buckets are not directly accessible, but are only accessible through CloudFront - explained in this article: https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-s3-amazon-cloudfront-a-match-made-in-the-cloud.

We should look to separate out some logic (fix duplicate data in the pipelines files), update the naming (especially buckets) and place buildspecs in a buildspecs directory.

The directory structure of ./infra/ should look something like this:

|--- infra/
       |--- README.md
       |--- buildspecs
             (|--- update_stacks.yml - coming later)
              |--- test.yml
              |--- deploy.yml
       |--- cloudfront.yml
       |--- route53.yml
       |--- s3.yml
       |--- codebuild.yml
       |--- master_pipeline.yml

Explained here: samhstn/my-config#70

This will involve setting up an s3 bucket with the script contents

Longer term plan for how the directory structure should look.

|-- .eslint-browser.js
|-- .eslint-node.js
|-- .eslint-lambda.js
|-- .npmrc
|-- .gitignore
|-- README.md
|-- package.json
|-- elm.json
|-- infra
    |-- README.md
    |-- Dockerfile
    |-- s3.yaml
    |-- cloudfront.yaml
    |-- keys.yaml
    |-- master_pipeline.yaml
    |-- route53.yaml
    |-- codebuild.yaml
    |-- ecr_repo.yaml
    |-- buildspecs
        |-- test.yaml
        |-- update_stacks.yaml
        |-- update_buckets.yaml
        |-- update_baseimage.yaml
        |-- deploy.yaml
    |-- lambda
        |-- email.js
|-- dev
    |-- README.md
    |-- index.js
|-- src
    |-- static
        |-- favicon.ico
        |-- logo.ico
    |-- js
        |-- index.js
    |-- elm
        |-- README.md
        |-- Main.elm
    |-- index.html
|-- dist

We don't need any js at the moment

It would be good to separate out the current README.md like so

|- README.md
|- infra/
    |- README.md

as to not clutter the initial view and to separate out concerns a little.

I ran the setup from scratch and found that:

• the badge now references our codebuild via a different url.
• there was a typo in the Configure our Codepipeline pipeline section in referencing our infra/master_pipeline.yaml file.

Our codebuild currently shows the following warning:

[Container] 2019/03/27 20:05:17 Running command aws s3 sync static s3://samhstn.com
/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:369: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning

This might be updated when we reference our custom docker image (#55) with python upgraded.

• We currently have 2 sids in our infra/s3.yaml file, but sids aren't use elsewhere.
• Our CodePipeline is named by aws, we should control this for consistency.

(todo: add other inconsistencies to this list)

We currently have this file: https://s3.amazonaws.com/codefactory-us-east-1-prod-default-build-badges/unknown.svg in our source code as static/repo/samhstn/badge.svg and added it to s3 - it isn't clear where this file came from and we have cluttered our static directory.

We should instead not include this svg in our source code and download it using curl if required.

The buildspec should instead run something like the following:

aws s3 sync static s3://samhstn.com --delete --exclude badge
CODEBUILD_BADGE_URL=$(\
  aws codebuild batch-get-projects \
    --name CodeBuild \
    --query "projects[0].badge.badgeRequestUrl" \
    --output text \
)
if aws s3 ls s3://samhstn.com/badge.svg || \
  [ $(\
    aws s3api head-object \
      --bucket samhstn.com \
      --key repo/samhstn/badge.svga \
      --query "Metadata.\"website-redirect-location\"" \
      --output text\
    ) != $CODEBUILD_BADGE ];then

  curl https://s3.amazonaws.com/codefactory-us-east-1-prod-default-build-badges/unknown.svg > ./badge.svg
  aws s3 cp ./badge.svg s3://samhstn.com/badge.svg --metadata "Website-Redirect-Location=$CODEBUILD_BADGE"

We have a few "*" references in our templates which don't follow the least privilege best practice.

These should be addressed and will make for more semantic pull request changes and more readable statements.

We currently use a aws/codebuild/nodejs:10.14.1 or aws/codebuild/ubuntu-base:14.04 image.

We should reference our own image after #33 has been addressed.

currently on clicking the Codebuild badge at the top of the README.md, we redirect to https://console.aws.amazon.com/codesuite/codebuild/projects/CodeBuild.

This no longer exists and should be https://console.aws.amazon.com/codesuite/codebuild/projects/CodeBuild due to the name change here: https://github.com/samhstn/samhstn/pull/60/files#diff-e41b13d89f238bc2b9c937098b983ac5R87

We should redirect to: https://console.aws.amazon.com/codesuite/codebuild/projects which doesn't depend on naming.

On every merge to master we should update our cloudformation stacks.

We should address #40 before we hack around with passing in parameters to each update-stack command.

After deleting a branch, we should also delete the relevant codebuild buckets.

From comment: samhstn/my-config#43 (comment), my bio website link is broken 😞

Once the site is up and running, we should look to update this to https://www.samhstn.com

We currently run just Source and Build stages in our pipeline. It would be good to add a Test stage.

The section: https://github.com/samhstn/samhstn/blob/master/infra/README.md#configure-our-ssl-certificate is too long.

There is a large 5 command bash section which we should remove. For simplicity here, we should just use the AWS console.

Currently I am the only one who view/access the different aws pipelines.

We should think of a way to allow easy collaboration and at least READ_ONLY access to everything (except secret keys) on AWS.

Update instructions in ./infra/README.md and separate out base.yml into more digestible files.

In our infra/cloudfront.yml template, we require a parameter of Region to be passed in the cli.

This isn't necessary, we should instead simply reference Region like so:

!Ref "AWS::Region"

We currently just use the BuildRole role which has access to more resources than it needs.

We should follow the Grant Least Privilege best practice principle.

This would involve having 2 or 3 roles in ./infra/codebuild.yaml with finer grained permissions.

As part of our tests, we should also do a dry run on our cloud-formation templates.

This should occur before our Build stage which updates our cloudformation #19

It would be good to set up the @samhstn.com email addresses.

We don't want to be paying $4 per month with using WorkMail.

Instead, it would be best to opt for a cheaper setup using just lambda and s3.

• Configure lambda function which logs email in cloudwatch
• Configure lambda function which writes email to s3.
• Configure lambda function to also send an email notification when an email arrives.

For now all emails can be read using the command line, or viewing them through the s3 console. We can worry about a more maintainable way to manage emails later.

We should deploy a dev.samhstn.com domain which can be pushed to for testing deployment changes.

For more info see #51

The changes documented here: https://github.com/samhstn/samhstn/pull/69/files say:

The dev environment should be almost identical to the master environment

Instead of this dev environment, we should actually strive for an environment which is exactly the same as master and this environment is dynamically created every pull request.

This will require major rewrite, where each deployment will be the same (including master) and will be architected in the following way:

We will configure the following templates

root-iam (allows cross account to root account)
project-iam (allows admin access for project account)
acm (configured for all *.samhstn.com and samhstn.com domains)
(this will be followed by a manual DNS validation of acm certificate)
keys (our personal access tokens and secrets)
waf (our firewall to be referenced by every cloudfront distribution)
dci (contains the following templates):

• webhook (listens to code changes from Github)
• codepipeline (facilitates the running of our codebuild)
• codebuild (creates/updates our branch codepipeline)

each push event will trigger

update of the above infrastructure if branch is master
create/update codebuild template
create/update codepipeline template
run our codepipeline which will:

• create/update our s3 template
• create/update our cloudfront template
• create/update our route53 template as root user

This will configure the url: https://.samhstn.com

each pr delete event will trigger

delete route53 template as root user
delete cloudfront template
empty s3 bucket and delete s3 template
delete codepipeline template
delete codebuild template

Infra folder structure

|- README.md
|- test.Dockerfile
|- deploy.Dockerfile
|- root
   |- README.md
   |- email.yaml
   |- project-iam.yaml # admin user
   |- root-iam.yaml # cross account access for route53 configuration
   |- route53.yaml # record set to be used for each deployment
|- buildspecs
   |- infra.yaml # only run if branch is `master` - will update our infrastructure
   |- dci.yaml # creates/updates and triggers our push/codepipeline
   |- test.yaml
   |- deploy.yaml
|- base.yaml
|- base
   |- public-bucket-policy.yaml
   |- private-bucket-policy.yaml
   |- acm.yaml
   |- keys.yaml
   |- waf.yaml
   |- dci.yaml
   |- dci
      |- codebuild.yaml
      |- codepipeline.yaml
      |- webhook.yaml
|- push.yaml
|- push
   |- README.md
   |- codebuild.yaml
   |- codepipeline.yaml
   |- s3.yaml
   |- cloudfront.yaml

CI is now running on every branch build (as of #25), look to add a badge for this to the README.md

We currently do this through the console, but there is no real reason why this shouldn't instead be a cloudformation template.

Implementation of #50 in our infra/buildspecs/*.yaml files.

We should add steps to be able to delete all templates.

Currently running aws cloudformation delete-stack --stack-name master-pipeline fails - for example.

There should be a nice series of bash commands which should nicely achieve this.

We should add a pipeline which is triggered on any branch change and runs just our tests.

There are some minor inconsistencies emerging in the infra/*.yml files, adding a yml linter would nip this in the bud.

The official aws cloudformation examples use yaml and yaml is encouraged over yml

We current are using ssm excessively and not making use of cloudformation parameters.

We should use ssm for values which won't change and cloudformation parameters for dynamic values.

We could also look to hardcode certain values (such as GithubOwner and GithubRepo).

We should consolidate every key that we are using and decide how it should be managed.

The following is one better way our parameters could be managed:

Hardcoded in template

• GithubOwner - will be samhstn and unlikely to change.
• GithubRepo - will be samhstn and unlikely to change.

Cloudformation parameter

• GithubBranch - should be overridable, for example when deploying a dev branch.
• Namespace - should be overridable, for example when deploying a dev stack.
• DomainName - should be overridable when deploying to url dev.samhstn.com.

Ssm parameter

• AcmCertArn - will rarely be updated, but should be configured once.
• DockerfileDigest - see #50

Secrets manager parameter

• GithubPAToken
• GithubSecret

We use !Join everywhere, !Sub makes for more readable templates.

We should follow the tutorials as explained in this SO answer

In the Authorize Github section we say to tick the repo scope checkbox.

We should also tick the admin:repo_hook to allow for the curl command to run at the bottom of our Configure CodePipeline section.

Currently we have some * permissions for our iam roles. It would be better to specify exactly which permissions we want to allow.

Places this should be addressed:

https://github.com/samhstn/samhstn/blob/1bcaf9deb2814ecb1c7695aebd3cba74964f70e8/infra/root/route53role.yml#L24-L27
https://github.com/samhstn/samhstn/blob/1bcaf9deb2814ecb1c7695aebd3cba74964f70e8/infra/samhstn/main.yml#L307-L311

I cannot currently build the master branch.

After some investigation, my creation of the kms key has denied permission of my subsequent stages of the master-pipeline.yaml pipeline to the pipeline S3 bucket.

It is a similar issue to this and this

I am unable to delete the kms key for 7 to 30 days, so need to find a solution.