Hi,

I have discovered that it is currently impossible to reliably run multiple buildkite-agents with docker on the same machine. If a couple of tasks start up shortly after each other, the following lines in the buildkite "environment" hook will lead to running docker-containers (from earlier tasks) being killed... In the build output this leads to failed tasks with exit status 137.

if [[ $(docker ps -q | wc -l) -gt 0 ]] ; then
  echo "~~~ Stopping existing :docker: containers"
  docker stop $(docker ps -q)
fi

I guess solutions would be to either rely on build-scripts to clean up after themselves (i.e. no cleanup in the environment hook at all), or to scope the killing to the specific buildkite-agent somehow...

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html

We ran into issues today when the spot price for the instance type that we're using skyrocketed. We ended up just spinning up a separate stack with an alternative instance type in the meantime. It would be cool to be able to specify a few different instance types and let spot fleet handle issues like these.

I have a put a password-less key in my secret bucket. The file is correctly downloaded from S3, but adding the key fails. Here's my build log.

Agent pid 2946
2016-08-04 08:01:33    1.6 KiB private_ssh_key
Downloading ssh key from s3://ss-bk-elastic-secrets-us-east-1/private_ssh_key
Enter passphrase for /dev/fd/63:

It hangs waiting for standard in.

It would be nice to launch these in e.g. us-west-2 in my case.

Now that we have KMS integration into the BuildKite agents for secret management I would like to have that as the default way of copying from the S3 secrets bucket instead of having to specify the environment variable BUILDKITE_USE_KMS.

This implies that by default (without having to configure pipelines) BuildKite agents would attempt to decrypt the secrets from S3 using KMS, we could still allow the use of PASSPHRASE for encryption with the same approach it has now (if the ENV variable exists than use it).

Happy to create the pull request if you would like.

I see these on builds occasionally:

$ docker-compose -f docker-compose.yml -p buildkite9c0e505a3f5844e4bde33ee36c928627 run node pug-lint app/assets/javascripts
Creating network "buildkite9c0e505a3f5844e4bde33ee36c928627_default" with the default driver
ERROR: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network

Here's the Dockerfile I'm using:

FROM node:4.5.0

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/

RUN npm install
RUN npm install -g eslint babel-eslint pug-lint

COPY . /usr/src/app

and the pipeline.yml:

steps:
  - name: ":docker:"
    plugins:
      docker-compose:
        build: node

  - wait

  - name: ":eslint:"
    command: "eslint --cache app/assets/javascripts spec/javascripts"
    plugins:
      docker-compose:
        run: node

  - name: ":dog:"
    command: "pug-lint app/assets/javascripts"
    plugins:
      docker-compose:
        run: node

It's not a major issue, but since I deploy my BK agents in a private subnet with NAT gateways, public IPs are unnecessary (and potentially confusing to anyone looking at the agents). Perhaps a configuration option as a stack parameter?

Slack comment

My understanding of the current scale down policy is that scaling down on ScheduledJobsCount <= 0 means that even if all workers are 100% busy running jobs, that some may still get killed.

I'm thinking that makes more sense to tie scaling down to the # of idle agents (perhaps being above 0 for X number of minutes).

E.g

An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts::sdfsdfsd:assumed-role/buildkite-docker-builders-Role/i-sdfsdf is not authorized to perform: ecr:GetAuthorizationToken on resource: *

At the moment the hourly cleaning of Docker images is a bummer for caches… especially on our builder machine. I'd be neat to support a S3 file containing the gc-docker rules… or perhaps just paste them in as a stack parameter?

The BUILDKITE_SECRETS_KEY is required to get objects from the secrets bucket. However if you have it set incorrectly builds will block when adding an SSH key. The build blocks waiting for input on STDIN. Is there some way we can prevent this so builds fail?

Reverting back to old versions of the stack is a little tricky. In #138 at least there's a URL outside of Buildkite that people can use. But it'd be nicer if there was a public list of releases where you could download the stack JSON file.

One idea would be to have a <major-ver>.<auto-incrementing-number> auto-tagging + release step as part of publishing, so you could go to GitHub releases and find previous releases of the stack. The major version could be checked into the repo, the minor version could just be the Buildkite build number?

When using the job based autoscaling, you can get into a (common) situation where there aren't any executors (so RunningJobsCount == 0), and there are queued jobs (so ScheduledJobsCount > 0). Then whenever the scale up hook adds a server, in the time it takes for the server to boot, the scale down hook kills it.

At the moment there's no way to login to ECR to use the docker-compose build/push support.

If you add an env hook that performs:

$(aws ecr get-login --region us-west-2)

then because of our pre-command hook the creds get removed before any commands are run.

The pre-command hook basically undoes the env hook.

I propose we:

• Don't rm the docker config files (but if we must keep it, at least move it into the conditional)
• Add built-in support for AWS_ECR_LOGIN=<region> which performs the login for you

We could also add support for logins in https://github.com/buildkite-plugins/docker-compose-buildkite-plugin itself?

It ends up missing the bucket-level permission to list objects. That is, given a prefix like foo/bar, we need IAM permissions for: arn:aws:s3:::foo/bar/* & arn:aws:s3:::foo, but the IAM config misses the second one.

Or some other way to give the machines extra IAM permissions that the builds may need.

Related to buildkite/buildkite-agent-metrics#10, to ease up the RPM on the API, it'd be nice to increase the PollInterval from 15s. Perhaps 30s? 60s?

The termination policy is currently Default, however I think it should be ClosestToNextInstanceHour- to get the best $ utilisation for when we scale down. The Default policy cares about AZs and launch configurations which probably aren't of concern to ephemeral CI instances

I think you were right on this one @lox — seeing as there's no monit/upstart, if you remotely stop the buildkite-agent process then you're going to get the instance just sitting there, but without any agent running on it.

I imagine what should happen is that the instance is marked as unhealthy and then replaced as soon as that happens.

I am having an issue similar to #109 where the ssh key downloads correctly from the default secrets s3 bucket, but then is hanging on user input when adding the key. My build output is below. I have also tried setting BUILDKITE_SECRETS_KEY and BUILDKITE_USE_KMS and it has given the same result.

Sourcing CloudFormation environment...
 
Starting an SSH Agent...
Agent pid 2940
 
Downloading secrets...
 
2016-11-09 09:15:29    2.1 KiB private_ssh_key
Downloading ssh key from s3://signpost-buildkite-trial-secrets/private_ssh_key
Enter passphrase for /dev/fd/63:

Thanks for your help.

Matt

If you have a Rails app that takes a long time to do a docker build on a fresh machine, and a pipeline with 50+ parallel spec steps, and an auto-scaling group that goes from 0 to x, having cold Docker caches can be a total bummer. This is because each of the test steps ends up running docker-compose build which can take a huge amount of time for non-trivial apps… basically creating these massive outlier build times that developers would tear that hair out over.

Now you can use the beta buildkite agent and https://github.com/buildkite/docker-compose-buildkite-plugin you're able to add a "build" step at the start of the pipeline and have all subsequent steps pick up this built docker image, instead of having to build it again on all your machines. This is very similar to how most people would structure their own pipelines anyhow, and it means you end up pushing a Docker image somewhere for each build which is also nice for later debugging. And it works with the docker hub credentials support that's already built into this stack. For example:

- command: build-and-push.sh
- wait
- command: parallel_spec.sh
  name: "Spec %n"
  parallelism: 50

The way I'm doing this at the moment is by creating two separate CloudFormation stacks: buildkite-builders and buildkite-runners. buildkite-builders is a t2.medium with a min size of 1 and max size of 2 and scale-in/scale-out of 0 to disable auto-scaling (you can't have it min 1 max 1, or else stack updates fail because of the stack update policy that requires at least one instance to be in service). buildkite-runners is a c4.medium with spot pricing, min size of 0, max size of 50, scale-out/in of +-25. This setup means you get an always-on warm cache for building+pushing, and relatively fast spin up for runners for the parallel tests (fast in both auto-scaling, and in terms starting the job on a new instance). This keeps most builds < 5 minutes if there's enough agents, or an extra few minutes if new agents need to be spun up.

I think this sort of thing is a pretty common pattern, and it'd be nice to build this into the stack somehow.

One idea would be to enable you to set a fixed size of "builders" instances which stay on at all times, which end up with their own queue which you can target using the new docker plugin. It could be that for a given stack you end up with {stack-name}-builders and {stack-name}-runners. There would be no auto-scaling of builders (but they could still be managed via an ASG). If we built this into the own stack, you'd end up with parameters to configure the two different ASGs, builders and runners.

It appears that a first job can execute before other services are fully ready, with the result that the S3 cp that happens in environment hook just infinitely hangs (perhaps the instance meta-data store isn't yet available)

We should check the init.d run levels:
http://stackoverflow.com/a/16159642

https://github.com/buildkite/elastic-ci-stack-for-aws/releases

We could mention the master builds URL in there too for people that want cutting edge.

The template tried to provision IOPS - which is doing nothing, since it specifies general purpose SSD (gp2) which doesn't support provisioned IOPS. Did you intend for a "volume type" argument to go with this?

Right now updating a cloudformation stack is quite buggy and (for me) almost always fails. Even something as simple as updating the spot price for instances is not possible.

As a workaround, we end up creating a new stack and deleting the old one. This is error prone, requires access to API keys, results in failed builds, and is time consuming.

Preferably the process for the cloudformation updates is not as buggy. Unfortunately I have no recommendations on how to improve this at this time.

The error is "Value of property ManagedPolicyArns must be of type List of String"

If you update the buildkite agent queue name, API key, agents per instance or release stream (from stable to beta) you need to manually set the ASG desired size to 0 and force a scale down for it to apply changes, because it's all done in cloud-init. And some people might try just deleting the stack and recreating.

It'd be preferable if you could just update the stack parameters and it's handled for you. This could mean performing a rolling update somehow?

This mightn't be an issue with this repo, but when I try and spin up a cluster in N. Virginia, it consistently gets rolled back.

AWS::EC2::Subnet
Subnet2
Value (us-east-1c) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1e, us-east-1a, us-east-1d, us-east-1b.

Googling around looks like it's a common problem but mightn't have any immediate outcomes. Perhaps some documentation on what to do if people run into this scenario?

Because https://github.com/buildkite/elastic-ci-stack-for-aws/blob/418fa67a948c0a86b318ff7dc94d4750510201a1/packer/conf/buildkite-agent/scripts/fix-buildkite-agent-builds-permissions is run within each build, but is run from every build's environment hook, you can get nasty clashes resulting in:

chown: changing ownership of '/var/lib/buildkite-agent/builds/buildkite-i-cd2747fc-8/myorg/myapp/buildkite-script-5a7f015d-4583-4bfd-8371-e947b35437c8': No such file or directory

For example, this was a job on buildkite-i-cd2747fc-10 trying to chown a job running on buildkite-i-cd2747fc-8.

This should be scoped to $BUILDKITE_BUILD_CHECKOUT_PATH

Sometimes stack creation fails with:

Value (us-east-1a) for parameter AvailabilityZones is invalid. Subnets can currently only be created in the following availability zones...

This is because if no AvailabilityZones is provided, the list of all availability zones is used, but only of these can have subnets created. Because each account is different (the availability zones are randomly named), a general solution is very difficult.

https://stackoverflow.com/questions/21390444/is-there-a-way-for-cloudformation-to-query-available-zones-for-subnet-creation

The correct terminology for the agent releases should be beta, not unstable.

Is it possible to simply launch a prepared AMI using this stack?

If builds run docker commands and mount in volumes, files are written as root, due to Docker's root-only-ness.

This means that subsequent builds fail with permission errors because they can't git clean the files.

Can you guys release the new version with global env support?

Thanks

Love the set up. One thing I was hoping to do was add some extra parameters to the cloud formation template to pass though to the individual buildkite agents, for consistency with our existing agents.

eg. we user an os=osx and os=linux agent parameter.

We could deduce this from the existing parameters you create for us, but it'd be great to tweak them as well.

I've tweaked the generated CF JSON to get this work, but I'm not sure where to edit the original templates to get this working. I assume around here in the buildkite-elastic.yml

I'm happy to have a crack if there's an easy way to test I haven't broken anything... on condition that I get to add devops to my LinkedIn ;)

For our application our main app requires 27 agents, but other pipelines only require 1 or 2. The only way we can make the stack responsive at the start of the day is to make the scaling policy +27 -27 scale out/in, but it's a waste of $ to do that for just a couple of scheduled jobs.

It'd be much better if we could use stepped scaling, and have it scale in/out in smaller increments

Currently the best way to handle a large app that takes ages to build is to have two separate instances of the stack: builders and runners, and https://github.com/buildkite-plugins/docker-compose-buildkite-plugin to have a build step that runs before the steps.

The builder doesn't autoscale, sits there with a warm docker cache, and pushes to a registry. The test steps run on the runners queue, and pulls from docker hub without doing any git clone at all.

The pipeline looks like this:

steps:

  - name: ":docker: :package:"
    plugins:
      docker-compose:
        build: app
        image-repository: index.docker.io/myorg/myapp
    agents:
      queue: elastic-builders

  - wait

  - name: ":spec: %n"
    command: ".buildkite/steps/knapsack"
    plugins:
      docker-compose:
        run: app
    agents:
      queue: elastic-runners
    parallelism: 75

A simpler setup would be to use something like http://blog.runnable.com/post/145362675491/distributing-docker-cache-across-hosts to make the cache mostly invisible, and require no separate queue or pipeline config changes.

A few bugs / tough things I've encountered:

• "Must specify a region" when using AWS_ECR_LOGIN=true. I got around this by force setting the env var AWS_DEFAULT_REGION.
• ECR Permissions. I continually get:

An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts::053722134931:assumed-role/buildkite-IAMRole-XXX is not authorized to perform: ecr:GetAuthorizationToken on resource: *

I think either more documentation around ECR or having the CloudFormation script add an ECR would make this way easier.

Hi there,

I notice that the AuthorziedSshUsers parameter can take a url or an s3 key, and the cfn-init script checks if it starts with s3:\\ to determine if its a curl or a aws s3 cp. Can this same logic be applied to the bootstrap script as well? I want to run a script and would put it in s3 but use an instance profile to allow access to it...

Many thanks.

Especially in the morning, you may see: ERROR: denied: Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one. When this happens, it's an agent that has been running for more than a day.

Here are some excerpts from a build that had this happen. I modified it and removed parts so let me know if this is sufficient (removed any identifying characteristics of what this job is doing).

$ /etc/buildkite-agent/hooks/environment
Setting up the environment	1s
Sourcing CloudFormation environment...
 
Starting an SSH Agent...
Agent pid 3354

Running global pre-command hook	0s
$ /etc/buildkite-agent/hooks/pre-command
Running build script	1s
$ docker-compose -f docker-compose.yml down
docker-compose -f docker-compose.yml run our_command bash -c "command is here"
docker-compose -f docker-compose.yml down
Removing network daily_default
WARNING: Network daily_default not found.
Creating network "daily_default" with the default driver
Creating daily-server_1
Pulling container_name (container details)...
ERROR: denied: Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one.

We are using aws ci stack with multiple agents per instance. It looks like the buildkite scheduler picks (seemingly) random agents from the pool of instances when picking containers for builds. This seems to result in the auto-scaler having more capacity than needed because even if we are only using 20 agents, they are spread out across all 10 instances, each instance having 2 agents being used.

Preferably the scheduler would pick agents from the instances that are already being used so that the other instances can scale down and save $$.

Right now, you must define your SecretsBucket as part of the CloudFormation configuration and only then are you able to override it with an environment variable.

It would be great to be able to set this using only environment variables, so that I could use the default CloudFormation stack and only need to use buildkite to configure this.

It appears that #167 has introduced a bug to and now managed policies no longer work.

Stack updates with a comma delimited list now produce this error:
Value of property ManagedPolicyArns must be of type List of String

I have a copy of a prior stack and the diff between them looks like this:

$ diff -uN stack.json buildkite-stack.json
--- stack.json	2016-11-18 14:37:18.000000000 +1100
+++ buildkite-stack.json	2016-11-22 09:06:49.000000000 +1100
@@ -113,17 +113,36 @@
     "IAMRole": {
       "Type": "AWS::IAM::Role",
       "Properties": {
-        "ManagedPolicyArns": {
-          "Fn::If": [
-            "UseSpecifiedIamPolicies",
-            {
-              "Ref": "ManagedPolicyArns"
-            },
-            {
-              "Ref": "AWS::NoValue"
-            }
-          ]
-        },
+        "ManagedPolicyArns": [
+          {
+            "Fn::If": [
+              "UseSpecifiedIamPolicies",
+              {
+                "Ref": "ManagedPolicyArns"
+              },
+              {
+                "Ref": "AWS::NoValue"
+              }
+            ]
+          },
+          {
+            "Fn::If": [
+              "UseECR",
+              {
+                "Fn::FindInMap": [
+                  "ECRManagedPolicy",
+                  {
+                    "Ref": "ECRAccessPolicy"
+                  },
+                  "Policy"
+                ]
+              },
+              {
+                "Ref": "AWS::NoValue"
+              }
+            ]
+          }
+        ],

This looks to be the cause:
https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/templates/buildkite-elastic.yml#L242

ManagedPolicies changes from a hash to an array.

cc:
@lox

It's possible this will be resolved by #57, but... I also have some instances which are no longer running the Buildkite agent, but are still part of the ASG. I found one of these and grabbed this from its logs:

May  6 17:31:51 ip-10-0-1-24 lifecycled: #033[34m  INFO#033[0m[0368] Received message          #033[34minstanceid#033[0m=i-8e00f214 #033[34mtransition#033[0m=autoscaling:EC2_INSTANCE_TERMINATING
May  6 17:31:51 ip-10-0-1-24 lifecycled: #033[34m  INFO#033[0m[0368] Executing handler         #033[34mhandler#033[0m=/usr/bin/buildkite-lifecycle-handler #033[34minstanceid#033[0m=i-8e00f214 #033[34mtransition#033[0m=autoscaling:EC2_INSTANCE_TERMINATING
May  6 17:31:51 ip-10-0-1-24 lifecycled: stopping buildkite-agent gracefully
May  6 17:31:52 ip-10-0-1-24 lifecycled: Stopping buildkite-agent...
May  6 17:31:52 ip-10-0-1-24 lifecycled: Stopped
May  6 17:31:52 ip-10-0-1-24 lifecycled: buildkite-agent stopped!
May  6 17:31:52 ip-10-0-1-24 lifecycled: #033[34m  INFO#033[0m[0370] Handler finished successfully #033[34mhandler#033[0m=/usr/bin/buildkite-lifecycle-handler #033[34minstanceid#033[0m=i-8e00f214 #033[34mtransition#033[0m=autoscaling:EC2_INSTANCE_TERMINATING
May  6 17:31:52 ip-10-0-1-24 lifecycled: #033[34m  INFO#033[0m[0370] Lifecycle action completed successfully #033[34minstanceid#033[0m=i-8e00f214 #033[34mtransition#033[0m=autoscaling:EC2_INSTANCE_TERMINATING

Looks like it thinks it terminated correctly?

buildkite-elastic.yml includes this in the UserData script:

/usr/local/bin/cfn-signal -e \$? -r 'cfn-init finished' \
  --stack $(AWS::StackName) --resource 'AgentAutoScaleGroup' --region $(AWS::Region)

However, /usr/local/bin/cfn-signal doesn't exist:

Cloud-init v. 0.7.7 running 'modules:config' at Tue, 07 Jul 2015 00:56:00 +0000. Up 50.92 seconds.
#!/bin/bash -xv
/usr/local/bin/cfn-init -s arn:aws:cloudformation:us-east-1:349538293217:stack/ci-20150706-175140/55799120-2442-11e5-b551-50fa1dbb2c64 -r AgentLaunchConfiguration --region us-east-1
+ /usr/local/bin/cfn-init -s arn:aws:cloudformation:us-east-1:349538293217:stack/ci-20150706-175140/55799120-2442-11e5-b551-50fa1dbb2c64 -r AgentLaunchConfiguration --region us-east-1
/var/lib/cloud/instance/scripts/part-001: line 2: /usr/local/bin/cfn-init: No such file or directory
/usr/local/bin/cfn-signal -e $? -r 'cfn-init finished' \
  --stack ci-20150706-175140 --resource 'AgentAutoScaleGroup' --region us-east-1
+ /usr/local/bin/cfn-signal -e 127 -r 'cfn-init finished' --stack ci-20150706-175140 --resource AgentAutoScaleGroup --region us-east-1
/var/lib/cloud/instance/scripts/part-001: line 3: /usr/local/bin/cfn-signal: No such file or directory

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=15.04
DISTRIB_CODENAME=vivid
DISTRIB_DESCRIPTION="Ubuntu 15.04"

$ curl http://169.254.169.254/latest/meta-data/ami-id ; echo
ami-55986a3e

When AWS Stack goes to create the MetricsStack instance, it uses the Default VPC, instead of the VpcId defined in the properties (or the VPC is creates for you). If the AWS Account doesn't have a Default VPC, the cloud formation script exits with a cryptic error.

Thanks @ryanbigg for helping us find this problem!

I tried to set BUILDKITE_SECRETS_BUCKET with a new bucket name "buildkite-pipelines-secrets" but apparently the old bucket name (SecretsBucket property set in cloudformation) is still used by the pipeline to look for a private key. This can be verified in the pipeline log and environment variables.

1. click the failed pipeline

2. select the Log tab. The log shows
   "Downloading secrets...
   No SSH keys found in s3://{old bucket name}"

3. go to the "Environment" tab, find that
   BUILDKITE_SECRETS_BUCKET="buildkite-pipelines-secrets"

The current rules seem to give a hard scale up to max, and a hard scale down to min when idle. This might fit a typical workday in one office, but because we have a few developers in different timezones using the elastic stack, this doesn't really fit because we have a lot of unused resources when this hard scale up occurs in the middle of the night. I'd prefer a more reactive stack that could scale to fit the demand at the time.

Typically the way I've used autoscaling is to peg average CPU utilisation, adding instances at 60%, and removing instances at 40%. Agents are a little different as we've literally got a queue being processed, which means our stack stats are very spiky. Builds that come in spawn multiple jobs, so the RunningJobsCount and UnfinishedJobs bounces around quite a bit.

So the metric I think will give us better results would be a "agents busy" percentage averaged over 5 minutes or so. Then we could trigger a more gentle scale up maybe at 70%, and a gentle scale down at maybe 20%. Hopefully that would give the elastic stack room to bounce around between 20-70%, but will nudge the cluster size in the right direction once utilisation increases or decreases past those points. And we would keep the existing scale up trigger if the ScheduledJobsCount rises above 0 to make sure we scale in the first place and on demand.

Thoughts @toolmantim @lox ?

When a scale down happens I'm finding that a job is aborting mid-test-run with ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?, and the last thing in the buildkite-agent CloudWatch logs is "Gracefully shutting down" but no more logs, but the job successfully uploads artifacts etc after failing, and gracefully shuts down via our API, but is missing any more log lines in CloudWatch.

My guess is that we're not preventing awslogs and docker daemons being shutdown in the autoscale terminating event. lifecycled appears only to be helping buildkite-agent from being shut down.

When performing a stack update with running instances it's very common for it to fail, making for a long 50 minutes. #126 was meant to help this, but it appears not really to help at all, but only to make the wait longer.

The stack event log:

21:07:41 UTC+10 UPDATE_ROLLBACK_COMPLETE  AWS::CloudFormation::Stack  elastic-runners 
21:07:41 UTC+10 UPDATE_COMPLETE AWS::CloudFormation::Stack  MetricsStack  
21:07:30 UTC+10 DELETE_COMPLETE AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  
21:07:29 UTC+10 UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  MetricsStack  
21:07:29 UTC+10 DELETE_IN_PROGRESS  AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  
21:07:26 UTC+10 UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS  AWS::CloudFormation::Stack  elastic-runners 
21:07:25 UTC+10 UPDATE_COMPLETE AWS::CloudFormation::Stack  MetricsStack  
21:07:02 UTC+10 UPDATE_COMPLETE AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup 
21:06:58 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-2e3ce11f
21:06:58 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-293ce118
21:06:58 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-3502df04
21:06:58 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-0b77820a
21:06:58 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-c07580c1
21:04:52 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup New instance(s) added to autoscaling group - Waiting on 5 resource signal(s) with a timeout of PT15M.
21:04:52 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Successfully terminated instance(s) [i-2ae73a1b,i-64d52065,i-278b2026,i-73e43942,i-ddd520dc] (Progress 100%).
21:04:50 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Terminating instance(s) [i-2ae73a1b,i-64d52065,i-278b2026,i-73e43942,i-ddd520dc]; replacing with 5 new instance(s).
21:04:49 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Temporarily setting autoscaling group MinSize and DesiredCapacity to 10.
21:04:49 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Rolling update initiated. Terminating 5 obsolete instance(s) in batches of 5. Waiting on resource signals with a timeout of PT15M when new instances are added to the autoscaling group.
21:04:45 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup 
21:04:43 UTC+10 UPDATE_COMPLETE AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  
21:04:42 UTC+10 UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  MetricsStack  
21:04:36 UTC+10 UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack  elastic-runners The following resource(s) failed to update: [AgentAutoScaleGroup].
21:04:34 UTC+10 UPDATE_FAILED AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Failed to receive 5 resource signal(s) within the specified duration
20:24:31 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup New instance(s) added to autoscaling group - Waiting on 5 resource signal(s) with a timeout of PT40M.
20:24:30 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Successfully terminated instance(s) [i-47e93476,i-3fed183e,i-6bb61d6a,i-58ea3769,i-4ffa0f4e] (Progress 50%).
20:24:29 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Terminating instance(s) [i-47e93476,i-3fed183e,i-6bb61d6a,i-58ea3769,i-4ffa0f4e]; replacing with 5 new instance(s).
20:24:28 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-278b2026
20:24:28 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-73e43942
20:24:28 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-64d52065
20:24:28 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-2ae73a1b
20:24:06 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Received SUCCESS signal with UniqueId i-ddd520dc
20:20:14 UTC+10 UPDATE_COMPLETE AWS::CloudFormation::Stack  MetricsStack  
20:18:16 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup New instance(s) added to autoscaling group - Waiting on 5 resource signal(s) with a timeout of PT40M.
20:18:15 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Temporarily setting autoscaling group MinSize and DesiredCapacity to 10.
20:18:15 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup Rolling update initiated. Terminating 10 obsolete instance(s) in batches of 5. Waiting on resource signals with a timeout of PT40M when new instances are added to the autoscaling group.
20:18:11 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  AgentAutoScaleGroup 
20:18:04 UTC+10 UPDATE_COMPLETE AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  
20:18:03 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  Resource creation Initiated
20:18:02 UTC+10 UPDATE_IN_PROGRESS  AWS::AutoScaling::LaunchConfiguration AgentLaunchConfiguration  Requested update requires the creation of a new physical resource; hence creating one.
20:17:56 UTC+10 UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  MetricsStack  
20:17:48 UTC+10 UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  elastic-runners User Initiated

The ASG log:

21:04:57 UTC+10 Successful Launching a new EC2 instance: i-2e3ce11f
21:04:57 UTC+10 Successful Launching a new EC2 instance: i-293ce118
21:04:57 UTC+10 Successful Launching a new EC2 instance: i-3502df04
21:04:57 UTC+10 Successful Launching a new EC2 instance: i-0b77820a
21:04:57 UTC+10 Successful Launching a new EC2 instance: i-c07580c1
21:04:51 UTC+10 Successful Terminating EC2 instance: i-64d52065
21:04:51 UTC+10 Successful Terminating EC2 instance: i-2ae73a1b
21:04:51 UTC+10 Successful Terminating EC2 instance: i-ddd520dc
21:04:51 UTC+10 Successful Terminating EC2 instance: i-73e43942
21:04:50 UTC+10 Successful Terminating EC2 instance: i-278b2026
20:24:29 UTC+10 Successful Terminating EC2 instance: i-3fed183e
20:24:29 UTC+10 Successful Terminating EC2 instance: i-47e93476
20:24:29 UTC+10 Successful Terminating EC2 instance: i-4ffa0f4e
20:24:29 UTC+10 Successful Terminating EC2 instance: i-58ea3769
20:24:29 UTC+10 Successful Terminating EC2 instance: i-6bb61d6a
20:22:00 UTC+10 Successful Launching a new EC2 instance: i-2ae73a1b
20:22:00 UTC+10 Successful Launching a new EC2 instance: i-278b2026
20:22:00 UTC+10 Successful Launching a new EC2 instance: i-73e43942

Our code that reports if the instance came up successfully is: