Giter Club home page Giter Club logo

aws-devops-essential's Introduction

AWS DevOps Essentials

An Introductory Workshop on CI/CD Practices

In few hours, quickly learn how to effectively leverage various AWS services to improve developer productivity and reduce the overall time to market for new product capabilities. In this session, we will demonstrate a prescriptive approach to incrementally adopt and embrace some of the best practices around continuous integration & delivery using AWS Developer Tools and 3rd party solutions including, AWS CodeCommit (a managed source control service), AWS CodeBuild (a fully managed build service), Jenkins (an open source automated build server), CodePipeline (a fully managed continuous delivery service), and CodeDeploy (an automated application deployment service). We will also highlight some best practices and productivity tips that can help make your software release process fast, automated, and reliable.

See the diagram below for a depiction of the complete architecture.

DevOps Workshop Architecture

Prerequisites

  • IAM Permissions: Ensure your user has sufficient privileges in the AWS account of your choice. You must have permissions for the following services:

AWS Identity and Access Management

Amazon Simple Storage Service

AWS CodeCommit

AWS CodeBuild

AWS CloudFormation

AWS CodeDeploy

AWS CodePipeline

AWS Cloud9

Amazon EC2

Amazon SNS

Amazon CloudWatch Events


Important:

Preferred regions for lab

  • North Virginia US-EAST-1
  • Oregon US-WEST-2

If you want to your region choice for the lab. Kindly the select the region which has all four Code* services and Cloud9 service. You can find the region services list. Stick to the same region throughout all labs. Make sure you have not reached the VPC or Internet Gateway limits for that region. If you already have 5 VPCs/IGWs, delete at least one before you proceed or choose an alternate region.

Labs

This workshop is broken into multiple labs. You must complete each Lab before proceeding to the next.

  1. Lab 1 - Build project on the cloud
  2. Lab 2 - Automate deployment for testing
  3. Lab 3 - Setup CI/CD using AWS CodePipeline
  4. Lab 4 (Optional) - Using Lambda as Test Stage in CodePipeline

Clean up

  1. Visit CodePipeline console, select the created pipeline. Select the Edit and click Delete.
  2. Visit CodeDeploy console, select the created application. In the next page, click Delete Application.
  3. Visit CodeBuild console, select the created project. Select the Action and click Delete.
  4. Visit CodeCommit console, select the created repository. Go to setting and click Delete repository.
  5. Visit Lambda console, select the created function. Select the Action and click Delete.
  6. Visit Cloudformation console, select the created stacks. Select the Action and click Delete Stack.
  7. Visit Cloud9 console, select the created Environment. Select the Action and click Delete.
  8. Visit Simple Notification Service console, select Topics. Select the created topic. Select the Action and click Delete topics. Next select Subscriptions. Select the created subscription. Select the Action and click Delete subscriptions.

License

This library is licensed under the Apache 2.0 License.

aws-devops-essential's People

Contributors

ashishgore avatar carlosafonso avatar dennydaugherty avatar erikbryant avatar jejikenwogu avatar kakakakakku avatar karthiksambandam avatar karthiksambandam-zz avatar leozhad avatar nvaidya1 avatar tbrown-wordstream avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-devops-essential's Issues

Bugs in Lab1 and Lab2 buildspec.yml

Was able to complete Lab 1 and Lab 2 but there are a couple of incorrect settings in the artifacts block that caused build and deploy to fail, respectively.

Are these intentional as a troubleshooting exercise to the user?

Lab1 > Stage 5 > buildspec.yml

   discard-paths: no  

FIX: discard-paths should be yes to work.

Lab2 > Stage 3 > buildspec.yml

artifacts:
  files:
    - appspec.yml
    - scripts/**/*
    - target/javawebdemo.war

FIX: scripts/**/* should be scripts/* to be included in the bundle (otherwise the scripts are missing in CodeDeploy).

Create kubernetes cluster - step now produces error

Is there a more current version of this workshop code? This uses K8s 1.10, unsupported?

aws eks create-cluster \
>   --name k8s-workshop \
>   --role-arn $EKS_SERVICE_ROLE \
>   --resources-vpc-config subnetIds=${EKS_SUBNET_IDS},securityGroupIds=${EKS_SECURITY_GROUPS} \
>   --kubernetes-version 1.10

An error occurred (InvalidParameterException) when calling the CreateCluster operation: unsupported Kubernetes version

AWS SDK dependency is old, causing regional failures for CMH and possibly others

The project currently has a dependency on 1.10.10 in the pom.xml; CMH wasn't released until 1.11.44. If you deploy to CMH you will get a failure in ValidateService and browsing to the application will return a 500 error referencing the fact that no region was selected.

https://aws.amazon.com/releasenotes/release-aws-sdk-for-java-1-11-44/?tag=releasenotes%23keywords%23java

This is easily resolved by modifying the pom.xml with support for the latest sdk (1.11.319). The concern here is that when a user gets to the point of failure they are pretty far along in the exercise and changing the dependency at this point is a little more work on clean-up and outside the scope of training, catching it before this phase is much easier.

Suggestion for clearer instructions

Lab 4, stage 2:

For User parameters, specify the Public DNS address for the Amazon EC2 DevWebApp01 instance you copied earlier (for example, http://ec2-52-62-36-220.ap-southeast-2.compute.amazonaws.com), and then choose Add action.

Please consider changing to:
For User parameters, specify **http://** and the Public DNS address for the Amazon EC2 DevWebApp01 instance you copied earlier (for example, http://ec2-52-62-36-220.ap-southeast-2.compute.amazonaws.com), and then choose Add action.

The Lambda function fails without this, requiring cycling through the entire pipeline again. Yes, the example shows http://, but explicitly calling it out would help.

Thanks

Can't access files on https://s3.amazonaws.com/devops-workshop-0526-2051/v1/

Lab 1, stage 3 asks to download https://s3.amazonaws.com/devops-workshop-0526-2051/v1/Web-App-Archive.zip, but I receive a 403 - Access Denied error. I worked around it using the file https://github.com/awslabs/aws-devops-essential/raw/master/sample-app/Web-App-Archive.zip, but then at the Stage 4 the aws cloudformation create-stack fails with
Error parsing parameter '--template-body': Unable to retrieve https://s3.amazonaws.com/devops-workshop-0526-2051/v1/01-aws-devops-workshop-roles.template: received non 200 status code of 403.
It seems an issue with the bucket holding the files needed by the workshop.

Choosing Docker runtime via inserting build command in Buildspec

During this step in lab:

Add the Build stage:
...
    Select runtime as Docker

I don't have an option for Docker, so build fails. To get this working, From AWS console I needed to:
In CodeBuild edit the eks-build-project, and choose Edit Buildspec, and chose insert build commands
I changed lines 12-17 to:

phases:
  install:
    runtime-versions:
      docker: 18

Consider wrapping all of this in a new VPC

A lot of people may not realize what resources do or do not exist in their current environments provided by their company. Might be worth creating a new VPC for all of the resources in this workshop as a first step. This could have the added benefit of making the cleanup process a bit easier at the end as well. Thanks for the well-written tutorial and materials!

No Outputs in CloudFormation for Stack DevopsWorkshop-roles

This is a continuation of Issue #8.

I decided to completely start over. This time I created an new IAM user for AWS Labs, using the Aministrative IAM on this account, which I was using last time. I delete everything that I had created in us-west-2, following the direction at the bottom of the main page. I got to Lab 1 Step 4. After running

DevOps_Labs:~/environment/WebAppRepo (master) $ aws cloudformation create-stack --stack-name DevopsWorkshop-roles \
--template-body https://github.com/awslabs/aws-devops-essential/raw/master/templates/01-aws-devops-workshop-roles.template \
--capabilities CAPABILITY_IAM
{
    "StackId": "arn:aws:cloudformation:us-west-2:539564704580:stack/DevopsWorkshop-roles/036dd070-734b-11e8-b47f-50a68a0bca36"
}

I opened CloudFormation looking for BuildRoleArn-Value-FROM-CLOUDFORMATION-OUTPUT, but no Output was found, see attached.
nooutputfordevopsworkshop-roles

I repeated the process of deleting the DevopsWorkshop-roles stack, and got the same results.

Permissions listed in README.md are not accurate

Hi guys, you're suggesting adding several permissions to a newly create IAM user, however the permissions listed do not 100% line up with what we have available in AWS IAM console.

Here's the summary of the permissions I'm applying, but I'm not sure if that's the best choice

screen shot 2018-11-29 at 1 39 07 pm

Inability to Stop and Start an DevWebApp01 and ProdWebApp01 Instances

At the end of the AWS DevOps Day class last week, I stop the DevWebApp01 and ProdWebApp01 instances that I creased in labs 1 and 2. When I got home, I started them and started on Lab #3. In Stage 1 step 9 the Staging event fails because the ApplicationStop Event failed on the DevWebApp01 instance here is the log:

Error CodeScriptFailed
Script Namescripts/stop_application
MessageScript at specified location: scripts/stop_application failed with exit code 1
Log TailLifecycleEvent - ApplicationStop
Script - scripts/stop_application

I stopped the server and ran it again, this also failed.

Sam Hennessy replied to my e-mail, and suggested that "easiest way to troubleshoot is to sign on to the machine, and look into the /opt/code-deploy directory. There will be a folder that matches up with the code deploy deployment. You can take a look in that folder, and run the script that seems to be failing." I soon discovered that I was not able to login the instances, because I do not have the Key Pair for the DevWebApp01 and ProdWebApp01 instances. I check my download folder and no PEN file was download during the tutorial.

Can you please make the default server creation methods, use a Key Pair that the student already has or creates inappropriate lab, thus allow in the student access to the server such that they can examine it. This may not be appropriate for a fully optimized DevOps system, but for someone trying to learn the system it would be highly beneficial.

Sincerely yours,
Marc Whitlow

Type

  1. Infrastructure is misspelled in the tutorial as Insfrastrture in Additional Exercises.
    "Stage 1: Pipeline + Infrastrture as code"

  2. Add the following step in your $git commands so that you don't have to enter gnarly user id/password every time.
    Store your git credentials:
    $ git config credential.helper store
    // thanks to the instructor

Lab5 ~ Stage2 - Step8 - Deploying Java "webapp" application issue

Hello team,

Hope you're doing well. I was setting this AWS DevOps Lab on my personal account, and during the Lab5, Stage2, Step8, I'm getting the following error while trying to deploy java application "webapp". Looks like the mirror for installing Apache Tomcat is no longer valid:

2022-08-24T03:20:32.801-05:00 | Step 8/22 : RUN java -version
| 2022-08-24T03:20:32.801-05:00 | ---> Running in f13afa63f92e
| 2022-08-24T03:20:36.338-05:00 | �[91mopenjdk version "1.8.0_342"
| 2022-08-24T03:20:36.338-05:00 | OpenJDK Runtime Environment (build 1.8.0_342-b07�[0m�[91m)
| 2022-08-24T03:20:36.338-05:00 | OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode)
| 2022-08-24T03:20:36.338-05:00 | �[0mRemoving intermediate container f13afa63f92e
| 2022-08-24T03:20:36.338-05:00 | ---> 14062452e9e2
| 2022-08-24T03:20:36.338-05:00 | Step 9/22 : RUN wget http://mirror.linux-ia64.org/apache/tomcat/tomcat-7/v7.0.96/bin/apache-tomcat-7.0.96.tar.gz && tar -xvf apache-tomcat-7.0.96.tar.gz && rm apache-tomcat*.tar.gz && mv apache-tomcat* ${CATALINA_HOME}
| 2022-08-24T03:20:36.338-05:00 | ---> Running in a1bf2b989760
| 2022-08-24T03:20:36.338-05:00 | �[91m--2022-08-24 08:20:33-- http://mirror.linux-ia64.org/apache/tomcat/tomcat-7/v7.0.96/bin/apache-tomcat-7.0.96.tar.gz
| 2022-08-24T03:20:36.338-05:00 | Resolving mirror.linux-ia64.org (mirror.linux-ia64.org)... �[0m�[91m89.189.177.241, 37.193.156.169
| 2022-08-24T03:20:36.338-05:00 | Connecting to mirror.linux-ia64.org (mirror.linux-ia64.org)|89.189.177.241|:80... �[0m�[91mconnected.
| 2022-08-24T03:20:36.338-05:00 | HTTP request sent, awaiting response... �[0m�[91m404 Not Found
| 2022-08-24T03:20:36.338-05:00 | �[0m�[91m2022-08-24 08:20:33 ERROR 404: Not Found.
| 2022-08-24T03:20:36.338-05:00 |  
| 2022-08-24T03:20:36.338-05:00 | �[0mThe command '/bin/sh -c wget http://mirror.linux-ia64.org/apache/tomcat/tomcat-7/v7.0.96/bin/apache-tomcat-7.0.96.tar.gz && tar -xvf apache-tomcat-7.0.96.tar.gz && rm apache-tomcat*.tar.gz && mv apache-tomcat* ${CATALINA_HOME}' returned a non-zero code: 8
| 2022-08-24T03:20:36.338-05:00 |  
| 2022-08-24T03:20:36.338-05:00 | [Container] 2022/08/24 08:20:34 Command did not exit successfully docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . exit status 8
| 2022-08-24T03:20:36.338-05:00 | [Container] 2022/08/24 08:20:34 Phase complete: BUILD State: FAILED
| 2022-08-24T03:20:36.338-05:00 | [Container] 2022/08/24 08:20:34 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .. Reason: exit status 8
| 2022-08-24T03:20:36.338-05:00 | [Container] 2022/08/24 08:20:34 Entering phase POST_BUILD
| 2022-08-24T03:20:36.338-05:00 | [Container] 2022/08/24 08:20:34 Running command echo Build completed on date
| 2022-08-24T03:20:36.338-05:00 | Build completed on Wed Aug 24 08:20:34 UTC 2022

Cloud9 CLI Command executed

carlos.lemus:~/environment $ ecs-cli compose --project-name ecsdemo-javawebapp service up --timeout 15 --create-log-groups --target-group-arn $target_group_arn --private-dns-namespace service --enable-service-discovery --container-name ecsdemo-javawebapp --container-port 8080 --cluster-config ecs-demo --vpc $vpc
INFO[0000] Using ECS task definition TaskDefinition="ecsdemo-javawebapp:4"
WARN[0000] Failed to create log group ecsdemo-javawebapp in us-east-1: The specified log group already exists
WARN[0000] Service Discovery can not be enabled on an existing ECS Service. Skipping this flag...
WARN[0000] You cannot update the load balancer configuration on an existing service. serviceName=ecsdemo-javawebapp
INFO[0000] Updated ECS service successfully desiredCount=1 force-deployment=false service=ecsdemo-javawebapp
INFO[0025] (service ecsdemo-javawebapp) has started 1 tasks: (task c893e6cba06c4045bb54be5865d3f321). timestamp="2022-08-24 08:36:53 +0000 UTC"
INFO[0086] (service ecsdemo-javawebapp) has started 1 tasks: (task aa8684ed1b794d06b15633a6727404b3). timestamp="2022-08-24 08:37:52 +0000 UTC"
INFO[0146] (service ecsdemo-javawebapp) has started 1 tasks: (task fdb4a3cb2c0c483a8e9a7a8f8d798f3f). timestamp="2022-08-24 08:38:53 +0000 UTC"
INFO[0487] (service ecsdemo-javawebapp) has started 1 tasks: (task d3b7cc10c4324cb6994cc278c68b41bc). timestamp="2022-08-24 08:44:33 +0000 UTC"
ERRO[0758] Error describing service error="ExpiredTokenException: The security token included in the request is expired\n\tstatus code: 400, request id: 376c5a6e-d73a-4822-8c87-c36457ce069e" service=ecsdemo-javawebapp
FATA[0758] ExpiredTokenException: The security token included in the request is expired
status code: 400, request id: 376c5a6e-d73a-4822-8c87-c36457ce069e

Stopped reason CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "docker.io/javawebapp/latest:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

NOTE: I ensured myself to cover the issue by adding permissions to the repository "javawebappdemo" just in case, but still getting the same problem with the URL mirror for installing Apache Tomcat. The error message "ExpiredTokenException" is due to multiple times the java app deployment process was tried, so this should be expected after all.
Those warning messages showing in the Cloud9 CLI command output is expected. This should not affect the process of deploying the java application. The whole lab was retried, but still getting the same results.

IMPORTANT: Most of the roles that were deployed are facing issues by not declaring the version of the json file, actions by not having an applicable resource, and actions not support resource level permissions, which requires a wildcard for the resource. It's worth it to update those roles, in order to avoid updating them manually (which this was my case). I'll ensure to upload the updates done on every role. Hopefully, this contributes for the community and all audience.

Any feedback or guidance is appreciated. Thanks in advance.
Carlos Lemus

Role01 - Before
Role01 - After
Role02 - Before
Role02 - After

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.