aws-samples / jenkins-on-aws Goto Github PK

License: MIT No Attribution

Python 89.95% Dockerfile 3.00% Shell 1.51% Jinja 5.54%

jenkins-on-aws's Introduction

Jenkins on AWS

What is this?

This project will build and deploy an immutable, fault tolerant, and cost effective Jenkins environment in AWS using ECS. All Jenkins images are managed within the repository (pulled from upstream) and fully configurable as code. Plugin installation is automated, including versioning, as well as configured through the Configuration as Code plugin.

Why

Managing Jenkins is hard. From plugin management and configuration details, to ensuring that the nodes are up to date and running, it's a lot of work and susceptible to issues.

Please note, this is an example and not production ready. You are still responsible for securing Jenkins (enabling TLS at the ALB, adding authentication, etc). This intent of this project is to provide an example of running Jenkins on ECS.

Architecture

Project Structure

./

./cdk/ <-- Deployment configuration

./cdk/docker/ <-- Dockerfiles and associated config files for Leader and Worker images. The cdk will build these images on a cdk deploy.

./cdk/app.py <-- cdk application file where all stacks are instantiated and built

./cdk/requirements.txt <-- Python module requirements

./cdk/jenkins/ <-- cdk stacks to deploy Jenkins environment

Requirements

To deploy this environment, we will use the aws-cdk

Please follow the requirements to install from the cdk github repo
Tested with the following version: 1.53.0 (build 6c326cb)

Fargate Jenkins (Leader and Workers)

Set config.ini

fargate_enabled = yes

EC2 Backed Leader and Fargate Workers

Set config.ini

ec2_enabled = yes

Validate configs and deploy

Navigate to the cdk directory, and run:

cdk synth

Output should look something like:

[user@computer cdk (cdk)]$ cdk synth
Successfully synthesized to jenkins-on-aws/cdk/cdk.out
Supply a stack name (JenkinsOnAWSNetwork, JenkinsOnAWSECS, JenkinsOnAWSWorker, JenkinsOnAWSJenkinsLeader) to display its template.

Feel free to check out the CloudFormation templates created by the cdk in the cdk.out directory

Let's deploy the environment! The below command will deploy all of the stacks required to get the environment up and running:

cdk deploy Jenkins*

Note: You will be prompted for approval during the stages of the deploy. Follow the instructions on the prompt when asked.

That's it! You now have a fully automated Jenkins implementation running on AWS Fargate with worker nodes automatically configured to run on an as needed basis.

jenkins-on-aws's People

Contributors

Stargazers

Watchers

jenkins-on-aws's Issues

InvalidParameterException: The new ARN and resource ID format must be enabled to work with ECS managed tags. Opt in to the new format and try again

Hello!

I received the following error when attempting to deploy:

The new ARN and resource ID format must be enabled to work with ECS managed tags. Opt in to the new format and try again. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException;)
Full error log:

 18/19 | 2:12:44 PM | CREATE_FAILED        | AWS::ECS::Service                         | JenkinsMasterService/Service/Service (JenkinsMasterService4ADB0000) The new ARN and resource ID format must be enabled to work with ECS managed tags. Opt in to the new format and try again. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 70fd53d5-20d6-4c6d-8220-b6a28e2887c4)
	new BaseService (/private/var/folders/fj/yy0y1r6n0vxgpfxjzpqvtdqc0000gn/T/jsii-kernel-PQtaNw/node_modules/@aws-cdk/aws-ecs/lib/base/base-service.js:110:25)
	\_ new FargateService (/private/var/folders/fj/yy0y1r6n0vxgpfxjzpqvtdqc0000gn/T/jsii-kernel-PQtaNw/node_modules/@aws-cdk/aws-ecs/lib/fargate/fargate-service.js:35:9)
	\_ new ApplicationLoadBalancedFargateService (/private/var/folders/fj/yy0y1r6n0vxgpfxjzpqvtdqc0000gn/T/jsii-kernel-PQtaNw/node_modules/@aws-cdk/aws-ecs-patterns/lib/fargate/application-load-balanced-fargate-service.js:49:24)
	\_ /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7762:49
	\_ Kernel._wrapSandboxCode (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8222:20)
	\_ Kernel._create (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7762:26)
	\_ Kernel.create (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7509:21)
	\_ KernelHost.processRequest (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7296:28)
	\_ KernelHost.run (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:14)
	\_ Immediate._onImmediate (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7239:37)
	\_ processImmediate (internal/timers.js:445:21)

I think this is half a problem of the Jenkins-On-AWS project and the AWS CDK:

It's not documented in this project's README.md that it's required to Opt-In to the new ARN/ID format (for using ECS Managed Tags)
The CDK assumes the account / principal has opted-in if using enable_ecs_managed_tags=True and it's not readily apparent this is required until deploy-time (when it fails)

The issue, and resolution, is summarized here: aws/aws-cdk#3844 (comment)

For those who come here experiencing the same problem, the resolution is (roughly) below. See the issue above if using CodePipeline for additional fixes.

#!/bin/bash
aws ecs put-account-setting-default --name serviceLongArnFormat --value enabled
aws ecs put-account-setting-default --name taskLongArnFormat --value enabled
aws ecs put-account-setting-default --name containerInstanceLongArnFormat --value enabled
aws ecs put-account-setting-default --name awsvpcTrunking --value enabled  # optional 
aws ecs put-account-setting-default --name containerInsights --value enabled  # optional

I'm happy to submit a PR with the appropriate fix, which may be one of:

Flip the default enable_ecs_managed_tags=True to False
Improve documentation / README: Add a note indicating that the new ARN / ID format Opt-In is required, and add the above manual AWS CLI fix so users are aware of an easy fix without Googling for a possible solution, or resorting to manual actions in the AWS Console.
Add a config.ini setting to allow the user to toggle this prior to deployment (for example:config['DEFAULT']['enable_ecs_managed_tags'])

Is there an accompanied workshop or a tutorial?

If yes I can't see any link in README

JenkinsOnAWSJenkinsLeader failed: failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 15: unknown instruction: APT-GET

failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 15: unknown instruction: APT-GET
[100%] fail: docker build --tag cdkasset-d12f541e7b96471a70fe4ea2a428cab368a57665a9a7b50012e7bdd167191f84 . exited with error code 1: #1 [internal] load build definition from Dockerfile
#1 sha256:9753d503d4088abb969cf72b19530c1376d24569c0b9f1969dbe37ab49f44497
#1 transferring dockerfile: 767B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:35cddb48ade69452f38462f468a179e1e047a4176a35f9e9d402b2cd8605f44e
#2 transferring context: 2B done
#2 DONE 0.0s
failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 15: unknown instruction: APT-GET

❌ JenkinsOnAWSJenkinsLeader failed: Error: Failed to publish one or more assets. See the error messages above for more information.
at Object.publishAssets (/opt/homebrew/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
at Object.deployStack (/opt/homebrew/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:232:3)
at CdkToolkit.deploy (/opt/homebrew/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:184:24)
at initCommandLine (/opt/homebrew/lib/node_modules/aws-cdk/bin/cdk.ts:213:9)
Failed to publish one or more assets. See the error messages above for more information.

503 Errors when using UI

Originally thought this was an issue with Fargate, but even with EC2 as the master 503 pages come up quite often. Back, refresh doesn't work. But if you wait around 60 seconds you can connect to the interface again. Almost seems like a restart is happening in the background.

Improve on the readme.md

For those that aren't familiar with CDK and Python, it would be nice to put this command line on the instructions, because the CDK output a dependency error message if we just type CDK Synth:

Lines to add to the documentation:

1- Install project dependencies:

python -m pip install -r requirements.txt

The runtime parameter of python3.6 is no longer supported for creating or updating AWS Lambda functions

Running cdk deploy Jenkins* --profile DEFAULT on Windows and I get this error message:
The runtime parameter of python3.6 is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (python3.9) while creating or updating functions.

Solution is not working, throwing errors related to plugins installation in the logs

Tried to spin-up the solution as-is but its failing with the below error screen, from looking at the logs looks like plugins are not able to install properly.

CloudWatch Logs

No hudson.slaves.Cloud implementation found for ecs

Opening the URL reported by cdk deploy ... displays Jenkins' Web UI. But it is not usable and only shows an error message with stacktrace:
java.lang.IllegalArgumentException: No hudson.slaves.Cloud implementation found for ecs (complete stacktrace attached).

I was only able to deploy the stack with my fix from #13 - but I do not think that this is the reason for this error in Jenkins.

[Feature] Add a tutorial in README

Presently the README is lacking details on how to deploy (to say the least). The repo would benefit from a tutorial on how to get the environment up and running with a step by step walkthrough.

Modify jinja2 environment set autoescape to True

Jinja2 is a Python HTML templating system. It is typically used to build web applications, though appears in other places well, notably the Ansible automation system. When configuring the Jinja2 environment, the option to use autoescaping on input can be specified. When autoescaping is enabled, Jinja2 will filter input strings to escape any HTML content submitted via template variables. Without escaping HTML input the application becomes vulnerable to Cross Site Scripting (XSS) attacks.

Unfortunately, autoescaping is False by default.

For more information see Bandit documentation here

Default security needed

Security needs to be applied on deployment.

Jenkins Master stack has no output URL

The Jenkins Master stack is missing the output URL for the app endpoint

CDK updates fail when going from Fargate to ec2

Changing the configuration from fargate_enabled = yes to no, and ec2_enabled = no to yes causes cdk deploy to fail.

Cloudformation Errors:
The key pair 'jenkinsonaws' does not exist (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: 4dbf68f0-10bb-11ea-9d3d-dbae4188b871)

Not sure this is really an issue and maybe the advice is when switching from Fargate to EC2 or vice-a-versa should be a destroy, then deploy. Or create a new set of stacks and treat it like a brand new deployment.

Building docker images?

This is continuation of #15

Is there a suggested approach for building Docker images from Jenkins running on Docker? Especially on Fargate.

To run docker inside docker the general approach is to mount /var/run/docker.sock of the host on the container, this is not allowed in Fargate
If we are running docker daemon as root user then we have to run Jenkins also as root user inside docker, this is not recommended
Docker rootless mode needs —privileged mode for the container, privileged mode is not allowed in Fargate

[Feature] Add EFS support to Jenkins leader node when running Fargate

Once there is CloudFormation support for Fargate/EFS, add a config option to enable EFS for the Leader node if running on Fargate (or EC2).

facing issue with jenkins-plugin.txt

Hi Team,

I spun up Jenkins farget ecs cluster using CDK approach but i'm facing while installing Jenkins plugin using leader/plugin.txt file in the repo.
Can you help me

Building docker images?

Can you demonstrate or add to the readme how to build docker images inside this setup?

I'm looking at kaniko, but it's not immediately obvious how to integrate it with jenkins using the amazon-ecs plugin to run agents in fargate.

Thanks!

PS: Please update your "master" references to jenkins server?

Master hangs for hours on deployment

I have tried in two environments, one with the new arn settings enabled, and another where I set managed tags to false to get around the issue. Each are stuck (for several hours) on steps 16 or 17:

JenkinsOnAWSJenkinsMaster: deploying...
[0%] start: Publishing 052c74adeda218bd870c6412340cccf4f1807d8d795be5284e510895cf5ed2b3:current
[100%] success: Published 052c74adeda218bd870c6412340cccf4f1807d8d795be5284e510895cf5ed2b3:current
JenkinsOnAWSJenkinsMaster: creating CloudFormation changeset...
[███████████████████████████████████████████████████▉······] (17/19)

1:38:49 PM | CREATE_IN_PROGRESS   | AWS::ECS::Service                         | JenkinsMasterService/Service/Service

I also don't see any logstreams either. The other stacks are deployed correctly.

CDK updates fail when going from ec2_enabled=yes

ec2_enabled = yes, fargate_enabled = yes in config.ini results in The key pair 'jenkinsonaws' does not exist (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError;