In this tutorial you will learn how to add an automated repair tool to your Jenkins Pipeline. Don't have a Pipeline yet? Don't worry we'll go through every step to set you up.
Let's first take a moment to look into the different tools we will be using in this tutorial.
You will setup parts of a ci/cd pipeline containing jenkins and sonarqube using docker. This will be connected to a github repository and you will add a plugin which runs automated code repair on the code.
Docker is a tool that makes the whole process of getting your application out there easier. I helps you all the steps, creating, deploying and running the applications. This is done by using containers, containers makes it possible for the developers to put everything that is needed to run your application in one simple package.
Jenkins is a a tool making it easy to set up a CI/CD, continoues integration or continoues delivery pipeline for almost all languages out there. It fetches your code from version control, builds it, packages it, and pushes it to the next application in line which can deploy it.
Sonarqube is a static code analysis tool, it contains a set of predefined rules which are usually checked during the build time of the application. It gives feedback to the developer by presenting the possible bugs in their code and also showing some generic solutions to the known issue.
- You will learn some basic Docker scripting.
- You will learn how to setup your basic Jenkins using docker.
- You will learn how to add plugins to your Jenkins pipeline. SonarQube and Repairnator in this case.
- You will learn some pipeline building using jenkinsfiles in your code repository.
- Docker
To minimize the complexity of many prerequisites, we'll be using Docker in this tutorial to setup our pipeline. Docker enables us to boot-up and run all our components of our pipeline in containers, avoiding installing a lot of dependencies on our local machines.
Here you can find and install a docker version suiting you Get Docker
Assuming you've installed Docker let's move on to getting our jenkins with sonarqube up and running.
If you already got a docker installation with some containers already running you should either stop your current containers or make sure that the ports are not colliding throughout this tutorial.
docker ps // List our containers
docker stop <my_container> // Stop any container using their name
docket rm <my_container> // Removes the container
Disclaimer: I've only tested the installation using Windows
Installing on macOS and Linux
- Create a bridge network in Docker which all our applications will connect to:
docker network create jenkins
More on networks
Containers on docker are exposed on a specific port from the main gateway docker creates. Networks in docker enables containers to communicate with each other with out leaving the docker network. If you boot-up many containers where you want to enable communication between only some of them, you can segregate them by having multiple networks in your docker environment and connecting the containers to different networks.
- Create volumes to persist the jenkins data if you start/stop the containers and so that Jenkins can get a hold of Docker client TLS certificates can be shared which will be used by jenkins
docker volume create jenkins-docker-certs
docker volume create jenkins-data
More on volumes
Volumes are used by docker as a memory slot. They are persisted over start/stop/restart of containers and may also be shared between multiple containers.
- To be able to run Docker commands inside of our Jenkins nodes we need to download and run the Docker in Docker image (DinD).
docker container run --name jenkins-docker --rm --detach \
--privileged --network jenkins --network-alias docker \
--env DOCKER_TLS_CERTDIR=/certs \
--volume jenkins-docker-certs:/certs/client \
--volume jenkins-data:/var/jenkins_home \
--volume "$HOME":/home docker:dind
- Finally we'll download and run the jenkins application
docker container run --name jenkinsci --rm --detach \
--network jenkins --env DOCKER_HOST=tcp://docker:2376 \
--env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 \
--volume jenkins-data:/var/jenkins_home \
--volume jenkins-docker-certs:/certs/client:ro \
--volume "$HOME":/home \
--publish 8080:8080 jenkinsci/blueocean
Note: as you can see there are some environment variables defined in the run command, these enable the jenkins to actually run the docker in docker commands.
Installing on Windows
- Create a bridge network in Docker which all our applications will connect to:
docker network create jenkins
More on networks
Containers on docker are exposed on a specific port from the main gateway docker creates. Networks in docker enables containers to communicate with each other with out leaving the docker network. If you boot-up many containers where you want to enable communication between only some of them, you can segregate them by having multiple networks in your docker environment and connecting the containers to different networks.
- Create volumes to persist the jenkins data if you start/stop the containers and so that Jenkins can get a hold of Docker client TLS certificates can be shared which will be used by jenkins
docker volume create jenkins-docker-certs
docker volume create jenkins-data
More on volumes
Volumes are used by docker as a memory slot. They are persisted over start/stop/restart of containers and may also be shared between multiple containers.
- To be able to run Docker commands inside of our Jenkins nodes we need to download and run the Docker in Docker image (DinD).
docker container run --name jenkins-docker --rm --detach ^
--privileged --network jenkins --network-alias docker ^
--env DOCKER_TLS_CERTDIR=/certs ^
--volume jenkins-docker-certs:/certs/client ^
--volume jenkins-data:/var/jenkins_home ^
--volume "%HOMEDRIVE%%HOMEPATH%":/home ^
docker:dind
- Finally we'll download and run the jenkins application
docker container run --name jenkinsci --rm --detach ^
--network jenkins --env DOCKER_HOST=tcp://docker:2376 ^
--env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 ^
--volume jenkins-data:/var/jenkins_home ^
--volume jenkins-docker-certs:/certs/client:ro ^
--volume "%HOMEDRIVE%%HOMEPATH%":/home ^
--publish 8080:8080 --publish 50000:50000 jenkinsci/blueocean
Note: as you can see there are some environment variables defined in the run command, these enable the jenkins to actually run the docker in docker commands.
It might take a while until the jenkins server is up and running. You should however be able to visit http://localhost:8080/ and see your jenkins application when it is ready.
Jenkins now tells us to find our password in the file /var/jenkins_home/secrets/initialAdminPassword
to unlock it. Either we could use the exec
command which lets us use any shell command towards a container we would like. We could also check the logs of the jenkins container:
docker exec jenkinsci cat var/jenkins_home/secrets/initialAdminPassword
docker logs jenkinsci
- Use the password to unlock the Jenkins application
- Follow the installation guide and install the suggested plugins
- Create a new admin account, e.g. using
user: admin
,pwd: admin
and some dummy email.
Assuming you've followed the guide for starting up Jenkins you should have a network called jenkins
which we will connect our SonarQube application to:
docker run -d --name sonarqube --network jenkins -p 9000:9000 sonarqube:lts
More about `docker run` command
The `docker run` command starts up an application in a container. The -d flag stands for detach which makes the container run in the background. At the end we define which Image we want to use, in this case "sonarqube:lts". If there is no image present locally matching this name, docker will try and pull it from dockerhub.You should be able to go to http://localhost:9000/ and see your sonarqube instance.
If you have issues reaching the application it is either still being started or you might be using a quite old computer or setup with docker. In my case hitting the wall when trying it out on an old mac(read: "very old"). I had to visit http://192.168.99.100:9000/ since the legacy Docker Toolbox runs with VirtualBox which used this ip-adress.
After successfully following the start up guides we should see three containers up and running using the docker ps
command. Looking something like this:
We'll now configure jenkins so that we can run a build which executes a sonarqube scan on our files.
- Click
Manage Jenkins
->Manage Plugins
->Available
- Search for
Sonarqube scanner
- Tick the box for the tool and then click the
Install without restarting
- On the next screen tick the box for restarting the Jenkins when the installation is done and no jobs are running.
- When the system has rebooted go back to
Manage Jenkins
again - This time click the
Configure (System)
- Scroll down to the SonarQube Servers section and click the
Add SonarQube
- Here you can specify your SonarQube server adress, name and auth-token
- In our case running sonarqube on the same docker network as our jenkins we can use the command
docker inspect sonarqube
and in this json structure find theNetworks->Jenkins->IpAddress
and put the value in the URL, in my case it was:http://172.18.0.4:9000/
- Generate token for Sonarqube
- Go to your SonarQube instance - localhost:9000
- Now login to the SonarQube using
username: admin
andpwd: admin
- Click on the user icon at the top right corner and click
my account
- Click
Security
-> fill in the Token name:My-SonarQube-Tutorial-Token
->copy
the generated token.
- Add the token to our Jenkins
- Back in Jenkins click the Add-button next to the "Server Authentication Token"
- Set
Kind
tosecret text
and add the paste the copied token into thesecret
input field. - Add any name and description of your liking.
Our pipeline is done and we're now ready to make our first build!
We'll be doing so by forking this repository and adding a new Jenkinsfile to it which will define what our builds should do.
At the top of this page you can fork this repository and then do a git clone
to your local machine:
Your jenkinsfile will tell Jenkins how to build your code in the given folder.
Go to the root of the repository and create a file named: Jenkinsfile
Now open the Jenkinsfile and add the following:
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Test Maven') {
steps ('Check maven version') {
sh 'mvn -version'
}
}
}
}
To begin with, we'll check if our maven used with docker is working, this you can see in the stages -> stage -> steps -> sh 'mvn -version'
section.
The agent part of the json file defines the node which will run the commands, here being a docker container running the image maven:3-alpine.
Now you should git add .
, git commit -a -m "Added first Jenkinsfile"
and git push
your changes.
-
Back to jenkins
"add new item" -> Enter an item name -> choose Pipeline -> click OK at the bottom
. -
Scroll down to pipeline and change the definition to:
Pipeline script from SCM
, chooseGIT
as your SCM, put the url to your forked repository and save. -
On the left hand side you should now see and click
start build now
. -
Refresh the page and you'll see a blinking dot with
#1
on the left hand-side underBuild History
, click on the number and then click Console Output.
You've now initialized your build and can see the console output from the pipeline. If successful we can carry on, otherwise something is incorrectly configured and you'll have to revisit previous steps, google similar issuer or contact me.
Edit the Jenkinsfile and make the following changes:
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Sonarscan') {
steps{
withSonarQubeEnv('SonarQube') {
sh "mvn clean package sonar:sonar"
}
}
}
}
}
Commit and push your changes and try and run the build on Jenkins again. If successful you should be able to visit your local sonarqube host and see that a new project is showing with some possible issues.
Navigate to the bug clicking on the project name -> the number of bugs -> the clickable red dscription of the bug
.
Sonarqube now shows you what exactly is the issue and where in the code. You can further dig deeper into how it could possibly be solved by clicking the see rule
box. In this case I chose quite a simple bug which also already has a functioning code repair tool for it. Should look something like this:
We'll be using a tool called repairnator which enables the user to choose from a variaty of modules conducting repair on your code.
- Go to
Manage Jenkins -> Manage Plugins
- Click on the
Advanced
-tab. - Choose upload file and find the .hpi file in the root of this repository. (Extra credit to the team behind with link to the tool and possible updated version.)
- Tick the box for restarting Jenkins and wait until it's all finished.
Now the idea behind the repairnator is to scan through your code for possible bugs and solutions to these and then create a PR to your Github repository with the suggested solution to the bug.
- Firstly, go back to jenkins main screen and create a new item, now using the
freestyle project
module. - In here you can add the URL to your repository if you click the
Git
radio button under version management. - Click the
Add post-build action
, and chooserun repairnator
from the given list. - Now you will have to generate a GitOAuthToken if you do not have one. You can go to this link, press
generate new token
and tick the "public_repo" box. - Now you can choose which repairtools to run, and if you want to get a Pull Request with the repairs you can click the
advanced
-button and fill in the GitUrl and Branch. If this doesn't work you might have to add the plugingithub pull request builder
.
There we have it! You've added an automated repair tool to your Jenkins Pipeline!
Stop our container:
docker container stop <container_name>
If you just want to clean up everything dockery:
docker system prune --volumes
If you would like to be a bit more granular:
docker container prune -- removes all stopped containers.
docker image prune -a -- removes all images that are not used by any existing container.
docker volumes prune -- removes all volumes not used by at least one container.
docker network prune -- removes all networks not used by at least one container.
- Repairnator plugin - The team behind creating the repairnator and the plugin for jenkins
-
Repairnator - "Software development bot that automatically repairs build failures on continuous integration."
-
SpoonLabs - "The laboratory for Java source code analysis and transformation using Spoon."