A hands-on DevOps course.
A reveal.js presention written to accompany this course can found at https://nemonik.github.io/hands-on-DevOps/.
This course will
- Discuss DevOps,
- Have you spin up a DevOps toolchain and development environment, and then
- Author two applications and their accompanying pipelines, the first a continuous integration (CI) and the second a continuous delivery (CD) pipeline.
After this course you will
- Be able to describe and have hands on experience the DevOps methods and repeated practices (e.g., use of Agile methods, configuration management, build automations, test automation and deployment automation orchestrated under continuous integration and delivery orchestrator), and why it matters;
- Identify what diverse tools and resources that exist in the DevOps ecosystem;
- Be able to address challenges sponsors face by transitioning to DevOps methods and repeated practices;
- Have had hands-on experience with Infrastructure as Code( Vagrant and Ansible ) to provision and configure an entire DevOps toolchain and development environment on VirtualBox including Docker Registry, GitLab, Drone CI, and SonarQube;
- Have had hands-on experience authoring code to include authoring and running automated tests in a CI/CD pipeline all under Configuration Management to insure an application follows style, adheres to good coding practices, builds, identify security issues, and functions as expected;
- Have had hands-on experience with (a) using Infrastructure as Code (IaC) in Vagrant and Ansible; (b) creating and using Kanban board in Taiga; (c) code configuration in git and GitLab; (d) authoring code in Go; (e) using style checkers and linters; (f) authoring a Makefile; (g) various commands in Docker (e.g., building a container image, pushing a container into a registry, creating and running a container); (h) authoring a pipeline for Drone CI; (i) using Sonar Scanner CLI to perform static analysis; (j) authoring security test in InSpec; (k) author an automated functional test in Selenium; (l) authoring a dynamic security test in OWASP Zap; an (m) using container platform to author and scale services;
- Have had hands-on experience authoring code to include authoring and running automated tests in a CI/CD pipeline all under Configuration Management to insure an application follows style, adheres to good coding practices, builds, identify security issues, and functions as expected.
We will be spending most of the course hands-on working with the tools and in the Unix command line making methods and repeated practices of DevOps happen, so as to grow an understanding of how DevOps actually works.
Don't fixate on the tools used, nor the apps we develop in the course of learning how and why. How and why is far more important. This course like DevOps is not about tools although we'll be using them. You'll spend far more time writing code. (Or at the very least cutting-and-pasting code.)
- Michael Joseph Walsh [email protected], [email protected]
- Walter Hiranpat for running through the course material for my first class prior to me teaching it.
- Eric McCann for assisting Walter and I work through Vagrant install issues on Microsoft Windows.
- My first class participants for "beta" testing my class.
- Jonathan Thomas for TA'ing my second class.
- David Trang for TA'ing my class countless times.
- Walter Hiranpat, David Trang, and Rony Xavier for prepping the MITRE Institute classroom for my class countless times.
- Rony Xavier and Aaron Lippold for assisting me mature the InSpec section.
See the License file at the root of this project.
The following skills would be useful in following along, but aren't strictly necessary.
What you should bring:
- Managing Linux or Unix-like systems would be tremendously helpful, but not necessary, as we will living largely within the terminal.
- A basic understanding of Vagrant, Docker, and Ansible would also be helpful, but not necessary.
- Mostly being a software engineer, but not necessary.
- 1. DevOps
- 2. Author
- 3. Thank you to
- 4. Copyright and license
- 5. Prerequisites
- 6. Table of contents
- 7. DevOps unpacked
- 7.1. What is DevOps?
- 7.2. What DevOps is not
- 7.3. DevOps is really about
- 7.4. How is it related to the Agile?
- 7.5. How do they differ?
- 7.6. Why?
- 7.7. What are the principles of DevOps?
- 7.8. How is this achieved?
- 7.9. What is Continuous Integration (CI)?
- 7.10. How?
- 7.11. CI best practices
- 7.12. What is Continuous Delivery?
- 7.13. But wait. What's a pipeline?
- 7.14. How is a pipeline manifested?
- 7.15. What underlines all of this?
- 7.16. But really why do we automate err. code?
- 7.17. Monitoring
- 7.18. What is DevOps culture?
- 7.19. Crawl, walk, run
- 8. Reading list
- 9. Now the hands-on part
- 9.1. Configuring environmental variables
- 9.2. VirtualBox
- 9.3. Git Bash
- 9.4. Retrieve the course material
- 9.5. Infrastructure as code (IaC)
- 9.5.1. Vagrant
- 9.5.1.1. Vagrant documentation and source
- 9.5.1.2. Installing Vagrant
- 9.5.1.3. Installing Vagrant plugins
- 9.5.1.4. The Vagrantfile explained
- 9.5.1.4.1. Modelines
- 9.5.1.4.2. Setting Software version for Ansible roles
- 9.5.1.4.3. Inserting Proxy setting via host environmental variables
- 9.5.1.4.4. Inserting enterprise CA certificates
- 9.5.1.4.5. Configuring the cache plugin to speed things along
- 9.5.1.4.6. Configuring the disksize plugin to increase the disk size
- 9.5.1.4.7. Requiring vagrant-vbquest for Windows
- 9.5.1.4.8. Configuring the development vagrant
- 9.5.1.4.9. Configuring the toolchain vagrant
- 9.5.2. Ansible
- 9.5.1. Vagrant
- 9.6. Docker image and containers
- 9.7. Docker-compose
- 9.8. Spinning up the toolchain vagrant
- 9.9. Spin up the development vagrant
- 9.10. Golang helloworld project
- 9.10.1. Create the project's backlog
- 9.10.2. Create the project in GitLab
- 9.10.3. Setup the project on the development Vagrant
- 9.10.4. Author the application
- 9.10.5. Align source code with Go coding standards
- 9.10.6. Lint your code
- 9.10.7. Build the application
- 9.10.8. Run your application
- 9.10.9. Author the unit tests
- 9.10.10. Automated the build (i.e., write the Makefile)
- 9.10.11. Author Drone-based Continuous Integration
- 9.10.12. The completed source for helloworld
- 9.11. Golang helloworld-web project
- 9.11.1. Create the project's backlog
- 9.11.2. Create the project in GitLab
- 9.11.3. Setup the project on the development Vagrant
- 9.11.4. Author the application
- 9.11.5. Build and run the application
- 9.11.6. Run gometalinter.v2 on application
- 9.11.7. Fix the application
- 9.11.8. Author unit tests
- 9.11.9. Perform static analysis (i.e., sonar-scanner) on the command line
- 9.11.10. Automated the build (i.e., write the Makefile)
- 9.11.11. Dockerize the application
- 9.11.12. Run the Docker container
- 9.11.13. Push the container image to the private Docker registry
- 9.11.14. Configure Drone to execute your CI/CD pipeline
- 9.11.15. Add Static Analysis (SonarQube) step to pipeline
- 9.11.16. Add the build step to the pipeline
- 9.11.17. Add container image publish step to pipeline
- 9.11.18. Add container deploy step to pipeline
- 9.11.19. Add complaince and policy automation (InSpec) test to the pipeline
- 9.11.20. Add automated funtional test to pipeline
- 9.11.21. Add DAST step (OWASP ZAP) to pipeline
- 9.11.22. All the source for helloworld-web
- 9.12. Microservices
- 9.13. Using what you've learned
- 9.14. Shoo away your vagrants
- 9.15. That's it
DevOps (a clipped compound of the words "development" and "operations") is a software development methodology with an emphasis on a more reliable release pipeline, automation, and stronger collaboration across all stakeholders with the goal of deploying working functionality into the hands of users (i.e., production) faster.
Yeah, that's the formal definition. I've grown to prefer the axiom:
You are what you code.
For example,
- If you're a developer or software engineer, you're at least coding the application, its build automation, unit tests, and CI/CD (the combined practices of Continuouse Integrarion and Continuous Delivery) automation.
- If a tester, you're coding typically the functional test automation.
- If a security engineer, you're coding the compliance and policy test automation.
- If ops, you're coding the deployment and infrastructure configuration management automation.
And since all these disciplines are coding, they're essentially using the same methods and repeated practices to ensure they're producing good code.
And they're all collectively doing all this coding left in the delivery pipeline, collaborating and from this springs forth culture.
DevOps is not entirely about tools.
Or as I like to put it, "Its not about the tools."
DevOps will also not entirely stop all bugs nor all vulnerabilities from making it into production, but that's not really the point.
There are countless vendors out there, who want to sell you their crummy tool.
Providing the culture and delivery/release pipeline that once a bug or vulnerability is discovered, the concern can to be quickly remediated and functionality returned back to the user.
Agile Software Development is an umbrella term for a set of methods and practices based on the values and principles expressed in the Agile Manifesto.
Agile Software Development shares the same goal, but DevOps extends Agile methods and practices by adding communication and collaboration between
- development,
- quality assurance, and
- technology operations
- to ensure software systems are delivered in a rapid, reliable, low-risk manner.
For Agile, solutions evolve through collaboration between self-organizing, cross-functional teams utilizing the appropriate practices for their context.
Again, in DevOps everyone is developing software, so it is my view DevOps builds on Agile.
While Agile Software Development encourages collaboration between cross-functional teams, the focus in DevOps is on the
- inclusion of analysis,
- design,
- development, and
- quality assurance functionaries as stakeholders into the development effort.
In Agile Software Development, there is rarely an integration of these individuals outside the immediate application development team with members of technology operations (e.g., network engineers administrators, testers, cyber security engineers.)
As DevOps matures, several principles have emerged, namely the necessity for product teams to:
- Apply holistic thinking to solve problems,
- Develop and test against production-like environments,
- Deploy with repeatable, and reliable processes,
- Remove the drudgery through automation,
- Validate and monitor operational quality, and
- Provide rapid, automated feedback to the stakeholders
Achieved through the repeated practices of Continuous Integration (CI) and Continuous Delivery (CD) often conflated into simply "CI/CD".
After tools, this is what is commonly (albeit mistakenly) thought to be the totality of DevOps.
It is a repeated Agile software development practice, lifted specifically from Extreme programming, where members of a development team frequently integrate their work in order to detect integration issues as quickly as possible thereby shifting discovery of issues "left" (i.e., early) in the software release.
Each integration is orchestrated through a CI service/orchestrator (e.g., Jenkins CI, Drone CI, GitLab Runners, Concourse CI) that essentially assembles a build, runs unit and integration tests every time a predetermined trigger has been met; and then reports with immediate feedback.
For the software's source code, where the mainline (i.e., master branch) is the most the most recent working version, past releases held in branches, and new features not yet merged into the mainline branch being worked on their own branches.
By accompanying build automation (e.g., Gradle, Apache Maven, Make) alongside the source code.
Perform source code analysis via automate formal code inspection and assessment.
In other words, ingrain testing by including unit and integration tests (e.g., Spock, JUnit, Mockito, SOAPUI, go package Testing) with the source code so as to be executed by the build automation to be execute by the CI service.
Or untested source code to the CMS mainline or otherwise risk breaking a build.
Prior to committing source code in their own workspace.
On the success or fail of a build integration to all its stakeholders.
It is a repeated software development practice of providing a rapid, reliable, low-risk product delivery achieved through automating all facets of building, testing, and deploying software.
With additional stages/steps aimed to provide ongoing validation that a newly assembled software build meets all desired requirements and thereby is releasable.
Is achieved through delivering applications into production via individual repeatable pipelines of ingrained system configuration management and testing
A pipeline automates the various stages/steps (e.g., Static Application Security Testing (SAST), build, unit testing, Dynamic Application Security Testing (DAST), secure configuration acceptance compliance, integration, function and non-functional testing, delivery, and deployment) to enforce quality conformance.
Each delivery pipeline is manifested as Pipeline as Code (i.e., software automation) accompanying software's source code in its version control repository.
I'd argue, a ubiquitous access to shared pools of configurable system resources and higher-level services that can be rapidly provisioned with minimal management effort without the repeated practices of DevOps will struggle. Although, it is possible to DevOps on mainframes.
In 2001, I think Larry Wall in his 1st edition of Programming Perl book put it best with "We will encourage you to develop the three great virtues of a programmer:
laziness,
impatience, and
hubris."
The second edition of the same book provided definitions for these terms
Well...
Once you have established yourself as an icon in your field it is important that you pay tribute to some of the great legends that came before you. This kind of gesture will create the illusion that you’re still humble and serve as a preemptive strike against anyone who has noticed what a callus and delusional ass you have become.
The opening monolog to the Blue Man Group’s I Feel Love https://www.youtube.com/watch?v=8vBKI3ya-l0
I kid, but in all serious the sentimanet of this seminal book still holds true.
Let me explain.
The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. (p.609)
The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least pretend to. Hence, the second great virtue of a programmer. (p.608)
Excessive pride, the sort of thing Zeus zaps you for. Also, the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. (p.607)
- Faster, coordinated, repeatable, and therefore more reliable deployments.
- Discover bugs sooner. Shifting their discovery left in the process.
- To accelerates the feedback loop between Dev and Ops.
- Reduce tribal knowledge, where one group or person holds the keys to how things get done. Yep, this is about making us all replaceable.
- Reduce shadow IT (i.e., hardware or software within an enterprise that is not supported by IT. Just waiting for its day to explode.)
Once deployed, the work is done, right?
A development team's work is not complete once a product leaves CI/CD and enters production; especially, under DevOps where the development teams include members of operations.
Is working software is the primary, but not the only, measure of progress. The key to successful DevOps is knowing how well the methodology and the software it produces are performing.
Is achieved by collecting and analyzing data produced by environments used for CI/CD and production.
So, that improvements can be gauged and anomalies detected.
To formulate and prioritize reactions weighting factors, such as, the frequency at which an anomaly arises and who is impacted.
A reaction could be as simple as operations instructing users through training to not do something that triggers the anomaly, or more ideally, result in an issue being entered into the product's backlog culminating in the development team delivering a fix into production.
Monitoring will also inform development teams of gaps in CI/CD resulting in additional testing for the issue that triggered the necessity for the improvement.
Further, monitoring may result in the re-scoping of requirements, re-prioritizing of a backlog, or the deprecation of un-used features.
culture noun \ ˈkəl-chər
the set of shared attitudes, values, goals, and practices that characterizes an institution or organization
Now that everyone is a coder using the same tools, methods and repeated practices for their particular discipline you have the foundations of a culture.
- With DevOps one does not simply hit the ground running.
- One must first crawl, walk and then ultimately run as you embrace the necessary culture change, methods and repeated practices.
- Collaboration and automation are expected continually improve so to achieve more frequent and more reliable releases.
AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis William J. Brown, Raphael C. Malveau, Hays W. "Skip" McCormick, and Thomas J. Mowbray ISBN: 978-0-471-19713-3 Apr 1998
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) David Farley and Jez Humble ISBN-13: 978-0321601919 August 2010
The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations Gene Kim Jez Humble, Patrick Debois, and John Willis ISBN-13: 978-1942788003 October 2016
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Nicole Forsgren PhD, Jez Humble, and Gene Kim ISBN-13: 978-1942788331 March 27, 2018
Site Reliability Engineering: How Google Runs Production Systems 1st Edition Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy ISBN-13: 978-1491929124 April 16, 2016 Also, available online at https://landing.google.com/sre/book/index.html
Release It!: Design and Deploy Production-Ready Software 2nd Edition Michael T. Nygard ISBN-13: 978-1680502398 January 18, 2018
The SPEED of TRUST: The One Thing That Changes Everything Stephen M .R. Covey ISBN-13: 978-1416549000 February 5, 2008 The gist of the book can be found at SlideShare https://www.slideshare.net/nileshchamoli/the-speed-of-trust-13205957
How to Deal With Difficult People Ujjwal Sinha Oct 25, 2014 The SlideShare can be found here https://www.slideshare.net/abhiujjwal/how-2-deal-wid-diiclt-ppl
In this class you will spin up the following developent and toolchain environment.
NOTE
- If your web-based Git-repository manager is paired with a PlantUML server you will see a diagram rendered below otherwise you will only the PlantUML source code for the diagram. Don't worry the course's Ansible automation will spin up both a GitLab and PlantUML.
- This class makes use of NOTE sections to call out things that are important to know or will drop a few tidbits. Reading these notes may save you some aggrevation.
@startuml
skinparam shadowing false
skinparam actor {
BorderColor #0B5C92
BackgroundColor none
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 18
}
skinparam node {
BorderColor #0B5C92
BackgroundColor #ffffff
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam rectangle {
roundCorner 25
BorderColor #0B5C92
BackgroundColor #ffffff
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam component {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam database {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
actor "You" as you
rectangle "Host"{
rectangle "Toolchain Vagrant" {
rectangle "Docker CE" as docker1 {
rectangle "Taiga" {
component "nemonik/taiga:4.0.4" as taiga
component "sameersbn/postgresql:9.6-4" as taiga_postgresql
}
rectangle "GitLab" {
component "nemonik/gitlab:11.8.3"" as gitlab
component "sameersbn/postgresql:10" as gitlab_postgresql
component "redis:4.0.9-1" as gitlab_redis
}
rectangle "Drone CI" {
component "drone/server:1" as drone_server
component "drone/agent:1" as drone_agent
component "sameersbn/postgresql:9.6-4" as drone_postgresql
}
rectangle "SonarQube" {
component "sonarqube:7.1" as sonarqube
}
rectangle "PlantUML Server" {
component "plantuml/plantuml-server:latest" as plantuml_server
}
rectangle "Registry" {
component "registry:2.7.1" as registry
}
}
}
rectangle "Development Vagrant" {
rectangle "Docker CE" as docker2 {
component "helloworld" as helloworld
component "helloworld-web" as helloworld_web
}
}
}
taiga -[#0B5C92]-> taiga_postgresql
gitlab -[#0B5C92]-> gitlab_postgresql
gitlab -[#0B5C92]-> gitlab_redis
drone_server -[#0B5C92]-> drone_agent
drone_server -[#0B5C92]-> drone_postgresql
you .[#0B5C92].> sonarqube
you .[#0B5C92].> gitlab
you .[#0B5C92].> taiga
you .[#0B5C92].> plantuml_server
you .[#0B5C92].> drone_server
you .[#0B5C92].> registry
you .[#0B5C92].> helloworld
you .[#0B5C92].> helloworld_web
@enduml
On Mac OS X or *NIX environments
Example settings (Adjust for your environment.) can be set by running the BASH script in the root of the project, set_env.sh
:
#!/usr/bin/env bash
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
# run in shell via
#
# ```
# . ./set_env.sh
# ```
export PROXY=http://gatekeeper.mitre.org:80
export proxy=$PROXY
export https_proxy=$PROXY
export http_proxy=$PROXY
export HTTP_PROXY=$PROXY
export ALL_PROXY=$PROXY
export NO_PROXY="127.0.0.1,localhost,.mitre.org,.local,$(echo 192.168.0.{1..255} | sed 's/ /,/g')"
#,172.30.1.1"
export no_proxy=${NO_PROXY}
export CA_CERTIFICATES=https://raw.githubusercontent.com/nemonik/hands-on-DevOps/master/certs/MITRE%20BA%20NPE%20CA-3(1).crt,https://raw.githubusercontent.com/nemonik/hands-on-DevOps/master/certs/MITRE%20BA%20ROOT.crt
export VAGRANT_ALLOW_PLUGIN_SOURCE_ERRORS=0
Execute in terminal session via
. ./set_env.sh
The alternative is unset.sh
bash script that unsets all these values:
#!/usr/bin/env bash
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
# run in shell via
#
# ```
# . ./unset.sh
# ```
unset no_proxy
unset NO_PROXY
unset ALL_PROXY
unset PROXY
unset proxy
unset https_proxy
unset http_proxy
unset HTTP_PROXY
unset ca_certificates
unset CA_CERTIFICATES
Execute in terminal session via
. ./unset.sh
On Windows
Example settings (Adjust for your environment.):
- In the Windows taskbar, enter
edit the system environment variables
intoSearch Windows
and select the icon with the corresponding name. - The
Systems Property
window will likely open in the background, so you will likely need to go find it and bring it forward. - In the
Systems Property
'sAdvanced
tab selectEnvironment Variables...
button. - In
Environment Variables
windows that opens, underUser variables for...
pressNew ...
to open aNew User Variable
window, enter eachVariable Name
and and its respectiveValue
for each pair in the table below
Variable Name | Value |
---|---|
proxy | http://gatekeeper.mitre.org:80 |
http_proxy | http://gatekeeper.mitre.org:80 |
https_proxy | http://gatekeeper.mitre.org:80 |
no_proxy | 127.0.0.1,localhost,.mitre.org,.local,192.168.0.10,192.168.0.11 |
CA_CERTIFICATES | http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt,http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20NPE%20CA-3%281%29.crt |
VAGRANT_ALLOW_PLUGIN_SOURCE_ERRORS | 0 |
NOTE
- The certificate URLs need to be encoded for parentheses to work.
- One Windows, you may inadvertantly cut-and-paste blank space characters (e.g., tabs, spaces) and the subsequent Ansible automation may fail.
VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at server, desktop and embedded use.
For the class, it is assumed VirtualBox is installed, but below are the instructions for installing it on Windows 10.
- Open your browser to https://www.virtualbox.org/wiki/Downloads
- Click
Windows hosts
link underVirtualBox 6.0.4 platform packages
. - Find and click the installer to install.
Then turn for Windows 10 turn off Hyper-V
- Click Windows
Start
and then typeturn Windows features on or off
into the search bar. - Select the icon with the corresponding name.
- This will open the
Windows Features
page and then unselect theHyper-V
checkbox if it is enabled and then clickOkay
.
The same site has the Mac OS X download. The install is less involved.
Git Bash is git
packaged for Windows with bash (a command-line shell) and a collection of other, separate *NIX utilities, such as, ssh
, scp
, cat
, find
and others compiled for Windows.
If you are on Windows, you'll need to install git
.
-
Download from https://git-scm.com/download/win
-
Click the installer.
-
Click
next
until you reach theConfiguring the line ending conversions
page selectCheckout as, commit Unix-style line endings
. -
Then
next
,next
,next
... -
Don't open git-bash from the final window as it will not have the environmental variables set. Go onto step-6.
-
On the Windows task bar, enter
git
intoSearch Windows
then selectGit Bash
. UseGit Bash
instead ofCommand
orPowershell
.
If you are reading this on paper and have nothing else, you only have a small portion of the class material. You will need to download the class project containing all the automation to spin up a DevOps toolchain and development, etc.
In a shell, for the purposes of the class, this means in Git Bash
, clone the project from https://github.com/nemonik/hands-on-DevOps.git via git like so:
git -c http.sslVerify=false clone https://github.com/nemonik/hands-on-DevOps.git
Output will resemble (i.e., not being precisely the same):
$ git -c http.sslVerify=false clone https://github.com/nemonik/hands-on-DevOps.git
Cloning into 'hands-on-DevOps'...
remote: Counting objects: 1184, done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 1184 (delta 207), reused 411 (delta 178)
Receiving objects: 100% (1184/1184), 235.51 MiB | 21.53 MiB/s, done.
Resolving deltas: 100% (480/480), done.
This class uses Infrastructure as code (IaC) to set up the class environment (i.e., two virtual machines that will later be referred to as "vagrants".) IaC is the process of provisioning, and configuring (i.e., managing) computr systems through code, rather than directly manipulating the systems by hand (i.e., manual processes).
This class uses Vagrant and Ansible IaC framewoorks and the following sections will unpack each.
This class uses Vagrant, a command line utility for managing the lifecycle of virtual machines as a vagrant in that the VMs are not meant to hang around in the same place for long.
Unless you want to pollute your machine with every imaginable programming language, framework and library version you'll find yourself often creating a virtual machine (VM) for each software project. Sometimes more than one. And if you're like me of the past you'll end up with a VirtualBox full of VMs. If you haven't gone about this the right way, you'll end up wondering what VM went with which project and now how did I create it? The anti-pattern around this problem is to write documentation. A better way that aligns with DevOps repeatable practices is to create automation to provision and configure your development VMs. This is where Vagrant comes in as it is "a command line utility for managing the lifecycle of virtual machines."
Vagrant's documentation can be found at
https://www.vagrantup.com/docs/index.html
It's canonical (i.e., authoritative) source can be found at
https://github.com/hashicorp/vagrant/
Vagrant is written in Ruby. In fact, a Vagrantfile is written in a Ruby DSL.
-
Download Vagrant
https://releases.hashicorp.com/vagrant/2.2.3/
As of April 2nd 2014,
vagrant-ca-certificates
plugin has an open issue (#34), where the plugin fails to run on the recent 2.2.4 release of Vagrant, but runs fine on prior 2.2.3 release. Please, install the 2.2.3 release of Vagrant. -
Click on the installer once downloaded and follow along. The installer may stall calculating for a bit on Windows and for this same OS may bury modals you'll need to respond to in the Windows Task bar, so keep an eye out for that. The installer will automatically add the
vagrant
command to your system path so that it is available on the command line. If it is not found, the documentation advises to try logging out and logging back into your system. This is particularly necessary sometimes for Windows. Windows will require a reboot, so remember to come back and complete step-3. -
If you're not on the MITRE corporate network skip this step. On Windows, use the File Explorer to replace the existing
C:\Hashicorp\vagrant\embedded\cacert.pem
file with the project'svagrant_files/cacert.pem
by using the File Explorer.Or when on Mac OS X copy it to
/opt/vagrant/embedded
as root using:sudo cp vagrant_files/cacert.pem /opt/vagrant/embedded/.
Vagrant plugins extend the functionality of Vagrant, and you'll need a few of them for this course.
In the command line (in Git Bash
, if on Windows) on the host
vagrant plugin install vagrant-ca-certificates
vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-disksize
vagrant plugin install vagrant-proxyconf
vagrant plugin install vagrant-vbguest
Verify the version installed with
vagrant plugin list
Whose output resembles this
vagrant-ca-certificates (1.3.0, global)
vagrant-cachier (1.2.1, global)
vagrant-disksize (0.1.3, global)
vagrant-proxyconf (2.0.1, global)
vagrant-vbguest (0.17.2, global)
The versions should be these or newer. Note difference in the event there ar problems later, becaus these versions have been verified to work.
The Vagrantfile
found at the root of the project describes how to provision and configure one or more virtual machines.
Vagrant's own documentation puts it best:
Vagrant is meant to run with one Vagrantfile per project, and the Vagrantfile is supposed to be committed to version control. This allows other developers involved in the project to check out the code, run vagrant up, and be on their way. Vagrantfiles are portable across every platform Vagrant supports.
If we were instead provisioning production Virtual Machines we'd alternatively use Terraform, a tool for building, changing, and versioning infrastructure
The following sub sections enumerate the various sections of the Vagrantfile
broken apart in order to discuss.
# -*- mode: ruby -*-
# vi: set ft=ruby :
When authoring, tells your text editor (e.g. emacs or vim) to choose a specific editing mode for the Vagrantfile. Line one is a modeline for emacs and line two is a modeline for vim.
# Holds the version of software for the Ansible roles to install
software_versions = {
drone_version: "1",
drone_cli_version: "1.0.7",
golang_version: "1.12.1",
gitlab_version: "11.8.3",
inspec_version: "3.9.0",
inspec_rpm_version: "3.9.0/el/7/inspec-3.9.0-1.el7.x86_64",
python_version: "2.7.16",
registry_version: "2.7.1",
plantuml_server_version: "latest",
sonarqube_version: "7.1", # 7.6-community is different enough i will need to look into in order to patch
sonar_scanner_cli_version: "3.3.0.1492",
selenium_standalone_chrome_version: "3.14.0",
selenium_standalone_firefox_version: "3.14.0",
taiga_version: "4.0.4",
zap2docker_stable_version: "2.7.0"
}
Contains all the version numbers for the software installed by the Ansible roles found in ansible/roles
.
The proxy setting section above uses the vagrant-proxyconf
plugin to set vagrants (i.e., managed VMs) to use specified proxies by using http_proxy and no_proxy environment variable.
Vagrant.configure('2') do |config|
# Set proxy settings for all vagrants
#
# Depends on install of vagrant-proxyconf plugin.
#
# To use:
#
# 1. Install `vagrant plugin install vagrant-proxyconf`
# 2. Set environmental variables for `http_proxy`, `https_proxy`, `ftp_proxy`, and `no_proxy`
#
# For example:
#
# ```
# export http_proxy=
# export https_proxy=
# export ftp_proxy=
# export no_proxy=
# ```
if (ENV['http_proxy'] || ENV['https_proxy'])
if Vagrant.has_plugin?('vagrant-proxyconf')
config.proxy.http = ENV['http_proxy']
config.proxy.https = ENV['https_proxy']
config.proxy.ftp = ENV['ftp_proxy']
config.proxy.no_proxy = ENV['no_proxy']
config.proxy.enabled = true
puts "HTTP Proxy variables set. http_proxy = #{ config.proxy.http }, https_proxy = #{ config.proxy.https }, ftp_proxy = #{ config.proxy.ftp }, no_proxy = #{ config.proxy.no_proxy }"
else
raise "Missing vagrant-proxyconf plugin. Install via: vagrant plugin install vagrant-proxyconf"
end
else
puts "No http_proxy or https_proxy environment variables are set."
config.proxy.http = nil
config.proxy.https = nil
config.proxy.ftp = nil
config.proxy.no_proxy = nil
config.proxy.enabled = false
end
# To add Enterprise CA Certificates to all vagrants
#
# Depends on the install of the vagrant-ca-certificates plugin
#
# To use:
#
# 1. Install `vagrant plugin install vagrant-ca-certificates`.
# 2. Set environement variable for `CA_CERTIFICATES` containing a comma separated list of certificate URLs.
#
# For example:
#
# ```
# export CA_CERTIFICATES=http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt,http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20NPE%20CA-3%281%29.crt
# ```
#
# The Root certificate *must* be denotes as the root certificat like so:
#
# http://employeeshare.mitre.org/m/mjwalsh/transfer/MITRE%20BA%20ROOT.crt
#
if ENV['CA_CERTIFICATES']
if Vagrant.has_plugin?('vagrant-ca-certificates')
puts "CA Certificates set to #{ ENV['CA_CERTIFICATES'] }"
config.ca_certificates.enabled = true
config.ca_certificates.certs = ENV['CA_CERTIFICATES'].split(',')
else
raise "Missing vagrant-ca-certificates plugin. Install via: vagrant plugin install vagrant-ca-certificates"
end
else
puts "No CA_CERTIFICATES environment variable set."
config.ca_certificates.certs = nil
config.ca_certificates.enabled = false
end
This section uses vagrant-ca-certificates
to as the plugin's documentation puts it configures the virtual machine to inject the specified certificates into the guest's root bundle. This is useful, for example, if your enterprise network has a firewall (or appliance) which utilizes SSL interception. So, we aren't the only ones that have to deal with the havoc SSL interception brings to development.
if Vagrant.has_plugin?("vagrant-cachier")
# Configure cached packages to be shared between instances of the same base box.
# More info on http://fgrehm.viewdocs.io/vagrant-cachier/usage
config.cache.scope = :box
else
raise "Missing vagrant-cachier plugin. Install via: vagrant plugin install vagrant-cachier"
end
if !Vagrant.has_plugin?("vagrant-disksize")
raise "Missing vagrant-disksize plugin. Install via: vagrant plugin install vagrant-disksize"
end
This section uses the vagrant-disksize
plugin to to resize disks in VirtualBox.
if Vagrant::Util::Platform.windows? and !Vagrant.has_plugin?('vagrant-vbguest')
raise "Missing vagrant-vbguest plugin. Install via: vagrant plugin install vagrant-vbguest"
end
vagrant-vbguest
automatically installs the host's VirtualBox Guest Additions on the guest system. Not needed on Mac OS X, but needed on Windows.
## Provision development vagrant
config.vm.define "development", primary: true do |development|
development.vm.box = "centos/7"
# development.disksize.size = "80GB"
development.vm.network "private_network", ip: "192.168.0.10"
development.vm.network :forwarded_port, guest: 22, host: 2222, id: 'ssh'
development.vm.hostname = "development"
development.vm.synced_folder ".", "/vagrant", type: "virtualbox"
development.vm.provider :virtualbox do |virtualbox|
virtualbox.name = "DevOps Class - development"
virtualbox.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10]
virtualbox.memory = 2048
virtualbox.cpus = 2
virtualbox.gui = false
development_docker_disk = './development_docker.vdi'
unless File.exist?(development_docker_disk)
virtualbox.customize ['createmedium', '--filename', development_docker_disk, '--size', 40 * 1024]
end
# the value of storage_system_bus depends on your platform
storage_system_bus = "IDE"
# Provisions a drive for Docker storage
virtualbox.customize ['storageattach', :id, '--storagectl', storage_system_bus, '--port', 1, '--device', 0, '--type', 'hdd', '--medium', development_docker_disk]
end
config.vm.provision "ansible_local" do |ansible|
ansible.playbook = "ansible/development-playbook.yml"
ansible.inventory_path = "hosts"
ansible.limit = "development"
ansible.install_mode = :default
ansible.compatibility_mode = "2.0"
ansible.verbose = true # true (equivalent to v), vvv, vvvv
# Pass top-level variables into Ansible from Vagrant
ansible.extra_vars = software_versions
if development.proxy.enabled
ansible.extra_vars[:http_proxy] = (!config.proxy.http ? "" : config.proxy.http)
ansible.extra_vars[:https_proxy] = (!config.proxy.https ? "" : config.proxy.https)
ansible.extra_vars[:ftp_proxy] = (!config.proxy.ftp ? "" : config.proxy.ftp)
ansible.extra_vars[:no_proxy] = (!config.proxy.no_proxy ? "" : config.proxy.no_proxy)
end
if development.ca_certificates.enabled
ansible.extra_vars[:CA_CERTIFICATES] = config.ca_certificates.certs
end
end
end
This section provisions and configures a development
vagrant used for development
-
The
config.vm.provision
block usesansible_local
to configure thedevelopment
vagrant. I've written Ansible roles underansible/roles
to automate the configuration of the vagrants.development.vm.box
loads the vagrant with thiscentos/7
vagrant base box. Vagrant curates a listing of base boxes here https://app.vagrantup.com/boxes/searchdevelopment.vm.synced_folder
mounts the class's project folder to/vagrant
path in the vagrant.- The
development.vm.provider
section tells the hypervisor how to configure the vagrant (i.e., how much memory, how many procesors)
-
ansible_local
does not require one to have Ansible installed on the host. Vagrant handles installing it on the vagrant and executing the specified playbook.- In this instance, I have
ansible.verbose
jacked to its highest setting providing loads ofdevelopment
logging. If too much information about what is going on unnerves you changing this tofalse
will disable verbose logging.
- In this instance, I have
## Provision the pipeline vagrant
config.vm.define "toolchain", autostart: false do |toolchain|
toolchain.vm.box = "centos/7"
# toolchain.disksize.size = "80GB"
toolchain.vm.network "private_network", ip: "192.168.0.11"
toolchain.vm.network :forwarded_port, guest: 22, host: 2223, id: 'ssh'
toolchain.vm.hostname = "toolchain"
toolchain.vm.synced_folder ".", "/vagrant", type: "virtualbox"
toolchain.vm.provider :virtualbox do |virtualbox|
virtualbox.name = "DevOps Class - toolchain"
virtualbox.customize ["guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10]
virtualbox.memory = 6144 #4096
virtualbox.cpus = 4
virtualbox.gui = false
toolchain_docker_disk = './toolchain_docker.vdi'
unless File.exist?(toolchain_docker_disk)
virtualbox.customize ['createmedium', '--filename', toolchain_docker_disk, '--size', 40 * 1024]
end
# the value of storage_system_bus depends on your platform
storage_system_bus = "IDE"
# Provisions a drive for Docker storage
virtualbox.customize ['storageattach', :id, '--storagectl', storage_system_bus, '--port', 1, '--device', 0, '--type', 'hdd', '--medium', toolchain_docker_disk]
end
config.vm.provision "ansible_local" do |ansible|
ansible.playbook = "ansible/toolchain-playbook.yml"
ansible.inventory_path = "hosts"
ansible.limit = "toolchain"
ansible.install_mode = :default
ansible.compatibility_mode = "2.0"
ansible.verbose = 'vvvv' # true (equivalent to v), vvv, vvvv
# Pass top-level variables into Ansible from Vagrant
ansible.extra_vars = software_versions
if toolchain.proxy.enabled
ansible.extra_vars[:http_proxy] = (!config.proxy.http ? "" : config.proxy.http)
ansible.extra_vars[:https_proxy] = (!config.proxy.https ? "" : config.proxy.https)
ansible.extra_vars[:ftp_proxy] = (!config.proxy.ftp ? "" : config.proxy.ftp)
ansible.extra_vars[:no_proxy] = (!config.proxy.no_proxy ? "" : config.proxy.no_proxy)
end
if toolchain.ca_certificates.enabled
ansible.extra_vars[:CA_CERTIFICATES] = config.ca_certificates.certs
end
end
end
This section provisions and configures the toolchain
vagrant. This is the beefy vagrant running GitLab. Drone CI, the private Docker registry, SonarQube, Selenium, Taiga...
Ansible is a "configuration management" tool that automates software provisioning, configuration management, and application deployment. Configuration management, deployment automation are two core repeated practices in DevOps, so for the purposes of the class Ansible addresses this concern in configuring the two vagrants.
Ansible was open source and then subsumed by Red Hat.
There are other "configuration management" tools, such as Chef and Puppet. There are of course even more, but they hold little or no market share (e.g., BOSH, Salt.)
Since, Ansible will work against multiple systems in the infrastructure at the same time it does this via what it refers to as an inventory.
The project's inventory, hosts
is located at the root of the project and contains:
controller ansible_connection=local
development ansible_connection=local
toolchain ansible_connection=local
[nodes]
development ansible_host=192.168.0.10
toolchain ansible_host=192.168.0.11
[developments]
development ansible_host=192.168.0.10
[toolchains]
toolchain ansible_host=192.168.0.11
# Used by Ansible roles to configure Docker DNS server settings
[dns]
# My home network DNS
#ns1 ansible_host=10.0.1.1
# MITRE DNS
#ns1 ansible_host=10.20.100.53
#ns2 ansible_host=10.20.200.53
# Google / Starbux DNS
#ns1 ansible_host=8.8.8.8
#ns2 ansible_host=8.8.4.4
# HAMP LAB DNS
#ns1 ansible_host=192.168.1.1
# OuterNET
ns1 ansible_host=192.52.194.138
ns2 ansible_host=198.49.146.138
The inventory collects all the vagrants under a [nodes]
group, and then defines each vagrant under their respective group either [developments]
or [toolchains]
.
NOTE
- The inventory also defines a
[dns]
group. DNS (domain name service) hosts will vary between environment. Please configure, for your environment.
In Ansible one defines playbooks to manage configurations of and deployments to remote machines. The playbooks I'm using to configure the vagrants exist in ansible/
.
The development
vagrant Ansible playbook (ansible/development-playbook.yml
) contains:
---
# Development Ansible playbook
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
- hosts: [developments]
remote_user: vagrant
roles:
- common
- docker
- docker-compose
- golang
- golint
- inspec
- drone-cli
- sonar-scanner-cli
# - openshift-cli
The file is written in a YAML-based DSL (domain specific language.)
The roles exist in ansible/roles
and permi the sharing of bits of configuration content with other users. Roles can also be found in the Ansible Galaxy, retrieved and placed into the ansible/roles
folder to be used, but I wrote all the roles for the class.
A role, such as a Taiga role is comprised of many components (e.g., files, templates), but at its core is the main task. The main task for the taiga
roles contains
---
# tasks file for taiga
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
- name: "copy to taiga files to /home/{{ ansible_user_id }}/taiga"
become: yes
copy:
src: files/
dest: "/home/{{ ansible_user_id }}/taiga"
owner: "{{ ansible_user_id }}"
group: "{{ ansible_user_id }}"
- name: "template files into /home/{{ ansible_user_id }}/taiga"
become: yes
template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
owner: "{{ ansible_user_id }}"
group: "{{ ansible_user_id }}"
with_items:
- { src: "templates/README.j2", dest: "/home/{{ ansible_user_id }}/taiga/README.MD" }
- { src: "templates/Dockerfile.j2", dest: "/home/{{ ansible_user_id }}/taiga/Dockerfile" }
- { src: "templates/dockerfile_build.j2", dest: "/home/{{ ansible_user_id }}/taiga/dockerfile_build.sh" }
- { src: "templates/docker-compose.j2", dest: "/home/{{ ansible_user_id }}/taiga/docker-compose.yml" }
- name: "make /home/{{ ansible_user_id }}/taiga/dockerfile_build.sh executable"
become: yes
file:
path: "/home/{{ ansible_user_id }}/taiga/dockerfile_build.sh"
mode: "u=rwx,g=r,o=r"
owner: "{{ ansible_user_id }}"
group: "{{ ansible_user_id }}"
- name: ensure user 'default' with a 1001 uid and a primary group of 'ROOT' exists
become: yes
user:
name: default
uid: 1001
group: root
- name: ensure ./volumes exist
become: yes
file:
path: "{{ item.path }}"
state: directory
owner: "{{ item.owner }}"
group: "{{ item.group }}"
with_items:
- { path: "/home/{{ ansible_user_id }}/taiga/volumes/static/" , owner: root, group: root }
- { path: "/home/{{ ansible_user_id }}/taiga/volumes/static/admin", owner: default, group: root }
- { path: "/home/{{ ansible_user_id }}/taiga/volumes/media/", owner: root, group: root }
- { path: "/home/{{ ansible_user_id }}/taiga/volumes/media/user", owner: default, group: root }
- name: spin up taiga via docker-compose
become: yes
docker_service:
build: yes
debug: yes
project_src: "/home/{{ ansible_user_id }}/taiga"
The Taiga
role dependes on docker-compose
, docker
, and common
having configured the vagrant.
So, what is Docker? What are Containers?
Whats a container?
- Containers are a form of lightweight virtualization first existing in 1979’s Version 7 UNIX operating system when the chroot command was developed.
- Since chroot, containers have continued to evolve within the Linux kernel to decouple applications from the operating system and run them in an isolated manner.
- A container is essentially an isolated processes running in user space.
- The benefit over Virtual Machines in that multiple containers can run on the same machine (in case of this class, a vagrant) sharing the OS kernel with other containers. Whereas a VM would require a full copy of an operating system in order to run the application. This makes containers insanely light-weight.
- Container support has thrived and seen popular adoption across various operating systems outside of the Linux including Windows Containers and Window’s direct ability to run Linux containers from the Hyper-V isolation work done by Microsoft.
- The Open Container Initiative (OCI), formed in 2015 maintains open industry standards for containers that focus on Runtime Specification (runtime-spec) and its partner project, the Image Specification (image-spec).
What are container images?
A container image is a lightweight, self-contained, executable package that includes everything needed to run your application including runtime, system tools, system libraries, and settings.
More can be read on the topic at
https://www.docker.com/what-container
You will build a couple Docker images and spin up a containers in this class.
And what is docker-compose?
Doceker-compose is a tool and domain specific language based on YAML used to define and run multi-container Docker applications.
What is YAML? YAML bills itself as a human-friendly data serialization standard for all programming languages. YAML also follows in in the computing tradition of being a recursive acronym, YAML Ain't Markup Language. Many of the tools used in this course make use of YAML, so you will see plenty examples of it.
You will edit a docker-compose.yml
file in this class in order to complete the configuration of one the key CI/CD tools.
In the command line of the host in the root of the class project, open ansible/tool-chain-playbook.yml
and make sure the roles are as so:
---
# Toolchain Ansible playbook
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
- hosts: toolchains
remote_user: vagrant
roles:
- common
- docker
- docker-compose
- golang
- golint
- docker-registry
# CI/CD Tools
- taiga
- gitlab
- plantuml-server
- drone
- drone-cli
- sonarqube
- golang-container-image
- inspec
- python-container-image
- golang-sonarqube-scanner-image
- standalone-firefox-container-image
- owasp-zap2docker-stable-image
# - inspec-container-image
# - standalone-chrome-container-image
# - gnome-desktop
# - firefox
# - chrome
# - openshift
Then enter into the command-line of the host at the root of the class project and enter into shell (I'll drop from time to time stating "into the shell" as it should understood.)
vagrant up toolchain
but presently as of December 18th, 2018 there is an issue downstream with thecentos/7
box requiring one to work around.
You will see a good deal of output and on the Windows OS, it will pester you to approve certain things. As a trust exercise blindly approve everything.
Once complete, open a secure shell (ssh) to the toolchain
vagrant
vagrant ssh toolchain
The command line will open a prompt a bash shell on the vagrant
[vagrant@toolchain ~]$
On this vagrant will be running a number of DevOps tools. This will take a while, so let's discuss what is being installed.
NOTE
- It is very possible a network anomaly may result in Ansible failing, if you can determine the role the automation failed in, you could try re-running the role the failure occured in and the subsequent roles like so:
Otherwise, re-run the automation from the start:
ANSIBLE_ARGS='--tags "python-container-image,golang-sonarqube-scanner-image, standalone-firefox-container-image,owasp-zap2docker-stable-image"' vagrant provision toolchain
vagrant up toolchain --provision
- Caution, it is easy to forget DevOps is as much about culture as it is about a methodology and repeated practices (often further mistakenly thought as "tools and automation"), so keep this in mind.
- The tools, methodology and repeated practices exist to support the culture.
- Again, I'll drop from time to time stating "into the shell" when instruction you to enter things in the CLI as it should understood.
Taiga is an Open Source project management platform for agile development.
There are many project management platforms for Agile.
Typically, Agile teams work using a visual task management tool such as a project board, task board or Kanban or Scrum visual management board. These boards can be implemented using a whiteboard or open space on a wall or in software. The board is at a minimum segmented into a few columns To do, In process, and Done, but the board can be tailored. I've personally seen boards for very large projects consume every bit of wallspace of a very large cavernous room, but as Lean-Agile has matured, teams have grown larger and more disparate, tools have emerged to provide a clear view into a project's management to all levels of concern (e.g., developers, managers, product owner, and the customer) answering:
- Are deadlines being achieved?
- Are team members overloaded?
- How much is complete?
- What's next?
Further, the Lean-Agile Software tools should provide the following capabilities:
- Dividing integration and development effort into multiple projects.
- Defining, allocating, and viewing resources and their workload across each product.
- Defining, maintaining, and prioritizing the accumulation of an individual product's requirements, features or technical tasks which, at a given moment, are known to be necessary and sufficient to complete a project's release.
- Facilitating the selection and assignment of individual requirements to resources, and the tracking of progress for a release.
- Permit collaboration with external third parties.
The 800lb Gorilla in this market segment is JIRA Software. Some of my co-workers hate it. It is part of the Atlassian suite providing provides collaboration software for teams with products including JIRA Software, Confluence, Bitbucket, and Stash.
NOTE
- Lean-Agile Project Management software's primary purpose is to integrare people and really not much else.
Taiga's documentation can be found at
It's canonical source can be found at
https://github.com/taigaio/taiga-front-dist/
dedicated to the front-end, and
https://github.com/taigaio/taiga-back/
dedicated to the back-end.
Taiga doesn't directly offer a Docker container image for, but I've authored an image that collapses both taiga-front-dist and -back onto one container behind an NGINX reverse proxy.
Once, stood up your instance of Taiga will reachable at
The default admin accont username and password are
admin
123123
GitLab is installed on the toolchain
vagrant, where it will be accessible at http://192.168.0.11:10080/.
GitLab Community Edition (CE) is an open source end-to-end software development platform with built-in version control, issue tracking, code review, and CI/CD.
For Agile teams to collaborate, a Configuration management (CM) is necessary to coordinate the development of new feature, changes, and experimentation. Also, a CM system (CMS) provides a history of changes, and thereby, the ability to roll back to a version known to be acceptable.
At a minimum, the following items will be placed under revision control in CM:
-
Source code,
-
If a database is needed, schema initialization and the migration between versions,
-
Text documentation containing
-
a synopsis (i.e., project name, overview, etc.),
-
version description,
-
guidance covering
- build,
- unit testing, and
- installation
-
a contributor enumeration,
-
license and/or ownership declaration with contacts, etc.,
-
A single CMS and the associated workflow (e.g., GitHub Workflow) can serve as the focal point for the entire enterprise thereby provide centralized version control if all documentation is authored in a lightweight markup language with plain text formatting syntax (e.g., Markdown, PlantUML).
A CMS must facilitate best practices, not limited to:
-
A means for developers to copy and work off a complete repository thereby permitting
- Private individual work to later be synchronized via exchanging sets of changes (i.e., patches) through a means described as "distributed version control", and
- Pre-flight build and test of their source code in their own private workspace, so as to minimize the chance of committing broken or untested source code thereby encouraging
- The committing of completed source code only.
-
Granular commits that communicate the motivation for the commit (i.e., the what and why). For example, for a change these could be:
- the inclusion of a new feature,
- a bug fix,
- the removal of dead code
-
Reducing the risk breaking a build by
- Utilize branching to separate different lines of development, and
- Standardize on CMS workflows (e.g., GitHub Workflow),
-
Make builds be self-testing (i.e., ingrain testing) by including unit and integration test with the source code so that it can be executed by
- the build automation, and
- the Continuous Integration service.
-
Trigger follow-on activities orchestrated by the Continuous Integration Service.
GitLab's documentation can be found at
It's canonical source can be found at
https://hub.docker.com/r/sameersbn/gitlab/
https://gitlab.com/gitlab-org/gitlab-ce
I'm using Sameer Naik's Docker container image for GitLab. The image can be found at
https://hub.docker.com/r/sameersbn/gitlab/
And its canonical source is located at
https://github.com/sameersbn/docker-gitlab
Once, stood up your instance of GitLab will reachable at
GitLab requires you to enter a password for its root account. You will be using the root account to host your repositories vice creating your own, but if you want you can. There is nothing stopping you.
I would suggest for the purposes of the class choosing something simple, but at least the same number of characters as "password".
Perform the following steps as the GitLab adminstrator:
- Log into GitLab as root with the password you entered in the prior step
- Click the wrench at the top of the page to enter the
Admin area
- In the
side bar
running the length of the left-side of the page, click thegear
-icon, selectNetwork
, and expandOutbound requests
- Under
Outbound requests
, checkAllow requests to the local network from hooks and services
and then clickSave Changes
- Log into GitLab as root with the password you entered in the prior step
- Click the wrench at the top of the page to enter the
Admin area
- In the
side bar
running the length of the left-side of the page, click thegear
-icon, selectCI/CD
, and expandContinuous Integration and Deployment
- Uncheck
Default to Auto DevOps pipeline for all projects
and then clickSave Changes
Drone CI often referred to simply as "Drone" is installed on the toolchain
vagrant, where it will be accessible at http://192.168.0.11/ after some user configuration that will be explained later in the Integrate Drone CI with GitLab section.
Drone is essentially a Continuous Delivery system built on container technology.
Drone is distributed as a Docker image. Drone CI can be run with an internal SQLite database, but it is advisable to run with an external database and this the configuration the class uses. It also integrates with multiple version control providers (i.e., GitHub, GitLab, BitBucket, Stash, and Gogs). Both CMS and database are configured using environment variables passed when the Drone CI container is first to run. Drone plugins (really just containers) can be used to deploy code, publish artifacts, send a notification, etc. Drone's approach to plugins is novel as plugins are really just Docker containers distributed in a typical manner. Each plugin is designed to perform pre-defined tasks and is configured as steps in your pipeline. Plugins are executed with read/write/execution access at the root of the source branch, therefore, permitting the pipeline to interact with the specific branch of source to build, test, bundle, deliver, and deploy. Developers create a Drone CI pipeline by placing a .drone.yml
file in the root of the repository. The .drone.yml is authored in a domain specific language (DSL) that is a superset of the docker-compose DSL to describe the build with multiple named steps executed in a separate Docker container having shared disk access to the specific branch of the source repository.
Drone and its brethren (e.g., Jenkins CI, GitLab CI/CD) is used to facilitate Continuous Integration (CI), a software development practice where members of an Agile team frequently integrate their work in order to detect integration issues as soon as possible. Each integration is orchestrated through a service that essentially assembles a build and runs tests every time a predetermined trigger has been met; and then reports with immediate feedback.
I don't use Jenkins unless I have to. Why? I'm simply not a fan. Initially, because its plugin architecture is painful to manage and your pipelines existed entirely in the Jenkins CI tool itself. Later, Jenkins CI introduced Groovy-based Jenkin Pipelines that are CMed with your project's source. Every orchestrator has based their DSL on YAML and although I love the Groovy language for its power, I don't find it makes for a good orchestration language. Your opinion may differ. I'm okay with that. Really. I am.
There are also SaaS CI/CD tools, such as Travis CI and Circle CI. These are great, free CI/CD orchestrators.
My java-stix project hosted on GitHub.com uses both Travis CI and Circle CI as part of its continuous integration.
The Travis CI orchestration contains
language: java
dist: precise
jdk:
- oraclejdk7
before_install:
- chmod +x gradlew
env: GRADLE_OPTS=-Dorg.gradle.daemon=true
env: CI_OPTS=--stacktrace
install: /bin/true
script: "./gradlew -x signArchives"
Whereas, the Circle CI orchestration contains
machine:
java:
version: oraclejdk7
environment:
GRADLE_OPTS: -Dorg.gradle.daemon=true
CI_OPTS: --stacktrace --debug
test:
override:
- ./gradlew -x signArchives
They are essentially similar both requiring the Oracle JDK and use Gradle to build and unit test the code. Both SaaS under the covers uses containers to run the builds.
In another GitHub hosted project, the java-stix-validator the Travis CI orchestration contains
language: java
jdk:
- oraclejdk8
before_install:
- chmod +x gradlew
env: CI_OPTS=--stacktrace
install: "/bin/true"
script: "./gradlew build -d"
deploy:
provider: heroku
api_key:
secure: m9Gbt0Oyqtjwyu4Y8CVobNNnj1q5mFt+Ygi2wiDWlf/RunLOj2CE8YAYuRyEAbpCOd1lrmrhmQb8uQAfiydauYBcQE5yyOlyIhNkrLi2m1+we0VeWWr6gxIVz57VuAhfbzoMtvkhmxl/Ey0U+1vI7tYurK0thzFUyQFqZh1wNq6EldIfHxVNxDZbEVtkDzFtK5cmVnPE8HM9xaQmuV7k3NhrvRS4pzN87uvndfFVb0vDhLmg5DulF+PLkdpP9UC5jsAE1HXMBL0cTtsvSHUkIyO7qhLb0RFAzVdRMvn7kEW2Q0ekoK09sPR13VwmfjewzHSNrWIf+rjJx7EzoBzbq5/VmC9nxH1oiGpXxoAG08pJjcQYMSxsa2JZLH8dSIEaMgOFNOxkrAhcqP59xXWZ9WVLCYPSN4atmg4L6etJOzFqfz3jAp40AB4Eu2QU49c60r6BH31Xj8ymjKMKqnlL199qCoqfZtv7FYqOFG3keLeWvL/F7JhmtV+JdvuqVPEvNq2D1b3kdCKk2cw4lmRCwC9hdT2oXTCwhjQvYwSm0sHQ98aeV55FkE7DuH4B+CuzYw4N9K78j3eQtW2Oas1lLCoHSDXgA/4O79RlM8p0nLa3MjdVq5OSIjbcCqhDLBe8nc5ucSpMMjnjNvhAKvcyrc5AbXdIVaLVvE2azMuZJLo=
app: agile-journey-9583
on:
repo: STIXProject/java-stix-validator
To deploy the web application to Heroku, one of the first PaaS (Platforms-as-Service). The code was last committed in 2015 and the code is still running free on Heroku at http://agile-journey-9583.herokuapp.com/#/
More details on Drone is sprinkled across the class. As you can see I favor being a polyglot when it comes to CI/CD.
Drone's main site is at
Its documentation is at
Its plugin market is at
Drone's canonical source can be found at
https://github.com/drone/drone
Drone is distributed as a container image and can be found respectfully at
https://hub.docker.com/r/drone/drone/
and
https://hub.docker.com/r/drone/agent/
Once, stood up your instance of Drone CI will reachable at
Drone will authenticate you off GitLab once integrated.
Perform the follwing tasks.
This README.MD has embedded PlantUML diagrams for you visual learners without the integration the diagrams will remain in the source code.
You'll need to enable the integration between GitLab and the PlantUML-Server by accomplishing the following:
- As
root
log into GitLab with the password you entered in the prior step, - Click on the
wrench
to enter theAdmin Area
, - Then click
Settings
or thegear
-icon and then selectIntegrations
. - Expand PlantUML and then check Enable PlantUML checkbox, set the
PlantUML URL
to http://192.168.0.11:8081 - Then click
Save changes
.
Once, Drone is configured, accomplish the steps.
The Ansible automation will install patched Docker containers for Drone and template the docker-compose file used to spin it up, but the automation will not spin up Drone as it requires human intervention in the absence of automation.
So, you will need to accomplish the following steps, while at the same time becoming familiar with GitLab.
- As
root
log into GitLab with the password you entered in the prior step, - Click on the
wrench
to enter theAdmin Area
, - Click
Applications
, - Click
New Application
, - Enter
Drone CI
forName
, - Enter
http://192.168.0.11/login
, - Check off
api
andread_user
, - Click
submit
, - Open a secure shell to the
toolchain
vagrant at the root of the project with
vagrant ssh toolchain
The command line will open a prompt to the vagrant
[vagrant@toolchain ~]$
Enter the drone
directory with
cd ~/drone
- Open
docker-compose.yml
in a text editor.
- DRONE_GITLAB_CLIENT=Your Application Id
- DRONE_GITLAB_SECRET=Your Secret
Around line 50, replce Your Application Id
and Your Secret
with Application Id
and Secret
provided by GitLab, respectively. Save your changes and exit the editor.
- Then enter into the command line
docker-compose up -d
Command line output will resemble
[vagrant@toolchain drone]$ docker-compose up -d
drone_drone-postgresql_1 is up-to-date
Recreating drone_drone-server_1 ... done
Recreating drone_drone-agent_1 ... done
You can drop -d
on the docker-compose
so as to not daemonize the drone, so you can monitor its logging for issues.
- Exit the root session.
exit
- Open http://192.168.0.11 in the browser and click
Authorize
to permit Drone to use your GitLab account.
You will only have to do these step in regards to Drone once. Drone should always restart in the toolchain vagrant even if you halt
the vagrant and later bring it up
.
- In GitLab (http://192.168.0.11:10080/) click on
Projects
in the upper left. a. SelectCreate Project
.
b. Or click http://192.168.0.11:10080/projects/new - Leave the
Project path
defaulted tohttp://192.168.0.11:10080/root/
. - Enter
hands-on-DevOps
(the form field the page defaults to) for theProject name
. - Provide an optional
Project description
. Something descriptive, such as, "An awesome DevOps class". - Make the application
public
to save yourself from entering your username and password when cloning. - Click the green
Create Project
button on the lower left.
The UI will refresh to show you a landing page for the project that should be accessible from http://192.168.0.11:10080/root/hands-on-DevOps
On your host in the DevOps class project:
- Enter into the following into the command line
git config --global user.name "Administrator"` git config --global user.email "[email protected]"
- Then enter the following
git remote add toolchain http://192.168.0.11:10080/root/hands-on-DevOps.git git push -u toolchain --all git push -u toolchain --tags
You now have a clone of the project hosted in the GitLab running on the toolchain
vagrant. This way you can open the README.MD and follow along.
SonarQube provides the capability to show the health of an application's source code, highlighting issues as they are introduced. SonarQube can be extended by language-specific extensions/plugins to report on duplicated code, coding standards, unit tests, code coverage, code complexity, comments, bugs, and security vulnerabilities.
SonarQube's main site is at
Its documentation is at
https://docs.sonarqube.org/display/SONAR/Documentation
SonarQube's canonical source can be found here
https://github.com/SonarSource/sonarqube
I'm using the container image provided at
https://hub.docker.com/_/sonarqube/
Once, stood up your instance of SonarQube will reachable at
The default admin account username and password is
admin
In another command line of the host in the root of the class project
Open ansible/development-playbook.yml
and make sure these roles are uncommented:
---
# Development Ansible playbook
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
- hosts: [developments]
remote_user: vagrant
roles:
- common
- docker
- docker-compose
- golang
- golint
- drone-cli
- sonar-scanner-cli
# - openshift-cli
Then in the command-line of the host (not while ssh'ed into the toolchain vagrant) at the root of the class project
./up_development.sh
Typically, all one has to do is type
vagrant up development
but presently as of December 18th, 2018 there is an issue downstream with thecentos/7
box requiring one to work around.
You will see a good deal of output.
Once complete open a secure shell to the development
vagrant
vagrant ssh development
Command line will open a prompt to the vagrant
[vagrant@development ~]$
NOTE
- Again, it is very possible a network anomaly may result in Ansible failing, if you can determine the role automation failed in you could try re-running the role the failure occured in and the subsequent roles like so:
Otherwise, re-run the automation from the start:
ANSIBLE_ARGS='--tags "inspec,drone-cli,sonar-scanner-cli"' vagrant provision development
vagrant up development --provision
The prior toolchain
and development
vagrants are required to be up and running for the following sections.
In this next part, we will create a simple helloworld GoLang project to demonstrate Continuous Integration. GoLang lends itself well to DevOps and underlines almost every new tool you can think of related to DevOps and cloud (e.g., golang / go, docker / docker-ce, kubernetes / kubernetes, openshift / origin ,hashicorp / terraform, coreos / etcd, hashicorp / vault, hashicorp / packer, hashicorp / consul, gogits / gogs, drone / drone.)
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog” #FFFFFF
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based Continuous Integration”
-left-> (*)
A backlog is essentially your to-do list, a prioritized list of work derived from the roadmap (e.g., the outline for future product functionality and when new features will be released) and its requirements.
Open Taiga in your web browser
The default admin account username and password are
admin
123123
Complete the follow to track your progress in completing the Golang helloworld project
-
Click
Create Project
. -
Select
Kanban
. In a Kanban board work moves from left to right with each column represents a stage within the value stream. -
Give your project a name. For example,
Helloworld
and a description, such as,My Kanban board for this awesome helloworld app
and then clickCREATE PROJECT
. -
You can skip this step and opt to chose to click
><
to foldREADY
,USER STORY STATUS
andARCHIVED
only after completing step 6. Otherwise, you can edit your Kanban board to justNEW
,IN PROGRESS
, andDONE
by a. On the bottom left, click theADMIN
gear.
b. ClickATTRIBUTES
.
c. Scroll down toUSER STORY STATUS
.
d. Hover overReady
, click the trash icon to delete and clickACCEPT
.
c. Do the same forReady for test
andArchived
.
d. Click theKANBAN
icon on the far left. It Looks like columns. And then reload the browser to get the changes to take. -
In the
NEW
column selectAdd New bulk
icon that looks like a list and when the page updates cut-and-paste the lines below into the text box and clickSAVE
.Create the project’s backlog Create the project in GitLab Setup your project on the development Vagrant Author the application Align source code with Go coding standards Lint the code Build the application Run your application Author the unit tests Write the Makefile Author Drone-based Continuous Integration
Track your progress in Taiga as you work through each section.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab” #FFFFFF
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
- In GitLab (http://192.168.0.11:10080/) click on
Projects
in the upper left. a. SelectCreate Project
.
b. Or click http://192.168.0.11:10080/projects/new - Leave the
Project path
defaulted tohttp://192.168.0.11:10080/root/
. - Enter
helloworld
(the form field the page defaults to) for theProject name
. - Provide an optional
Project description
. Something descriptive, such as, "GoLang helloworld application for the hands-on DevOps class.". - Make the application
public
to save yourself from entering your username and password when cloning. - Click the green
Create Project
button on the lower left.
The UI will refresh to show you a landing page for the project that should be accessible from http://192.168.0.11:10080/root/helloworld
NOTE
- Sometimes GitLab with release
11.5.1
will throw an error on the step 6 above. Simply click the greenCreate Project
button again.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant” #FFFFFF
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
On your host open a shell to development
vagrant, and configure your user name and email:
git config --global user.name "Administrator"
git config --global user.email "[email protected]"
The Ansible golang
role (ansible/roles/golang
) in the class project on your host will already have configured:
- Your
$GOPATH
and$GOBIN
environmental variable. - Added
$GOBIN
to your PATH environmental variable. - Created
go
workspace in vagrant user's home directory (i.e.,/home/vagrant/go
directory containingbin
,pkg
, andsrc
.)
Go source code is placed in the src
directory under a namespace (i.e., a unique base path to avoid naming collisions under which all your go code will reside.) In open source software development, it is typical to use github.com account path. Mine is github.com/nemonik
, so I would create this base path via (You do the same., so we are all on the same page.)
mkdir -p $GOPATH/src/github.com/nemonik
Now lets create the GoLang helloworld
project to demonstrate Continuous Integration via
cd $GOPATH/src/github.com/nemonik
git clone http://192.168.0.11:10080/root/helloworld.git
cd helloworld
NOTE
- Ignore the
warning: You appear to have cloned an empty repository.
warning. This is perfectly normal. Then move into the clone of your repository via
So that you do not commit certian files to GitLab when you push, create a .gitignore
file with your editor with the following contents
# OS-specific
.DS_Store
.scannerwork
*.out
*.json
helloworld
NOTE
- Make sure you pre-pend that dot (
.
) at the start of.gitignore
. In *NIX Dot-files are hidden files. .gitignore
will not show up if you simply list the file system via thels
command, but they do if you usels -a
orls --all
. Either arguments configuresls
to not ignore entries starting with.
.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application” #FFFFFF
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
In the project follder (i.e., /home/vagrant/go/src/github.com/nemonik/helloworld
), create main.go
in emacs, nano, vi, or vim with this content:
package main
import "fmt"
func main() {
fmt.Println(HelloWorld())
}
func HelloWorld() string {
return "hello world"
}
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards” #FFFFFF
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
Format source code according to Go coding standards using
go fmt
Results in the code being formatted to
package main
import "fmt"
func main() {
fmt.Println(HelloWorld())
}
func HelloWorld() string {
return "hello world"
}
You'll see the difference if you cat
your source
cat main.go
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code” #FFFFFF
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
Already installed on your development
vagrant is golint
. Where go fmt
reformatted the code to GoLang standards, golint
prints style mistakes.
To run golint
, in the root of the helloworld
project execute
golint
Command line out will be
[vagrant@development helloworld]$ golint
hello.go:9:1: exported function HelloWorld should have comment or be unexported
Fix the error by editing main.go
to
package main
import "fmt"
func main() {
fmt.Println(HelloWorld())
}
// HelloWorld returns "hello world"
func HelloWorld() string {
return "hello world"
}
Run golint
again and it should return no output indicating it sees nothing wrong.
Build the project by executing
go build -o helloworld .
Success returns no command line output. What? Did you want a cookie? No cookie for you. This is GoLangs way of doing things. Silence is golden and means things went fine. Otherwise, go back and fix the mistakes in your code.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application” #FFFFFF
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
Running your application
./helloworld
The command line output will be
[vagrant@development hello]$ ./helloworld
hello world
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests” #FFFFFF
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
GoLang ships with a built-in testing
package
https://golang.org/pkg/testing/
for automated unit testing of Go packages. Unit testing is a software development process where the smallest testable components of an application are individually tested for proper operation. Unit testing offers the biggest return for dollars spent in comparison to integration and functional testing.
For more on this topic read Martin Fowler's
https://martinfowler.com/bliki/TestPyramid.html
In your editor create main_test.go
with this content:
package main
import (
"os"
"testing"
)
func TestMain(m *testing.M) {
os.Exit(m.Run())
}
func TestHelloWorld(t *testing.T) {
if HelloWorld() != "hello world" {
t.Errorf("got %s expected %s", HelloWorld(), "hello world")
}
}
Execute the unit test by entering
go test -v -cover
The command line returns
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok github.com/nemonik/helloworld 0.002s
This step and all the proceeding follows of a DevOps tenant of Devops, "Developers are expected to pre-flight new code."
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile” #FFFFFF
-left->“Author Drone-based\nContinuous Integration”
-left-> (*)
Build automation is a key practice of CI. So, let's make the build reproducible by automating everything we've done this far via authoring a Makefile.
In the root of the project create Makefile
and add the following contents
GOCMD=go
GOFMT=$(GOCMD) fmt
GOBUILD=$(GOCMD) build
GOCLEAN=$(GOCMD) clean
GOTEST=$(GOCMD) test
GOLINT=$(GOCMD)lint
GOGET=$(GOCMD) get
BINARY_NAME=helloworld
all: build
clean:
$(GOCLEAN)
rm -f $(BINARY_NAME)
fmt:
$(GOFMT)
lint: fmt
$(GOGET) golang.org/x/lint/golint
golint
test: lint
$(GOTEST) -v -cover ./...
build: test
$(GOBUILD) -o $(BINARY_NAME) -v
run:
$(GOBUILD) -o $(BINARY_NAME) -v ./...
./$(BINARY_NAME)
NOTE
- Each line indentation is a
tab
and not a series ofspace
characters.Make
will fail to execute if these tabs are converted to a series of space characters.
Save the file and exit your editor.
Okay, let's try it out
make all
The output will resemble
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok github.com/nemonik/helloworld (cached) coverage: 50.0% of statements
go build -o helloworld -v
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->“Create\nthe project’s\nbacklog”
-right->“Create\nthe project\nin GitLab”
-right->“Setup\nyour project on\nthe development\nVagrant”
-right->“Author\nthe application”
-right->“Align source\ncode with Go\ncoding standards”
-right->“Lint\nthe code”
-right->“Build\nthe application”
-down->“Run\nyour application”
-left->“Author\nthe unit tests”
-left->“Write the Makefile”
-left->“Author Drone-based\nContinuous Integration” #FFFFFF
-left-> (*)
CI integrates all of the steps we have worked to ensure a high quality build into a pipeline, so let's do that.
We're going to author a continuous integration pipeline for our application and execute it on Drone. Drone expects a .drone.yml
to exist at the root of the project and will execute the pipeline it contains when the project is committed to GitLab.
A pipeline is broken up into multiple named steps, where each step executes in an ephemeral (i.e., does its job and then poof it is gone) Docker container with shared disk access to the project's workspace. The benefit of this approach is that it relieves you from having to create and maintain slaves to execute your pipelines.
Drone automatically clones your project's repo (Short for "repository.") into a volume (referred to as the workspace) shared by each Docker container (including plugins service containers).
In your editor create .drone.yml
file at the root of the helloworld
project:
kind: pipeline
name: default
steps:
- name: build
image: 192.168.0.11:5000/nemonik/golang:1.12.1
commands:
- make lint
- make test
- make build
- name: run
image: 192.168.0.11:5000/nemonik/golang:1.12.1
commands:
- make run
NOTE
- Make sure you pre-pend that dot (
.
) at the start of.drone.yml
. - Make sure to leave two consecutive returns to indicate the end of the
.drone.yml
file or a parse error when executed in Drone will result.
The pipeline is authored in YAML like almost all the CI orchestrators out there except for Jenkin's Pipelines, whose you author in Groovy-based DSL.
steps:
- defines a list of steps to build, test and deploy your code.build
andrun
- are the names of the step. These are yours to name. Name steps something meaningful as to what the step is orchestrating. Each step is executed serially, in the order defined.image: 192.168.0.11:5000/nemonik/golang:1.12.1
- The build step is using the nemonik/golang container tagged1.12.1
retrieved from private Docker registry located at192.168.0.11:5000
. Drone uses Docker images for the build environment, plugins and service containers. Drone spins them up for the execution of the pipeline and when no longer needed they go poof.commands
- Is a collection of terminal commands to be executed. These are all the same commands we executed previously in the command line. If anyone of these commands were to fail returning a non-zero exit code, the pipeline will immediately end resulting in a failed build.
- Open http://192.168.0.11/ in your browser and authenticate through GitLab on into Drone, if you need to.
- Then select
SYNC
. The arrows will chase each other for a bit. - Then click
root/helloworld
repo andACTIVATE REPOSITORY
, thenSAVE
under theMain
section to enable Drone orchestration for the project. - Then click the Drone logo in the upper left of the page to return home.
You won't have any builds to start, but when you do the builds will increment starting from 1. Red is failed the build. Yellow-orange is a presently executing build. Green is a build that passed. When a build does start, click on its row to open and monitor it. The UI will update as the build proceeds informing you as to its progress.
To trigger the build, in your ssh connection to development
simply commit your code:
git add .
git commit -m "Added Drone pipeline"
git push origin master
Immediately after you enter your GitLab username/password open http://192.168.0.11/root/hello in your browser, if you re-use an existing tab to this page refresh the page.
The execution of this pipeline will follow as so:
- A new build will appear. Click on it.
- Drone will clone your project's repository in a
clone
step. - And then will execute a
build
andrun
steps in order each spinning upnemonik/golang:1.12.1
container, whose container image was patched if proxy environmental variable are set to work behind MITRE's http proxy and SSL introspection. (Later, you may want examine the contents of ansible/roles/golang-container-image/files/Dockerfile for more details as to how this was accomplished.) - These steps execute the commands in the same way you executed them yourself: a. make lint b. make test c. make build d. make run
The output of the build
(An arbitrary name. You could use "skippy".) step will resemble:
+ make lint
go fmt
go get golang.org/x/lint/golint
golint
+ make test
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok _/drone/src 0.004s coverage: 50.0% of statements
+ make build
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
coverage: 50.0% of statements
ok _/drone/src (cached) coverage: 50.0% of statements
go build -o helloworld -v
_/drone/src
The output of the run
step will resemble:
+ make run
go build -o helloworld -v ./...
./helloworld
hello world
Our build was successful. Drone uses a container's exit code to determine success or failure. A container's non-zero exit code will cause the pipeline to exit immediately.
That's it. This is essentially CI. Remember, CI stands for Continuous Integration. Scintillating isn't it?
The helloworld
project can be viewed completed at
https://github.com/nemonik/helloworld
Like helloworld
, the helloworld-web
project is a very simple application that we will use to explore Continuous Deliver. Remember, Continuous Delivery builds upon Continuous Integration.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog” #FFFFFF
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Open Taiga in your web browser
Complete the follow to track your progress in completing the helloworld-web project
-
Click
Create Project
. -
Select
Kanban
. A Kanban board shows how work moves from left to right, each column represents a stage within the value stream. -
Give your project a name. For example,
Helloworld-web
and a description, such as,My Kanban board for this awesome helloworld-web app
and then clickCREATE PROJECT
. -
You can skip this step and opt to chose to click
><
to foldREADY
,USER STORY STATUS
andARCHIVED
only after completing step 6. Otherwise, you can edit your Kanban board to justNEW
,IN PROGRESS
, andDONE
by a. On the bottom left, click theADMIN
gear. b. ClickATTRIBUTES
. c. Scroll down toUSER STORY STATUS
. d. Hover overReady
, click the trash icon to delete and clickACCEPT
. c. Do the same forReady for test
andArchived
. d. Click theKANBAN
icon on the far left. It Looks like columns. And then reload the browser to get the changes to take -
In the
NEW
column selectAdd New bulk
icon that looks like a list and when the page updates cut-and-paste the lines below into the text box and clickSAVE
.Create the project's backlog Create the project in GitLab Setup the project in the development Vagrant Author the application Build and run the application Run gometalinter.v2 on application Fix the application Author unit tests Perform static analysis on the CLI Write the Makefile Dockerize the application Run the Docker container Push the container image to the private registry Configure Drone to execute the CI/CD pipeline Add Static Analysis (SonarQube) step to pipeline Add build step to pipeline Add container image publish step to pipeline Add container deploy step to pipeline Add complaince automation test (InSpec) step to pipeline Add automated funtional test (Selenium) to pipeline Add DAST step (OWASP ZAP) to the pipeline
Track your progress in Taiga as you work through each section.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab" #FFFFFF
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
- In GitLab (http://192.168.0.11:10080/) click on
Projects
in the upper left. a. SelectYour projects
from the dropdown. b. Click the greenNew Project
on the far right underProjects
, or c. Click http://192.168.0.11:10080/projects/new - Leave the
Project path
defaulted tohttp://192.168.0.11:10080/root/
. - Enter
helloworld-web
for theProject name
. Be careful with the spelling. - Provide an optional
Project description
. Something descriptive, such as, "GoLang helloworld application for the hands-on DevOps class.". - Save yourself a headache, and make the application
public
. - Click the green
Create Project
button on the lower left.
The UI will refresh to show you landing page for the project that should be accessible from
http://192.168.0.11:10080/root/helloworld-web
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant" #FFFFFF
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Create a helloworld-web
GitLab hosted repo as you did prior for the helloworld
project.
If you do not already have a shell open on development
vagrant, on your host in a shell at the root of the class project enter the following
vagrant ssh development
Once connected to your development
vagrant enter
cd ~/go/src/github.com/nemonik
git clone http://192.168.0.11:10080/root/helloworld-web.git
cd helloworld-web
NOTE
- Again, ignore the "warning: You appear to have cloned an empty repository." warning.
- The
git clone
will fail if you did not name your project correctly while in GitLab.
So, that you do not commit certian files to GitLab when you push, create a .gitignore
file with your editor with this content
# OS-specific
.DS_Store
.scannerwork
*.out
*.json
helloworld-web
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application" #FFFFFF
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Create main.go
in emacs, nano, vi, or vim with this content:
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", handler)
fmt.Print("listening on :3000\n")
http.ListenAndServe(":3000", logRequest(http.DefaultServeMux))
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello world!\n")
}
func logRequest(handler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Printf("%s %s %s\n", r.RemoteAddr, r.Method, r.URL)
handler.ServeHTTP(w, r)
})
}
Format and lint the code like you did previously.
go fmt
golint
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application" #FFFFFF
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Run the application
go build -o helloworld-web .
No output means things are peachy. Otherwise fix your mistakes.
Now run
./helloworld-web
Command line output will be
[vagrant@development helloworld-web]$ ./helloworld-web
listening on :3000
To run the application, either
- Open http://192.168.0.10:3000 in a web browser, or
- Enter
curl http://192.168.0.10:3000
into the command-line.
Both will return:
Hello world!
The command line will output
192.168.0.1:63201 GET /
192.168.0.1:63201 GET /favicon.ico
crl-c
will stop your application.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application" #FFFFFF
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
The following command line steps are optional as you have installed these dependencies for the helloworld
application
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Output will resemble
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
Run our linters
gometalinter.v2
And the output like the follwing outout will be returned after some time
main.go:11:21:warning: error return value not checked (http.ListenAndServe(":3000", logRequest(http.DefaultServeMux))) (errcheck)
main.go:15:13:warning: error return value not checked (fmt.Fprintf(w, "Hello world!\n")) (errcheck)
main.go:11::warning: Errors unhandled.,LOW,HIGH (gosec)
Oops. Line 11 and line 15 have problems:
-
http.ListenAndServe(":3000", logRequest(http.DefaultServeMux))
on line 11 returns anerr
if it runs into problems. We need to handle the problem by logging theerr
and exit. -
fmt.Fprintf
on line 17 returns the number of bytes written and any write error encountered. We need to handle that too. We can swallow the number of bytes return via_
, collect theerr
, log it, and exit.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application" #FFFFFF
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Let's fix the problems by importing fmt
and logging the err
s:
package main
import (
"log"
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", handler)
fmt.Print("listening on :3000\n")
log.Fatal(http.ListenAndServe(":3000", logRequest(http.DefaultServeMux)))
}
func handler(w http.ResponseWriter, r *http.Request) {
_, err := fmt.Fprintf(w, "Hello world!\n")
if err != nil {
log.Fatal(err)
}
}
func logRequest(handler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Printf("%s %s %s\n", r.RemoteAddr, r.Method, r.URL)
handler.ServeHTTP(w, r)
})
}
Run our linters again
gometalinter.v2
And after some time, nothing is returned. Problem solved.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests" #FFFFFF
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
We don't have any unit tests, so let's fix that.
Create the text file main_test.go
in emacs, nano, vi, or vim with this content:
package main
import (
"io/ioutil"
"net/http"
"net/http/httptest"
"os"
"testing"
)
func TestLogRequest(t *testing.T) {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(handler))
l := logRequest(mux)
// Create an http request
req, _ := http.NewRequest("GET", "/", nil)
// Create http.ResponseWriter for test inspection
recorder := httptest.NewRecorder()
// Capture Stdout
rescueStdout := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
l.ServeHTTP(recorder, req)
// Stop capturing Stdout
w.Close()
out, _ := ioutil.ReadAll(r)
os.Stdout = rescueStdout
// Compare
if string(out) != " GET /\n" {
t.Errorf("logRequest didn't log the expected \"GET /\"")
}
}
func TestHandler(t *testing.T) {
// Create an http request
req, _ := http.NewRequest("GET", "/", nil)
// Create http.ResponseWriter for test inspection
recorder := httptest.NewRecorder()
// Call the handler
handler(recorder, req)
// Inspect the http.ResponseWriter
if recorder.Code != http.StatusOK {
t.Errorf("Server did not return %v", http.StatusOK)
}
if recorder.Body.String() != "Hello world!\n" {
t.Errorf("Body contain \"%v\" instead of expected \"Hello world!\"", recorder.Body.String())
}
}
func TestMain(m *testing.M) {
os.Exit(m.Run())
}
Execute the unit test by entering
go test -v -cover
The command line will return something like
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web 0.005s
Notice we only scored 55.6% coverage, but had unit tests for all our methods? This where dicernment comes in.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI" #FFFFFF
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
SonarQube provides the capability to show the health of an application's source code, highlighting issues as they are introduced.
This section is optional as the sonar-scanner
command line tool will do all this automagically for you.
Open in your browser
http://192.168.0.11:9000/admin/projects_management
If you have to authenticate, your username and password is
admin
Click Create project
.
Provide the name
of
helloworld-web
Provide the key
of
helloworld-web
- Open you browser to http://192.168.0.11:9000/admin/marketplace
- If you need to login as
admin
with the paswordadmin
. - Search for
SonarGo
. - Click
Install
. - Click
Restart
. - Yes, your sure. Click
Restart
in the modal.
Generate the reports and submit to SonarQube
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
Output will look like
[vagrant@development helloworld-web]$ gometalinter.v2 > gometalinter-report.out
[vagrant@development helloworld-web]$ golint > golint-report.out
[vagrant@development helloworld-web]$ go test -v ./... -coverprofile=coverage.out
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web 0.004s coverage: 55.6% of statements
[vagrant@development helloworld-web]$ go test -v ./... -json > report.json
[vagrant@development helloworld-web]$ sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=97ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=41ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=92ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=22ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=51ms
INFO: Load active rules
INFO: Load active rules (done) | time=275ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=37ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=22ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 6 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=178ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=10ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=0ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=14ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 74ms, dir size=3 KB
INFO: Analysis reports compressed in 7ms, zip size=3 KB
INFO: Analysis report uploaded in 360ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngSRM9HEJ11KtPiWR1
INFO: Task total time: 1.771 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 3.839s
INFO: Final Memory: 7M/108M
INFO: ------------------------------------------------------------------------
NOTE
-
If you see this outputed
ERROR: Error during SonarQube Scanner execution ERROR: No quality profiles have been found, you probably don't have any language plugin installed.
you didn't enable the SonarGo plugin in SonarQube.
Let me unpack what the above commands are doing
The first line go get github.com/alecthomas/gometalinter
installs the gometalinter
dependency for concurrently run Go lint tools and normalise their output
Gometalinter
then runs all these Go linters:
go vet
- Reports potential errors that otherwise compile.go tool vet --shadow
- Reports variables that may have been unintentionally shadowed.gotype
- Syntactic and semantic analysis similar to the Go compiler.gotype -x
- Syntactic and semantic analysis in external test packages (similar to the Go compiler).deadcode
- Finds unused code.gocyclo
- Computes the cyclomatic complexity of functions.golint
- Google's (mostly stylistic) linter.varcheck
- Find unused global variables and constants.structcheck
- Find unused struct fields.maligned
- Detect structs that would take less memory if their fields were sorted.errcheck
- Check that error return values are used.megacheck
- Run staticcheck, gosimple and unused, sharing work.dupl
- Reports potentially duplicated code.ineffassign
- Detect when assignments to existing variables are not used.interfacer
- Suggest narrower interfaces that can be used.unconvert
- Detect redundant type conversions.goconst
- Finds repeated strings that could be replaced by a constant.gosec
- Inspects source code for security problems by scanning the Go AST.
gometalinter.v2 > gometalinter-report.out
executes all the above linters and consolidates their results into a single gometalinter-report.out
report. Go is a very quiet language. As you may have noticed. So, when gometalinter.v2
runs and everything passes the output of report will contain nothing if there are no problems. If you want to be assured it called the linters it claims to call, you run gometalinter.v2
if debug like so gometalinter.v2 --debug
.
go test -coverprofile=coverage.out
executes your unit tests and generate the coverage.out
report.
go test -json > report.json
executes your unit tests and generate a unit tests execution report, report.json
sonar-scanner
through -D
parameters is configured to submit the reports. If you skipped the prior section, SonarQube will automatically create the project for you.
Open in your host's web browser
http://192.168.0.11:9000/dashboard?id=helloworld-web
to view your SonarQube report.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile" #FFFFFF
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
In your editor create a Makefile
to ensure the build and the steps leading to are repeatable.
GOCMD=go
GOFMT=$(GOCMD) fmt
GOBUILD=$(GOCMD) build
GOCLEAN=$(GOCMD) clean
GOTEST=$(GOCMD) test
GOLINT=$(GOCMD)lint
GOGET=$(GOCMD) get
BINARY_NAME=helloworld-web
SONAR_DEPENDENCIES := \
gopkg.in/alecthomas/gometalinter.v2
all: sonar build
clean:
$(GOCLEAN)
rm -f $(BINARY_NAME)
fmt:
$(GOFMT)
lint: fmt
$(GOGET) golang.org/x/lint/golint
golint
test: lint
$(GOTEST) -v -cover ./...
sonar: test
$(GOGET) -u $(SONAR_DEPENDENCIES)
-gometalinter.v2 --install
-gometalinter.v2 > gometalinter-report.out
-$(GOLINT) > golint-report.out
-$(GOTEST) -v ./... -coverprofile=coverage.out
-$(GOTEST) -v ./... -json > report.json
-sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
build:
$(GOBUILD) -o $(BINARY_NAME) -v
run:
./$(BINARY_NAME)
NOTE
- The indents are
tab
characters and notspaces
characters. otherwise youmake
will fail to execute. - And cut-and-paste may split the single line beginning with
sonar-scanner
into multiple lines. You want this to be single line of text.
Test out your Makefile
make all
And so, you have build and test automation whose output resembles
[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web (cached) coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web 0.004s coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=76ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=46ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=10ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=69ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=29ms
INFO: Load active rules
INFO: Load active rules (done) | time=88ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=38ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=15ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 7 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=158ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=14ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=0ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=1ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=25ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 67ms, dir size=3 KB
INFO: Analysis reports compressed in 7ms, zip size=3 KB
INFO: Analysis report uploaded in 32ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngTpT4HEJ11KtPiWR3
INFO: Task total time: 1.271 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.371s
INFO: Final Memory: 7M/112M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application" #FFFFFF
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Build a Docker image named nemonik/helloworld-web
for the application and all its dependencies to ensure a repeatable deployment on Docker by opening the file Dockerfile
in an editor at the root of the project and copy the is content into it
FROM 192.168.0.11:5000/nemonik/golang:1.12.1
MAINTAINER Michael Joseph Walsh <[email protected]>
RUN mkdir /app
ADD main.go /app/
WORKDIR /app
RUN go build -o helloworld-web .
CMD ["/app/helloworld-web"]
EXPOSE 3000
Then create the Docker image
docker build -t nemonik/helloworld-web .
NOTE
- Don't miss that last period (
.
) at the end the line above.
Command line output will resemble
[vagrant@development helloworld-web]$ docker build -t nemonik/helloworld-web .
Sending build context to Docker daemon 10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.1
1.12.1: Pulling from nemonik/golang
e79bb959ec00: Pull complete
d4b7902036fe: Pull complete
1b2a72d4e030: Pull complete
d54db43011fd: Pull complete
963c818ebafc: Pull complete
2c6333e9b74a: Pull complete
3b0c71504fac: Pull complete
eb7d59297445: Pull complete
6335f01326ba: Pull complete
bfcbc32359e9: Pull complete
76c51a60b271: Pull complete
Digest: sha256:e5f086026391553344abba61bcf184618f1ff6e6c32dc392f645bbd8971ab403
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/golang:1.12.1
---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
---> Running in f68f90915e36
Removing intermediate container f68f90915e36
---> 096e6501ab10
Step 3/8 : RUN mkdir /app
---> Running in b0ded1b2fead
Removing intermediate container b0ded1b2fead
---> d4752c51206f
Step 4/8 : ADD main.go /app/
---> ecfbd5db128d
Step 5/8 : WORKDIR /app
---> Running in 022f248d47f0
Removing intermediate container 022f248d47f0
---> 8bfa4710fdc1
Step 6/8 : RUN go build -o helloworld-web .
---> Running in 622b81b17191
Removing intermediate container 622b81b17191
---> 269d7f9c41b0
Step 7/8 : CMD ["/app/helloworld-web"]
---> Running in 78e6b23e9fc3
Removing intermediate container 78e6b23e9fc3
---> 37c8c1ed4ced
Step 8/8 : EXPOSE 3000
---> Running in fc138b7e2b5b
Removing intermediate container fc138b7e2b5b
---> d279e0618ec7
Successfully built d279e0618ec7
Successfully tagged nemonik/helloworld-web:latest
What just happened?
- The
FROM
line instructs Docker to retreive thenemonik/golang:1.12.1
from the private Docker registry running on our Toolchain vagrant, which it did. - Then the rest of the commands in the
Dockerfile
are executed building a new docker image. docker build
then places the image with the namenemonik/helloworld-web
in theDevelopment
's local Docker registry so that containers can be created off this image locally.
Check the local Docker registry via
docker images
The command line output will resemble
[vagrant@development helloworld-web]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nemonik/helloworld-web latest facd0af56fe5 2 minutes ago 1.01GB
192.168.0.11:5000/nemonik/golang 1.11.2 32c900b4b4da 2 hours ago 1GB
NOTE:
- The registy will also contain the
192.168.0.11:5000/nemonik/golang:1.12.1
on whichnemonik/helloworld-web:latest
is based, so the next time re-build image you won't have to wait for192.168.0.11:5000/nemonik/golang:1.12.1
to be retrieved.
We can update the project's Makefile adding a docker-build
target
docker-build: build
docker build --no-cache -t nemonik/helloworld-web .
Also, add the docker-build
to the all
target like so:
all: sonar docker-build
Remember to us tab characters vice space characters.
And then run in the command lin
make all
Whose output will be
[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web (cached) coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web 0.004s coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=97ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=56ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=13ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=79ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=30ms
INFO: Load active rules
INFO: Load active rules (done) | time=82ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=32ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=15ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 8 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=166ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=25ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=1ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=16ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 61ms, dir size=3 KB
INFO: Analysis reports compressed in 8ms, zip size=3 KB
INFO: Analysis report uploaded in 26ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngWOBIHEJ11KtPiWR4
INFO: Task total time: 1.280 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.733s
INFO: Final Memory: 7M/114M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v
docker build --no-cache -t nemonik/helloworld-web .
Sending build context to Docker daemon 10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.1
---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
---> Running in f1da2f04877b
Removing intermediate container f1da2f04877b
---> 0eaaae49e98c
Step 3/8 : RUN mkdir /app
---> Running in 1a248ef09da9
Removing intermediate container 1a248ef09da9
---> 3ebe65b2cb75
Step 4/8 : ADD main.go /app/
---> 7de0547ee25e
Step 5/8 : WORKDIR /app
---> Running in b38d14f60f18
Removing intermediate container b38d14f60f18
---> 5c662ba39229
Step 6/8 : RUN go build -o helloworld-web .
---> Running in 28966ff6604d
Removing intermediate container 28966ff6604d
---> a2c445d6bb55
Step 7/8 : CMD ["/app/helloworld-web"]
---> Running in 90b64a9fd5e7
Removing intermediate container 90b64a9fd5e7
---> 241184d5dc62
Step 8/8 : EXPOSE 3000
---> Running in d5172565bd5d
Removing intermediate container d5172565bd5d
---> 3c1cdf32fbfd
Successfully built 3c1cdf32fbfd
Successfully tagged nemonik/helloworld-web:latest
NOTE
- As noted before the base image already existed in the local Docker registry thereby saving us time.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container" #FFFFFF
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Spin up a new nemonik/helloworld-web
container by either entering
docker run -p 3000:3000 --name helloworld-web nemonik/helloworld-web
and hitting http://192.168.0.10:3000 in your web browser, ordocker run -d -p 3000:3000 --name helloworld-web nemonik/helloworld-web
and hitting the same URL viacurl http://192.168.0.10:3000
.
Where
run
messages Docker you are running a new container-d
in the second option, runs the container in background and prints container ID.-p 3000:3000
published the container's port as 3000 to the host.--name helloworld-web
names the running containernemonik/helloworld-web
states what container image to use.
The command line output for the first option will be
[vagrant@development helloworld-web]$ docker run -p 3000:3000 --name helloworld-web nemonik/helloworld-web
listening on :3000
192.168.0.1:54740 GET /
192.168.0.1:54740 GET /favicon.ico
^C
For the second option there will be no output written to the screen, but you can see the same output if you run
docker logs <the container id output when started the container> -f
or
docker logs helloworld-web -f
returning
[vagrant@development helloworld-web]$ docker logs 31257f287d26bdb8ae02d9d444f8ebb1478da183f7d6f908767e006c212fcbd8 -f
listening on :3000
192.168.0.1:55159 GET /
192.168.0.1:55159 GET /favicon.ico
ctrl-c
to exit the logs.
To kill the container
docker rm -f <the container id output when started the container>
and the container id returned when you ran the container.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry" #FFFFFF
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Push the nemonik/hellowrold-web
container image into the private Docker registry running on the Toolchain
vagrant, so that both vagrants can create containers from the image with the commands
docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
docker push 192.168.0.11:5000/nemonik/helloworld-web
Command line output will be
[vagrant@development helloworld-web]$ docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
[vagrant@development helloworld-web]$ docker push 192.168.0.11:5000/nemonik/helloworld-web
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
f7ac261b7c4d: Pushed
67624b1e99ab: Pushed
53ce894cd0ad: Pushed
f3956851d408: Mounted from nemonik/golang
dea8f6ac2dd7: Mounted from nemonik/golang
303fb5379543: Mounted from nemonik/golang
d6ae08d86ad3: Mounted from nemonik/golang
49c92637f9e9: Mounted from nemonik/golang
9ccbb4ca29f8: Mounted from nemonik/golang
f4907c4e3f89: Mounted from nemonik/golang
b17cc31e431b: Mounted from nemonik/golang
12cb127eee44: Mounted from nemonik/golang
604829a174eb: Mounted from nemonik/golang
fbb641a8b943: Mounted from nemonik/golang
latest: digest: sha256:0d7508fadaa570ac261906c95054efc96663ab6c9c7131a13cbd225a2e77eb0c size: 3267
We can update the project's Makefile adding a docker-push
target
docker-push: docker-build
docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
docker push 192.168.0.11:5000/nemonik/helloworld-web
Also, add the docker-push
to the all
target like so:
all: sonar docker-push
And then run in the command line
make all
The output will be something like
[vagrant@development helloworld-web]$ make all
go fmt
go get golang.org/x/lint/golint
golint
go test -v -cover ./...
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web (cached) coverage: 55.6% of statements
go get -u gopkg.in/alecthomas/gometalinter.v2
gometalinter.v2 --install
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
gometalinter.v2 > gometalinter-report.out
golint > golint-report.out
go test -v ./... -coverprofile=coverage.out
=== RUN TestLogRequest
--- PASS: TestLogRequest (0.00s)
=== RUN TestHandler
--- PASS: TestHandler (0.00s)
PASS
coverage: 55.6% of statements
ok github.com/nemonik/helloworld-web 0.004s coverage: 55.6% of statements
go test -v ./... -json > report.json
sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /home/vagrant/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=78ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /home/vagrant/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=67ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=12ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=78ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=26ms
INFO: Load active rules
INFO: Load active rules (done) | time=83ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=34ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=12ms
INFO: Base dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /home/vagrant/go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 8 files indexed
INFO: 1 file ignored because of inclusion/exclusion patterns
INFO: Quality profile for go: Sonar way
INFO: Sensor SonarGo [go]
INFO: Load coverage report from '/home/vagrant/go/src/github.com/nemonik/helloworld-web/coverage.out'
INFO: Sensor SonarGo [go] (done) | time=159ms
INFO: Sensor Go Unit Test Report [go]
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestLogRequest
WARN: Failed to find test file for package github.com/nemonik/helloworld-web and test TestHandler
INFO: Sensor Go Unit Test Report [go] (done) | time=14ms
INFO: Sensor Import of Golint issues [go]
ERROR: GoLintReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of Golint issues [go] (done) | time=16ms
INFO: Sensor Import of GoMetaLinter issues [go]
ERROR: GoMetaLinterReportSensor: Import of external issues requires SonarQube 7.2 or greater.
INFO: Sensor Import of GoMetaLinter issues [go] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=15ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 1 file
INFO: CPD calculation finished
INFO: Analysis report generated in 70ms, dir size=3 KB
INFO: Analysis reports compressed in 8ms, zip size=3 KB
INFO: Analysis report uploaded in 31ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngW6IWHEJ11KtPiWR5
INFO: Task total time: 1.371 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2.879s
INFO: Final Memory: 8M/115M
INFO: ------------------------------------------------------------------------
go build -o helloworld-web -v
docker build --no-cache -t nemonik/helloworld-web .
Sending build context to Docker daemon 10.64MB
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.1
---> 8bd76ee17c41
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
---> Running in 4de3c6d49980
Removing intermediate container 4de3c6d49980
---> 1f58e6e0faf3
Step 3/8 : RUN mkdir /app
---> Running in 7cfdef36f806
Removing intermediate container 7cfdef36f806
---> 29ec163e00eb
Step 4/8 : ADD main.go /app/
---> af9663389a72
Step 5/8 : WORKDIR /app
---> Running in a3fd471cf2b9
Removing intermediate container a3fd471cf2b9
---> ae03915db2cc
Step 6/8 : RUN go build -o helloworld-web .
---> Running in d49a8cc11d8a
Removing intermediate container d49a8cc11d8a
---> 5838172f19ba
Step 7/8 : CMD ["/app/helloworld-web"]
---> Running in 26f26efb5e11
Removing intermediate container 26f26efb5e11
---> d0390a3e0c83
Step 8/8 : EXPOSE 3000
---> Running in 84d2f0fa920d
Removing intermediate container 84d2f0fa920d
---> 79aa7a9c6392
Successfully built 79aa7a9c6392
Successfully tagged nemonik/helloworld-web:latest
docker tag nemonik/helloworld-web 192.168.0.11:5000/nemonik/helloworld-web
docker push 192.168.0.11:5000/nemonik/helloworld-web
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
636decb58204: Pushed
166b3d7a9755: Pushed
fa87d269f747: Pushed
f3956851d408: Layer already exists
dea8f6ac2dd7: Layer already exists
303fb5379543: Layer already exists
d6ae08d86ad3: Layer already exists
49c92637f9e9: Layer already exists
9ccbb4ca29f8: Layer already exists
f4907c4e3f89: Layer already exists
b17cc31e431b: Layer already exists
12cb127eee44: Layer already exists
604829a174eb: Layer already exists
fbb641a8b943: Layer already exists
latest: digest: sha256:0e30de168363aa683b78cc010ef6cd88427579e18ba2b757e4283bb6bec5f5a0 size: 3267
The Docker registry container image shipped by Docker does not provide a GUI, but we can verify by querying the catalog of the private registry through a web browser or Unix command line tool curl
by entering into the command line
curl -X GET http://192.168.0.11:5000/v2/_catalog
Returns in the command line
[vagrant@development helloworld-web]$ curl -X GET http://192.168.0.11:5000/v2/_catalog
{"repositories":["nemonik/golang","nemonik/golang-sonarqube-scanner","nemonik/helloworld-web","nemonik/python","nemonik/standalone-firefox","nemonik/zap2docker-stable"]}
The Docker registry on our Toolchain
vagrant doesn't have a human readable web UI, but you can query for the container you pushed and read the JSON:
curl -X GET http://192.168.0.11:5000/v2/nemonik/helloworld-web/tags/list
Returns in the command line
[vagrant@development helloworld-web]$ curl -X GET http://192.168.0.11:5000/v2/nemonik/helloworld-web/tags/list
{"name":"nemonik/helloworld-web","tags":["latest"]}
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline" #FFFFFF
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
As you did for the purpose of CI (Continuous Integration) of the prior application, you will need to configure Drone to perform CI/CD (a combination of Continuous Integration and Continuous Delivery) on the helloworld-web
application.
Complete the following:
- Open http://192.168.0.11/ in your browser and authenticate through GitLab on into Drone, if you need to.
- Then select
SYNC
. The arrows will chase each other for a bit. - Then click
root/helloworld-web
repo andACTIVATE REPOSITORY
, thenSAVE
under theMain
section to enable Drone orchestration for the project. - Then click the Drone logo in the upper left of the page to return home.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline" #FFFFFF
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
So, lets create our pipeline starting with a sonar-scan
step to update SonarQube with code quality scans automatically by opening .drone.yml
in our text editor and adding the following step right after our build
step, so that we run the step prior to a deploy.
kind: pipeline
name: default
steps:
- name: sonarqube
image: 192.168.0.11:5000/nemonik/golang-sonarqube-scanner:3.3.0.1492
commands:
- export DRONESRC=`pwd`
- export GOBIN=$GOPATH/bin
- mkdir -p $GOPATH/src/github.com/nemonik
- cd $GOPATH/src/github.com/nemonik
- ln -s $DRONESRC helloworld-web
- cd helloworld-web
- go get -u golang.org/x/lint/golint
- go get -u gopkg.in/alecthomas/gometalinter.v2
- gometalinter.v2 --install
- gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
- go lint > golint-report.out || true
- go test -v ./... -coverprofile=coverage.out || true
- go test -v ./... -json > report.json || true
- sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true
Things to note in the above
- This step uses an image,
nemonik/golang-sonarqube-scanner:3.3.0.1492
, built ontop of thenemonik/golang:1.12.1
image to speed builds along. - Cut-and-pasting may split the last
-sonar-scanner
line into multiple line resulting in your build failing. Correct in the editor. - Make sure to leave two returns to indicate the end of the
.drone.yml
file or a parse error will result in Drone.
- export DRONESRC=`pwd`
- export GOBIN=$GOPATH/bin
- mkdir -p $GOPATH/src/github.com/nemonik
- cd $GOPATH/src/github.com/nemonik
- ln -s $DRONESRC helloworld-web
What follows handles running the scan absorbing errors as they arise, so as to not break the build.
- gometalinter.v2 --install
- gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
- go lint > golint-report.out || true
- go test -v ./... -coverprofile=coverage.out || true
- go test -v ./... -json > report.json || true
- sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true
Commit the code to GitLab hosted remote repo
git add .drone.yml
git commit -m "added sonar step to pipeline"
git push origin master
Open on your host
http://192.168.0.11/root/helloworld-web
And monitor the progress of the build. The pipeline should execute in a few minutes.
Typical success resembles
+ export DRONESRC=/drone/src
+ export GOBIN=$GOPATH/bin
+ mkdir -p $GOPATH/src/github.com/nemonik
+ cd $GOPATH/src/github.com/nemonik
+ ln -s $DRONESRC helloworld-web
+ cd helloworld-web
+ go get -u golang.org/x/lint/golint
+ go get -u gopkg.in/alecthomas/gometalinter.v2
+ gometalinter.v2 --install
Installing:
deadcode
dupl
errcheck
gochecknoglobals
gochecknoinits
goconst
gocyclo
goimports
golint
gosec
gosimple
gotype
gotypex
ineffassign
interfacer
lll
maligned
megacheck
misspell
nakedret
safesql
staticcheck
structcheck
unconvert
unparam
unused
varcheck
+ gometalinter.v2 --deadline=120s --checkstyle > gometalinter-report.out || true
+ go lint > golint-report.out || true
go lint: unknown command
Run 'go help' for usage.
+ go test -v ./... -coverprofile=coverage.out || true
go: warning: "./..." matched no packages
no packages to test
+ go test -v ./... -json > report.json || true
go: warning: "./..." matched no packages
no packages to test
+ sonar-scanner -D sonar.host.url=http://192.168.0.11:9000 -D sonar.projectKey=helloworld-web -D sonar.projectName=helloworld-web -D sonar.projectVersion=1.0 -D sonar.sources=. -D sonar.go.gometalinter.reportPaths=gometalinter-report.out -D sonar.go.golint.reportPaths=golint-report.out -D sonar.go.coverage.reportPaths=coverage.out -D sonar.go.tests.reportPaths=report.json -D sonar.exclusions=**/*test.go || true
INFO: Scanner configuration file: /usr/local/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-957.5.1.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: SonarQube server 7.1.0
INFO: Default locale: "en_US", source code encoding: "US-ASCII" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=83ms
INFO: Server id: AWngQ8AWiugn-Qnmvhkr
INFO: User cache: /root/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=49ms
INFO: Load/download plugins
INFO: Load/download plugins (done) | time=89ms
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=79ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=27ms
INFO: Load active rules
INFO: Load active rules (done) | time=84ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=33ms
WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.
INFO: Project key: helloworld-web
INFO: Project base dir: /go/src/github.com/nemonik/helloworld-web
INFO: ------------- Scan helloworld-web
INFO: Load server rules
INFO: Load server rules (done) | time=19ms
INFO: Base dir: /go/src/github.com/nemonik/helloworld-web
INFO: Working dir: /go/src/github.com/nemonik/helloworld-web/.scannerwork
INFO: Source paths: .
INFO: Source encoding: US-ASCII, default locale: en_US
INFO: Index files
INFO: Excluded sources:
INFO: **/*test.go
INFO: 3 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=11ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: Calculating CPD for 0 files
INFO: CPD calculation finished
INFO: Analysis report generated in 65ms, dir size=1 KB
INFO: Analysis reports compressed in 10ms, zip size=1 KB
INFO: Analysis report uploaded in 29ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.0.11:9000/dashboard/index/helloworld-web
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.0.11:9000/api/ce/task?id=AWngY_tOHEJ11KtPiWR6
INFO: Task total time: 1.169 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 3.359s
INFO: Final Memory: 11M/132M
INFO: ------------------------------------------------------------------------
Add a build step to our .drone.yml
- name: build
image: nemonik/golang:1.12.1
commands:
- make build
NOTE
- Make sure to leave two consecutive returns to indicate the end of the
.drone.yml
file or a parse error will result in Drone.
Then commit to GitLab
git add .
git commit -m "added build step to pipeline"
git push origin master
Since we have not registered an SSH key with GitLab, we will need to enter a Username
and Password
when prompted.
In the browser open Drone at http://192.168.0.11/root/helloworld-web and click on the build being executed to monitor progress.
Output for build
resembles
+ make build
go build -o helloworld-web -v
_/drone/src/192.168.0.11/root/helloworld-web
Mirroring what you saw in development in your local environment.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline" #FFFFFF
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Add the publish step to your .drone.yml
so that the container image is publish to the private registry via the pipeline. The publish:
step must be indented the same as the prior build:
step.
- name: publish
image: plugins/docker
settings:
storage_driver: overlay
insecure: true
registry: 192.168.0.11:5000
repo: 192.168.0.11:5000/nemonik/helloworld-web
force_tag: true
tags:
- latest
custom_dns:
- "10.20.100.53"
- "10.20.200.53"
Note
- Your DNS server entries for
custom_dns
may vary for your environment or this line may not be needed. In this case, the DNS servers were needed to be pass otherwise the build will fail to resolve the enterprise proxy when it attempts to retrieve external dependency to include in the container image. - Make sure to leave two consecutive returns to indicate the end of the
.drone.yml
file or a parse error will result in Drone.
Push your code into GitLab
git add .
git commit -m "added publish step to pipeline"
git push origin master
The publish step takes quite a bit of time and resembles:
+ /usr/local/bin/dockerd -g /var/lib/docker -s overlay --insecure-registry 192.168.0.11:5000 --dns 10.20.100.53 --dns 10.20.200.53
Registry credentials not provided. Guest mode enabled.
+ /usr/local/bin/docker version
Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:05:38 2017
OS/Arch: linux/amd64
Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:12:29 2017
OS/Arch: linux/amd64
Experimental: false
+ /usr/local/bin/docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.12.0-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: Alpine Linux v3.7 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.67GiB
Name: 5f313f5c5c13
ID: 5H5L:Z6X5:IPNQ:YTB3:GBVC:RV2O:3BP3:KAWS:JQZA:MKIT:HI3K:AWGR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
192.168.0.11:5000
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
+ /usr/local/bin/docker build --rm=true -f Dockerfile -t 05cfa544b6fc3082f61155a9797d8ae5628a8d74 . --pull=true --label org.label-schema.schema-version=1.0 --label org.label-schema.build-date=2018-12-01T06:59:05Z --label org.label-schema.vcs-ref=05cfa544b6fc3082f61155a9797d8ae5628a8d74 --label org.label-schema.vcs-url=http://192.168.0.11:10080/root/helloworld-web.git
Sending build context to Docker daemon 6.653MB
Step 1/9 : FROM 192.168.0.11:5000/nemonik/golang:1.12.1
1.11.2: Pulling from nemonik/golang
54f7e8ac135a: Pulling fs layer
d6341e30912f: Pulling fs layer
087a57faf949: Pulling fs layer
5d71636fb824: Pulling fs layer
f368dba6a331: Pulling fs layer
f2e97ec220f9: Pulling fs layer
e4c39f2ed8c2: Pulling fs layer
a2992ac95eb6: Pulling fs layer
b6347f061623: Pulling fs layer
624cedf99a02: Pulling fs layer
b3caf54db51f: Pulling fs layer
e4c39f2ed8c2: Waiting
a2992ac95eb6: Waiting
b6347f061623: Waiting
624cedf99a02: Waiting
b3caf54db51f: Waiting
5d71636fb824: Waiting
f368dba6a331: Waiting
f2e97ec220f9: Waiting
087a57faf949: Verifying Checksum
087a57faf949: Download complete
d6341e30912f: Verifying Checksum
d6341e30912f: Download complete
54f7e8ac135a: Verifying Checksum
54f7e8ac135a: Download complete
5d71636fb824: Verifying Checksum
5d71636fb824: Download complete
e4c39f2ed8c2: Verifying Checksum
e4c39f2ed8c2: Download complete
f368dba6a331: Verifying Checksum
f368dba6a331: Download complete
a2992ac95eb6: Verifying Checksum
a2992ac95eb6: Download complete
b6347f061623: Verifying Checksum
b6347f061623: Download complete
624cedf99a02: Verifying Checksum
624cedf99a02: Download complete
f2e97ec220f9: Verifying Checksum
f2e97ec220f9: Download complete
b3caf54db51f: Verifying Checksum
b3caf54db51f: Download complete
54f7e8ac135a: Pull complete
d6341e30912f: Pull complete
087a57faf949: Pull complete
5d71636fb824: Pull complete
f368dba6a331: Pull complete
f2e97ec220f9: Pull complete
e4c39f2ed8c2: Pull complete
a2992ac95eb6: Pull complete
b6347f061623: Pull complete
624cedf99a02: Pull complete
b3caf54db51f: Pull complete
Digest: sha256:5f5b7b244da69c56a95bb1c924c2557e08b60b703c7f3131884f73d02f907bab
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/golang:1.12.1
---> 32c900b4b4da
Step 2/9 : MAINTAINER Michael Joseph Walsh <[email protected]>
---> Running in ac61aba08f1c
Removing intermediate container ac61aba08f1c
---> 33a70df94816
Step 3/9 : RUN mkdir /app
---> Running in 52c411171688
Removing intermediate container 52c411171688
---> ce4ef1288d28
Step 4/9 : ADD main.go /app/
---> 8adeb9f66464
Step 5/9 : WORKDIR /app
Removing intermediate container 8e92749cfe8d
---> 8f164d7c66c9
Step 6/9 : RUN go build -o helloworld-web .
---> Running in dbc17c5c9e1c
Removing intermediate container dbc17c5c9e1c
---> 807609ef0468
Step 7/9 : CMD ["/app/helloworld-web"]
---> Running in a01670b6a0aa
Removing intermediate container a01670b6a0aa
---> 549af300f148
Step 8/9 : EXPOSE 3000
---> Running in 2455b914519f
Removing intermediate container 2455b914519f
---> fa6831463c27
Step 9/9 : LABEL "org.label-schema.build-date"='2018-12-01T06:59:05Z' "org.label-schema.schema-version"='1.0' "org.label-schema.vcs-ref"='05cfa544b6fc3082f61155a9797d8ae5628a8d74' "org.label-schema.vcs-url"='http://192.168.0.11:10080/root/helloworld-web.git'
---> Running in cff7286ce8c2
Removing intermediate container cff7286ce8c2
---> 495506e61a7a
Successfully built 495506e61a7a
Successfully tagged 05cfa544b6fc3082f61155a9797d8ae5628a8d74:latest
+ /usr/local/bin/docker tag 05cfa544b6fc3082f61155a9797d8ae5628a8d74 192.168.0.11:5000/nemonik/helloworld-web:latest
+ /usr/local/bin/docker push 192.168.0.11:5000/nemonik/helloworld-web:latest
The push refers to repository [192.168.0.11:5000/nemonik/helloworld-web]
b30658e0f8ca: Preparing
ec0ffa96b9d6: Preparing
65083988c970: Preparing
382408b8ce47: Preparing
48302eac2a97: Preparing
274449f3d60f: Preparing
b65d59e413b9: Preparing
652fefcd9ea8: Preparing
8019683f918e: Preparing
e7b9eaeca217: Preparing
f75e64f96dbc: Preparing
8f7ee6d76fd9: Preparing
c23711a84ad4: Preparing
90d1009ce6fe: Preparing
8019683f918e: Waiting
e7b9eaeca217: Waiting
f75e64f96dbc: Waiting
8f7ee6d76fd9: Waiting
274449f3d60f: Waiting
c23711a84ad4: Waiting
b65d59e413b9: Waiting
90d1009ce6fe: Waiting
652fefcd9ea8: Waiting
48302eac2a97: Layer already exists
382408b8ce47: Layer already exists
274449f3d60f: Layer already exists
652fefcd9ea8: Layer already exists
b65d59e413b9: Layer already exists
e7b9eaeca217: Layer already exists
8019683f918e: Layer already exists
65083988c970: Pushed
ec0ffa96b9d6: Pushed
f75e64f96dbc: Layer already exists
8f7ee6d76fd9: Layer already exists
90d1009ce6fe: Layer already exists
c23711a84ad4: Layer already exists
b30658e0f8ca: Pushed
latest: digest: sha256:e692b9f876d463755ab58cc07c124139d2d99fb4b8da4a55832a8b9ff9665b3a size: 3267
+ /usr/local/bin/docker rmi 05cfa544b6fc3082f61155a9797d8ae5628a8d74
Untagged: 05cfa544b6fc3082f61155a9797d8ae5628a8d74:latest
+ /usr/local/bin/docker system prune -f
Total reclaimed space: 0B
Indicating the publish
step executed, successfully.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline" #FFFFFF
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
Usually, one adds a Drone secret and the appleboy/ssh
plugin, but the last I looked there appeared to be a bug in the plugin, but one can accomplish the same goal using the plugin slightly differently.
Enable the Trusted
setting for the repository in Drone by opening
-
Under
Main
and thenProject Settings
check offTrusted
. -
Leave the others to their defaults and click
SAVE
and look for Drone to float a modal at the bottom left denotingSuccessfully saved
indicating success.
-
Then click the Drone icon in the upper left of the page to return home.
Since we let Vagrant manage the creation and configuration of each Vagrant's SSH keys for each of our variants (e.g., dev, toolchain) vagrant will have to create a unique key pair for each vagrant. Vagrant store each vagrant's private inside the .vagrant
path created in the class project.
The toolchain
vagrant's private keys exists at
/vagrant/.vagrant/machines/toolchain/virtualbox/private_key
We can access each private key though from inside each vagrant, because we configured each to mount the class project to the path /vagrant
with the line
dev.vm.synced_folder ".", "/vagrant", type: "virtualbox"
placed inside of each vagrant's config.vm.define
block.
For example, while on the development
vagrant enter the following command
ls /vagrant
It will show you class project.
Open root/helloworld-web
's .drone.yml
in your editor and add an additional deploy:
step below the ones you already have with the deploy:
being indented the same as the prior build:
and publish:
steps.
- name: deploy
image: appleboy/drone-ssh
volumes:
- name: private_key
path: /root/ssh/drone_rsa
settings:
host: 192.168.0.11
port: 22
username: vagrant
key_path: /root/ssh/drone_rsa
command_timeout: 5m
script:
- docker stop helloworld-web 2>/dev/null
- docker rm helloworld-web 2>/dev/null
- docker rmi 192.168.0.11:5000/nemonik/helloworld-web 2>/dev/null
- docker run -d --restart=always --name helloworld-web --publish 3000:3000 192.168.0.11:5000/nemonik/helloworld-web
volumes:
- name: private_key
host:
path: /vagrant/.vagrant/machines/toolchain/virtualbox/private_key
NOTE
- Make sure to leave two consecutive returns to indicate the end of the
.drone.yml
file or a parse error will result.
Commit the code to GitLab hosted remote repo
git add .drone.yml
git commit -m "added deploy step to pipeline"
git push origin master
The deploy step output resembles:
======CMD======
docker stop helloworld-web 2>/dev/null
docker rm helloworld-web 2>/dev/null
docker rmi 192.168.0.11:5000/nemonik/helloworld-web 2>/dev/null
docker run -d --restart=always --name helloworld-web --publish 3000:3000 192.168.0.11:5000/nemonik/helloworld-web
======END======
out: f56fa19cf749134fb8e1a53f8e4706612a7956fe0756d25e4d0a2284f3b5fa4f
err: Unable to find image '192.168.0.11:5000/nemonik/helloworld-web:latest' locally
err: latest: Pulling from nemonik/helloworld-web
err: 54f7e8ac135a: Already exists
err: d6341e30912f: Already exists
err: 087a57faf949: Already exists
err: 5d71636fb824: Already exists
err: f368dba6a331: Already exists
err: f2e97ec220f9: Already exists
err: e4c39f2ed8c2: Already exists
err: a2992ac95eb6: Already exists
err: b6347f061623: Already exists
err: 624cedf99a02: Already exists
err: b3caf54db51f: Already exists
err: a9520f945746: Pulling fs layer
err: 19de2ec0e886: Pulling fs layer
err: aa0e691fa83d: Pulling fs layer
err: 19de2ec0e886: Verifying Checksum
err: 19de2ec0e886: Download complete
err: a9520f945746: Download complete
err: aa0e691fa83d: Verifying Checksum
err: aa0e691fa83d: Download complete
err: a9520f945746: Pull complete
err: 19de2ec0e886: Pull complete
err: aa0e691fa83d: Pull complete
err: Digest: sha256:208ecc85c106a81d67bca9cd81146403568b16a4e66996c996f622c1b803c734
err: Status: Downloaded newer image for 192.168.0.11:5000/nemonik/helloworld-web:latest
==========================================
Successfully executed commands to all host.
==========================================
Open on your host
http://192.168.0.11/root/helloworld-web
And monitor the progress of the build. The pipeline should execute in a few minutes.
Once completed sucessfully, on your host open secure shell to your toolchain
vagrant via
vagrant ssh toolchain
Then list the running Docker containers filtering for the one the pipeline just created
docker ps --filter "name=helloworld-web"
Command line output will resemble
[root@toolchain drone]# docker ps --filter "name=helloworld-web"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9d08909cc2ee 192.168.0.11:5000/nemonik/helloworld-web "/app/helloworld-web" 6 seconds ago Up 5 seconds 0.0.0.0:3000->3000/tcp helloworld-web
Then either on your docker host in a browser open
or in the command-line of your development
vagrant enter
curl http://192.168.0.11:3000
Either way the container will return
Hello world!
So, now we have beginnings of a real CI/CD pipeline. There are no strings on me.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline" #FFFFFF
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
InSpec tests permit you to author compliance as code thereby turing compliance, security, and other policy requirements into automated tests.
InSpec has a very robust support for both the docker container and its host, see: https://lollyrock.com/posts/inspec-for-docker/ for more information for using InSpec with Docker.
More on InSpec in general can be found here
And these specific sectons are relevant to Docker:
https://www.inspec.io/docs/reference/resources/docker_container/
https://www.inspec.io/docs/reference/resources/docker/
https://www.inspec.io/docs/reference/resources/docker_image/
At the root of the helloworld-web
project create a directory to hold your inspec test
mkdir inspec-tests
cd inspec-tests
Then author a simple Ruby-based tests to verify the expected state of the nemonik/helloworld-web:latest
Docker container by creating inspec.rb
in an editor with this content:
control 'docker-checks-1.1' do
impact 0.7 # High Impact
tag "nist": ['CM-6', 'Rev_4']
title 'Verify Docker Container exists and is running'
describe docker_container(name: 'helloworld-web') do
it { should exist }
it { should be_running }
its('repo') { should eq '192.168.0.11:5000/nemonik/helloworld-web' }
its('ports') { should eq '0.0.0.0:3000->3000/tcp' }
its('command') { should eq '/app/helloworld-web' }
end
end
And then create the profile's metadata file, inspec.yml
with the contents:
name: helloworld-tests
title: InSpec Profile
maintainer: Michael Joseph Walsh
copyright: Michael Joseph Walsh
copyright_email: [email protected]
license: New BSD License
summary: A minimal InSpec Compliance Profile for helloworld-web
version: 0.0.1
supports:
platform: os
attributes:
- name: 'application_url'
type: string
default: 192.168.0.11
- name: 'application_port'
type: string
First, we'll execute the test against container running locally by first spinning up a 192.168.0.11:5000/nemonik/helloworld-web:latest
container via
docker run -d -p 3000:3000 --name helloworld-web 192.168.0.11:5000/nemonik/helloworld-web:latest
If this command fails you likely already have a instance of the container running, you can kill the previous instance via
docker stop helloworld-web
docker rm helloworld-web
And then in the root of your helloworld-web
project run the inspec test
inspec exec inspec.rb
Note
-
Remember to stop your
helloworld-web
docker container viadocker rm -f helloworld-web
The result will be a comparsion of the expected state against the current state of the container and will look like
[vagrant@development inspec-tests]$ inspec exec inspec.rb
Profile: tests from inspec.rb (tests from inspec.rb)
Version: (not specified)
Target: local://
✔ docker-checks-1.1: Verify Docker Container exists and is running
✔ Docker Container helloworld-web should exist
✔ Docker Container helloworld-web should be running
✔ Docker Container helloworld-web repo should eq "192.168.0.11:5000/nemonik/helloworld-web"
✔ Docker Container helloworld-web ports should eq "0.0.0.0:3000->3000/tcp"
✔ Docker Container helloworld-web command should eq "/app/helloworld-web"
Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 5 successful, 0 failures, 0 skipped
Open in an editor .drone.yml
and add the following step
- name: inspec
image: appleboy/drone-ssh
volumes:
- name: private_key
path: /root/ssh/drone_rsa
settings:
host: 192.168.0.11
port: 22
username: vagrant
key_path: /root/ssh/drone_rsa
command_timeout: 5m
script:
- cd /tmp
- rm inspec.*
- wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
- wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
- inspec exec inspec.rb
Then commit the your code
git add .
git commit -m "added inspec step to pipeline"
git push origin master
The output of the inspec
step of the pipeline will resemble
======CMD======
cd /tmp
rm inspec.*
wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
wget http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
inspec exec inspec.rb
======END======
out:
out: Profile: tests from inspec.rb (tests from inspec.rb)
out: Version: (not specified)
out: Target: local://
out:
out: ✔ docker-checks-1.1: Verify Docker Container exists and is running
out: ✔ Docker Container helloworld-web should exist
out: ✔ Docker Container helloworld-web should be running
out: ✔ Docker Container helloworld-web repo should eq "192.168.0.11:5000/nemonik/helloworld-web"
out: ✔ Docker Container helloworld-web ports should eq "0.0.0.0:3000->3000/tcp"
out: ✔ Docker Container helloworld-web command should eq "/app/helloworld-web"
out:
out:
out: Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
out: Test Summary: 5 successful, 0 failures, 0 skipped
err: --2018-12-06 02:07:49-- http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.rb
err: Connecting to 192.168.0.11:10080... connected.
err: HTTP request sent, awaiting response... 200 OK
err: Length: 449 [text/plain]
err: Saving to: ‘inspec.rb’
err:
err: 0K 100% 73.3M=0s
err:
err: 2018-12-06 02:07:50 (73.3 MB/s) - ‘inspec.rb’ saved [449/449]
err:
err: --2018-12-06 02:07:50-- http://192.168.0.11:10080/root/helloworld-web/raw/master/inspec-tests/inspec.yml
err: Connecting to 192.168.0.11:10080... connected.
err: HTTP request sent, awaiting response... 200 OK
err: Length: 416 [text/plain]
err: Saving to: ‘inspec.yml’
err:
err: 0K 100% 77.1M=0s
err:
err: 2018-12-06 02:07:50 (77.1 MB/s) - ‘inspec.yml’ saved [416/416]
err:
==========================================
Successfully executed commands to all host.
==========================================
MITRE maintains two projects for viewing of InSpec profiles and evaluations in a convenient interface.
We'll use Heimdall-lite, a single page JavaScript implementation of the MITRE Heimdall InSpec results viewer, to view:
In the Development vagrant, use inspec
to report out to JSON via
inspec exec inspec.rb --reporter json > /vagrant/inspec_hellworld.json
Open Heimdall-lite https://mitre.github.io/heimdall-lite/, select Load JSON
, the Browse
to navigate to inspec_hellworld.json
in the root of the class project and upload to view the results.
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline" #FFFFFF
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline"
-left-> (*)
We're going to write an automated functional test of helloworld-web
instead of relying on a manual functional test by using Selenium, a portable software-testing framework for web applications. Essentially, Selenium automates web browsers.
More can be found here
You'll need a couple of shells open to your development
vagrant to complete this section.
In the first vagrant ssh development
enter
docker run -d -p 3000:3000 --name helloworld-web nemonik/helloworld-web:latest
NOTE
- If you don't mind running more than a couple shells you can drop the
-d
argument and not daemonize the container, so you can watch its log.
In another vagrant ssh
to development
enter
curl -X GET http://192.168.0.11:5000/v2/_catalog
curl -X GET http://192.168.0.11:5000/v2/nemonik/standalone-firefox/tags/list
docker pull 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
to search for and pull then pull the nemonik/standalone-firefox:3.14.0
container image frome our private Docker repository running on our toolchain
vagrant.
Then enter into the command line
docker run -p 4444:4444 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
This stands up Selenium specifically running Firefox. The container is running in the foreground so we can watch the log output.
A good start outputs to the command line
[vagrant@development ~]$ docker run -p 4444:4444 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
2018-12-01 08:15:45,146 WARN Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2018-12-01 08:15:45,148 INFO supervisord started with pid 7
2018-12-01 08:15:46,154 INFO spawned: 'xvfb' with pid 10
2018-12-01 08:15:46,158 INFO spawned: 'selenium-standalone' with pid 11
08:15:46.576 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.14.0', revision: 'aacccce0'
08:15:46.577 INFO [GridLauncherV3$1.launch] - Launching a standalone Selenium Server on port 4444
2018-12-01 08:15:46,578 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-01 08:15:46,578 INFO success: selenium-standalone entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-01 08:15:46.679:INFO::main: Logging initialized @487ms to org.seleniumhq.jetty9.util.log.StdErrLog
08:15:46.852 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 4444
In another vagrant ssh
to development
, we author and run our automated test.
In this other command line at the root of the helloworld-web
project (e.g., /home/vagrant/go/src/github.com/nemonik/helloworld-web
) enter
mkdir selenium-test
cd selenium-test
We're going to write our test in Python. Python is already installed on the development
vagrant. Ansible uses it.
We'll need to install a dependency, we can install by entering into the command line
sudo pip install 'selenium==3.14.0'
The output of looks like
[vagrant@development ~]$ sudo pip install 'selenium==3.14.0'
Collecting selenium==3.14.0
Downloading https://files.pythonhosted.org/packages/b8/53/9cafbb616d20c7624ff31bcabd82e5cc9823206267664e68aa8acdde4629/selenium-3.14.0-py2.py3-none-any.whl (898kB)
100% |████████████████████████████████| 901kB 3.1MB/s
Requirement already satisfied: urllib3 in /usr/lib/python2.7/site-packages (from selenium==3.14.0) (1.24.1)
Installing collected packages: selenium
Successfully installed selenium-3.14.0
But let's create a requirements.txt
file to hold our dependency with our text editor containing
selenium==3.14.0
This allow the test's dependencies to be enumerated in a file including the specififc version of the dependency, so their installation can be automated.
Then test out the requirements.txt file by entering
sudo pip install -r requirements.txt
Then in an editor create test_helloworld.py
Python-based unit test containing
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
import unittest
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import sys
import os
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"proxyType":"DIRECT"
# "noProxy":"os.environ['no_proxy']," + selenium_host # Selenium cannot handle no_proxy
}
self.driver = webdriver.Remote(command_executor="http://" + selenium_host + ":4444/wd/hub", desired_capabilities=webdriver.DesiredCapabilities.FIREFOX)
def test_helloworld_web(self):
driver = self.driver
driver.get(helloworld_web_url)
assert "Hello world!" in driver.page_source
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
if len(sys.argv) == 3:
selenium_host = sys.argv[1]
helloworld_web_url = sys.argv[2]
else:
print "python test_hello_world.py selenium_host helloworld_web_url"
sys.exit()
del sys.argv[1:]
unittest.main()
Save the file and exit.
You might notice the commented out section about Selenium cannot handle no_proxy.
Yeah. This will later present a problem.
For now, now let us run our test by entering into the command line
python test_helloworld.py 192.168.0.10 http://192.168.0.10:3000
Successful command line output in this window will be
[vagrant@development selenium-test]$ python test_helloworld.py 192.168.0.10 http://192.168.0.10:3000
.
----------------------------------------------------------------------
Ran 1 test in 1.699s
OK
The other windows will show logging.
ctrl-c
to stop the containers in your other shells.
NOTE
-
If you pushed the
helloword-web
to the background, you'll have to remove it viadocker rm -f helloworld-web
In one of your existing vagrant ssh
to development
enter into the commmand line
cd /home/vagrant/go/src/github.com/nemonik/helloworld-web
Edit the .drone.yml
file and add to the bottom the following with the selenum:
step having the same idention as the prior steps, and services:
starting the line.
- name: selenium
image: 192.168.0.11:5000/nemonik/python:2.7.16
commands:
- export NO_PROXY=$NO_PROXY,$(python selenium-test/resolve.py firefox)
- export no_proxy=$no_proxy,$(python selenium-test/resolve.py firefox)
- cd selenium-test
- pip install -r requirements.txt
- python test_helloworld.py firefox http://192.168.0.11:3000
volumes:
- name: private_key
host:
path: /vagrant/.vagrant/machines/toolchain/virtualbox/private_key
- name: shared_memory
host:
path: /dev/shm
services:
- name: firefox
image: 192.168.0.11:5000/nemonik/standalone-firefox:3.14.0
volumes:
- name: shared_memory
path: /dev/shm
NOTE
services
is not part of the prior steps, because it is not a step, but enumeration of services, so be careful when you edit the pipeline.
Drone uses the services:
section to spin up a patched version of selenium/standalone-firefox:3.14.0
exposed with the name firefox
. This is where the problem creeps in that I mentioned earlier, where the selenium
step will hit corporate proxy to resolve firefox
. We don't want that to happen, so the selenium:
step will makes use of another Python script to resolved firefox
service to its private IP and add it the no_proxy
and NO_PROXY
environmental variables, so that the Python selenium test code doesn't attempt to pass the request on to the corporate proxy and result in the test failing.
Create in a text editor selenium-test/resolve.py
with
#!/usr/bin/env python
# Copyright (C) 2019 Michael Joseph Walsh - All Rights Reserved
# You may use, distribute and modify this code under the
# terms of the the license.
#
# You should have received a copy of the license with
# this file. If not, please email <[email protected]>
import socket, sys
if __name__ == "__main__":
if len(sys.argv) == 2:
print socket.gethostbyname( sys.argv[1] )
Commit the code to GitLab hosted remote repo
git add .
git commit -m "added selenium step to pipeline"
git push origin master
Open on your host
http://192.168.0.11/root/helloworld-web
And monitor the progress of the build. The pipeline should execute in a few minutes.
Successful output for this stage resembles
+ export NO_PROXY=$NO_PROXY,$(python selenium-test/resolve.py firefox)
+ export no_proxy=$no_proxy,$(python selenium-test/resolve.py firefox)
+ cd selenium-test
+ pip install -r requirements.txt
Collecting selenium==3.14.0 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/b8/53/9cafbb616d20c7624ff31bcabd82e5cc9823206267664e68aa8acdde4629/selenium-3.14.0-py2.py3-none-any.whl (898kB)
Collecting urllib3 (from selenium==3.14.0->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Installing collected packages: urllib3, selenium
Successfully installed selenium-3.14.0 urllib3-1.24.1
+ python test_helloworld.py firefox http://192.168.0.11:3000
.
----------------------------------------------------------------------
Ran 1 test in 5.283s
OK
skinparam shadowing false
skinparam title {
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 30
}
skinparam activity {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityDiamond {
BorderColor #0B5C92
BackgroundColor #e0e59a
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
skinparam activityStart {
Color #0B5C92
}
skinparam activityEnd {
Color #0B5C92
}
skinparam arrow {
Color #0B5C92
}
skinparam note {
BorderColor #0B5C92
BackgroundColor #FEFECE
FontName "Yanone Kaffeesatz"
FontStyle "Thin"
FontSize 15
}
(*) -right->”Create\nthe project's\nbacklog”
-right->"Create\nthe project\nin GitLab"
-right->"Setup the project\nin the development\nVagrant"
-right->"Author\nthe application"
-right->"Build and run\nthe application"
-right->"Run gometalinter.v2\non application"
-down->"Fix\nthe application"
-left->"Author\nunit tests"
-left->"Perform\nstatic analysis\non the CLI"
-left->"Write\nthe Makefile"
-left->"Dockerize\nthe application"
-left->"Run the\nDocker container"
-down->"Push the container\nimage to the\nprivate registry"
-right->"Configure Drone\nto execute\nthe CI/CD pipeline"
-right->"Add\nStatic Analysis\n(SonarQube)\nstep to pipeline"
-right->"Add build step\nto pipeline"
-right->"Add container\nimage publish\nstep to pipeline"
-right->"Add container\ndeploy step\nto pipeline"
-down->"Add\ncomplaince automation\ntest (InSpec)\nstep to pipeline"
-left->"Add automated\nfuntional test\n(Selenium)\nto pipeline"
-left->"Add DAST step\n(OWASP ZAP)\nto the pipeline" #FFFFFF
-left-> (*)
Dynamic application security testing (DAST) is used to detect security vulnerabilities in an application while it is running, so as to help you remediate these concerns while in development.
The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free DAST tools actively maintained by hundreds of international volunteers, so add a step to test the application.
First exercise ZAP locally opening vagrant ssh
to development
at the root of the project
docker pull 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
docker run 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0 zap-baseline.py -t http://192.168.0.11:3000
Success in the command line resembles the following for the docker pull
.
[vagrant@development helloworld-web]$ docker pull 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
2.7.0: Pulling from nemonik/zap2docker-stable
660c48dd555d: Pull complete
4c7380416e78: Pull complete
421e436b5f80: Pull complete
e4ce6c3651b3: Pull complete
be588e74bd34: Pull complete
2e09786d5b2f: Pull complete
197722f79cca: Pull complete
bd33ded2095e: Pull complete
4b06ea4ab95b: Pull complete
5549c936d696: Pull complete
c2bfc8c8addb: Pull complete
1a4b29516a99: Pull complete
8ce1a9f9c46c: Pull complete
8808d1ac4805: Pull complete
24eed6f4cce3: Pull complete
2cd87b08c44e: Pull complete
5ae7b6620997: Pull complete
d6d847cf31e3: Pull complete
1bd5a2ca7470: Pull complete
737316e32acc: Pull complete
6379bbb7b3df: Pull complete
Digest: sha256:6787b975227b4e559fc029e48c0fbf3181ef2567abaae8c5b60b50d5948de9ec
Status: Downloaded newer image for 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
And the following for the docker run
[vagrant@development helloworld-web]$ docker run 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0 zap-baseline.py -t http://192.168.0.11:3000
Dec 01, 2018 8:52:48 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Password Autocomplete in Browser [10012]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Web Browser XSS Protection Not Enabled [10016]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header Scanner [10020]
PASS: Secure Pages Include Mixed Content [10040]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Application Error Disclosure [90022]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 4
http://192.168.0.11:3000/
http://192.168.0.11:3000/robots.txt
http://192.168.0.11:3000/sitemap.xml
http://192.168.0.11:3000
FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 1 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 13
NOTE
- OWASP ZAP is not the fastest tool, so give it time.
Great, now lets add another step to our pipeline after the selenium
step, but before services:
.
- name: owasp-zaproxy:
image: 192.168.0.11:5000/nemonik/zap2docker-stable:2.7.0
commands:
- zap-baseline.py -t http://192.168.0.11:3000 || true
Commit the code to GitLab hosted remote repo
git add .
git commit -m "added owasp-zaproxy step to pipeline"
git push origin master
Open on your host
http://192.168.0.11/root/helloworld-web
And monitor the progress of the build. The pipeline should execute in a few minutes.
Successful output for this stage resembles
+ zap-baseline.py -t http://192.168.0.11:3000 || true
Apr 03, 2019 2:39:30 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Total of 3 URLs
PASS: Cookie No HttpOnly Flag [10010]
PASS: Cookie Without Secure Flag [10011]
PASS: Password Autocomplete in Browser [10012]
PASS: Incomplete or No Cache-control and Pragma HTTP Header Set [10015]
PASS: Web Browser XSS Protection Not Enabled [10016]
PASS: Cross-Domain JavaScript Source File Inclusion [10017]
PASS: Content-Type Header Missing [10019]
PASS: X-Frame-Options Header Scanner [10020]
PASS: Secure Pages Include Mixed Content [10040]
PASS: Private IP Disclosure [2]
PASS: Session ID in URL Rewrite [3]
PASS: Script Passive Scan Rules [50001]
PASS: Application Error Disclosure [90022]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 4
http://192.168.0.11:3000/
http://192.168.0.11:3000/robots.txt
http://192.168.0.11:3000/sitemap.xml
http://192.168.0.11:3000
FAIL-NEW: 0 FAIL-INPROG: 0 WARN-NEW: 1 WARN-INPROG: 0 INFO: 0 IGNORE: 0 PASS: 13
The application is relatively simple, so it was doubtful anything would be found, but there is a warning that a X-Content-Type-Options Header is missing that could be resolved by adding addtional handlers to the helloworld-web's main.go.
The helloworld-web
project can be viewed completed at
https://github.com/nemonik/helloworld-web
So, you may not realize it, but with the helloworld-web
application you've actually written a microservice.
What's a microservice?
- Microservices perform a single function (i.e., a single business capability or feature.) In our case, our application's single function is to return, "Helloworld!"
- Communicate with other services via well-defined APIs.
- Can be written using different frameworks and programming languages.
- Can be deployed independently or as a group.
The benefit
- Traditional monolithic architectures don’t lend themselves to cloud deployments.
- Nor do they scale.
- They’re simple not “cloud-native.” (More on that word later.)
- In monolithic architecture, if a single feature experiences a demand, the entire architecture must be scaled.
- In Microservice Architecture each feature runs as a separate service within its own container.
- Thereby, permitting each service to be scaled and maintained independently.
- Providing crash Isolation, so if a single piece of your service is crashing, then the rest of your application continues to work
- Providing Isolation for Security, so a compromise of one service only gains the adversary access to the resources of that single service
- Providing independent scaling, so each microservice’s compute utilization can be independently scaled up and down
- Increasing development velocity, lowering development risks thereby permitting teams to build faster in comparison to monolithic development.
So, with the helloworld-web
application, you've developed a micrsoservice, packaged it into a container (i.e., containerized it), and by doing so essentially created a cloud-native application.
Cloud-native computing deploys applications as microservices, packaging each into its own container, and by doing so have opened yourself up to the capability of being able to dynamically orchestrate the container to optimize resource utilization, elastic scaling, security issolation...
Granted our microservice doesn't do much, but we can get a peak of the possibilities without even installing a full blown Container Platform, such as, Kubernetes, Docker Swarm, or OpenShift.
On the development
vagrant
docker swarm init --advertise-addr=192.168.0.10
Edit the go/src/github.com/nemonik/helloworld-web/main.go
package main
import (
"fmt"
"log"
"net/http"
"os"
)
var hostname string
func main() {
hostname, _ = os.Hostname()
http.HandleFunc("/", handler)
fmt.Print("listening on :3000\n")
log.Fatal(http.ListenAndServe(":3000", logRequest(http.DefaultServeMux)))
}
func handler(w http.ResponseWriter, r *http.Request) {
_, err := fmt.Fprintf(w, "Hello world! %s\n", hostname)
if err != nil {
log.Fatal(err)
}
}
func logRequest(handler http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Printf("%s %s %s\n", r.RemoteAddr, r.Method, r.URL)
handler.ServeHTTP(w, r)
})
}
Changes made:
- Added
os
package. - Added the
hostname
global variable. - Set
hostname
global inmain()
. - Append
hostname
to output of thehandler
.
Why these changes were made will become apparent.
At the root of the hellworld-web
application create a new file named docker-compose.yml
and add to it the following content
version: "3"
services:
helloworld:
build: .
image: nemonik/helloworld-web:latest
deploy:
replicas: 1
resources:
limits:
cpus: "0.1"
memory: 4M
restart_policy:
condition: on-failure
ports:
- "3000:3000"
networks:
- web
networks:
web:
The docker-compose.yml
file tells Docker to do the following:
- Use the
nemonik/helloworld-web:latest
Docker image. - Run 1 instance of that image as a service called
service
, limiting each replica to at most 10% of the CPU and 4MB of RAM (the smallest amount one can in this instance). - Restart immediately if container fails.
- Share port 3000 via a load-balanced network called
web
. - All running on the load-balanced overlay
web
network.
To spin up our app
docker-compose build
docker stack deploy -c docker-compose.yml helloworld
The output will resemeble
WARNING: Some services (helloworld) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Building helloworld
Step 1/8 : FROM 192.168.0.11:5000/nemonik/golang:1.12.1
---> 6aa2d9de5f78
Step 2/8 : MAINTAINER Michael Joseph Walsh <[email protected]>
---> Using cache
---> 17abd315c43e
Step 3/8 : RUN mkdir /app
---> Using cache
---> 3ae5aef96fce
Step 4/8 : ADD main.go /app/
---> Using cache
---> dffac9d00543
Step 5/8 : WORKDIR /app
---> Using cache
---> b1e5d8c118fb
Step 6/8 : RUN go build -o helloworld-web .
---> Using cache
---> a8d30237b993
Step 7/8 : CMD ["/app/helloworld-web"]
---> Using cache
---> b2668738c4ea
Step 8/8 : EXPOSE 3000
---> Using cache
---> e999d0f3737d
Successfully built e999d0f3737d
Successfully tagged nemonik/helloworld-web:latest
Ignoring unsupported options: build
Creating network helloworld_web
Creating service helloworld_helloworld
Now list the services Docker is running via
docker service list
The output will look like
ID NAME MODE REPLICAS IMAGE PORTS
mgra793avso7 helloworld_helloworld replicated 1/1 nemonik/helloworld-web:latest *:3000->3000/tcp
Now test your service
[vagrant@development helloworld-web]$ curl http://192.168.0.10:3000
The output will look like
Hello world! b6835470176e
That random hex number is the result of our code change. It is the hostname of the container that responded to your request.
Now lets scale from 1 to 5 replicas via
docker service scale helloworld_helloworld=5
The output will resemble
helloworld_helloworld scaled to 5
overall progress: 5 out of 5 tasks
1/5: running
2/5: running
3/5: running
4/5: running
5/5: running
verify: Service converged
Re-running
docker service list
Will also show we've scale to 5 replicas.
ID NAME MODE REPLICAS IMAGE PORTS
mgra793avso7 helloworld_helloworld replicated 5/5 nemonik/helloworld-web:latest *:3000->3000/tcp
Now run curl several times via
for i in {1..10}
do
curl http://192.168.0.10:3000
done
The output will resemble
Hello world! 6a0e64e07f27
Hello world! 899d588cae34
Hello world! 3fd46049e8bb
Hello world! 743630bf2f8b
Hello world! b6835470176e
Hello world! 6a0e64e07f27
Hello world! 899d588cae34
Hello world! 3fd46049e8bb
Hello world! 743630bf2f8b
Hello world! b6835470176e
Notice how the curl
requests are Round-robin load balanced across each replica container. Typically, these containers would be spread across a container platform's cluster thereby permiting you to scale your application to accommodate shifting capacity needs. In other words, to be become elastic. Further, the approach permits iterative software development and deployment (thereby benefitting from DevOps methods and repeated practices), isolation, and resiliency.
Git clone the Python+Flask Magic Eight Ball web app
https://github.com/nemonik/magiceightball
and develop a pipeline for.
If you're done with your vagrants, shoo them away from the root of the project on the host
vagrant destroy -f
And they're gone.
That's a wrap.