Jenkins master and agent cluster utilizing the Jenkins Kubernetes plugin

• https://plugins.jenkins.io/kubernetes

Inspired by the awesome work of Carlos Sanchez mailto:[email protected]

This chart will do the following:

• 1 x Jenkins Master with port 8080 exposed on an external LoadBalancer
• All using Kubernetes Deployments

To install the chart with the release name my-release:

$ helm install my-release stable/jenkins

A major chart version change (like v0.40.0 -> v1.0.0) indicates that there is an incompatible breaking change needing manual actions.

Breaking changes:

• values have been renamed to follow helm chart best practices for naming conventions so that all variables start with a lowercase letter and words are separated with camelcase https://helm.sh/docs/chart_best_practices/#naming-conventions
• all resources are now using recommended standard labels https://helm.sh/docs/chart_best_practices/#standard-labels

As a result of the label changes also the selectors of the deployment have been updated. Those are immutable so trying an updated will cause an error like:

Error: Deployment.apps "jenkins" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/component":"jenkins-master", "app.kubernetes.io/instance":"jenkins"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

In order to upgrade, delete the Jenkins Deployment before upgrading:

kubectl delete deploy jenkins

The following tables list the configurable parameters of the Jenkins chart and their default values.

Some third-party systems, e.g. GitHub, use HTML-formatted data in their payload sent to a Jenkins webhooks, e.g. URL of a pull-request being built. To display such data as processed HTML instead of raw text set master.enableRawHtmlMarkupFormatter to true. This option requires installation of OWASP Markup Formatter Plugin (antisamy-markup-formatter). The plugin is not installed by default, please update master.installPlugins.

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install my-release -f values.yaml stable/jenkins

Tip: You can use the default values.yaml

Your Jenkins Agents will run as pods, and it's possible to inject volumes where needed:

agent:
  volumes:
  - type: Secret
    secretName: jenkins-mysecrets
    mountPath: /var/run/secrets/jenkins-mysecrets

The supported volume types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret. Each type supports a different set of configurable attributes, defined by the corresponding Java class.

To make use of the NetworkPolicy resources created by default, install a networking plugin that implements the Kubernetes NetworkPolicy spec.

For Kubernetes v1.5 & v1.6, you must also turn on NetworkPolicy by setting the DefaultDeny namespace annotation. Note: this will enforce policy for all pods in the namespace:

kubectl annotate namespace default "net.beta.kubernetes.io/network-policy={\"ingress\":{\"isolation\":\"DefaultDeny\"}}"

Install helm chart with network policy enabled:

$ helm install stable/jenkins --set networkPolicy.enabled=true

master.securityRealm in values can be used to support custom security realm instead of default LegacySecurityRealm. For example, you can add a security realm to authenticate via keycloak.

securityRealm: |-
  <securityRealm class="org.jenkinsci.plugins.oic.OicSecurityRealm" plugin="[email protected]">
    <clientId>testId</clientId>
    <clientSecret>testsecret</clientSecret>
    <tokenServerUrl>https:testurl</tokenServerUrl>
    <authorizationServerUrl>https:testAuthUrl</authorizationServerUrl>
    <userNameField>email</userNameField>
    <scopes>openid email</scopes>
  </securityRealm>

master.additionalConfig can be used to add additional config files in config.yaml. For example, it can be used to add additional config files for keycloak authentication.

additionalConfig:
  testConfig.txt: |-
    - name: testName
      clientKey: testKey
      clientURL: testUrl

master.serviceLabels can be used to add custom labels in jenkins-master-svc.yaml. For example,

ServiceLabels:
  expose: true

The Jenkins image stores persistence under /var/jenkins_home path of the container. A dynamically managed Persistent Volume Claim is used to keep the data across deployments, by default. This is known to work in GCE, AWS, and minikube. Alternatively, a previously configured Persistent Volume Claim can be used.

It is possible to mount several volumes using persistence.volumes and persistence.mounts parameters.

1. Create the PersistentVolume
2. Create the PersistentVolumeClaim
3. Install the chart

$ helm install my-release --set persistence.existingClaim=PVC_NAME stable/jenkins

It is possible to define which storage class to use:

$ helm install my-release --set persistence.storageClass=customStorageClass stable/jenkins

If set to a dash (-, as in persistence.storageClass=-), the dynamic provision is disabled.

If the storage class is set to null or left undefined (persistence.storageClass=), the default provisioner is used (gp2 on AWS, standard on GKE, AWS & OpenStack).

Jenkins Configuration as Code is now a standard component in the Jenkins project. Add a key under configScripts for each configuration area, where each corresponds to a plugin or section of the UI. The keys (prior to | character) are just labels, and can be any value. They are only used to give the section a meaningful name. The only restriction is they must conform to RFC 1123 definition of a DNS label, so may only contain lowercase letters, numbers, and hyphens. Each key will become the name of a configuration yaml file on the master in /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin during Jenkins startup. The lines after each | become the content of the configuration yaml file. The first line after this is a JCasC root element, eg jenkins, credentials, etc. Best reference is the Documentation link here: https://<jenkins_url>/configuration-as-code. The example below creates ldap settings:

configScripts:
  ldap-settings: |
    jenkins:
      securityRealm:
        ldap:
          configurations:
            configurations:
              - server: ldap.acme.com
                rootDN: dc=acme,dc=uk
                managerPasswordSecret: ${LDAP_PASSWORD}
              - groupMembershipStrategy:
                  fromUserRecord:
                    attributeName: "memberOf"

Further JCasC examples can be found here.

Config as Code changes (to master.JCasC.configScripts) can either force a new pod to be created and only be applied at next startup, or can be auto-reloaded on-the-fly. If you choose master.sidecars.autoConfigReload.enabled: true, a second, auxiliary container will be installed into the Jenkins master pod, known as a "sidecar". This watches for changes to configScripts, copies the content onto the Jenkins file-system and issues a CLI command via SSH to reload configuration. The admin user (or account you specify in master.adminUser) will have a random SSH private key (RSA 4096) assigned unless you specify a key in master.adminSshKey. This will be saved to a k8s secret. You can monitor this sidecar's logs using command kubectl logs <master_pod> -c jenkins-sc-config -f If you want to enable auto-reload then you also need to configure rbac as the container which triggers the reload needs to watch the config maps.

master:
  JCasC:
    enabled: true
  sidecars:
    configAutoReload:
      enabled: true
rbac:
  install: true

When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist. Since the admin account is used by the sidecar to reload config, in order to use auto-reload, you must change the .master.adminUser to a valid username on your LDAP (or other) server. If you use the matrix-auth plugin, this user must also be granted Overall\Administer rights in Jenkins. Failure to do this will cause the sidecar container to fail to authenticate via SSH and enter a restart loop. You can enable LDAP using the example above and add a Config as Code block for matrix security that includes:

configScripts:
  matrix-auth: |
    jenkins:
      authorizationStrategy:
        projectMatrix:
          grantedPermissions:
          - "Overall/Administer:<AdminUser_LDAP_username>"

You can instead grant this permission via the UI. When this is done, you can set master.sidecars.configAutoReload.enabled: true and upon the next Helm upgrade, auto-reload will be successfully enabled.

RBAC is enabled by default if you want to disable it you will need to do the following:

• helm install stable/jenkins --set rbac.create=false

Adds a backup CronJob for jenkins, along with required RBAC resources.

To restore a backup, you can use the kube-tasks underlying tool called skbn, which copies files from cloud storage to Kubernetes. The best way to do it would be using a Job to copy files from the desired backup tag to the Jenkins pod. See the skbn in-cluster example for more details.

The default settings of this helm chart let Jenkins run as root user with uid 0. Due to security reasons you may want to run Jenkins as a non root user. Fortunately the default jenkins docker image jenkins/jenkins contains a user jenkins with uid 1000 that can be used for this purpose.

Simply use the following settings to run Jenkins as jenkins user with uid 1000.

master:
  runAsUser: 1000
  fsGroup: 1000

Jobs can be created (and overwritten) by providing jenkins config xml within the values.yaml file. The keys of the map will become a directory within the jobs directory. The values of the map will become the config.xml file in the respective directory.

Below is an example of a values.yaml file and the directory structure created:

master:
  jobs:
    test-job: |-
      <?xml version='1.0' encoding='UTF-8'?>
      <project>
        <keepDependencies>false</keepDependencies>
        <properties/>
        <scm class="hudson.scm.NullSCM"/>
        <canRoam>false</canRoam>
        <disabled>false</disabled>
        <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
        <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
        <triggers/>
        <concurrentBuild>false</concurrentBuild>
        <builders/>
        <publishers/>
        <buildWrappers/>
      </project>
    test-job-2: |-
      <?xml version='1.0' encoding='UTF-8'?>
      <project>
        <keepDependencies>false</keepDependencies>
        <properties/>
        <scm class="hudson.scm.NullSCM"/>
        <canRoam>false</canRoam>
        <disabled>false</disabled>
        <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
        <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
        <triggers/>
        <concurrentBuild>false</concurrentBuild>
        <builders/>
        <publishers/>
        <buildWrappers/>

.
├── _test-job-1
|   └── config.xml
├── _test-job-2
|   └── config.xml

Docs taken from https://github.com/jenkinsci/docker/blob/master/Dockerfile: Jenkins is run with user jenkins, uid = 1000. If you bind mount a volume from the host or a data container,ensure you use the same uid

The master pod uses an Init Container to install plugins etc. If you are behind a corporate proxy it may be useful to set master.initContainerEnv to add environment variables such as http_proxy, so that these can be downloaded.

Additionally, you may want to add env vars for the Jenkins container, and the JVM (master.javaOpts).

master:
  initContainerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
    - name: no_proxy
      value: ""
  containerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
  javaOpts: >-
    -Dhttp.proxyHost=192.168.64.1
    -Dhttp.proxyPort=3128
    -Dhttps.proxyHost=192.168.64.1
    -Dhttps.proxyPort=3128

The following configuration method is deprecated and will be removed in an upcoming version of this chart. We recommend you use Jenkins Configuration as Code to configure instead. When creating a new parent chart with this chart as a dependency, the customConfigMap parameter can be used to override the default config.xml provided. It also allows for providing additional xml configuration files that will be copied into /var/jenkins_home. In the parent chart's values.yaml, set the jenkins.master.customConfigMap value to true like so

jenkins:
  master:
    customConfigMap: true

and provide the file templates/config.tpl in your parent chart for your use case. You can start by copying the contents of config.yaml from this chart into your parent charts templates/config.tpl as a basis for customization. Finally, you'll need to wrap the contents of templates/config.tpl like so:

{{- define "override_config_map" }}
    <CONTENTS_HERE>
{{ end }}

This configuration enable jenkins to use keystore inorder to serve https: https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777
Here is the value file section related to keystore configuration.
Keystore itself should be placed in front of jenkinsKeyStoreBase64Encoded key and in base64 encoded format. To achive that after having keystore.jks file simply do this: cat keystore.jks | base64 and paste the output in front of jenkinsKeyStoreBase64Encoded .
After enabling httpsKeyStore.enable make sure that httpPort and targetPort are not the same as targetPort will serve https.
Do not set master.httpsKeyStore.httpPort to -1 because it will cause readiness and liveliness prob to fail.
If you already have a kubernetes secret that has keystore and its password you can specify its' name in front of jenkinsHttpsJksSecretName, You need to remember that your secret should have proper data key names jenkins-jks-file and https-jks-password.

master:
   httpsKeyStore:
       enable: true
       jenkinsHttpsJksSecretName: ''
       httpPort: 8081
       path: "/var/jenkins_keystore"
       fileName: "keystore.jks"
       password: "changeit"
       jenkinsKeyStoreBase64Encoded: ''