Giter Club home page Giter Club logo

figwasp's Introduction

Keep Kubernetes Deployments up-to-date with the :latest* container images

Practising continuous integration in a highly-automated environment means that new versions of multiple build artifacts are generated with every incremental change to the codebase. In the context of cloud-native microservices, these build artifacts often take the form of Linux container images that are deployed and managed via an orchestrator such as Kubernetes. Resources would be provisioned for a test cluster, to which artifacts built from the latest commit pushed to the trunk of the code repository would be deployed to undergo acceptance tests. Artifacts that pass the test would be tagged as releasable for deployment to a production cluster.

While Kubernetes can be configured to pull the latest image with a given tag when a Deployment is restarted, it does not provide a means of watching container image repositories for changes in tags. A restart had to be triggered by an administrator using the command-line interface, or by build automation calling the Kubernetes API. The former is a manual bottleneck in an otherwise automated workflow, while the latter necessitates the delegation of cluster administrator privileges and credentials.

Figwasp offers an elegant alternative to the existing "push" model and can easily be employed as a Kubernetes CronJob with minimal configuration. It works by comparing the digests of deployed images to those in image repositories and triggering a rolling update of the Deployment if it detects a difference.

* or more commonly a mutable :<major>.<minor> tag following the SemVer convention e.g. :1.2

User Story

As a Kubernetes administrator deploying container applications to a cluster,
I want a rolling restart of a deployment to be automatically triggered
whenever the tag of a currently-deployed image is inherited by another image
so that the deployment is always up-to-date without manual intervention.

Kubernetes issue #33664 "Force pods to re-pull an image without changing the image tag"

@yissachar:

Problem

A frequent question that comes up on Slack and Stack Overflow is how to trigger an update to a Deployment/RS/RC when the image tag hasn't changed but the underlying image has.

Consider:

  1. There is an existing Deployment with image foo:latest
  2. User builds a new image foo:latest
  3. User pushes foo:latest to their registry
  4. User wants to do something here to tell the Deployment to pull the new image and do a rolling-update of existing pods

The problem is that there is no existing Kubernetes mechanism which properly covers this.

StackOverflow question "Kubernetes: --image-pull-policy always does not work"

DenCowboy:

I have a Kubernetes deployment which uses image: test:latest (not real image name but it's the latest tag). This image is on docker hub. I have just pushed a new version of test:latest to dockerhub. I was expecting a new deployment of my pod in Kubernetes but nothing happends.

I've created my deployment like this:

kubectl run sample-app --image=`test:latest` --namespace=sample-app --image-pull-policy Always

Why isn't there a new deployment triggered after the push of a new image?

StackOverflow question "How to automatically restart pods when a new image ready"

user1739211:

I was expecting that every time I would push a new image with the tag latest, the pods would be automatically killed and restart using the new images.

I tried the rollout command

kubectl rollout restart deploy simpleapp-direct

The pods restart as I wanted.

However, I don't want to run this command every time there is a new latest build. How can I handle this situation ?.

StackOverflow question "Kubernetes how to make Deployment to update image"

Andriy Kopachevskyy:

I do have deployment with single pod, with my custom docker image like:

containers:
  - name: mycontainer
    image: myimage:latest

During development I want to push new latest version and make Deployment updated.

StackOverflow question "Update a Deployment image in Kubernetes"

Yuval Simhon:

I saw this in the documentation:

Note: a Deployment’s rollout is triggered if and only if the Deployment’s pod template (i.e. .spec.template) is changed.

I'm searching for an easy way/workaround to automate the flow: Build triggered > a new Docker image is pushed (withoud version changing) > Deployment will update the pod > service will expose the new pod.

Usage

Example Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-app
    figwasp/target: "true" # Figwasp ignores Deployments without this label
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-repository/my-image:latest
        ports:
        - containerPort: 80
      imagePullPolicy: Always
      imagePullSecrets:
      - name: my-repository

The value of the label figwasp/target: "true" must be quoted, because Kubernetes allows only strings for label keys and values.

For Figwasp to work, it is important to set imagePullPolicy: Always if the image tag is anything other than :latest. (See relevant Kubernetes documentation.)

Figwasp makes use of imagePullSecrets when querying private container image repositories, eliminating the need for additional configuration and secret management.

Run Figwasp as a CronJob

Users should edit the merely illustrative spec.schedule to suit their needs.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: figwasp
spec:
  schedule: "*/5 9-18 * * 1-5" # https://crontab.guru/#*/5_9-18_*_*_1-5
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: figwasp
          containers:
          - name: figwasp
            image: ghcr.io/figwasp/figwasp:2
          # env:
          # - name: FIGWASP_TARGET_NAMESPACE
          #   value: "default"
          # - name: FIGWASP_CLIENT_TIMEOUT
          #   value: "30s"
          restartPolicy: Never

Figwasp must be run as a service account with the appropriate permissions to perform its functions.

Configure Permissions

apiVersion: v1
kind: ServiceAccount
metadata:
  name: figwasp
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default # must match value of env. var. `FIGWASP_TARGET_NAMESPACE`
  name: figwasp
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["list", "get", "update"]
- apiGroups: ["apps"]
  resources: ["replicasets"]
  verbs: ["list"]
- apiGroups: [""]
  resources: ["pods", "secrets"]
  verbs: ["list"]

Figwasp needs permissions to list (and get) Deployments, ReplicaSets and Pods so that it can collate the image references and digests of deployed images. Permission to list secrets is required for Figwasp to obtain credentials necessary when querying private container image repositories for image digests. To initiate a rolling restart of a Deployment, Figwasp must be granted permission to update the Deployment.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: default # must match value of env. var. `FIGWASP_TARGET_NAMESPACE`
  name: figwasp
subjects:
- kind: ServiceAccount
  name: figwasp
roleRef:
  kind: Role
  name: figwasp
  apiGroup: rbac.authorization.k8s.io

figwasp's People

Contributors

joel-ling avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

figwasp's Issues

Extend the functionality to StatefulSets

Currently only Deployments are supported, while it might be convenient to extend the functionality to StatefulSets as well, which are quite popular for applications.

Support multiple Deployments dinamically

Currently a single resource can be monitored and it's configured via the FIGWASP_TARGET_DEPLOYMENT environment variable in the Figwasp manifest.

Using annotations (such as figwasp/target: true or similar) attached to the monitored Deployments brings some benefits:

  • more Deployments can be monitored, without the need of setting a list in the same environment variable
  • Deployments can be added and removed naturally by adding/editing/removing the annotation
  • prevents to update Figwasp itself

Explicit namespace in Role (and RoleBinding)

In the README, a Role (and RoleBinding) is used in order to provide permissions to the CronJob.

Since a Role is scoped at namespace level, I would make namespace explicit in the Role and RoleBinding definitions.

Also, as alternative a ClusterRole (and a ClusterRoleBinding) can be used instead.

Prevent concurrent re-deployments

The README mentions to schedule the CronJob every minute, and this might cause a new Job instance to start before the previous one completes, which in turn means the same Deployment resource can be triggered once a re-deployment is already ongoing.

In order to prevent this, I'd to propose few options:

  • set concurrencyPolicy: Forbid to prevent concurrent Jobs
  • in addition to the above parameter, consider lowering the frequency
  • as alternative, replace the CronJob with a Deployment and an internal loop

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.