jetstack / version-checker Goto Github PK

View Code? Open in Web Editor NEW

645.0 11.0 75.0 605 KB

Kubernetes utility for exposing image versions in use, compared to latest available upstream, as metrics.

Home Page: https://jetstack.io

License: Apache License 2.0

Dockerfile 0.24% Makefile 0.56% Go 98.58% Mustache 0.62%

kubernetes prometheus image version utility sre go grafana grafana-dashboard docker

version-checker's Introduction

version-checker

version-checker is a Kubernetes utility for observing the current versions of images running in the cluster, as well as the latest available upstream. These checks get exposed as Prometheus metrics to be viewed on a dashboard, or soft alert cluster operators.

Registries

version-checker supports the following registries:

ACR
Docker Hub
ECR
GCR (inc gcr facades such as k8s.gcr.io)
Quay
Self Hosted (Docker V2 API compliant registries, e.g. registry, artifactory etc.). Multiple self hosted registries can be configured at once.

These registries support authentication.

Installation

version-checker can be installed as either static manifests;

$ kubectl apply -k ./deploy/yaml

Or through helm;

$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm install version-checker jetstack/version-checker
NAME: version-checker
LAST DEPLOYED: Wed Jul 12 17:47:41 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

The helm chart supports creating a Prometheus/ServiceMonitor to expose the version-checker metrics.

Grafana Dashboard

A grafana dashboard is also available to view the image versions as a table.

Grafana Dashboard

Options

By default, without the flag -a, --test-all-containers, version-checker will only test containers where the pod has the annotation enable.version-checker.io/*my-container*, where *my-container* is the name of the container in the pod.

version-checker supports the following annotations present on other pods to enrich version checking on image tags:

pin-major.version-checker.io/my-container: 4: will pin the major version to check to 4 (v4.0.0).
pin-minor.version-checker.io/my-container: 3: will pin the minor version to check to 3 (v0.3.0).
pin-patch.version-checker.io/my-container: 23: will pin the patch version to check to 23 (v0.0.23).
use-metadata.version-checker.io/my-container: "true": will allow to search for image tags which contain information after the first part of the semver string. For example, this can be pre-releases or build metadata (v1.2.4-alpha.0, v1.2.3-debian-r3).
use-sha.version-checker.io/my-container: "true": will check against the latest SHA tag available. Essentially, the latest image by date. This is silently set to true if no image tag, or "latest" image tag is set. Cannot be used with any other options.
match-regex.version-checker.io/my-container: ^v\d+\.\d+\.\d+-debian-: is used for only comparing against image tags which match the regex set. For example, the above annotation will only check against image tags which have the form of something like v1.3.4-debian-r30. use-metadata.version-checker.io is not required when this is set. All other options, apart from URL overrides, are ignored when this is set.
override-url.version-checker.io/my-container: docker.io/bitnami/etcd: is used to change the URL for where to lookup where the latest image version is. In this example, the current version of my-container will be compared against the image versions in the docker.io/bitnami/etcd registry.

Metrics

By default, version-checker will expose the version information as Prometheus metrics on 0.0.0.0:8080/metrics.

version-checker's People

Contributors

Stargazers

Watchers

version-checker's Issues

Wrong url path for token in selfhosted client

Hello,

I deployed version-checker with enabled self hosted registry (artifactory), but I got 405 error in the log (sensitive info replaced with stars):
error: failed to setup image registry clients: failed to create selfhosted client "https://*****": failed to setup token auth (405): { "errors" : [ { "status" : 405, "message" : "Method Not Allowed" } ] }

It seems that selfhosted client in the version-checker has wrong token url path for artifactory because in official documentation it is different.

I tested it with curl and got the same result:
$ curl -u*** -XPOST "https://***/v2/token" -d "username=***"
{ "errors" : [ { "status" : 405, "message" : "Method Not Allowed" } ] }

With correct url path it works:
$ curl -u*** -XPOST "https://***/artifactory/api/security/token" -d "username=***"
{ "scope" : "applied-permissions/groups:*** api:*", "access_token" : "***", "expires_in" : 3600, "token_type" : "Bearer" }

Gitlab support

Hello,

It appears that this tool doesn't support custom Gitlab repositories. It returns a 404.

Is this the case or I am doing something wrong?

Thank you in advance.

Add to helm repo/hub

Hi,

Currently it's not easy to use this chart with the helm-operator.
With the helm-operator you can either specify a Helm repo/hub or a git repo.

The git repo one is only safe to use with tags/branches as I don't want it linked with the master 😄.

I would like to contribute to the chart but this means the chart version/release won't be in sync with the app version. Which will result that the git repo tag approach is not useable anymore.

Adding the chart to https://charts.jetstack.io would solve this issue or use the chart-releaser? This will also result in an easier distribution of the chart 😃

Handle Paging in clients

We do not currently properly support paging in clients. This means that tag sets will only include tags which are returned in the first page from the registry. This may lead to incorrect results when reporting the latest image when options have been applied.

We should update our clients to implement proper HTTP paging, to ensure we get the fill list of tag results.

Support ECR image registry

Unauthorized error on public ECR repository

When attempting to check image tags against the public ECR repository I am met with the following error:

failed to check container image \"opensearch\": failed to get tags from remote registry for "public.ecr.aws/opensearchproject/opensearch\": {\"errors\":[{\"code\":\"DENIED\",\"message\":\"Not Authorized\"}]}\n, requeuing

Please add Pod Labels in Helm Chart

Hi there,

If could you add Pod Labels in the Helm Chart, and also an option to disable ClusterRole And ClusterRoleBinding, that would be great, as it a hard requirement for us to be able to use it, since we just need to access images running on namespace level only.

Thanks in advance!.

x509: certificate signed by unknown authority

curl -k -u docker_reg:0IFmcttx https://ol8-tst-k8s-master-node.com:5000/v2/int-xfr-dispatch/tags/list?n=5000
{"name":"int-xfr-dispatch","tags":["tst-20211102-J11-237-1183f99c","tst-20211117-J13-240-c3f64612"

version-checker
Args:
--test-all-containers
--selfhosted-username=docker_reg
--selfhosted-password=0IFmcttx
--insecure-skip-tls-verify=true
--log-level=debug
This is the Error that I am getting in the log file.
How are you calling the versions. What do I need to add to my config
I would seem that the x509 error could be from an incorrect certificate. Not sure which certificate could be incorrect as I do not understand how you are calling the Get https://

grafana dashboard display no data

Display no data in the Grafana Dashboard after deployment according to the install content in the README.

Remove garbage collection from version package initialization

Currently, we start the garbage collector for the version package pkg/version, as part of a call to New.

version-checker/pkg/version/version.go

Line 38 in 6266beb

go vg.garbageCollect(cacheTimeout / 2)

This isn't great for having this package be importable, and should move the start of the garbage collection to another func. The controller can start it as a go routine.

/assign

Terminated pods remain visible in metrics

Hey

For a certain application that regularly spins up in our CICD flow, I've noticed that some instances keep hanging around in the metrics long after the pod gets terminated. In the sample metrics below you'll see that there are different values for latest_version indicating old data. Restarting version_checker does clean these up.

I'm running:
version-checker:
Container ID: docker://581de433d72c6199d5194af1eb3c5fb8deff331acb932567ab92368afb4ae64c
Image: quay.io/jetstack/version-checker:v0.2.1
Image ID: docker-pullable://quay.io/jetstack/version-checker@sha256:5f6f8ba0b671db2382f811b58fad64a2fd22ca1caaf087f9708c8e4b8309d7e0

Metrics:

version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.2",namespace="jenkins",pod="release-bump-9371-hbjm2-z1g91-6sd8x"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.3",namespace="jenkins",pod="release-bump-9379-z9lfh-xp6gf-j22qn"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9442-gc98d-9rc8c-hzjld"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9450-9jc31-fr5f5-kzxch"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9451-rp62c-kt9fj-4msmb"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9483-5q6pn-c9951-pvk7l"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.5",namespace="jenkins",pod="release-bump-9493-bsnzm-2f3q9-0nczp"} 0

Inconsistent values for daemonset

Hey

I've been trying out this tool for a bit and found a few issues. Most of them have to do with the regex matching but this one I can't figure out.

For some reason there is one fluent-bit pod in the daemonset that gives a false value while being identical to the others. This behavior is replicated on another cluster (which is set up identical to this one).

Metrics:

version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-59hw7"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-7btfd"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-7fh2r"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-cpv5p"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-j5lh8"} 0
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-rhrxf"} 1

Support helm chart version checking

Support building multi-arch images of version-checker

We should publish multi-arch images, similar to cert-manager.

/assign

Wrong EnableAnnotationKey name in message and options

Hello,

When I start version-checker, the log indicates that the annotation to enable the check is enable.version-checker/${my-container} :

version-checker-6477448bf-4lvvn 
version-checker time="2020-11-20T18:08:42Z" 
level=info 
msg="flag --test-all-containers=false only containers with the annotation \"enable.version-checker/${my-container}=true\" will be parsed"

But in the documentation and in the code the annotation prefix is enable.version-checker.io

The .io is missing in the log message here and also in options message here, isn't it ?

Error prometheus metrics

Hi,
It would be great to have a prometheus metric with the number of errors, it would be helpful to detect possible issues with some images.
What do you think?

Thanks

There's a problem with the latest tag fetch logic for some of the mirrored tags.

Like docker.io/coredns/coredns

“Silently” fallback to comparing SHA when using `latest`

When a container image uses the latest tag, version-checker should fallback to comparing SHA “silently” (the log level should be relaxed).

At the moment, logs are filled with:

time="2020-08-18T18:38:35Z" level=warning msg="image using \"latest\" tag, comparing image SHA \"sha256:530e471dba7ee6995a230b43a1c8db4404873fb0402469397c8674dcf920d59f\"" container=foo
module=controller name=foobar-0 namespace=foobar

I believe this message should be emitted with log level debug.

version-checker fails with nginx-ingress controller new registry

I’m not sure why but version-checker seems to fail on the nginx-ingress controller image:

time="2020-08-18T18:38:33Z" level=error msg="error syncing 'nginx-ingress-9bbd564fc-8ktdv/nginx-ingress': failed to sync pod nginx-ingress-9bbd564fc-8ktdv/nginx-ingress: failed to test container image
\"nginx-ingress-controller\": \"us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller\": no tags found for given image URL: \"us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller\", requeuing"       
module=controller

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller is the new official image according to ingress-nginx release notes for version 0.34.0.

I believe the issue might be that some images have been published without a tag in that registry.

add nodeSelector to the chart

So in my cluster with a mix of arch's it doesn't try and run on a Windows node. :)

Support self hosted registries

Implementing Artifactory

First of all, thank you for this project. It's awesome.

For our stack I'm eager to combine this with Artifactory. I would not mind investing some time to get started on such feature. Yet I'm not sure if there is already a plan for this or any roadmap?

Basically, this "issue" is my request on how to help :)

Fixing already deprecated apiVersion for Role and RoleBinding of k8s manifests

Hi folks,

Now we use deprecated apiVersion ( "rbac.authorization.k8s.io/v1beta1" ) for Role and RoleBinding in Prometheus manifests
The current version of apiVersion is "rbac.authorization.k8s.io/v1".
Please fix it.

P.S.: I tried to create PR but your policy has dined my try.

missing contributing guide

Hello,

We were working on a similar tool so nice to see this repository. But in order to contribute to this repository, the jetstack-bot send the PR authors to the CONTRIBUTING.md file, which does not exist (#2 (comment))

False positives for strange tag versions

I have some strange version checks with popular images and I think there is something missing in the checking algorithm.

For the image grafana/grafana version-checker says the latest version is 9799770991, which is obviously correct if you compare just version numbers. But this tag is 8 months old and the actual latest tag currently would be 10.2.2. Another example would be quay.io/jetstack/cert-manager-webhook-arm64: v1.13.2 vs. 608111629.

I am wondering if it would be better to always take the publish date into account when checking versions?

Maybe there is another trick how to avoid these false positives?

Provide GHCR example

Hey! Thank you for making this terrific application. It's really useful.

I'm getting some errors in my use-case about unauthorized attempts to access the api.github.com URL for checking image tags. It looks like this cannot be accessed unauthenticated in v4 of the GitHub API spec (apparently it could be in v3?).

I'm looking for an example of how to configure a GHCR account. I see the PR where support was implemented but not really sure how to enable it via the helm chart.

Add kubernetes probes support

From looking at the deployment manifest, it currently doesn't seem to have any probes for readiness or liveness check.

It would be nice to get that fixed.

Add support for renewing client token

Currently, when supplying a username/password, we use the returned JWT forever.

We should add support for renewing the token, as a function of the expiry claim.

RenewAt(time.Now().Add(expiry - (duration / 3))

failed to get tags from remote registry for "docker.elastic.co/beats/filebeat"

I have the following error with the pods of the filebeat daemonset :

time="2021-01-20T17:48:18Z" level=error msg="error syncing 'filebeat-filebeat-v58xm/log-management': failed to sync pod filebeat-filebeat-v58xm/log-management: failed to check container image \"filebeat\": failed to get tags from remote registry for \"docker.elastic.co/beats/filebeat\": {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"beats/filebeat\",\"Action\":\"pull\"}]}]}\n, requeuing" module=controller

However, I can pull the docker image without issue :

docker pull docker.elastic.co/beats/filebeat:7.10.2  
7.10.2: Pulling from beats/filebeat
[...]

Here is the configuration of filebeat daemonset :

$ kubectl describe ds filebeat-filebeat
Name:           filebeat-filebeat
Selector:       app=filebeat-filebeat,release=filebeat
Labels:    app=filebeat-filebeat
                chart=filebeat-7.9.0
                heritage=Helm
                release=filebeat
Annotations:    deprecated.daemonset.template.generation: 9
Pods Status:  13 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:      app=filebeat-filebeat
                    chart=filebeat-7.9.0
                    heritage=Helm
                    release=filebeat
  Annotations:      configChecksum: 02443bbba38f244a1748e6769f4c91614189e4d5f598a687ec63b57389572df
                            enable.version-checker.io/filebeat: true
  Service Account:  filebeat-filebeat
  Containers:
   filebeat:
    Image:      docker.elastic.co/beats/filebeat:7.9.0
[...]

Helm Chart Upgrades Fail due to immutable selector fields in deployment.

I believe a bug was introduced in #115 when the selector fields were upgraded to use {{- include "version-checker.labels" . | nindent 6 }}. This is because version-checker.labels includes things like app.kubernetes.io/version which changes on newer versions. The selector field is immutable in Deployments and shouldn't change between the same release.

To replicate:

$ helm upgrade -i version-checker jetstack/version-checker --version v0.3.1
Release "version-checker" does not exist. Installing it now.
NAME: version-checker
LAST DEPLOYED: Mon Dec  4 15:49:49 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

$ helm upgrade -i version-checker jetstack/version-checker --version v0.3.2
Error: UPGRADE FAILED: cannot patch "version-checker" with kind Deployment: Deployment.apps "version-checker" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"version-checker", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"version-checker", "app.kubernetes.io/version":"v0.3.2", "helm.sh/chart":"version-checker-v0.3.2"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Add metrics for images failing version-check

Today version-checker keeps logging when it fails on some image.

Wouldn't it be nice having a metric for this?

Maybe something like version_checker_image_failed_count?
(with labels similar to the version_checker_is_latest_version metric — such as; container, image, exported_namespace, exported_pod)

Then you could be alerted when there's images that fail the version-check. That would prompt you to check the logs for further details.

Support existing secrets

I would like avoid storing my username/password in values.yaml and reuse an existing secret for my Artifactory server. An an example of a selfHosted configuration:

 - name: something
   url: something
   # username
   # password
   # token
   secretRef: something

My use case is that I use external-secrets for integration with AWS Secrets Manager and to avoid storing clear text passwords in GitHub.

Error when using the Harbor as selfhosted.

I have set up selfhosted for Harbor, but it is showing an error.
Here are my customized values:

selfhosted: 
- name: HARBOR
   host: harbor.mycompany.com
   username: <myUser>
   password: <myPassowrd>

Below is the log with the error:

Error: failed to setup image registry clients: failed to create selfhosted client "harbor.mycompany.com": failed to setup token auth: failed to send basic auth request "harbor.mycompany.com/v2/token": Post "harbor.mycompany.com/v2/token": unsupported protocol scheme ""
Usage:
  version-checker [flags]

App flags:

  -c, --image-cache-timeout duration     The time for an image version in the cache to be considered fresh. Images will be rechecked after this interval. (default 30m0s)
  -v, --log-level string                 Log level (debug, info, warn, error, fatal, panic). (default "info")
  -m, --metrics-serving-address string   Address to serve metrics on at the /metrics path. (default "0.0.0.0:8080")
  -a, --test-all-containers              If enabled, all containers will be tested, unless they have the annotation "enable.version-checker/${my-container}=false".

Auth flags:

      --acr-password string               Password to authenticate with azure container registry (VERSION_CHECKER_ACR_PASSWORD).
      --acr-refresh-token string          Refresh token to authenticate with azure container registry. Cannot be used with username/password (VERSION_CHECKER_ACR_REFRESH_TOKEN).
      --acr-username string               Username to authenticate with azure container registry (VERSION_CHECKER_ACR_USERNAME).
      --docker-password string            Password to authenticate with docker registry (VERSION_CHECKER_DOCKER_PASSWORD).
      --docker-token string               Token to authenticate with docker registry. Cannot be used with username/password (VERSION_CHECKER_DOCKER_TOKEN).
      --docker-username string            Username to authenticate with docker registry (VERSION_CHECKER_DOCKER_USERNAME).
      --ecr-access-key-id string          ECR access key ID for read access to private registries (VERSION_CHECKER_ECR_ACCESS_KEY_ID).
      --ecr-secret-access-key string      ECR secret access key for read access to private registries (VERSION_CHECKER_ECR_SECRET_ACCESS_KEY).
      --ecr-session-token string          ECR session token for read access to private registries (VERSION_CHECKER_ECR_SESSION_TOKEN).
      --gcr-token string                  Access token for read access to private GCR registries (VERSION_CHECKER_GCR_TOKEN).
      --quay-token string                 Access token for read access to private Quay registries (VERSION_CHECKER_QUAY_TOKEN).
      --selfhosted-password string        Password is authenticate with a selfhosted registry (VERSION_CHECKER_PASSWORD).
      --selfhosted-registry-host string   Full host of the selfhosted registry. Include http[s] scheme (VERSION_CHECKER_HOST
      --selfhosted-token string           Token to authenticate to a selfhosted registry. Cannot be used with username/password (VERSION_CHECKER_TOKEN).
      --selfhosted-username string        Username is authenticate with a selfhosted registry (VERSION_CHECKER_USERNAME).

Kubernetes flags:

      --as string                      Username to impersonate for the operation
      --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --cache-dir string               Default cache directory (default "/root/.kube/cache")
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
      --context string                 The name of the kubeconfig context to use
      --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kubeconfig string              Path to the kubeconfig file to use for CLI requests.
  -n, --namespace string               If present, the namespace scope for this CLI request
      --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
  -s, --server string                  The address and port of the Kubernetes API server
      --tls-server-name string         Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
      --token string                   Bearer token for authentication to the API server
      --user string                    The name of the kubeconfig user to use

error: failed to setup image registry clients: failed to create selfhosted client "harbor.mycompany.com": failed to setup token auth: failed to send basic auth request "harbor.mycompany.com/v2/token": Post "harbor.mycompany.com/v2/token": unsupported protocol scheme ""

Give info statement on runtime if test-all-containers is false

If the argument test-all-containers is false (default) , chances are that it won't process a thing. Therefore place a clear info message in the logs that it's running on false. That gives a better indication on what is going on.

Is this project still active? Container Image update?

Is this project still under maintenance?
I really like the project, however:
The image quay.io/jetstack/version-checker:v0.2.1 is now over 2 years old and has some security issues: https://trivy.dev/results/?image=quay.io/jetstack/version-checker:v0.2.1

I built a new image with alpine:3.17.1. Unfortunately, Trivy still finds some security issues regarding gobinary:

Would it be possible for someone to build an up-to-date container image without any security issues?

version-checker seemingly leaks memory and gets oom-killed

I am running version-checker on a single node, quite small cluster with ~60 pods. So far it is working nicely, but I do not understand the memory behavior it has.

I'm basically running the sample deployment file, plus the --test-all-containers flag and some cpu limits:

        resources:
          requests:
            cpu: 10m
            memory: 32M
          limits:
            cpu: 50m
            memory: 128M

kubectl get pod -o yaml

Over time, I see that version-checker approaches the memory limit and then stays near ~99% for a while. After some time, the kernel kills the ct due to OOM and k8s restarts the pod.

However, I do not see anything alarming in the logs, other than some failures and expected permission errors.

This doesn't seem to have any functional impact, but does fire some alerts and doesn't look good on my dashboards :)

Is this behavior intended, and/or is there any way to prevent it?

override-url does not seem to work

Hello,

I deployed version-checker with testAllContainers: true and try to override the registry for one of my image using the annotation :

apiVersion: v1
kind: Pod
metadata:
  annotations:
    override-url.version-checker.io/frontend: gcr.io/google-samples/microservices-demo/frontend
[...]
spec:
  containers:
    image: XXXX.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/frontend:v0.2.1
[...]

but the url is not overriden :

$ stern version -n version-checker | grep frontend

version-checker-58f996cdc5-tfffj version-checker time="2020-11-20T18:30:47Z" level=error msg="error syncing 'frontend-5c88f5f675-wccfx/microdemo': failed to sync pod frontend-5c88f5f675-wccfx/microdemo: failed to check container image \"server\": failed to get tags from remote registry for \"XXXX.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/frontend\": failed to describe images: EmptyStaticCreds: static credentials are empty, requeuing" module=controller

Minio scrape has odd error about constraints

Hi,

We have k8s running on Azure and the following (images) versions:
Prometheus: quay.io/prometheus/prometheus:v2.21.0
Version-checker: quay.io/jetstack/version-checker:v0.2.0
Grafana: grafana/grafana:7.2.0-ubuntu

Our config for the deployment of version-checker is:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: version-checker
  name: version-checker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: version-checker
  template:
    metadata:
      labels:
        app: version-checker
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
        enable.version-checker.io/version-checker: "true"
    spec:
      serviceAccountName: version-checker
      containers:
        - image: quay.io/jetstack/version-checker:v0.2.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          name: version-checker
          command: ["version-checker"]
          args:
            - "--log-level=debug"
            - "--test-all-containers=true"
          env:
            - name: VERSION_CHECKER_ACR_USERNAME
              valueFrom:
                secretKeyRef:
                  name: version-checker-pdokcr
                  key: ACR_USERNAME
            - name: VERSION_CHECKER_ACR_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: version-checker-pdokcr
                  key: ACR_PASSWORD
          resources:
            limits:
              cpu: 200m
              memory: 256Mi
            requests:
              cpu: 200m
              memory: 256Mi

On a side note, above config might look a bit odd, we use Kustomize to deploy our environments.

As per command-args above we want to scrape all pods, without exception. There are however several pods not being shown inside the table. Some of these are related to minio. We dont have minio running on our own ACR and use the "public" available images for that.

When I look at the logs of the version-checker, it has this to say about it:

time="2020-10-07T07:43:28Z" level=error msg="minio/mc: no tags found with these option constraints: {}" container=download-cleanup module=controller name=bgt-download-cleanup-1601942400-k7wlx namespace=download
time="2020-10-07T07:43:56Z" level=error msg="minio/minio: no tags found with these option constraints: {}" container=minio module=controller name=minio-695644c4b5-9dlq9 namespace=services
time="2020-10-07T07:43:56Z" level=error msg="pdok/mapserver: no tags found with these option constraints: {}" container=mapserver module=controller name=bzk-brogmwkenset-wms-v2-1-mapserver-5448896fd7-54bsh namespace=services

I have no idea what tags it is trying to figure out, as I have the command-arg set to "--test-all-containers=true". Shouldn't it pickup these pods this way or am I missing out on something?

Our pods have these images currently active and I kind of expect them to be shown in the table:
minio-etch: gcr.io/etcd-development/etcd:v3.3.10
minio-server: minio/minio:RELEASE.2020-10-03T02-19-42Z

Support Azure image registry

Version checker seems to pick the wrong latest version on multi-arch images when using SHA

What happened?
When I was running version-checker on KIND (amd64), the latest image version reported for the following container docker.io/ealen/echo-server:latest was 3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5, for arm/v6, instead of the tag for amd64 or the tag used for the manifests collection sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034 which is the actual ID assigned to the container when using :latest.

    Image:          docker.io/ealen/echo-server:latest
    Image ID:       docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034

Logs
Echo Image Manifests and digests:

docker buildx imagetools inspect docker.io/ealen/echo-server:latest
Name:      docker.io/ealen/echo-server:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
           
Manifests: 
  Name:      docker.io/ealen/echo-server:latest@sha256:397996a29090fde2eeb8866d972b683fd64afadbe7161d0e64c53350b4a83fd7
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/ealen/echo-server:latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6
             
  Name:      docker.io/ealen/echo-server:latest@sha256:80e5ed2796aa8242b965c80678bc512ae2396c58f13614be204e7e70ce4529b6
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      docker.io/ealen/echo-server:latest@sha256:1865bce901f1565d6262a730f0ae6f562a86e4cca04f65fea6cfe2c30896ec7f
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64
             
  Name:      docker.io/ealen/echo-server:latest@sha256:1f183d751287c846a8d1c5c21e8c635b175141b23af81df067631d8cd9906787
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

vc metrics:

 curl http://localhost:8083/metrics
# HELP version_checker_is_latest_version Where the container in use is using the latest upstream registry version
# TYPE version_checker_is_latest_version gauge
version_checker_is_latest_version{container="echo-2",current_version="sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034",image="docker.io/ealen/echo-server",latest_version="latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5",namespace="default",pod="echo-2-79fbc9bc76-dvggs"} 0
version_checker_is_latest_version{container="version-checker",current_version="v0.2.1",image="quay.io/jetstack/version-checker",latest_version="v0.2.1",namespace="version-checker",pod="version-checker-5677485967-b7czz"} 1

The targeted pod:

Name:                      echo-2-6dfbb54cfb-fxkrq
Namespace:                 default
Priority:                  0
Node:                      kind-control-plane/172.17.0.2
Start Time:                Tue, 10 Nov 2020 11:20:23 +0000
Labels:                    app=echo
                           pod-template-hash=6dfbb54cfb
Annotations:               enable.version-checker.io/echo-2: true
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  30s
IP:                        
IPs:
  IP:           
Controlled By:  ReplicaSet/echo-2-6dfbb54cfb
Containers:
  echo-2:
    Container ID:   containerd://45bc033cb1b6fef9a6d46633a7c99f5e22541e218a410e71f083541dc8a37804
    Image:          docker.io/ealen/echo-server:latest
    Image ID:       docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
    Port:           8081/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 10 Nov 2020 11:20:25 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>

Enable Registries with certificates signed by custom authorities.

We use an on premise registry (artifactory), with certificates signed by our private authority. When we try to run version-checker, we get certificate signed by unknown authority errors:

time="2021-01-18T12:48:29Z" level=error msg="error syncing 'version-checker-597b645b-mpw6b/kube-version-checker': failed to sync pod version-checker-597b645b-mpw6b/kube-version-checker: failed to check container image \"version-checker\": failed to get tags from remote registry for \"quay-docker-remote.repo.example.com/jetstack/version-checker\": failed to get docker image: Get \"https://quay-docker-remote.repo.example.com/v2/jetstack/version-checker/tags/list?n=500\": x509: certificate signed by unknown authority, requeuing" module=controller

It would be great to add a flag like --ca=path/to/truststore.pem which configures go's transport.

Error when setting override to docker.io/busybox

When setting annotation:
override-url.version-checker.io/fsgroup-volume: docker.io/busybox
I get the following error:

time="2023-10-17T16:42:30Z" level=error msg="error syncing 'opensearch-us5-data-0-25/opensearch': failed to sync pod opensearch-us5-data-0-25/opensearch: failed to check container image \"fsgroup-volume\": failed to get tags from remote registry for \"docker.io/busybox\": failed to parse image timestamp: parsing time \"\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"\" as \"2006\",failed to check container image \"sysctl\": failed to get tags from remote registry for \"docker.io/busybox\": failed to parse image timestamp: parsing time \"\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"\" as \"2006\", requeuing" module=controller

Fails on registry.opensource.zalan.do/teapot/external-dns

Enabled this for the external-dns helm-chart, but it fails with:

time="2020-10-01T12:13:14Z" level=error msg="error syncing 'external-dns-87d6cdc68-z9fdn/external-dns': failed to sync pod external-dns-87d6cdc68-z9fdn/external-dns: failed to check container image \"external-dns\": failed to get tags from remote registry for \"registry.opensource.zalan.do/teapot/external-dns\": failed to get docker image: Get \"https:///v2/teapot/external-dns/tags/list?n=500\": http: no Host in request URL, requeuing" module=controller

Allow for excluding specific containers

When enabling version-checker to check all applications there should be a way to exclude specific containers. This would be extremely useful for containers hosted in places that version-checker does not yet support.

Support Image URLs with both version tags and SHAs

We need to be able to support image tags which include both a version semvar, as well as a sha tag. E.g.

us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.0@sha256:56633bd00dab33d92ba14c6e709126a762d54a75a6e72437adefeaaca0abb069

/assign

ACR Tokens support for authentication

The feature is still in preview, but you can generate tokens for ACR that are (optionally) scoped to specific repositories and with limited permissions.

Having the ability to use these tokens for authentication would be a huge improvement.

Docs:
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions

Error on OCI style repositories

Seeing the following error when version-checker attempts to check the fluent-bit repository:

time="2023-10-16T21:36:44Z" level=error msg="cr.fluentbit.io/v2/fluent/fluent-bit/manifests/2.0: failed to get manifest response for tag, skipping (404): {\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"OCI index found, but accept header does not support OCI indexes\"}]}\n" client=

time="2023-10-16T21:40:19Z" level=error msg="cr.fluentbit.io/v2/fluent/fluent-bit/manifests/sha256-9b386368a776f8f8a25ee7113ef8f39ae9004d3b6e8a80d0848bd9e37a9c6aae.sig: failed to get manifest response for tag, skipping (404): {\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"OCI manifest found, but accept header does not support OCI manifests\"}]}\n" client=

Image of the container is cr.fluentbit.io/fluent/fluent-bit:2.1.9

does not seem to support kube2iam for ECR access

Hello,

I have a K8S cluster deployed in AWS with kubeadm.
Some of my images comes from the ECR of the K8S AWS account and I wanted to use kube2iam annotation on version-checker pod to allow it to check for image tags but it does not seem to work :

version-checker pod :

apiVersion: v1
kind: Pod
metadata:
  annotations:
    enable.version-checker.io/version-checker: "true"
    iam.amazonaws.com/role: ecr-read-profile
[...]

version-checker logs :

time="2020-12-07T14:47:39Z" level=error msg="error syncing 'checkoutservice-78b576896d-9pk6z/microdemo': failed to sync pod checkoutservice-78b576896d-9pk6z/microdemo: failed to check container image \"server\": failed to get tags from remote registry for \"<AWS_ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/checkoutservice\": failed to describe images: EmptyStaticCreds: static credentials are empty, requeuing" module=controller

Does the ECR authent only work with static credentials ?
Would it be possible to support kube2iam to avoid giving the pod static key and password ?
Thanks

Rename controller "imageCache"

We should rename the imageCache in pkg/controller. It's not really an image cache, it's a searchCache.

/assign

jetstack / version-checker Goto Github PK

version-checker's Introduction

version-checker

Registries

Installation

Grafana Dashboard

Options

Metrics

version-checker's People

Contributors

Stargazers

Watchers

Forkers

version-checker's Issues

Recommend Projects

Recommend Topics

Recommend Org