jetstack / version-checker Goto Github PK
View Code? Open in Web Editor NEWKubernetes utility for exposing image versions in use, compared to latest available upstream, as metrics.
Home Page: https://jetstack.io
License: Apache License 2.0
Kubernetes utility for exposing image versions in use, compared to latest available upstream, as metrics.
Home Page: https://jetstack.io
License: Apache License 2.0
Today version-checker keeps logging when it fails on some image.
Wouldn't it be nice having a metric for this?
Maybe something like version_checker_image_failed_count
?
(with labels similar to the version_checker_is_latest_version
metric — such as; container, image, exported_namespace, exported_pod)
Then you could be alerted when there's images that fail the version-check. That would prompt you to check the logs for further details.
I am running version-checker
on a single node, quite small cluster with ~60 pods. So far it is working nicely, but I do not understand the memory behavior it has.
I'm basically running the sample deployment file, plus the --test-all-containers
flag and some cpu limits:
resources:
requests:
cpu: 10m
memory: 32M
limits:
cpu: 50m
memory: 128M
Over time, I see that version-checker
approaches the memory limit and then stays near ~99% for a while. After some time, the kernel kills the ct due to OOM and k8s restarts the pod.
However, I do not see anything alarming in the logs, other than some failures and expected permission errors.
This doesn't seem to have any functional impact, but does fire some alerts and doesn't look good on my dashboards :)
Is this behavior intended, and/or is there any way to prevent it?
We should rename the imageCache
in pkg/controller
. It's not really an image cache, it's a searchCache
.
/assign
We should publish multi-arch images, similar to cert-manager.
/assign
curl -k -u docker_reg:0IFmcttx https://ol8-tst-k8s-master-node.com:5000/v2/int-xfr-dispatch/tags/list?n=5000
{"name":"int-xfr-dispatch","tags":["tst-20211102-J11-237-1183f99c","tst-20211117-J13-240-c3f64612"
version-checker
Args:
--test-all-containers
--selfhosted-username=docker_reg
--selfhosted-password=0IFmcttx
--insecure-skip-tls-verify=true
--log-level=debug
This is the Error that I am getting in the log file.
How are you calling the versions. What do I need to add to my config
I would seem that the x509 error could be from an incorrect certificate. Not sure which certificate could be incorrect as I do not understand how you are calling the Get https://
Hello,
We were working on a similar tool so nice to see this repository. But in order to contribute to this repository, the jetstack-bot send the PR authors to the CONTRIBUTING.md file, which does not exist (#2 (comment))
Display no data in the Grafana Dashboard after deployment according to the install content in the README.
Hello,
When I start version-checker, the log indicates that the annotation to enable the check is enable.version-checker/${my-container}
:
version-checker-6477448bf-4lvvn
version-checker time="2020-11-20T18:08:42Z"
level=info
msg="flag --test-all-containers=false only containers with the annotation \"enable.version-checker/${my-container}=true\" will be parsed"
But in the documentation and in the code the annotation prefix is enable.version-checker.io
The .io
is missing in the log message here and also in options message here, isn't it ?
I have some strange version checks with popular images and I think there is something missing in the checking algorithm.
For the image grafana/grafana
version-checker says the latest version is 9799770991, which is obviously correct if you compare just version numbers. But this tag is 8 months old and the actual latest tag currently would be 10.2.2. Another example would be quay.io/jetstack/cert-manager-webhook-arm64
: v1.13.2 vs. 608111629.
I am wondering if it would be better to always take the publish date into account when checking versions?
Maybe there is another trick how to avoid these false positives?
Hi there,
If could you add Pod Labels in the Helm Chart, and also an option to disable ClusterRole And ClusterRoleBinding, that would be great, as it a hard requirement for us to be able to use it, since we just need to access images running on namespace level only.
Thanks in advance!.
We need to be able to support image tags which include both a version semvar, as well as a sha tag. E.g.
us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.0@sha256:56633bd00dab33d92ba14c6e709126a762d54a75a6e72437adefeaaca0abb069
/assign
Hey! Thank you for making this terrific application. It's really useful.
I'm getting some errors in my use-case about unauthorized attempts to access the api.github.com
URL for checking image tags. It looks like this cannot be accessed unauthenticated in v4 of the GitHub API spec (apparently it could be in v3?).
I'm looking for an example of how to configure a GHCR account. I see the PR where support was implemented but not really sure how to enable it via the helm chart.
When attempting to check image tags against the public ECR repository I am met with the following error:
failed to check container image \"opensearch\": failed to get tags from remote registry for "public.ecr.aws/opensearchproject/opensearch\": {\"errors\":[{\"code\":\"DENIED\",\"message\":\"Not Authorized\"}]}\n, requeuing
I would like avoid storing my username/password in values.yaml and reuse an existing secret for my Artifactory server. An an example of a selfHosted configuration:
- name: something
url: something
# username
# password
# token
secretRef: something
My use case is that I use external-secrets for integration with AWS Secrets Manager and to avoid storing clear text passwords in GitHub.
Hi folks,
Now we use deprecated apiVersion ( "rbac.authorization.k8s.io/v1beta1" ) for Role and RoleBinding in Prometheus manifests
The current version of apiVersion is "rbac.authorization.k8s.io/v1".
Please fix it.
P.S.: I tried to create PR but your policy has dined my try.
We use an on premise registry (artifactory), with certificates signed by our private authority. When we try to run version-checker, we get certificate signed by unknown authority
errors:
time="2021-01-18T12:48:29Z" level=error msg="error syncing 'version-checker-597b645b-mpw6b/kube-version-checker': failed to sync pod version-checker-597b645b-mpw6b/kube-version-checker: failed to check container image \"version-checker\": failed to get tags from remote registry for \"quay-docker-remote.repo.example.com/jetstack/version-checker\": failed to get docker image: Get \"https://quay-docker-remote.repo.example.com/v2/jetstack/version-checker/tags/list?n=500\": x509: certificate signed by unknown authority, requeuing" module=controller
It would be great to add a flag like --ca=path/to/truststore.pem
which configures go's transport.
First of all, thank you for this project. It's awesome.
For our stack I'm eager to combine this with Artifactory. I would not mind investing some time to get started on such feature. Yet I'm not sure if there is already a plan for this or any roadmap?
Basically, this "issue" is my request on how to help :)
What happened?
When I was running version-checker on KIND (amd64), the latest image version reported for the following container docker.io/ealen/echo-server:latest
was 3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5
, for arm/v6, instead of the tag for amd64 or the tag used for the manifests collection sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
which is the actual ID assigned to the container when using :latest
.
Image: docker.io/ealen/echo-server:latest
Image ID: docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
Logs
Echo Image Manifests and digests:
docker buildx imagetools inspect docker.io/ealen/echo-server:latest
Name: docker.io/ealen/echo-server:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest: sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
Manifests:
Name: docker.io/ealen/echo-server:latest@sha256:397996a29090fde2eeb8866d972b683fd64afadbe7161d0e64c53350b4a83fd7
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/amd64
Name: docker.io/ealen/echo-server:latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/arm/v6
Name: docker.io/ealen/echo-server:latest@sha256:80e5ed2796aa8242b965c80678bc512ae2396c58f13614be204e7e70ce4529b6
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/arm/v7
Name: docker.io/ealen/echo-server:latest@sha256:1865bce901f1565d6262a730f0ae6f562a86e4cca04f65fea6cfe2c30896ec7f
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/arm64
Name: docker.io/ealen/echo-server:latest@sha256:1f183d751287c846a8d1c5c21e8c635b175141b23af81df067631d8cd9906787
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/386
vc metrics:
curl http://localhost:8083/metrics
# HELP version_checker_is_latest_version Where the container in use is using the latest upstream registry version
# TYPE version_checker_is_latest_version gauge
version_checker_is_latest_version{container="echo-2",current_version="sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034",image="docker.io/ealen/echo-server",latest_version="latest@sha256:3f390e3cb38a1bb7a5169b27ae56285335ffede3cf334156e4d826173ce54ae5",namespace="default",pod="echo-2-79fbc9bc76-dvggs"} 0
version_checker_is_latest_version{container="version-checker",current_version="v0.2.1",image="quay.io/jetstack/version-checker",latest_version="v0.2.1",namespace="version-checker",pod="version-checker-5677485967-b7czz"} 1
The targeted pod:
Name: echo-2-6dfbb54cfb-fxkrq
Namespace: default
Priority: 0
Node: kind-control-plane/172.17.0.2
Start Time: Tue, 10 Nov 2020 11:20:23 +0000
Labels: app=echo
pod-template-hash=6dfbb54cfb
Annotations: enable.version-checker.io/echo-2: true
Status: Terminating (lasts <invalid>)
Termination Grace Period: 30s
IP:
IPs:
IP:
Controlled By: ReplicaSet/echo-2-6dfbb54cfb
Containers:
echo-2:
Container ID: containerd://45bc033cb1b6fef9a6d46633a7c99f5e22541e218a410e71f083541dc8a37804
Image: docker.io/ealen/echo-server:latest
Image ID: docker.io/ealen/echo-server@sha256:3419f7cf35d10c9096ce359df762283ccfa7af12095f9be51542566bef8e9034
Port: 8081/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 10 Nov 2020 11:20:25 +0000
Ready: True
Restart Count: 0
Environment: <none>
Hey
I've been trying out this tool for a bit and found a few issues. Most of them have to do with the regex matching but this one I can't figure out.
For some reason there is one fluent-bit pod in the daemonset that gives a false value while being identical to the others. This behavior is replicated on another cluster (which is set up identical to this one).
I'm running:
version-checker:
Container ID: docker://581de433d72c6199d5194af1eb3c5fb8deff331acb932567ab92368afb4ae64c
Image: quay.io/jetstack/version-checker:v0.2.1
Image ID: docker-pullable://quay.io/jetstack/version-checker@sha256:5f6f8ba0b671db2382f811b58fad64a2fd22ca1caaf087f9708c8e4b8309d7e0
Metrics:
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-59hw7"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-7btfd"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-7fh2r"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-cpv5p"} 1
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-j5lh8"} 0
version_checker_is_latest_version{container="fluent-bit",current_version="1.6.8@sha256:49e3fbd3e3a76b8e7088d618dab637d35d711a656d4f2a5e72244d66e88bd3e7",image="docker.io/fluent/fluent-bit",latest_version="1.6.8@sha256:de6a3f63d0723430cc0bc042b15b27cab8a4f8aeb9219aa1c13445650b1a49e9",namespace="logging",pod="fluent-bit-rhrxf"} 1
The feature is still in preview, but you can generate tokens for ACR that are (optionally) scoped to specific repositories and with limited permissions.
Having the ability to use these tokens for authentication would be a huge improvement.
Hello,
I deployed version-checker with testAllContainers: true
and try to override the registry for one of my image using the annotation :
apiVersion: v1
kind: Pod
metadata:
annotations:
override-url.version-checker.io/frontend: gcr.io/google-samples/microservices-demo/frontend
[...]
spec:
containers:
image: XXXX.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/frontend:v0.2.1
[...]
but the url is not overriden :
$ stern version -n version-checker | grep frontend
version-checker-58f996cdc5-tfffj version-checker time="2020-11-20T18:30:47Z" level=error msg="error syncing 'frontend-5c88f5f675-wccfx/microdemo': failed to sync pod frontend-5c88f5f675-wccfx/microdemo: failed to check container image \"server\": failed to get tags from remote registry for \"XXXX.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/frontend\": failed to describe images: EmptyStaticCreds: static credentials are empty, requeuing" module=controller
I have set up selfhosted for Harbor, but it is showing an error.
Here are my customized values:
selfhosted:
- name: HARBOR
host: harbor.mycompany.com
username: <myUser>
password: <myPassowrd>
Below is the log with the error:
Error: failed to setup image registry clients: failed to create selfhosted client "harbor.mycompany.com": failed to setup token auth: failed to send basic auth request "harbor.mycompany.com/v2/token": Post "harbor.mycompany.com/v2/token": unsupported protocol scheme ""
Usage:
version-checker [flags]
App flags:
-c, --image-cache-timeout duration The time for an image version in the cache to be considered fresh. Images will be rechecked after this interval. (default 30m0s)
-v, --log-level string Log level (debug, info, warn, error, fatal, panic). (default "info")
-m, --metrics-serving-address string Address to serve metrics on at the /metrics path. (default "0.0.0.0:8080")
-a, --test-all-containers If enabled, all containers will be tested, unless they have the annotation "enable.version-checker/${my-container}=false".
Auth flags:
--acr-password string Password to authenticate with azure container registry (VERSION_CHECKER_ACR_PASSWORD).
--acr-refresh-token string Refresh token to authenticate with azure container registry. Cannot be used with username/password (VERSION_CHECKER_ACR_REFRESH_TOKEN).
--acr-username string Username to authenticate with azure container registry (VERSION_CHECKER_ACR_USERNAME).
--docker-password string Password to authenticate with docker registry (VERSION_CHECKER_DOCKER_PASSWORD).
--docker-token string Token to authenticate with docker registry. Cannot be used with username/password (VERSION_CHECKER_DOCKER_TOKEN).
--docker-username string Username to authenticate with docker registry (VERSION_CHECKER_DOCKER_USERNAME).
--ecr-access-key-id string ECR access key ID for read access to private registries (VERSION_CHECKER_ECR_ACCESS_KEY_ID).
--ecr-secret-access-key string ECR secret access key for read access to private registries (VERSION_CHECKER_ECR_SECRET_ACCESS_KEY).
--ecr-session-token string ECR session token for read access to private registries (VERSION_CHECKER_ECR_SESSION_TOKEN).
--gcr-token string Access token for read access to private GCR registries (VERSION_CHECKER_GCR_TOKEN).
--quay-token string Access token for read access to private Quay registries (VERSION_CHECKER_QUAY_TOKEN).
--selfhosted-password string Password is authenticate with a selfhosted registry (VERSION_CHECKER_PASSWORD).
--selfhosted-registry-host string Full host of the selfhosted registry. Include http[s] scheme (VERSION_CHECKER_HOST
--selfhosted-token string Token to authenticate to a selfhosted registry. Cannot be used with username/password (VERSION_CHECKER_TOKEN).
--selfhosted-username string Username is authenticate with a selfhosted registry (VERSION_CHECKER_USERNAME).
Kubernetes flags:
--as string Username to impersonate for the operation
--as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--cache-dir string Default cache directory (default "/root/.kube/cache")
--certificate-authority string Path to a cert file for the certificate authority
--client-certificate string Path to a client certificate file for TLS
--client-key string Path to a client key file for TLS
--cluster string The name of the kubeconfig cluster to use
--context string The name of the kubeconfig context to use
--insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
--kubeconfig string Path to the kubeconfig file to use for CLI requests.
-n, --namespace string If present, the namespace scope for this CLI request
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
-s, --server string The address and port of the Kubernetes API server
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
error: failed to setup image registry clients: failed to create selfhosted client "harbor.mycompany.com": failed to setup token auth: failed to send basic auth request "harbor.mycompany.com/v2/token": Post "harbor.mycompany.com/v2/token": unsupported protocol scheme ""
Hello,
I deployed version-checker with enabled self hosted registry (artifactory), but I got 405 error in the log (sensitive info replaced with stars):
error: failed to setup image registry clients: failed to create selfhosted client "https://*****": failed to setup token auth (405): { "errors" : [ { "status" : 405, "message" : "Method Not Allowed" } ] }
It seems that selfhosted client in the version-checker has wrong token url path for artifactory because in official documentation it is different.
I tested it with curl and got the same result:
$ curl -u*** -XPOST "https://***/v2/token" -d "username=***"
{ "errors" : [ { "status" : 405, "message" : "Method Not Allowed" } ] }
With correct url path it works:
$ curl -u*** -XPOST "https://***/artifactory/api/security/token" -d "username=***"
{ "scope" : "applied-permissions/groups:*** api:*", "access_token" : "***", "expires_in" : 3600, "token_type" : "Bearer" }
I have the following error with the pods of the filebeat daemonset :
time="2021-01-20T17:48:18Z" level=error msg="error syncing 'filebeat-filebeat-v58xm/log-management': failed to sync pod filebeat-filebeat-v58xm/log-management: failed to check container image \"filebeat\": failed to get tags from remote registry for \"docker.elastic.co/beats/filebeat\": {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"beats/filebeat\",\"Action\":\"pull\"}]}]}\n, requeuing" module=controller
However, I can pull the docker image without issue :
docker pull docker.elastic.co/beats/filebeat:7.10.2
7.10.2: Pulling from beats/filebeat
[...]
Here is the configuration of filebeat daemonset :
$ kubectl describe ds filebeat-filebeat
Name: filebeat-filebeat
Selector: app=filebeat-filebeat,release=filebeat
Labels: app=filebeat-filebeat
chart=filebeat-7.9.0
heritage=Helm
release=filebeat
Annotations: deprecated.daemonset.template.generation: 9
Pods Status: 13 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=filebeat-filebeat
chart=filebeat-7.9.0
heritage=Helm
release=filebeat
Annotations: configChecksum: 02443bbba38f244a1748e6769f4c91614189e4d5f598a687ec63b57389572df
enable.version-checker.io/filebeat: true
Service Account: filebeat-filebeat
Containers:
filebeat:
Image: docker.elastic.co/beats/filebeat:7.9.0
[...]
Hello,
I have a K8S cluster deployed in AWS with kubeadm.
Some of my images comes from the ECR of the K8S AWS account and I wanted to use kube2iam annotation on version-checker pod to allow it to check for image tags but it does not seem to work :
version-checker pod :
apiVersion: v1
kind: Pod
metadata:
annotations:
enable.version-checker.io/version-checker: "true"
iam.amazonaws.com/role: ecr-read-profile
[...]
version-checker logs :
time="2020-12-07T14:47:39Z" level=error msg="error syncing 'checkoutservice-78b576896d-9pk6z/microdemo': failed to sync pod checkoutservice-78b576896d-9pk6z/microdemo: failed to check container image \"server\": failed to get tags from remote registry for \"<AWS_ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/google-samples/microservices-demo/checkoutservice\": failed to describe images: EmptyStaticCreds: static credentials are empty, requeuing" module=controller
Does the ECR authent only work with static credentials ?
Would it be possible to support kube2iam to avoid giving the pod static key and password ?
Thanks
When setting annotation:
override-url.version-checker.io/fsgroup-volume: docker.io/busybox
I get the following error:
time="2023-10-17T16:42:30Z" level=error msg="error syncing 'opensearch-us5-data-0-25/opensearch': failed to sync pod opensearch-us5-data-0-25/opensearch: failed to check container image \"fsgroup-volume\": failed to get tags from remote registry for \"docker.io/busybox\": failed to parse image timestamp: parsing time \"\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"\" as \"2006\",failed to check container image \"sysctl\": failed to get tags from remote registry for \"docker.io/busybox\": failed to parse image timestamp: parsing time \"\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"\" as \"2006\", requeuing" module=controller
I believe a bug was introduced in #115 when the selector fields were upgraded to use {{- include "version-checker.labels" . | nindent 6 }}
. This is because version-checker.labels
includes things like app.kubernetes.io/version
which changes on newer versions. The selector field is immutable in Deployments and shouldn't change between the same release.
To replicate:
$ helm upgrade -i version-checker jetstack/version-checker --version v0.3.1
Release "version-checker" does not exist. Installing it now.
NAME: version-checker
LAST DEPLOYED: Mon Dec 4 15:49:49 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
$ helm upgrade -i version-checker jetstack/version-checker --version v0.3.2
Error: UPGRADE FAILED: cannot patch "version-checker" with kind Deployment: Deployment.apps "version-checker" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"version-checker", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"version-checker", "app.kubernetes.io/version":"v0.3.2", "helm.sh/chart":"version-checker-v0.3.2"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Hello,
It appears that this tool doesn't support custom Gitlab repositories. It returns a 404.
Is this the case or I am doing something wrong?
Thank you in advance.
Enabled this for the external-dns
helm-chart, but it fails with:
time="2020-10-01T12:13:14Z" level=error msg="error syncing 'external-dns-87d6cdc68-z9fdn/external-dns': failed to sync pod external-dns-87d6cdc68-z9fdn/external-dns: failed to check container image \"external-dns\": failed to get tags from remote registry for \"registry.opensource.zalan.do/teapot/external-dns\": failed to get docker image: Get \"https:///v2/teapot/external-dns/tags/list?n=500\": http: no Host in request URL, requeuing" module=controller
Hi,
Currently it's not easy to use this chart with the helm-operator.
With the helm-operator you can either specify a Helm repo/hub or a git repo.
The git repo one is only safe to use with tags/branches as I don't want it linked with the master 😄.
I would like to contribute to the chart but this means the chart version/release won't be in sync with the app version. Which will result that the git repo tag approach is not useable anymore.
Adding the chart to https://charts.jetstack.io would solve this issue or use the chart-releaser? This will also result in an easier distribution of the chart 😃
Currently, when supplying a username/password, we use the returned JWT forever.
We should add support for renewing the token, as a function of the expiry claim.
RenewAt(time.Now().Add(expiry - (duration / 3))
When enabling version-checker to check all applications there should be a way to exclude specific containers. This would be extremely useful for containers hosted in places that version-checker does not yet support.
Seeing the following error when version-checker attempts to check the fluent-bit repository:
time="2023-10-16T21:36:44Z" level=error msg="cr.fluentbit.io/v2/fluent/fluent-bit/manifests/2.0: failed to get manifest response for tag, skipping (404): {\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"OCI index found, but accept header does not support OCI indexes\"}]}\n" client=
time="2023-10-16T21:40:19Z" level=error msg="cr.fluentbit.io/v2/fluent/fluent-bit/manifests/sha256-9b386368a776f8f8a25ee7113ef8f39ae9004d3b6e8a80d0848bd9e37a9c6aae.sig: failed to get manifest response for tag, skipping (404): {\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"OCI manifest found, but accept header does not support OCI manifests\"}]}\n" client=
Image of the container is cr.fluentbit.io/fluent/fluent-bit:2.1.9
Hi,
We have k8s running on Azure and the following (images) versions:
Prometheus: quay.io/prometheus/prometheus:v2.21.0
Version-checker: quay.io/jetstack/version-checker:v0.2.0
Grafana: grafana/grafana:7.2.0-ubuntu
Our config for the deployment of version-checker is:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: version-checker
name: version-checker
spec:
replicas: 1
selector:
matchLabels:
app: version-checker
template:
metadata:
labels:
app: version-checker
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
enable.version-checker.io/version-checker: "true"
spec:
serviceAccountName: version-checker
containers:
- image: quay.io/jetstack/version-checker:v0.2.0
imagePullPolicy: Always
ports:
- containerPort: 8080
name: version-checker
command: ["version-checker"]
args:
- "--log-level=debug"
- "--test-all-containers=true"
env:
- name: VERSION_CHECKER_ACR_USERNAME
valueFrom:
secretKeyRef:
name: version-checker-pdokcr
key: ACR_USERNAME
- name: VERSION_CHECKER_ACR_PASSWORD
valueFrom:
secretKeyRef:
name: version-checker-pdokcr
key: ACR_PASSWORD
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 200m
memory: 256Mi
On a side note, above config might look a bit odd, we use Kustomize to deploy our environments.
As per command-args above we want to scrape all pods, without exception. There are however several pods not being shown inside the table. Some of these are related to minio
. We dont have minio running on our own ACR and use the "public" available images for that.
When I look at the logs of the version-checker, it has this to say about it:
time="2020-10-07T07:43:28Z" level=error msg="minio/mc: no tags found with these option constraints: {}" container=download-cleanup module=controller name=bgt-download-cleanup-1601942400-k7wlx namespace=download
time="2020-10-07T07:43:56Z" level=error msg="minio/minio: no tags found with these option constraints: {}" container=minio module=controller name=minio-695644c4b5-9dlq9 namespace=services
time="2020-10-07T07:43:56Z" level=error msg="pdok/mapserver: no tags found with these option constraints: {}" container=mapserver module=controller name=bzk-brogmwkenset-wms-v2-1-mapserver-5448896fd7-54bsh namespace=services
I have no idea what tags it is trying to figure out, as I have the command-arg set to "--test-all-containers=true"
. Shouldn't it pickup these pods this way or am I missing out on something?
Our pods have these images currently active and I kind of expect them to be shown in the table:
minio-etch: gcr.io/etcd-development/etcd:v3.3.10
minio-server: minio/minio:RELEASE.2020-10-03T02-19-42Z
Hi,
It would be great to have a prometheus metric with the number of errors, it would be helpful to detect possible issues with some images.
What do you think?
Thanks
Is this project still under maintenance?
I really like the project, however:
The image quay.io/jetstack/version-checker:v0.2.1 is now over 2 years old and has some security issues: https://trivy.dev/results/?image=quay.io/jetstack/version-checker:v0.2.1
I built a new image with alpine:3.17.1. Unfortunately, Trivy still finds some security issues regarding gobinary:
Would it be possible for someone to build an up-to-date container image without any security issues?
Hey
For a certain application that regularly spins up in our CICD flow, I've noticed that some instances keep hanging around in the metrics long after the pod gets terminated. In the sample metrics below you'll see that there are different values for latest_version indicating old data. Restarting version_checker does clean these up.
I'm running:
version-checker:
Container ID: docker://581de433d72c6199d5194af1eb3c5fb8deff331acb932567ab92368afb4ae64c
Image: quay.io/jetstack/version-checker:v0.2.1
Image ID: docker-pullable://quay.io/jetstack/version-checker@sha256:5f6f8ba0b671db2382f811b58fad64a2fd22ca1caaf087f9708c8e4b8309d7e0
Metrics:
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.2",namespace="jenkins",pod="release-bump-9371-hbjm2-z1g91-6sd8x"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.3",namespace="jenkins",pod="release-bump-9379-z9lfh-xp6gf-j22qn"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9442-gc98d-9rc8c-hzjld"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9450-9jc31-fr5f5-kzxch"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9451-rp62c-kt9fj-4msmb"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9483-5q6pn-c9951-pvk7l"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.5",namespace="jenkins",pod="release-bump-9493-bsnzm-2f3q9-0nczp"} 0
Currently, we start the garbage collector for the version package pkg/version
, as part of a call to New
.
version-checker/pkg/version/version.go
Line 38 in 6266beb
This isn't great for having this package be importable, and should move the start of the garbage collection to another func. The controller can start it as a go routine.
/assign
When a container image uses the latest
tag, version-checker
should fallback to comparing SHA “silently” (the log level should be relaxed).
At the moment, logs are filled with:
time="2020-08-18T18:38:35Z" level=warning msg="image using \"latest\" tag, comparing image SHA \"sha256:530e471dba7ee6995a230b43a1c8db4404873fb0402469397c8674dcf920d59f\"" container=foo
module=controller name=foobar-0 namespace=foobar
I believe this message should be emitted with log level debug
.
So in my cluster with a mix of arch's it doesn't try and run on a Windows node. :)
If the argument test-all-containers
is false (default) , chances are that it won't process a thing. Therefore place a clear info message in the logs that it's running on false. That gives a better indication on what is going on.
I’m not sure why but version-checker
seems to fail on the nginx-ingress controller image:
time="2020-08-18T18:38:33Z" level=error msg="error syncing 'nginx-ingress-9bbd564fc-8ktdv/nginx-ingress': failed to sync pod nginx-ingress-9bbd564fc-8ktdv/nginx-ingress: failed to test container image
\"nginx-ingress-controller\": \"us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller\": no tags found for given image URL: \"us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller\", requeuing"
module=controller
us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller
is the new official image according to ingress-nginx release notes for version 0.34.0.
I believe the issue might be that some images have been published without a tag in that registry.
From looking at the deployment manifest, it currently doesn't seem to have any probes for readiness or liveness check.
It would be nice to get that fixed.
We do not currently properly support paging in clients. This means that tag sets will only include tags which are returned in the first page from the registry. This may lead to incorrect results when reporting the latest image when options have been applied.
We should update our clients to implement proper HTTP paging, to ensure we get the fill list of tag results.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.