What we did:
We introduced LifeCyclePolicy into ECR to avoid having cached really old images.
We set this to keep 3 latest tags, which in effect has removed a lot of tags including old ones and currently used.
After cleanup k8s-image-swapper still recognized image as existing in ECR and mutate pod to start with ECR cached image, which ends up with ImagePullBackOff

What is the issue:
Seems that there is some cache for skopeo, which see image even if it not exists.
After deleting/recreating image-swapper pod situation get backs to normal.

Steps to reproduce:

1. Start deployment with nginx:1:14.2
2. Wait until k8s-image-swapper will cache image
3. Restart nginx deployment - it will be started with cached image
4. Remove image tag from ECR
5. Restart nginx deployment - it will fall into ImagePullBackOff

Logs:

2023-12-14T12:07:18+01:00 11:07AM DBG github.com/estahn/[email protected]/pkg/webhook/image_swapper.go:285 > jmespath search results filter="obj.metadata.namespace == 'kube-system'" results=false
2023-12-14T12:07:18+01:00 11:07AM TRC github.com/estahn/[email protected]/pkg/registry/ecr.go:239 > found in cache kind="/v1, Kind=Pod" name= namespace=tcn-personal-1 ref=000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 uid=fe091146-17ba-42af-863f-f937b757365d
2023-12-14T12:07:18+01:00 11:07AM DBG github.com/estahn/[email protected]/pkg/webhook/image_swapper.go:251 > set new container image image=000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 kind="/v1, Kind=Pod" name= namespace=tcn-personal-1 uid=fe091146-17ba-42af-863f-f937b757365d
2023-12-14T12:07:18+01:00 11:07AM TRC github.com/estahn/[email protected]/pkg/registry/ecr.go:239 > found in cache kind="/v1, Kind=Pod" name= namespace=tcn-personal-1 ref=000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 source-image=docker.io/library/nginx:1.14.2 target-image=000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 uid=fe091146-17ba-42af-863f-f937b757365d
2023-12-14T12:07:18+01:00 11:07AM TRC github.com/estahn/[email protected]/pkg/webhook/image_copier.go:71 > image copy aborted: image already present in target registry kind="/v1, Kind=Pod" name= namespace=tcn-personal-1 source-image=docker.io/library/nginx:1.14.2 target-image=000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 uid=fe091146-17ba-42af-863f-f937b757365d

Additional info:
Prove that image-tag is missing

aws ecr list-images --repository-name docker.io/library/nginx --filter '{ "tagStatus": "TAGGED" }'
    "imageIds": [
        {
            "imageDigest": "sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
            "imageTag": "1.21.3"
        },
        {
            "imageDigest": "sha256:08bc36ad52474e528cc1ea3426b5e3f4bad8a130318e3140d6cfe29c8892c7ef",
            "imageTag": "latest"
        }
    ]
}

also running skopeo inspect from image-swapper pod:

skopeo inspect --retry-times 3 docker://000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2 --creds $TOKEN
FATA[0000] Error parsing image name "docker://000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx:1.14.2": reading manifest 1.14.2 in 000000000000.dkr.ecr.eu-west-1.amazonaws.com/docker.io/library/nginx: manifest unknown: Requested image not found 

W/A:
Restart image-swapper Deployment

Document:

• Getting started
• Usage
• Patterns for filters
• Production considerations
• Caveats
• Kustomize
• Contribution

Looks like v1.3.1, v1.3.2, and v1.3.3 don't have packages available. Could we have those (or at least for the latest release) generated? Reason is that v1.3.3 includes an update to the alpine base image that patches some vulnerabilities.

Hi, trying to setup a basic filter which is very similar to the one in the docs.

Setting:
jmsepath: "contains(container.image, 'some_repo_prefix')"

Receiving:
ERR Filter (idx 0) could not be evaluated. error="SyntaxError: Incomplete expression" filter=

On the first run, it sometimes shows up with this. This should cause panic and restart of the process so it can pick up the certificate. I assume a race condition with the helm hook. Need to investigate.

2021/01/04 03:06:21 http: TLS handshake error from 100.96.2.0:30084: remote error: tls: bad certificate
2021/01/04 03:06:25 http: TLS handshake error from 100.96.2.0:9203: remote error: tls: bad certificate
2021/01/04 03:06:27 http: TLS handshake error from 100.96.2.0:13730: remote error: tls: bad certificate
2021/01/04 03:06:29 http: TLS handshake error from 100.96.2.0:9523: remote error: tls: bad certificate
2021/01/04 03:06:29 http: TLS handshake error from 100.96.2.0:32653: remote error: tls: bad certificate
2021/01/04 03:06:30 http: TLS handshake error from 100.96.2.0:33301: remote error: tls: bad certificate
2021/01/04 03:06:30 http: TLS handshake error from 100.96.2.0:6242: remote error: tls: bad certificate

This issue provides visibility into Renovate updates and their statuses. Learn more

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency alpine to v3.15
• chore(deps): update docker/login-action action to v1.14.1
• chore(deps): update goreleaser/goreleaser-action action to v2.9.1
• fix(deps): update kubernetes packages to v0.23.5 (k8s.io/api, k8s.io/apimachinery, k8s.io/client-go)
• fix(deps): update module github.com/alitto/pond to v1.7.1
• fix(deps): update module github.com/aws/aws-sdk-go to v1.43.31
• fix(deps): update module github.com/go-co-op/gocron to v1.13.0
• fix(deps): update module github.com/gruntwork-io/terratest to v0.40.6
• fix(deps): update module github.com/prometheus/client_golang to v1.12.1
• fix(deps): update module github.com/rs/zerolog to v1.26.1
• fix(deps): update module github.com/slok/kubewebhook/v2 to v2.3.0
• fix(deps): update module github.com/spf13/cobra to v1.4.0
• fix(deps): update module github.com/spf13/viper to v1.10.1
• chore(deps): update actions/cache action to v3
• chore(deps): update actions/checkout action
• chore(deps): update actions/setup-go action to v3
• chore(deps): update actions/setup-node action to v3
• chore(deps): update actions/setup-python action
• chore(deps): update golangci/golangci-lint-action action to v3
• fix(deps): update module k8s.io/client-go to v1

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/stretchr/testify to v1.7.1
• chore(deps): update actions/checkout action to v2.4.0
• chore(deps): update actions/setup-go action to v2.2.0
• chore(deps): update actions/setup-node action to v2.5.1
• chore(deps): update actions/setup-python action to v2.3.2
• fix(deps): update module github.com/containers/image/v5 to v5.20.0
• Click on this checkbox to rebase all open PRs at once

• Check this box to trigger a request for Renovate to run again on this repository

think about the order of webhooks?

You would want this webhook to be considered last, so then other webhooks like the istio side car/init pod injector's images can also be managed by image-swapper.

I'm seeing this message:
ERR error fetching referenced service account, continue without service account imagePullSecrets error="serviceaccounts \"myapp-services\" is forbidden: User \"system:serviceaccount:k8s-image-swapper:k8s-image-swapper\" cannot get resource \"serviceaccounts\" in API group \"\" in the namespace \"myns\""
at every access.

To give needed permissions to k8s-image-swapper, I'm using an EC2 Role.

• Number of Admission requests
• Number of Image downloads
• Number of Image swaps (aka mutation)

https://github.com/estahn/k8s-image-swapper/blob/alpha/pkg/webhook/image_swapper.go#L81-L85

So you can only use Sydney ECR right now,

Hi we're using AWS ECR and not using image pull secrets, just IAM restrictions with IRSA. We have chart 1.0.1 with appVersion 1.1.0 installed and it is frequently dumping panics in the following form. Because we have the secretReader option disabled, its not creating a ClusterRole or ClusterRoleBinding so its not surprising it gets permission denied. It is however surprising that it is trying to access the 'default' serviceAccount in a different namespace when its not configured to do so.

7:35PM ERR error fetching referenced service account, continue without service account imagePullSecrets error="serviceaccounts "default" is forbidden: User "system:serviceaccount:image-swapper:image-swapper" cannot get resource "serviceaccounts" in API group "" in the namespace "myappnamespace""
Worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
Stack trace: goroutine 198 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:24 +0x65
github.com/alitto/pond.defaultPanicHandler({0x1946000, 0x2c8fa20})
github.com/alitto/[email protected]/pond.go:19 +0x27
github.com/alitto/pond.(*WorkerPool).executeTask.func1()
github.com/alitto/[email protected]/pond.go:364 +0x45
panic({0x1946000, 0x2c8fa20})
runtime/panic.go:1038 +0x215
os.(*File).Name(...)
os/file.go:57
github.com/estahn/k8s-image-swapper/pkg/webhook.(*ImageSwapper).Mutate.func1()
github.com/estahn/[email protected]/pkg/webhook/image_swapper.go:219 +0x317
github.com/alitto/pond.(*WorkerPool).executeTask(0xc00073d3b0, 0x1ebce10)
github.com/alitto/[email protected]/pond.go:371 +0x69
github.com/alitto/pond.worker(0xc000199200, 0x1ea74e8, 0xc00073d3ec, 0x0, 0xc0007e7d80)
github.com/alitto/[email protected]/pond.go:427 +0x79
created by github.com/alitto/pond.(*WorkerPool).maybeStartWorker

Problem

nfo msg=\"Warning: failed, retrying in 2s ... (2/3)\"\ntime=\"2021-01-28T02:55:01Z\" level=info msg=\"Warning: failed, retrying in 4s ... (3/3)\"\ntime=\"2021-01-28T02:55:40Z\" level=fatal msg=\"Error initializing source docker://prom/pushgateway:v0.6.0: Error reading manifest v0.6.0 in docker.io/prom/pushgateway: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\n"

Solutions

• Exponential backoff upon failure for each registry
• Allow authentication via config

The filters currently operate on pod-level which doesn't allow to filter based on image name, e.g. filter all images containing "nginx".

When an image pull secret is created like so :

kubectl create secret docker-registry regcred --docker-server="*.registry.com" --docker-username=MY_USERNAME --docker-password="MY_PASSWORD" --docker-email="my_email"

The k8s-image-swapper is not able to copy the image from the registry to ECR.

Kubernetes is able to pull the image successful using a secret like that but k8s-image-swapper doesn't like it.

Any idea how to resolve this?

Thank you

We're using k8s-image-swapper in conjunction with GitLab CI. GitLab provides a dependency proxy service, basically a Docker pull-through cache for images used during builds. The registry URL starts with gitlab.com:443, which cannot be translated to an AWS ECR repository name (doesn't allow colons).

Is this handled in the code (I don't seem to be able to find a registry normalization function) and if so what is a registry URL that includes a port translated into?

We currently rely on the registry being public. If imagePullSecrets is specified, this should be used for authentication.

• Image pull secrets from Pod ( see docs)
• Image pull secrets from ServiceAccount (see docs)
• Use informer to monitor secret changes

see https://goharbor.io/

Setup integration tests in CI:

• kind
• helm install
• run tests

I am trying to enable KMS encryption and have set the following in the config, however repositories are still being created with AES-256. :

target:
  aws:
    accountId: "123456"
    ecrOptions:
      encryptionConfiguration:
        encryptionType: "KMS"
        kmsKey: "arn:aws:kms:us-west-2:123456"
    region: us-west-2

No errors are logged. Please let me know if anyone has successfully set kms encryption. Any help is appreciated.
Also, I noticed there is no reference to encryptionConfiguration in the ecr.go.80:

client := &ECRClient{
		client:          ecrClient,
		ecrDomain:       ecrDomain,
		cache:           cache,
		scheduler:       scheduler,
		targetAccount:   clientConfig.AccountID,
		accessPolicy:    clientConfig.ECROptions.AccessPolicy,
		lifecyclePolicy: clientConfig.ECROptions.LifecyclePolicy,
		tags:            clientConfig.ECROptions.Tags,
	}

Hello! I am getting a parse error when trying to use the following policy:

    ecrOptions:
      accessPolicy: |
        {
          "Statement": [
            {
              "Sid": "AllowCrossAccountPull",
              "Effect": "Allow",
              "Principal": {
                "AWS": "*"
              },
              "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
              ],
              "Condition": {
                "StringEquals": {
                  "aws:PrincipalOrgID": "o-XXXX"
                }
              }
            }
          ],
          "Version": "2008-10-17"
        }

Logs:

k8s-image-swapper-576dbbb84b-4zrkd k8s-image-swapper 11:09AM ERR parsing image swap policy failed error="unknown image swap policy string: '', defaulting to exists" policy=
k8s-image-swapper-576dbbb84b-4zrkd k8s-image-swapper 11:09AM ERR parsing image copy policy failed error="unknown image copy policy string: '', defaulting to delayed" policy=

Please note that i am upgrading from 1.1.0 helm chart version to 1.8.0 and in the previous version everything was working fine.

Thanks

Hi,

We encountered errors when running Pods with multiple initContainers with different imagePullPolicy values, where the value of the latest would overwrite the preceding ones. This then causes possible erroneous check conditions in the Mutate function here :

copyFn := func() {
    if p.registryClient.ImageExists(targetImage) && container.ImagePullPolicy != corev1.PullAlways {
        return
    }
   ...

The variable container that is used inside copyFn is overwritten at every iteration of the loop on containers/initContainers and when the goroutine (that runs copyFn) is eventually run, the value has (very likely) already changed because the following iterations have been computed first. The value in container.ImagePullPolicy then does not correspond to the image in targetImage, but rather to the image of the following/last container.

Erroneous checks in the condition may, for example, lead to repeatedly try to copy images that are already present on the target because the last container has a pullPolicy or Always.

This issue only applies to delayed and immediate modes where the image copy is done in background.

Since 1.5.0 release k8s-image-swapper supposedly can handle private registry authentication, but I'm unable to make it work following documentation.

1. I'm using helm chart to install k8s-image-swapper with image.tag=1.5.0.
2. I have User in AWS with AmazonEC2ContainerRegistryFullAccess role attached.
3. aws_access_key_id and aws_secret_access_key are stored as a secret.
4. Config:

    dryRun: true
    logFormat: console
    logLevel: debug
    ImageSwapPolicy: exists
    imageCopyPolicy: delayed
    source:
      registries:
        - type: aws
          aws:
            accountId: XXX
            region: XXX
      filters:
      - jmespath: obj.metadata.namespace == 'kube-system'
    target:
      aws:
        accountId: XXX
        region: XXX

When I deployed hello-world application it seem to create an repo in a registry and successfully swapping image in pod, but image pulling is faling:

  Normal   Scheduled  21s                default-scheduler  Successfully assigned helloworld/hello-kubernetes-7c9b9588d4-5fq5p to node-01
  Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image "XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest"
  Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    8s (x2 over 21s)   kubelet            Pulling image "XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest"
  Warning  Failed     8s (x2 over 20s)   kubelet            Failed to pull image "XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest": rpc error: code = Unknown desc = failed to pull and unpack image "XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest": failed to resolve reference "XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest": pulling from host XXX.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized
  Warning  Failed     8s (x2 over 20s)   kubelet            Error: ErrImagePull

Debug log from swapper:

...
3:31AM DBG create repository kind="/v1, Kind=Pod" name= namespace=helloworld repository=docker.io/myoung34/armhf-hello-kubernetes source-image=docker.io/myoung34/armhf-hello-kubernetes:latest target-image=XXXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest uid=3fa07827-f775-479a-96d4-faf29a37d57b
3:31AM DBG set new container image image=XXX.dkr.ecr.us-west-2.amazonaws.com/docker.io/myoung34/armhf-hello-kubernetes:latest kind="/v1, Kind=Pod" name= namespace=helloworld uid=3fa07827-f775-479a-96d4-faf29a37d57b
...

The ECR auth token procurement was refactored to not fetch a token for every request. Now the token expires at a certain point.

Add a ticker to renew the ECR auth token after a certain period.

https://docs.aws.amazon.com/cli/latest/reference/ecr/get-authorization-token.html

Image State Keeper keeps the ideal representation of the image state in the registry, e.g. exists/not exists.

This is to minimise requests to registries checking if the image already exists, etc.

Some logging is too verbose and should go to tracing.

https://jfrog.com/artifactory/

I want to be able to specify 2 regions for my ECR, in case one of them is down.
Always copy to both, but use the first priority one for the actual replacement if it's up.

We started getting errors in our swaps w/ k8s-image-swapper around ecr:CreateRepository (seeing this in cloudtrail), because of the missing permission ecr:TagResource.

Adding the ecr:TagResource action to our IAM policy for IRSA did fix the problem, so I think the docs just need updated.

they should be in 'ecrOptions:' under 'aws:'

I've noticed that this project is using skopeo to copy images, that's very cool!
Would be great if this project supported validating and copying sigstore signatures as well.

Skopeo utilizes this config format to validate images: https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md
It can be instructed to look at signatures using this config format: https://github.com/containers/image/blob/main/docs/containers-registries.d.5.md#individual-configuration-sections

Ideally one should be able to pass the necessary configuration files to skopeo by allowing for custom skopeo args to be configured.

k8s-image-swapper is currently not handling initContainers.

k8s-image-swapper is currently using skopeo, a CLI, to copy images into the registry. This is being executed as an external command and can lead to several issues (OOM, security exposure via skopeo, ...).

skopeo is utilising github.com/containers/image/v5/copy, with some added mechanics like retry, which can be utilised directly in k8s-image-swapper.

Hi,
after many months of good work, k8s-image-swapper is no more working when source image is on docker.io:
unexpected status code [manifests 41]: 401 Unauthorized
Did we run Docker Hub limits? In case, we have a valid account but docs are not clear about setup.

Image Downloader manages the image downlod, e.g.

• transfer image from source to target registry
• dedupe download requests
• throttle number of concurrent downloads
• return early if already downloaded

The current implementation only checks if it's available via in-memory cache. If it is not then a download will be initiated – which effectively only syncs the latest layers. But ideally, no download should be initiated as this seems to count towards the rate-limiting of some providers.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>whitesource/merge-confidence:beta)

Hi,
Receiving the titled error when using image that includes tags and digest like k8s.gcr.io/ingress-nginx/controller:v0.43.0@sha256:9bba603b99bf25f6d117cf1235b6598c16033ad027b143c90fa5b3cc583c5713

Where this limitation is coming from? can't ECR support digest?
Can we somehow overcome that so the tool will support that?

Remove the unnecessary nesting in the configuration file:

    target:
      registry:
        aws:
          accountId: 123456789
          region: ap-southeast-2

to

    target:
      aws:
        accountId: 123456789
        region: ap-southeast-2

A helm chart is available in deploy/k8s-image-swapper but should be properly manged/versioned/published.

• Create repo charts
• Relocate chart
• Publish via https://artifacthub.io/ (or Github Pages?)

I have deployed k8s-image-swapper into my cluster and it looks like the webhook is being ignored. No log in webhook server regarding pod creation could be found.

Some details:
Platform: EKS, v1.24
swapper helm chart version: 1.8.0
swapper image version: 1.5.7

Values:

image:
  tag: "1.5.7"
awsSecretName: k8s-image-swapper-aws
config:
  ImageSwapPolicy: "always"
  ImageCopyPolicy: "immediate"
  source:
    filters:
      - jmespath: "contains(container.image, '.dkr.ecr.') && contains(container.image, '.amazonaws.com')"
      - jmespath: "obj.metadata.namespace == 'kube-system'"
  target:
    type: aws
    aws:
      accountId: "my-account-id-number"
      region: eu-west-1
      ecrOptions:
        tags:
          - key: cluster
            value: my-cluster

testing:

# executed:
kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml

# result
pod/nginx created

logs on webhook server:

{"level":"info","file":"/.k8s-image-swapper.yaml","time":"2023-11-20T18:53:24Z","message":"using config file"}
6:53PM DBG auth token set, schedule next token renewal expiryAt=2023-11-21T06:53:24Z renewalAt=2023-11-21T06:51:24Z
6:53PM INF Listening on :8443

All the image pull secret we use in our cluster are of type .dockercfg so when using k8s-image-swapper, it's not using the image pull secret.

Is there anyway to also use .dockercfg image pull secret type?

Thank you

$ helm install k8s-image-swapper estahn/k8s-image-swapper  --set config.target.aws.region=eu-south-1 --set config.target.aws.accountId=215492924011
Error: values don't meet the specifications of the schema(s) in the following chart(s):
k8s-image-swapper:
- config.target.aws.accountId: Invalid type. Expected: string, given: integer

I'm force to install without specifying account-id and then modify ConfigMap manually.

We might want to not swap some images, please allow to blacklist these.

I have run into this: kubernetes/kubernetes#79365 (comment)

We need to ensure the patch jobs have different labels/label values than the deployment so HPA does not consider the patch jobs in its calculations creating errors.

This issue tracks interest, discussion & progress in regards Azure Container Registry.

🤝 If anyone is keen on sponsoring this to be implemented please contact me at [email protected]

We are testing the image swapper and have deployed it to our eks cluster, and the dryRun flag does not seem to be working (the actions are still taken; repo is created, image is uploaded, image ref is updated).

When we set the helm chart with

config:
  dryRun: true
  logLevel: debug
  logFormat: console

and/or set --dry-run=true as arg in deployment spec the actions are still executed.

currently images are pushed to ecr in /docker.io/ syntax in some cases where multiple eks clusters are in the same account in the same region but may not share authorized images a parent repository path is appropriate add a prefix to the aws target config and prepend the value to the target path to ensure separation as needed