The gRPC liveness check is in alpha in Kubernetes v1.23 and behind a feature gate. Once it is not behind a gate it needs to be supported here in order for the checks to continue to be a superset of the Kubernetes probes.

It could also be implemented sooner and placed in a similar alpha state.

Description

canary validate fails if the container takes more than 10 seconds to start up.

That 10 second threshold is currently hard-coded.

That timeout should be configurable.

Benefits of this work

Would allow the use of canary validate with images that take longer than 10 seconds to start up, for example:

• containers that do some heavy work on startup, like starting a process and polling until it passes health checks)
• docker daemon that is running slowly, e.g. because it's performing I/O on a network filesystem or is in some other way resource-constrainted

Acceptance Criteria

• it's possible to modify, via command-line argument(s), how long canary validate waits for a container to start up before timing out

Approach

Modify this

such that that 10 second timeout can be altered via configuration.

For example, I'd like to be able to run the following:

canary validate \
   --file ./checks.yaml \
   --startup-timeout 30 \
   ${IMAGE_URI}

Notes

Created this after observing this exact timeout while testing RAPIDS images over in rapidsai/docker#670.

Create an example for Vertex AI Workbench.

https://cloud.google.com/vertex-ai/docs/training/create-custom-container

If the container image doesn't exist canary just exits without a helpful error message

If docker is not installed canary just exits badly with no useful output. We should check and give a useful error.

Add an example for databricks.

https://docs.databricks.com/clusters/custom-containers.html#option-2-build-your-own-docker-base

Description

container-canary should support setting the command: for running an image without modifying the file containing validation checks.

Benefits of this work

Would separate details of how the validation checks are run from what is being validated, making it easier to share validation manifests across many different images. See "Motivation" below for details.

Acceptance Criteria

• it is possible to set / override the command: used when container-canary starts up a container without modifying the file that check configurations are stored in

Approach

This might be accomplished by adding new command-line arguments to container-canary, like --cmd to set the command.

canary validate \
    --file tests.yaml \
    --cmd "sh -c 'sleep 9999'" \
    ${IMAGE_URI}

The more Kubernetes-thonic (is that a word?) but also more difficult to implement approach would be to allow providing multiple files and merging them all together.

For example, something like this:

canary validate \
    --file https://raw.githubusercontent.com/NVIDIA/container-canary/main/examples/kubeflow.yaml \
    --file ./override-command.yaml \
    ubuntu:22.04

Where override-command.yaml just contains something like this:

command:
  - /bin/sh
  - -c
  - "sleep 1234"

And where the implication is that later files are patched on top of earlier files.

Following the way that some of these other tools work:

• helm values files: https://helm.sh/docs/chart_template_guide/values_files/
• Kustomize overlays: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#bases-and-overlays

Notes

Motivation

container-canary starts up a container based on the image passed to it, then run checks inside that running container.

If that image's default CMD doesn't result in starting a process (e.g. like a webserver), then container-canary will fail to run on it.

Consider the following:

canary validate \
    --file https://raw.githubusercontent.com/NVIDIA/container-canary/main/examples/kubeflow.yaml \
    ubuntu:22.04

That kubeflow.yaml file doesn't have a command: entry, so the container will run whatever the default CMD is on the image.

In this case, it's just a shell.

docker inspect ubuntu:22.04 \
| jq '.[0].ContainerConfig.Cmd'

[
  "/bin/sh",
  "-c",
  "#(nop) ",
  "CMD [\"/bin/bash\"]"
]

And as a result, container-canary fails to run any checks.

Error: container failed to start

In the current state of container-canary, you have the following options:

1. change the default CMD to something long-running, like python -m http.server 80
2. copy the contents of that remote config and add a command: block tacking on some long-running command

If instead you could override just the command: but still reuse the remote config file, then that one config could be the source of truth for something like "what characteristics does a valid Kubeflow notebook image have" for many different images spread over many repositories, without them needing to manually hold their own copies of it or add unnecessary CMD entries in their images just for the sake of this validation.

It would be useful to make some conditions as optional and add a flag to enable them if desired.

For example in the databricks example folks may want Python OR R but not both. They may also want to check for CUDA toolkit for use on GPU nodes, but not everyone will want to test for that.

It would be nice to add an optional setting which still runs them but doesn't fail the test if they fail, but have a CLI flag that makes them mandatory.

E.g

# foo.yaml
apiVersion: container-canary.nvidia.com/v1
kind: Validator
name: foo
command:
  - /bin/sh
  - -c
  - "sleep 3600"
checks:
  - name: bash
    description: Has bash installed
    optional: true
    probe:
      exec:
        command:
          - /bin/sh
          - -c
          - "which bash"

$ canary validate --file foo.yaml --check-optional "bash" somecontainer

Consider the following:

#!/bin/sh

set -ex

cat > phony-tcp.Dockerfile <<EOF
FROM ubuntu:22.04

# It succeeds even without the EXPOSE command
# EXPOSE 8080

CMD /bin/bash -c 'while true; do sleep 60; done'
EOF

cat > phony-tcp.yaml <<EOF
apiVersion: container-canary.nvidia.com/v1
kind: Validator
name: phony-tcp
description: phony-tcp checks
ports:
  - port: 8080
    protocol: tcp
checks:
  - name: tcp
    probe:
      tcpSocket:
        port: 8080
EOF

docker build -t phony-tcp -f phony-tcp.Dockerfile .

container-canary validate --file phony-tcp.yaml phony-tcp

The check succeeds even though the container is clearly not listening to port 8080, because container-canary is connecting to the Docker proxy, rather than the actual process inside the container.

Unfortunately, I'm not sure on how to actually fix this. We may have to simply issue a warning for this particular check.

this was discovered on CBL-Mariner (https://github.com/microsoft/CBL-Mariner), but likely exists everywhere.

steps to reproduce are -

1. provision a fresh system
2. git clone container-canary
3. cd container-canary; make

i would expect the test runner to check for docker and check for docker to be running, and provide a clear next steps.

note: mariner uses moby, which includes docker porcelain.

Description

For Linux, this project is currently only publishing amd64 binaries on releases.

We should consider also publishing arm64 binaries.

Benefits of this work

• allows install-from-releases workflow to work in arm64 Linux environment

That would be useful because it allows use of this project without needing to set up Go and do go install.

Acceptance Criteria

• releases contain a canary_linux_arm64 binary

Approach

As described in https://www.digitalocean.com/community/tutorials/building-go-applications-for-different-operating-systems-and-architectures#using-your-local-goos-and-goarch-environment-variables, Go has builtin support for cross-compiling, so I think this should be achievable on the GitHub-hosted ubuntu-latest runner, without requiring an arm64 runner.

Like this:

GOOS=linux GOARCH=arm64 go build

Notes

Writing this up specifically from the perspective of RAPIDS. This would be helpful (but not critical) for rapidsai/docker#667, as RAPIDS builds both amd64 and arm64 images there and it's helpful to run those images on amd64 and arm64 runners directly (instead of using emulation).

If canary tries to expose a container port for testing and that port is already in use the container fails to start and canary fails to validate.

Works

# check-port.yaml
apiVersion: container-canary.nvidia.com/v1
kind: Validator
name: check-port
description: Check port
env: []
ports:
  - port: 80
    protocol: TCP
volumes: []
checks:
  - name: http
    description: Check port 80
    probe:
      httpGet:
        path: /
        port: 80
      failureThreshold: 30

$ canary validate --file check-port.yaml nginx
Validating nginx against check-port
 Check port 80                                      [passed]
validation passed

Reproducer

$ docker run -p 80:80 nginx  # Start a process that binds to port 80 in another terminal

$ canary validate --file /tmp/test.yaml nginx
Validating nginx against check-port
\ Starting container
Error: container failed to start after 10 seconds

The container also doesn't get cleaned up.

$ docker ps -a             
CONTAINER ID   IMAGE                                  COMMAND                  CREATED         STATUS         PORTS                               NAMES
e8d32b8f45aa   nginx                                  "/docker-entrypoint.…"   2 minutes ago   Created                                            canary-runner-d43137e8