tilt-dev / kind-local Goto Github PK

Hey,
Thanks for the inspiration. As you have mentioned in the readme that multinode is still todo.
I had to solve it in my case and below you can find a nix snippet to make multicluster work.
I was a bit unhappy to keep whole node config and registry in repo, so in my solution:

1. I’m taking the node config and appending local repo to all nodes
2. Using helm chart for registry docker-helm-chart instead of keeping custom definition within registry.
   In my case it works well, however I had to change local registry path to dev.local to make knative happy.

{pkgs, env-config}:
with pkgs;
let
  docker = "${pkgs.docker}/bin/docker";

  docker-cfg = env-config.docker;
  registry-alias = "${docker-cfg.registry}";            # dev.local - to make knative happy
  exposed-port = docker-cfg.local-registry.exposedPort; # 32001
  cluster-name = env-config.projectName;

  change-config = pkgs.writeScript "change-config" ''
    sed '/\[plugins.cri.registry.mirrors\]/s/.*/&\
        \    [plugins.cri.registry.mirrors."${registry-alias}"\]\
        \      endpoint = \["http:\/\/host.docker.internal:${toString exposed-port}"\]/' $1 > $1.toml
  '';

  get-kind-nodes = pkgs.writeScript "get-kind-nodes" ''
    ${docker} ps --filter 'name=${cluster-name}-*' --format "{{.Names}}"
  '';

  get-node-config = pkgs.writeScript "get-node-config" ''
    node=$1
    ${docker} exec $node cat /etc/containerd/config.toml
  '';

  copy-config = pkgs.writeScript "copy-config-to-node" ''
    config=$1
    node=$2
    ${docker} cp $config.toml $node:/etc/containerd/config.toml
  '';

  restart-node-containerd = pkgs.writeScript "restart-node-containerd" ''
    node=$1
    ${docker} exec $node systemctl restart containerd.service
    ${docker} exec $node systemctl restart kubelet.service
  '';
in
pkgs.writeScriptBin "append-local-docker-registry" ''
  TEMP=$(mktemp -d)

  for node in $(${get-kind-nodes}); do
    config="$(${get-node-config} $node)"
    config_location=$TEMP/$node

    if [[ "$config" != *"${registry-alias}"* ]]; then
      echo "$config" > $config_location

      ${change-config} $config_location
      ${copy-config} $config_location $node
      ${restart-node-containerd} $node
      echo "Adding local docker registry to node: $node"
    else
      echo "Containerd already aware of local registry: $node"
    fi
  done
''

To make picture clear and be able to automate all these things, flow which works for me is as follows:

1. create kind cluster
2. apply changes to configuration - to kind nodes (above script)
3. apply helm chart and wait to be ready on kubernetes (wait for pod)
4. upload images to localhost:32001
5. at this point all images will be available

In my local testing with latest master I found 2 issues, which I aim to submit solutions for immediately:

1. Workaround for MacOS when kind-control-plane container is not started after reboot.
2. A minor fix for proper string comparison in bash.

I wonder if this registry works with the scenario where you have:

1. a kind cluster with enabled local registry (set up on localhost:5000)
2. and then you build a custom image on some Pod container inside that kind of cluster using f.e., the kaniko tool.

I think it will not work.

The script is now added to upstream docs: https://github.com/kubernetes-sigs/kind/blob/master/site/static/examples/kind-with-registry.sh

It's already deviating, it maybe makes sense to just point to the upstream one 😄

Just a heads up, it looks like the networking refactor in KIND 0.8.0 may have broken the ability for the kind nodes to communicate directly to the registry over the docker interface.

Our setup has diverged a bit from the setup script here and we have the registry bind to 0.0.0.0:5000 on the host OS, so we were able to workaround the problem by having the kind node connect to the registry by going to the host OS instead.

After installing kind via kind local and run through the instructions here -

kind-with-registry.sh
docker tag alpine localhost:5000/alpine

my docker connectivity to registry is messed up. I get this error -

docker pull radial/busyboxplus:curl
Error response from daemon: Get https://registry-1.docker.io/v2/: Not Found

Any ideas what is going on here ?