Kubenix-managed namespaces about kubenix HOT 10 CLOSED

Could you share a code example and the command you ran?

from kubenix.

Here is a code example (imported as a module):

{
  kubernetes.resources.namespaces.test-namespace = { };
  kubernetes.resources.pods.example = {
    metadata.namespace = "test-namespace";
    spec.containers.nginx.image = "nginx";
  };
}

Running this gives the error (:

$ nix run .#kubenix.x86_64-linux
Error from server (NotFound): namespaces "test-namespace" not found

Doing something like this in YAML does work though:

apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: test-namespace
spec:
  containers:
  - name: nginx
    image: nginx

$ kubectl apply -f ./repro.yaml 
namespace/test-namespace created
pod/nginx created

Maybe this is caused by the diffs?

from kubenix.

Ah wait, changing the order in the YAML file does matter. If I put the pod definition before the namespace definition, you get the same error.

from kubenix.

Changing the kubernetes.resourceOrder option to the following also doesn't fix this issue:

kubernetes.resourceOrder = [
  "Namespace"
  "Pod"
];

from kubenix.

Default order should already put the namespace first, so maybe something messes up with the ordering on the way from nix to final yaml?

from kubenix.

Could you run nix eval —json on generated to see if the order is right? If it’s not then it’s something wrong with ordering logic

from kubenix.

How do I do that exactly? I'm using Kubenix as a flake output with outputs.kubenix = kubenix.packages.${system}.default .override {}.

from kubenix.

ok, so if you run:

nix build .#kubenix

you can then inspect the generated manifest with:

jq . result/manifest.json

which should show something like:

{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "v1",
      "kind": "Namespace",
      "metadata": {
        "annotations": {
          "kubenix/k8s-version": "1.27",
          "kubenix/project-name": "kubenix"
        },
        "labels": {
          "kubenix/hash": "dc73d66262ab324ab024e6f3258fff7ad526113b"
        },
        "name": "test-namespace"
      }
    },
    {
      "apiVersion": "v1",
      "kind": "Pod",
      "metadata": {
        "annotations": {
          "kubenix/k8s-version": "1.27",
          "kubenix/project-name": "kubenix"
        },
        "labels": {
          "kubenix/hash": "dc73d66262ab324ab024e6f3258fff7ad526113b"
        },
        "name": "example",
        "namespace": "test-namespace"
      },
      "spec": {
        "containers": [
          {
            "image": "nginx",
            "name": "nginx"
          }
        ]
      }
    }
  ],
  "kind": "List",
  "labels": {
    "kubenix/hash": "dc73d66262ab324ab024e6f3258fff7ad526113b",
    "kubenix/k8s-version": "1.27",
    "kubenix/project-name": "kubenix"
  }
}

with Namespace first in the array

this works for me:

kubectl apply -f result/manifest.json

with output:

namespace/test-namespace created
pod/example created

but I got the same error as you when running:

kubectl apply --dry-run=server -f result/manifest.json

the error you are getting is from the diff part of the script, see:

nix run .#kubenix diff

from kubenix.

This is actually the same problem @hall talked about here: #23 (comment)
And to quote upstream Kubectl:

Yeah, this is hard, I have no clue how we can fix this properly, not without very complicated machinery.

from kubenix.

Looked at Kapp as an alternative to plain kubectl. Kapp only really works when you use their kapp.k14s.io labels, which seems unreasonable to ask of users of Kubenix in my opinion. I think I will use this for my home lab though, as it is quite convenient :)

So yeah, I don't think we should/can fix this in kubectl.

from kubenix.