Kubernetes Setup Tutorial Doesn't Work in Clean Virtualbox VM about clear-linux-documentation HOT 16 CLOSED

@shrmrf Confirmed on proxy configuration. I work behind a proxy too. I had to add few IPs to proxy.conf file located here: /etc/systemd/system/crio.service.d/proxy.conf. ( I realize that your environmental setup is different.)

Just to be clear, per our tutorial (see Troubleshooting), I must run the command with -E flag to pass my environmental variables:
sudo -E kubeadm init --pod-network-cidr 10.244.0.0/16 --cri-socket=/run/crio/crio.sock

While we try to add some help for potential proxy configuration issues, we must limit it.

Thanks to your feedback--we're considering an extra command that you use in your vagrant script. Please check out our current PR on the k8s tutorial: #273

Feel free to offer feedback.

from clear-linux-documentation.

@mythi Thanks for your hints... I just tried my Vagrantfile with these changes and it works: sysarcher/k8s-resources@d65e360#diff-0e16f5f1bbe6083ea3521794a6b59b45

@mvincerx validation from my side of the new PR #273

from clear-linux-documentation.

@shrmrf the feedback makes sense, we can improve the notes on CRI selection.

from clear-linux-documentation.

@shrmrf Unless you're working behind a proxy, you should be able to safely omit -E from the command above. Also, we're working on follow up PR to correct a few items in the k8s tutorial. Please allow us a little time while we look into your scenario. Thanks.

from clear-linux-documentation.

@mvincerx thanks for the reply. I'm not working behind a proxy the -E is just a copy-paste thing :)

Yeah, let me know if there's something missing in my description above

from clear-linux-documentation.

@mvincerx I think I've found the culprit. I have added some systemctl restart lines to my Vagrantfile and now, the tutorial seems to run fine.

It seems that crio.service is required even if I'm just following the tutorial for docker + runc! It's part of the kubelet.service unit file in fact.

Is this a desired behavior?

I have added some comments in the Solution section of my README file

from clear-linux-documentation.

@mvincerx I just found out the hard way, that behind a proxy, we need to also configure the HTTP_PROXY environment for crio.service... Here is my modified file:

(the significant line below is: Environment="HTTP_PROXY=http://proxy-url.com:port")

[Unit]
Description=Open Container Initiative Daemon
Documentation=https://github.com/kubernetes-sigs/cri-o
After=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=6

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/crio-storage
EnvironmentFile=-/etc/sysconfig/crio-network
Environment=GOTRACEBACK=crash
Environment="HTTP_PROXY=http://proxy-url.com:port"
ExecStart=/usr/bin/crio \
          $CRIO_STORAGE_OPTIONS \
          $CRIO_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TasksMax=infinity
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
OOMScoreAdjust=-999
TimeoutStartSec=0
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

See also (for similar issue): containerd/cri#834

from clear-linux-documentation.

And, to prove that I did not cheat:

clear@clearlinux ~ $ sudo swupd verify                                                                                                                                                                             
Verifying version 25700
Verifying files
        ...26%Hash mismatch for file: /usr/lib/systemd/system/crio.service
        ...100%
Inspected 40584 files
  1 files did not match
Verify successful

from clear-linux-documentation.

If you want to use k8s+docker, make sure kubelet(s) connect to dockershim socket (not CRI-O socket which is the default). This seems to be missing in your Vagrantfile. In case Docker is set up, crio.service isn't needed.

If you want to use k8s+CRI-O, kubeletes are configured but you need to remember to pass --cri-socket=/run/crio/crio.sock

from clear-linux-documentation.

@mythi I want to use dockershim... Do I need to modify kubelet.service for that? Or just have docker installed?

What I found was kubelet.service was dependent on unix:///var/run/crio/crio.sock:

# /usr/lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/

[Service]
Environment="KUBELET_KUBECONFIG_ARGS= --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/usr/libexec/cni"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=30m"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS
Restart=on-failure
StartLimitInterval=300
StartLimitBurst=6
RestartSec=10

[Install]
WantedBy=multi-user.target

So if crio.sock isn't properly up, kubelet.service tends to fail. Do you mean, we should be changing the kubelet.service file to modify the --container-runtime-endpoint flag's value?

from clear-linux-documentation.

@shrmf good to hear it works. Any feedback how to improve the doc? Should some of the steps be explained or organized differently (e.g. docker vs crio)?

from clear-linux-documentation.

@mythi I think the docs are well written. Maybe some pointers to assumptions could be highlighted e.g. crio being default!.

I don't know if it's a sane default either because coming from any other distribution, I would just expect dockershim to be the correct default and it's interesting that the solution is to set KUBELET_EXTRA_ARGS="" in /etc/systemd/system/kubelet.conf.d/kubeadm.conf. Totally strange IMO.

from clear-linux-documentation.

@mythi and @mvincerx when using dockershim, I believe KUBELET_NETWORK_ARGS should also be reset otherwise, the node remains in a non-ready state because it can't find the cni network plugin.

i.e. in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf we have an additional line:

[Service]
Environment="KUBELET_EXTRA_ARGS="
Environment="KUBELET_NETWORK_ARGS="

see: https://github.com/shrmrf/k8s-resources/blob/500ed708463633064019874022b2332d9a559399/clear/Vagrantfile#L24

from clear-linux-documentation.

@shrmrf Environment="KUBELET_NETWORK_ARGS=" is wrong. It makes dockershim to use a "dummy" networking interface that results in docker to use its default networking (bridge). While it looks it works, that is not what users normally want.

The CNI network needs to be installed separately. Once you have that, the pods are no longer failing. We'll be improving the doc on the CNI setup side too.

from clear-linux-documentation.

@mythi thanks for the info. I was just proposing the easiest defaults.. It am wrong of course.

I'll try to apply a pod network and report back. (closing issue for now)

from clear-linux-documentation.

@mythi yup... This fixed the problem for me... Of course, we should document that this tutorial uses flannel so people deploy the correct pod network.

clear@clearlinux ~ $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

from clear-linux-documentation.