For building a multi-nodes k8s cluster environment with infra services.
Table Contents
- k8s-playground
The overall structure is shown in the figure below.
Install Ubuntu22 on HP Z440 Workstation and install common command.
Install common commands through apt.
sudo apt update
sudo apt install -y \
htop iftop \
vim \
curl wget \
makeRefer:
Install KVM through apt command.
sudo apt -y install \
bridge-utils \
cpu-checker \
libvirt-clients \
libvirt-daemon \
qemu \
qemu-kvmInstall Docker through apt command. It is recommended to install according to the latest official website introduction.
sudo apt-get update
sudo apt-get -y install \
ca-certificates \
curl \
gnupg \
lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get -y install \
docker-ce docker-ce-cli \
containerd.io \
docker-compose-pluginAfter installing docker, there is a quick way to allow a non-root user to use docker.
chown ${USER} /var/run/docker.sockRefer:
This repo uses a special version of minikube and helm.
Install Minikube:
curl -LO https://storage.googleapis.com/minikube/releases/v1.26.1/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
rm minikube-linux-amd64Install Helm:
curl -fsSL -o ./get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 ./get_helm.sh
./get_helm.sh
rm ./get_helm.shRefer:
Config network environment, includes creating virtual bridge and kvm virtual network. They will be bridge br0 as shown in the architecture diagram.
Create virtual bridge by netplan command. Pls check network config file first, and the default config will use a static ip which may not be right for you.
mv /etc/netplan/01-network-manager-all.yaml /etc/netplan/01-network-manager-all.yaml.backup
cp configs/network/01-network-manager-all.yaml /etc/netplan/01-network-manager-all.yaml
netplan applyCreate KVM virtual network.
virsh net-define configs/network/kvm-bridged-network.xml
virsh net-start bridged-network
virsh net-autostart bridged-network
virsh net-listSomeday, the network is down in virtual machine, and the root cause is still unknown.
You can also choose to use NAT mode.
You cannot access specific network through changing route ip in nat mode
virsh net-define configs/network/kvm-nat-network.xml virsh net-start nat-network virsh net-autostart nat-network virsh net-list
Refer:
- KVM: Creating a guest VM on a NAT network
- KVM: Creating a bridged network with NetPlan on Ubuntu 18.04 bionic
- JINGTAO: PVE 网络瞎折腾
- JINGTAO: 《Docker 容器与容器云》读书笔记 之 容器
It is need to create some infra to support k8s running well because this is a multi-nodes cluster.
Create NFS server through bash script, before running it, pls check NFS_DOMAIN variable which indicates the subnet that can access the NFS service.
Now, It is all ready for starting k8s cluster! Start it through below command.
For setting route for k8s cluster, run scripts/k8s_set_route.sh script which will set route when node is ready.
PROFILE_NAME='playground'
SOFT_ROUTE_IP='192.168.1.41'
KVM_NETWORK='nat-network'
NODE_NUM=3
# bash ${WORKING_DIR}/scripts/k8s_set_route.sh ${PROFILE_NAME} ${NODE_NUM} ${SOFT_ROUTE_IP} &
# export HTTP_PROXY=http://${SOFT_ROUTE_IP}:1080
# export HTTPS_PROXY=https://${SOFT_ROUTE_IP}:1080
# export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24
minikube config set WantUpdateNotification false
minikube \
--profile ${PROFILE_NAME} \
--driver=kvm2 \
--install-addons=false \
--kubernetes-version='v1.24.3' \
--auto-update-drivers=false \
--nodes=${NODE_NUM} \
--cpus=8 \
--memory=18g \
--disk-size=40g \
--kvm-network="${KVM_NETWORK}" \
--image-mirror-country='cn' \
--image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' \
startTo get nodes:
PROFILE_NAME="playground"
minikube kubectl --profile ${PROFILE_NAME} -- get pods -ATo access dashboard:
PROFILE_NAME="playground"
minikube dashboard --profile ${PROFILE_NAME} --urlRefer to k8s_start.sh.
Refer to k8s_infra_services_enable.sh for all deploy code.
Because this is a multi-nodes k8s, so the default storage class which using a certain host path can't satisfy the need. And the standard storage class can mount a nfs path which can be accessed by any node.
Deploy it using below command.
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm repo update
PROFILE_NAME="playground"
CONTEXT_NAME=${PROFILE_NAME}
NFS_STORAGE_NAMESPACE="storage-nfs"
IMAGE_MIRROR_SUFFIX=".registry.jingtao.fun"
# IMAGE_MIRROR_SUFFIX="" # Leave blank to not apply mirror service
# get host ip
BR0_IP=$(ip addr show br0 | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:")
BR0_IP=$(echo ${BR0_IP//\// } | awk '{print $1}')
echo "your host ip: ${BR0_IP}"
minikube kubectl --profile ${PROFILE_NAME} -- create namespace ${NFS_STORAGE_NAMESPACE} --dry-run=client -o yaml | minikube kubectl --profile ${PROFILE_NAME} -- apply -f -
helm upgrade --install nfs-subdir-external-provisioner \
--kube-context ${CONTEXT_NAME} \
--namespace ${NFS_STORAGE_NAMESPACE} \
--values configs/charts_values/nfs-values.yaml \
--set nfs.server=${BR0_IP} \
--set image.repository="k8s.gcr.io${IMAGE_MIRROR_SUFFIX}/sig-storage/nfs-subdir-external-provisioner" \
--wait \
--timeout 10m0s \
nfs-subdir-external-provisioner/nfs-subdir-external-provisionerRefer:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent