kubernetes-sigs / cluster-api-provider-packet Goto Github PK
View Code? Open in Web Editor NEWCluster API Provider Packet (now Equinix Metal)
Home Page: https://deploy.equinix.com/labs/cluster-api-provider-packet/
License: Apache License 2.0
Cluster API Provider Packet (now Equinix Metal)
Home Page: https://deploy.equinix.com/labs/cluster-api-provider-packet/
License: Apache License 2.0
Currently, the cluster-template.yaml
is tight to Ubuntu (maybe Debian as well)
because we use the BootstrapTemplate
to install packages like: kubeadm
,
docker
, kubectl
and so on:
kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
metadata:
name: "${CLUSTER_NAME}-control-plane1-config"
spec:
preKubeadmCommands:
- swapoff -a
- apt-get -y update
- DEBIAN_FRONTEND=noninteractive apt-get install -y apt-transport-https curl
- curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
- echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- apt-key fingerprint 0EBFCD88
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- apt-get update -y
- apt-get install -y ca-certificates socat jq ebtables apt-transport-https cloud-utils prips docker-ce docker-ce-cli containerd.io kubelet kubeadm kubectl
- systemctl daemon-reload
- systemctl enable docker
- systemctl start docker
As you can see this does not work for every operating system.
I am not 100% sure about how the other cluster-api-providers manage this, I
remember AWS, DigitalOcean generating their own images.
I think we should not put the effort of supporting multiple operating systems,
but we have to be clear about it.
In general the template can be modified in a lot of ways from the end user to
make the installation process to work for CentOS or ArchLinux, it is just matter
of changing the packege manager.
In any case I like the idea to maintain a list of images that people can grab
from Packet and use with the ClusterAPI mainly because it will decrease the
amount of runtime actions that can fail along the way.
So, this issue is to understand what you think about it.
First step should be to at least write a documentation about where we are to
avoid the pain for a user to discover that CentOS does not work when trying the
cluster-api.
the cluster is initialized with Weave's CNI but to really do anything on Packet you need Calico for BGP + MetalLB
generate-yaml.sh
currently blocks if the out/packet/
directory already exists. Have a -f
option to override it.
The readme file should be reviewed because there are some missing links such as the machine configs url under https://github.com/kubernetes-sigs/cluster-api-provider-packet#supported-node-os-and-versions.
Packet has support for cluster-autoscaler. The CAPP implementation should have an option to deploy this as part of provisioning clusters, just like it deploys CCM.
Right now the facility is part of the PacketMachine Spec. This has two issues:
When importing this package in other projects, the module path must match the source name.
$ go mod vendor
warning: ignoring symlink /Users/marques/src/openshift-installer/pkg/asset/store/data
go: sigs.k8s.io/[email protected]: parsing go.mod:
module declares its path as: github.com/packethost/cluster-api-provider-packet
but was required as: sigs.k8s.io/cluster-api-provider-packet
The source should be sigs.k8s.io/cluster-api-provider-packet.
Compare
cluster-api-provider-packet/go.mod
Line 1 in f333fec
Recently I added a new doc folder that I called docs/experiences
where the idea is to write about how people are using cluster-api. Because it is very flexible and it is not simple to document all its pieces by themself. I think it is nice to share what we do and I hope those use cases will drive personalized solutions for other people.
I would like to ask @rsmitty and @andrewrynhard from Talos to write their story using iPXE and custom os #131 .
Should we have a chat about it?
/kind documentation
While testing, 2 masters were created instead of just the one that I was intending to create. This worked as expected the 6 times before I tried but the 7th, it gave me this weird behavior. "cluster-api-provider-packet-controller-manager" logs are as follows:
2020-08-14T22:24:47.525Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine instance is pending {"packetmachine": "default/kluister07-control-plane-8rkkc", "machine": "kluister07-control-plane-7gz9d", "cluster": "kluister07", "packetcluster": "kluister07", "instance-id": "99a54cbf-13ec-4475-9cef-4e19fc1ee86b"}
2020-08-14T22:24:54.415Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine instance is pending {"packetmachine": "default/kluister07-control-plane-8rkkc", "machine": "kluister07-control-plane-7gz9d", "cluster": "kluister07", "packetcluster": "kluister07", "instance-id": "66343c24-7adb-45da-ae8b-5cedbc5b52b8"}
It had the same object name but as you can see, the instance-id was different. I am leaving these machines and logs up just in case but let me know if there are any other log locations you need.
For production control planes, the instances will be long running and using device reservations is a great cost saving measure.
During my testing of cluster-API i have been intermittently hitting what appears is a timing condition between the moment the bootstrap secret is created for a machine and the packet controller attempting to consume it. When this occurs the solution is to manually delete the machine resource; which I may have to do for several machines before it works again.
Controller log:
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.581Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine Controller has not yet set OwnerRef {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.581Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.613Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine Controller has not yet set OwnerRef {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.613Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.626Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine Controller has not yet set OwnerRef {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.626Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.639Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Machine Controller has not yet set OwnerRef {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.639Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.716Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Reconciling PacketMachine {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.716Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Bootstrap data secret is not yet available {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.729Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.729Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Reconciling PacketMachine {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.729Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Bootstrap data secret is not yet available {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.729Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.736Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Reconciling PacketMachine {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.736Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Bootstrap data secret is not yet available {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:38.736Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:39.166Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Reconciling PacketMachine {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:39.184Z ERROR controller-runtime.controller Reconciler error {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "error": "failed to create machine k8s-game-np-default-43e5-fz49q: impossible to retrieve bootstrap data from secret: failed to retrieve bootstrap data secret for PacketMachine dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q: Secret \"k8s-game-np-default-sqqlb\" not found"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager github.com/go-logr/zapr.(*zapLogger).Error
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager k8s.io/apimachinery/pkg/util/wait.JitterUntil
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager k8s.io/apimachinery/pkg/util/wait.Until
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:40.185Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Reconciling PacketMachine {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:40.185Z INFO controllers.PacketMachine.infrastructure.cluster.x-k8s.io/v1alpha3 Error state detected, skipping reconciliation {"packetmachine": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q", "machine": "k8s-game-np-default-d5c944fdf-7v8bq", "cluster": "dev-ewr2-mine-k8s-game", "packetcluster": "dev-ewr2-mine-k8s-game"}
cluster-api-provider-packet-controller-manager-66d9fc947d-zlqln manager 2020-07-29T04:10:40.186Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "packetmachine", "request": "dev-pkt-ewr2-mine-k8s-game/k8s-game-np-default-43e5-fz49q"}
Secret exists!
$ k get secret k8s-game-np-default-sqqlb -o yaml
apiVersion: v1
data:
value: ...
kind: Secret
metadata:
creationTimestamp: "2020-07-29T04:10:39Z"
labels:
cluster.x-k8s.io/cluster-name: dev-ewr2-mine-k8s-game
name: k8s-game-np-default-sqqlb
namespace: dev-pkt-ewr2-mine-k8s-game
ownerReferences:
- apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
controller: true
kind: KubeadmConfig
name: k8s-game-np-default-sqqlb
uid: 0428f42a-0282-4edb-91ed-970011d3c82b
resourceVersion: "9495969"
selfLink: /api/v1/namespaces/dev-pkt-ewr2-mine-k8s-game/secrets/k8s-game-np-default-sqqlb
uid: 560772d3-ad97-4760-93a3-d0e8b60ce34f
type: cluster.x-k8s.io/secret
I don't see a way to use iPXE os type.
ClusterAPi at the moment do not support default parameter for cluster.
To solve that we serve a make
utility called:
make cluster
That generates a cluster spec that can be used as a baseline.
The script that generates that does not place the k8s version in the right way
Is it feasible that Lokomotive could be the cluster that is stood up by the cluster api? That also would include FlatCar Linux OS for the nodes.
The URLs included in the cluster template are relative to the master
branch on the CCM and CSI repositories. If this branch is renamed or the files are removed or relocated, in either repository, this provider will break.
cluster-api-provider-packet/templates/cluster-template.yaml
Lines 28 to 30 in ab54c49
These URLs should be pinned to a specific tag to make them less fragile.
Consider using an Elastic IP for the cluster, which is assigned to the master, rather than the first master's public IP.
Might be possible to do via metallb and simplify it.
When it is ready, switch to v1alpha2
There's a big list of cluster-api implementations at
https://github.com/kubernetes-sigs/cluster-api/blob/master/README.md
Add a Packet one to this list, when this is ready to roll.
Ensure that maintainers aligns with OWNERS once transferred, and includes at least:
@gianarb
@deitch
@jacobsmith928
@mrmrcoleman
Anyone else we need
I would like to use kubectl get machine X -o jsonpath...
to parse out addresses assigned to a node. Unfortunately the values in the status.addresses
array are formatted golang data structures. Making these JSON would allow me to parse out whatever I needed from the entries.
status:
addresses:
...
- address: packngo.IPAddressAssignment{IpAddressCommon:packngo.IpAddressCommon{ID:"REDACTED",
Address:"10.100.2.17", Gateway:"10.100.2.16", Network:"10.100.2.16", AddressFamily:4,
Netmask:"255.255.255.254", Public:false, CIDR:31, Created:"2020-07-25T14:54:10Z",
Updated:"", Href:"/ips/REDACTED", Management:true,
Manageable:true, Project:packngo.Href{Href:"/projects/REDACTED"},
Tags:[], CustomData:map[]}, AssignedTo:packngo.Href{Href:"/devices/REDACTED"}}
type: InternalIP
I am trying to follow the steps in README.md, but the clusterctl init
step fails with the failure:
$ clusterctl --config=https://github.com/packethost/cluster-api-provider-packet/releases/latest/clusterctl.yaml init --infrastructure=packet
Error: failed to initialize the configuration reader: stat https://github.com/packethost/cluster-api-provider-packet/releases/latest/clusterctl.yaml: no such file or directory
The URL https://github.com/packethost/cluster-api-provider-packet/releases/latest/clusterctl.yaml is indeed invalid.
I also tried going back to the latest release v0.1.0. However, README of v0.1.0 does not seem to be compatible with the current clusterctl v0.3.6.
So I am not sure how to proceed.
To support Flatcar Container Linux for cluster-api for Packet, I have been following the quickstart guide.
Though it fails when reading ignition config from https://metadata.packet.net/userdata.
$ export PACKET_API_KEY="..."
$ export PROJECT_ID="2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4"
$ export FACILITY="ams1"
$ export NODE_OS="flatcar_stable"
$ export SSH_KEY="2020-06 [email protected]"
$ export POD_CIDR="172.25.0.0/16"
$ export SERVICE_CIDR="172.26.0.0/16"
$ export MASTER_NODE_TYPE="t1.small"
$ export WORKER_NODE_TYPE="t1.small"
$ kind create cluster
... (ok)
$ clusterctl init
... (ok)
$ clusterctl init --infrastructure packet
... (ok)
$ kubectl apply -f ./capi-dongsu.yaml
... (fail)
Error messages:
cat: unrecognized option '-------------------------------------------------------------------------------'
Try 'cat --help' for more information.
Ignition v0.33.0-1-ga7c8752-dirty
reading system config file "/usr/lib/ignition/base.ign"
parsing config with SHA512: 0131bd505bfe1b1215ca4ec9809701a3323bf448114294874f7249d8d300440bd742a7532f60673bfa0746c04de0bd5ca68d0fe9a8ecd59464b13a6401323cb4
parsed url from cmdline: ""
no config URL provided
reading system config file "/usr/lib/ignition/user.ign"
no config at "/usr/lib/ignition/user.ign"
GET https://metadata.packet.net/userdata: attempt #1
GET result: OK
parsing config with SHA512: d4e25b97bb56f8c6f56b052b48726ce2876d94c010911b3bb21099addf254545e2bdd9055aa37ca16c855eb9fb18f9bffccd598913d1b9023301069405126a1c
error at line 1, column 1
invalid character '#' looking for beginning of value
failed to fetch config: config is not valid
failed to acquire config: config is not valid
POST message to Packet Timeline
GET https://metadata.packet.net/metadata: attempt #1
GET result: OK
Ignition failed: config is not valid
capi-dongsu.yaml
:
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfig
metadata:
name: dongsu-capi-control-plane1-config
namespace: default
spec:
clusterConfiguration:
controllerManager:
extraArgs:
enable-hostpath-provisioner: "true"
initConfiguration:
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
postKubeadmCommands:
- 'kubectl --kubeconfig /etc/kubernetes/admin.conf create secret generic -n kube-system
packet-cloud-config --from-literal=cloud-sa.json=''{"apiKey": "{{ .apiKey }}","projectID":
"2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4"}'''
- kubectl apply --kubeconfig /etc/kubernetes/admin.conf -f https://raw.githubusercontent.com/packethost/packet-ccm/master/deploy/releases/v1.0.0/deployment.yaml
preKubeadmCommands:
- swapoff -a
- systemctl enable --now docker.service
- mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d
- curl -L "https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz" | tar -C /opt/cni/bin -xz
- curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz" | tar -C /opt/bin -xz
- curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/{kubeadm,kubelet,kubectl}
- chmod +x kubeadm kubelet kubectl
- mv kubeadm kubelet kubectl /opt/bin
- curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
- curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
name: dongsu-capi
namespace: default
spec:
clusterNetwork:
pods:
cidrBlocks:
- 172.25.0.0/16
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketCluster
name: dongsu-capi
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketCluster
metadata:
name: dongsu-capi
namespace: default
spec:
projectID: 2c850bb4-70b4-4dcd-b60f-2d0be86d0ae4
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Machine
metadata:
labels:
cluster.x-k8s.io/cluster-name: dongsu-capi
cluster.x-k8s.io/control-plane: "true"
name: dongsu-capi-master-0
namespace: default
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfig
name: dongsu-capi-control-plane1-config
clusterName: dongsu-capi
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachine
name: dongsu-capi-master-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachine
metadata:
name: dongsu-capi-master-0
namespace: default
spec:
OS: flatcar_stable
billingCycle: hourly
facility:
- ams1
machineType: t1.small
sshKeys:
- "2020-06 [email protected]"
tags: []
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
labels:
cluster.x-k8s.io/cluster-name: dongsu-capi
pool: worker-a
name: dongsu-capi-worker-a
namespace: default
spec:
clusterName: dongsu-capi
replicas: 3
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: dongsu-capi
pool: worker-a
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: dongsu-capi
pool: worker-a
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
name: dongsu-capi-worker-a
clusterName: dongsu-capi
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachineTemplate
name: dongsu-capi-worker-a
version: v1.18.3
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachineTemplate
metadata:
name: dongsu-capi-worker-a
namespace: default
spec:
template:
spec:
OS: flatcar_stable
billingCycle: hourly
facility:
- ams1
machineType: t1.small
sshKeys:
- "2020-06 [email protected]"
tags: []
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
metadata:
name: dongsu-capi-worker-a
namespace: default
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
preKubeadmCommands:
- swapoff -a
- systemctl enable --now docker.service
- mkdir -p /opt/bin /opt/cni/bin /etc/systemd/system/kubelet.service.d
- curl -L "https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz" | tar -C /opt/cni/bin -xz
- curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz" | tar -C /opt/bin -xz
- curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/v1.18.3/bin/linux/amd64/{kubeadm,kubelet,kubectl}
- chmod +x kubeadm kubelet kubectl
- mv kubeadm kubelet kubectl /opt/bin
- curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
- curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.18.3/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
As far as I understand, every Packet machine can accept "userdata" in cloud-init format. When simply creating a machine on the Packet UI, the machine gets configured without userdata. If we want to specify userdata, we need to configure via UI before creating the machine.
In the Packet UI for creating the machine, its help message shows:
Use this to execute script/package tasks or trigger more advanced configuration processes after the server is ready.
Paste your YAML or script here. Packet supports most cloud-init config formats. Support for formats other than #cloud-config and #! (script) are experimental.
So I think somewhere cluster-api seems to generate a config with unsupported characters in the beginning of cloud-init config.
But I am not sure exactly where/how it gets configured. The cluster-api has a cloud-init config for userdata. Though changing it does not seem to fix the issue.
Does anyone have an idea?
Thanks.
In theory you can specify the Kubernetes version you want your cluster to run.
I am saying in theory because right now we do not support that. I feel like this
is in some way related to #118 but not sure yet.
As you can see from cluster-template.yaml you can specify the kubernetes version
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
name: ${CLUSTER_NAME}-worker-a
labels:
cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
pool: worker-a
spec:
replicas: ${WORKER_MACHINE_COUNT}
clusterName: ${CLUSTER_NAME}
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
pool: worker-a
template:
metadata:
labels:
cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
pool: worker-a
spec:
version: ${KUBERNETES_VERSION}
clusterName: ${CLUSTER_NAME}
bootstrap:
configRef:
name: ${CLUSTER_NAME}-worker-a
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfigTemplate
infrastructureRef:
name: ${CLUSTER_NAME}-worker-a
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: PacketMachineTemplate
We have to dig into it to see how the other cluster-api-providers implemented
this feature.
ClusterAPI currently works with single control plane but not with multi-master
multi muster works via KubeadmControlPlane. As you can see from the graph it works a bit differently:
https://cluster-api.sigs.k8s.io/images/control-plane-controller.png
The controller that manages KubeadmControlPlane does not create any servers until the IP is set. That creates two options:
Machine
for the masterReconcile()
CAPP currently does the first. We want to enable the second.
Packet is blocked on the second one by an internal Packet API issue around cluster management known internally as 4090. When that is fixed, this issue will be addressed.
addons.yaml.template
includes CCM (basic necessity for getting the cluster to work) but not CSI. We should consider including it, based on the status of block storage support at Packet.
Currently, the implementation assumes one master Machine
. It is possible to do multi-master with kubeadm.
We have encountered a provisioning failure for your device test1-g7tqp-master-0.
The PacketAPI returns:
403 You are not authorized to view this device
Right now the MachineController
tries over and over without an end. it does not sound great. Should we treat the error as we do when the controller receives a 404? We assume that the server is not running anymore and we make the reconciliation as a success. It is not ideal because 403 will may be fixed by generating a new API key. I think the API is not returning the right status code here.
We have versions in too many places:
Makefile
templates/metadata-template.yaml
test/e2e/config/packet-dev.yaml
go.mod
These need to be centralized. We need one place to control versions, rather than duplicating them
Packet's cluster-template.yaml
has a clusterNetwork
block with only pods: in the array, so even if you export SERVICE_CIDR
and provide the range when generating the template, the service CIDR is not included in the spec
This issue was reported in other providers - Running several Cluster API pods (e.g. CAPI + CABPK + an infra provider) in the same namespace results in leader election errors.
Additional details here - kubernetes-retired/cluster-api-bootstrap-provider-kubeadm#271
Currently, we create an elastic IP for every cluster and we assign it to one of the control-plane (the first one that gets created).
There is no process to re-assign it in case of a failure from the control plane.
There are currently two possibilities:
Once the machine
resource is created, it contains several useful variables, most notably the address block:
Ex:
addresses:
- address: 147.75.194.145
type: ExternalIP
- address: 147.75.105.125
type: ExternalIP
- address: '2604:1380:1:1200::37'
type: ExternalIP
- address: 10.99.189.55
type: InternalIP
However, you can see that the type
is not unique enough to distinguish. Each respective type, e.g IPv4, IPv6, and private vs. public should have unique types to reference.
Also, it appears as though the creation of the master and the IP it's able to advertise is coupled with the ability to create + apply an elastic IP - if you can omit the `{{ .controlPlaneEndpoint }} and have it default to one of the hosts' addresses, then maybe just the docs need to be updated, otherwise it would be valuable to be able to reference these as variables during the cluster initialization.
The end goal being that one does not have to have an elastic IP assigned to the master, and could reference a variable like InternalIP
within the respective kubeadmConfigSpec
blocks; ex:
initConfiguration:
nodeRegistration:
localAPIEndpoint:
advertiseAddress: {{ InternalIP }}
kubeletExtraArgs:
cloud-provider: external
Once the Machines + Cluster have been created, the Cluster object should derive it's caKeyPair values from a kube secret
When doing clusterctl init
with packet provider the PACKET_API_TOKEN env var is required because the manager uses it to interact with PacketAPI.
We create the secret but its content is not base64 encoded, so it means that when kubernetes tries to attach it as env var in the pod it can't decode it
Create an option to deploy using k3s instead of upstream k8s and kubeadm
cluster-api v0.3.7
capp v0.3.2
packet-ccm v1.1.0
I am hitting an issue with cluster-api's ability to roll the control plane nodes. This appears to be because of how we bind the EIP to the node via the lo:0 interface and how cluster-api tears down the node's etcd instance before fully draining the workloads running on the node.
To reproduce, bring up a 1 node control plane with a 1 node worker node and then edit the KubeadmControlPlane's kubeadmConfigSpec changing the postKubeadmCommands to include an echo
or other innocuous addition.
The new control plane node will begin deployment and eventually come into service. At some point, cluster-api kills the etcd of the older control plane node and the Packet CCM EIP health check moves the EIP to the new node. Once the etcd goes away, the kube-apiserver panics and the node stalls being unable to reach the EIP which is bound locally with no running kube-apiserver.
After several minutes, on the new control plane you can see various pods stuck in Terminating and/or Pending state. The cluster-api will not progress past this point.
# k get -A pods -o wide | grep -v Running
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
core cert-manager-webhook-69c8965665-49cfh 1/1 Terminating 0 11h 240.0.18.144 k8s-game-cp-1d5ce5-6wnjj <none> <none>
kube-system cilium-operator-7597b4574b-bg94f 1/1 Terminating 0 11h 10.66.5.5 k8s-game-cp-1d5ce5-6wnjj <none> <none>
kube-system cilium-operator-7597b4574b-nlbjw 0/1 Pending 0 20m <none> <none> <none> <none>
kube-system cilium-sjtk8 0/1 Pending 0 28m <none> <none> <none> <none>
kube-system coredns-66bff467f8-jznm9 1/1 Terminating 0 11h 240.0.18.145 k8s-game-cp-1d5ce5-6wnjj <none> <none>
kube-system coredns-66bff467f8-s77cv 0/1 Pending 0 20m <none> <none> <none> <none>
topolvm-system controller-7d85c6bbbc-8ps5q 0/5 Pending 0 20m <none> <none> <none> <none>
topolvm-system controller-7d85c6bbbc-ppvvz 5/5 Terminating 0 11h 240.0.18.12 k8s-game-cp-1d5ce5-6wnjj <none> <none>
To get things moving again you have to go onto the old control plane node and ip addr del <EIP>/32 dev lo
. Once this is done, the local kubelet can talk again to the API, the cluster-api evicts the pods, and the old node is deleted.
I believe these issues may be related:
kubernetes-sigs/cluster-api#2937
kubernetes-sigs/cluster-api#2652
As a work around, I created the following script along with a systemd service which gets installed into all control plane nodes. This setup allows the rolling update to occur without manual interaction.
Script:
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
EIP=$1
while true; do
rc=0
curl -fksS --retry 9 --retry-connrefused --retry-max-time 180 https://$EIP:6443/healthz || rc=$?
if [[ $rc -eq 7 ]]; then
echo "removing EIP $EIP"
ifdown lo:0
ip addr del $EIP/32 dev lo || true
break
fi
echo ""
sleep $(($RANDOM % 15))
done
postKubeadmCommands addition:
cat <<EOT > /etc/systemd/system/packet-eip-health.service
[Unit]
Description=Packet EIP health check
Wants=kubelet.service
After=kubelet.service
[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/packet-eip-health.sh {{ .controlPlaneEndpoint }}
[Install]
WantedBy=multi-user.target
EOT
systemctl daemon-reload
systemctl enable packet-eip-health
systemctl start packet-eip-health
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.