Operator merges the flag from default pod spec and the custom flags provided by user in control plane CRD

KCM currently has no info about the cloud provider. KCM should be configured to be able to talk to cloud provider APIs.
As a result once the nodes are deleted in EC2 corresponding node objects in the kit cluster are not deleted

Today Kit operator limits the ASG size to 1000, which is unrealistic for Megaxl customers.

• Split out tests in the separate repo
• Have sample tests which are periodically run against a KIT cluster and owned by the scalability team
• Documents steps to configure Tekton to test repos

Testbed:

Either moved to a managed policy like AWSLoadBalancerControllerIAMPolicy when it's available in IAM (or) scope down the policies of AWS LB IRSA role using this - https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json

The control plane service name includes - <cluster-name>-controlplane-endpoint and the port referenced is <cluster-name>-controlplane-endpoint-port. In some cases cluster-name can be too long, which creates issues when creating DNS name >63 char.

• storage is GP2 for etcd with default IOPS KIT operator gives is 100 which is far from the EKS default we have today. No wonder, we run into slow disk issues on etcd. We need ability to mention this from KIT CRD spec.

**Additional Requirements that we uncovered for KIT operator needs documentation as part of it's installation process for users using KIT operator to not run into these below issues here :

• NAT Subnet IGW routing requirement
• Karpenter dependency for KIT and min version like 0.4.1
• Custom tags to subnets required as part for KIT/karpenter to launch nodes to schedule KIT CP pods and KIT DP pods
• DNS issue for dataplane nodes created by KIT where pods have wrong DNS IP in resolv.conf

All these need to be documented as part of README ?, as users need to know this at the time of installation of KIT operator for users wanting to use KIT on KOPS, Testbed etc

• Add a field in Dataplane spec to allow user to provide custom AMIs for worker nodes in KIT cluster. Ref- https://github.com/awslabs/kubernetes-iteration-toolkit/pull/47/files#r730299802
• Add a field in Dataplane spec to allow user to provider different instance types for worker nodes in KIT.
• Recycle instance every 30 days to take advantage for EC2 instance discounts.

• ipv6 mode for running guest clusters (more kit clusters per env)
• Use clusterip from Tekton tasks (avoid slow nlb)
• Don't wait in helm (avoid hanging on cluster creation)
• Rename substrate->environment
• Simplify overly parameterized pipelines

After adding the --insecure-port for monitoring in 1.21 API server crashes with this error-

Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 8080: only zero is allowed

When controlplane spec is updated with args, KIT operator recycles all of the System components like apiserver etc at once, this will lead to a outage during the update process.

See below all the apiserver pods for this controlplane are unavailable during update process:

dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2   0/1     ContainerCreating   0          54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt   0/1     ContainerCreating   0          54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg   0/1     ContainerCreating   0          54s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm   1/1     Running             0          64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp   0/1     Pending             0          64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq   0/1     Pending             0          64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0                     1/1     Running             0          100m
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1                     0/1     Pending             0          5s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2                     0/1     Error               1          62s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4   0/1     Pending             0          64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq   1/1     Running             0          64s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz   0/1     Pending             0          64s

dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods | grep c0955120-57bc-420f-bec0-f79d28d86fbe
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b57t5d2   1/1     Running     3          6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b58v2wt   1/1     Running     3          6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-apiserver-d6cb488b5zqqzg   1/1     Running     3          6m4s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-bfnrm   1/1     Running     0          6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-gf4lp   1/1     Running     0          6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-controller-manager-ms4vq   1/1     Running     0          6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-0                     1/1     Running     0          3m25s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-1                     1/1     Running     2          5m15s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-etcd-2                     1/1     Running     4          6m12s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787gwpl4   1/1     Running     0          6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787m57zq   1/1     Running     0          6m14s
s-c0955120-57bc-420f-bec0-f79d28d86fbe-scheduler-7557cb787p82fz   1/1     Running     0          6m14s

Expected: Pods should be updated based on rolling update strategy and should not disrupt current state of cluster.

If we apply the default ControlPlane object, we will launch one apiserver pod and Karpenter will launch one control plane node for this pod. If we scale up the number of replicas in the apiserver deployment from one to three, the two new apiserver pods will be launched on the existing node rather than being assigned their own nodes. Note that creating the ControlPlane object initially with three replicas would result in three nodes being launched.

Reproduction:

1. Create a default ControlPlane object which defaults to one apiserver replica:

$ cat <<EOF | kubectl apply -f -
apiVersion: kit.k8s.sh/v1alpha1
kind: ControlPlane
metadata:
  name: ${GUEST_CLUSTER_NAME} # Desired Cluster name
spec: {}
EOF

2. Edit the object and increase number of apiserver replicas from one to three:

$ kubectl get controlplane example -o yaml
apiVersion: kit.k8s.sh/v1alpha1
kind: ControlPlane
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kit.k8s.sh/v1alpha1","kind":"ControlPlane","metadata":{"annotations":{},"name":"example","namespace":"default"},"spec":{}}
  creationTimestamp: "2022-01-28T21:38:40Z"
  finalizers:
  - kit.k8s.sh/control-plane
  generation: 2
  name: example
  namespace: default
  resourceVersion: "925126"
  uid: 98474866-0d03-443d-a501-bff16a7f8a9b
spec:
  etcd:
    replicas: 3
  kubernetesVersion: "1.19"
  master:
    apiServer:
      replicas: 3
status:
...

3. Check placement of apiserver pods:

$ kubectl describe node ip-192-168-191-222.us-west-2.compute.internal
...
Non-terminated Pods:          (8 in total)
  Namespace                   Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                  ------------  ----------  ---------------  -------------  ---
  default                     example-apiserver-7f588874d9-6qjtq    1 (1%)        0 (0%)      0 (0%)           0 (0%)         25s
  default                     example-apiserver-7f588874d9-7ldgm    1 (1%)        0 (0%)      0 (0%)           0 (0%)         2m3s
  default                     example-apiserver-7f588874d9-j7l7d    1 (1%)        0 (0%)      0 (0%)           0 (0%)         25s
  default                     example-authenticator-wsvxp           0 (0%)        0 (0%)      0 (0%)           0 (0%)         119s
  default                     example-controller-manager-hqrck      1 (1%)        0 (0%)      0 (0%)           0 (0%)         119s
  default                     example-scheduler-5vf8g               1 (1%)        0 (0%)      0 (0%)           0 (0%)         119s
  kube-system                 aws-node-7h2kr                        25m (0%)      0 (0%)      0 (0%)           0 (0%)         77s
  kube-system                 kube-proxy-qwwvb                      100m (0%)     0 (0%)      0 (0%)           0 (0%)         77s

Health-checks are required for KIT control-plane components to not mask issues.

• Update the readme to include steps to install EBS CSI driver
• PVCs are not cleaned up after deleting the control plane object
  We will need to clean them in operator, until this feature is available https://kubernetes.io/blog/2021/12/16/kubernetes-1-23-statefulset-pvc-auto-deletion/
• [Document] PVCs are not configurable for an existing cluster, user will need to delete the cluster and recreate the cluster desired PVCs
  Statefulsets don't allow updating PVC spec - spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"
• Have an option in the control plane spec to let the user create etcd with host volume mounted to the pods

Our customer would like us to provide the following features:

• Expose the APIserver request duration metrics through Prometheus.
• Enable log data analytics

By default APIserver metrics endpoints exposes apiserver_request_duration_seconds_bucket, however, such metrics are not shown in Prometheus. This is likely due to the fact that prometheus add-on helm chart is configured to disable apiserver scraping. How do we enable apiserver scraping for guest cluster will be the tricky part.

To support log analytics, the kit substrate should also deploy ELK stack to collect logs and provide elasticsearch for data analysis.

Ref: #113 (comment)

Because we name DS aws-iam-authenticator server on kit controlplane as aws-iam-authenticator for each KIT control-plane , it's causing issues such as recycling of authenticator pods (as a result, it's causing high STS and ec2DescibeInstance calls )when we deploy more than one KIT controlplane CRD on host cluster testbed.
We need to have a unique name for each aws-iam-authenticator component to prevent such restarts from happening.

dev-dsk-hakuna-2c-dfc2b72b % kubectl get ds 
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                     AGE
aws-iam-authenticator   3         3         3       3            3           kit.k8s.sh/app=s-e74df950-2d04-42c7-9614-af831b0a3ef3-apiserver   3d4h

dev-dsk-hakuna-2c-dfc2b72b % kubectl get pods -o wide | grep aws-iam-authenticator 
aws-iam-authenticator-frhhh                                       1/1     Running     0          3m41s   10.22.237.119   ip-10-22-237-119.us-west-2.compute.internal   <none>           <none>
aws-iam-authenticator-h55sg                                       1/1     Running     0          3m41s   10.22.127.7     ip-10-22-127-7.us-west-2.compute.internal     <none>           <none>
aws-iam-authenticator-k5x55                                       1/1     Running     0          3m41s   10.22.201.10    ip-10-22-201-10.us-west-2.compute.internal    <none>           <none>

• Provision AWS infrastructure required to create Kubernetes clusters
• Create a substrate node with Kubernetes control plane
• Deploy addons such as
  • kube-proxy,
  • DNS
  • CNI
• Deploy required controllers such as Karpenter, KIT-operator

To run the tests successfully we will also need the following tools deployed-

• Prometheus and Grafana stack
• Tekton and Flux to sync the tests

Deploy https://github.com/kubernetes-sigs/aws-encryption-provider to encrypt secrets in the cluster created by kit-operator

Explore adding v1alpha5. Constraints to the dataplane spec for nodes creation.

[Tekton] Enhance Validate cluster step to validate cluster before kicking off test

Currently, we are using strategicpatch.StrategicMergePatch if a user needs to update a flag in etcd or any other component. They will need to provide all the flags in the controlPlane CR instances along with the updated value.

We should be able to merge the flags and use the user provided values for the flags.

[operator] KIT Controller restarting continuously, it couldn't come up healthy.

I1123 15:10:54.158002       1 request.go:655] Throttling request took 1.0440526s, request: GET:https://172.20.0.1:443/apis/authorization.k8s.io/v1?timeout=32s
2021-11-23T15:10:54.261Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-11-23T15:10:54.262Z	INFO	controller-runtime.builder	skip registering a mutating webhook, admission.Defaulter interface is not implemented	{"GVK": "kit.k8s.sh/v1alpha1, Kind=ControlPlane"}
2021-11-23T15:10:54.262Z	INFO	controller-runtime.builder	skip registering a validating webhook, admission.Validator interface is not implemented	{"GVK": "kit.k8s.sh/v1alpha1, Kind=ControlPlane"}
2021-11-23T15:10:54.262Z	INFO	controller-runtime.builder	skip registering a mutating webhook, admission.Defaulter interface is not implemented	{"GVK": "kit.k8s.sh/v1alpha1, Kind=DataPlane"}
2021-11-23T15:10:54.262Z	INFO	controller-runtime.builder	skip registering a validating webhook, admission.Validator interface is not implemented	{"GVK": "kit.k8s.sh/v1alpha1, Kind=DataPlane"}
I1123 15:10:54.263057       1 leaderelection.go:243] attempting to acquire leader lease kit/kit-leader-election...
2021-11-23T15:10:54.263Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I1123 15:11:11.689185       1 leaderelection.go:253] successfully acquired lease kit/kit-leader-election
2021-11-23T15:11:11.689Z	INFO	controller-runtime.manager.controller.dataplane	Starting EventSource	{"reconciler group": "kit.k8s.sh", "reconciler kind": "DataPlane", "source": "kind source: /, Kind="}
2021-11-23T15:11:11.689Z	INFO	controller-runtime.manager.controller.control-plane	Starting EventSource	{"reconciler group": "kit.k8s.sh", "reconciler kind": "ControlPlane", "source": "kind source: /, Kind="}
I1123 15:11:12.738906       1 request.go:655] Throttling request took 1.047619315s, request: GET:https://172.20.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s
2021-11-23T15:11:12.840Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "DataPlane.kit.k8s.sh", "error": "no matches for kind \"DataPlane\" in version \"kit.k8s.sh/v1alpha1\""}
2021-11-23T15:11:15.489Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "ControlPlane.kit.k8s.sh", "error": "no matches for kind \"ControlPlane\" in version \"kit.k8s.sh/v1alpha1\""}
2021-11-23T15:11:15.490Z	ERROR	controller-runtime.manager	error received after stop sequence was engaged	{"error": "no matches for kind \"ControlPlane\" in version \"kit.k8s.sh/v1alpha1\""}
2021-11-23T15:11:15.490Z	ERROR	controller-runtime.manager	error received after stop sequence was engaged	{"error": "leader election lost"}
panic: Unable to start manager, no matches for kind "DataPlane" in version "kit.k8s.sh/v1alpha1"

goroutine 1 [running]:
main.main()
	github.com/awslabs/kit/operator/cmd/controller/main.go:64 +0x86a

Depends on #102. Once this task is complete, we need to add functionality to kitcli to be able to provision KIT clusters

• Configure Prometheus to be able to scrape service and system metrics from tenant control plane components (API server, KCM, scheduler)

[operator] Need image field for Dataplane and Control plane components in the Spec instead of hardcoding them in code in ImageProvider.go, so it would help us from changing code every time we want to test with a new image.

Creating this issue for tracking purposes.

There are two ways to configure the etcd pod-

• When etcd data directory /var/lib/etcd node volume is mounted to the etcd pod, following issues are seen -

  • When an etcd cluster is created and quorum is established, If a node gets terminated new node will be created by karpenter and will have a different peer/member ID and quorum is not established.
  • When an etcd cluster is created and quorum is established, and if statefulset is deleted and recreated on the same set of nodes, new etcd pods come up with different IDs and quorum is never established.

• When etcd data directory /var/lib/etcd is not mounted to the etcd pod

  • If a pod gets restarted, quorum is not established and we see the following error -
    member 86adba6a3f2b2193 has already been bootstrapped

• Configure Prometheus to have persistent storage, in case the pod dies we shouldn't be losing metrics

To run the tests successfully we will also need the following tools deployed-

• Prometheus and Grafana stack
• Install Tekton
• Install Flux
• Configure Tekton

Currently the service cluster IP range in KIT cluster is 10.96.0.0/12, and DNS cluster IP is 10.100.0.10.

A user might have the same underlying subnet in VPC and it can conflict. At some point we should revisit this and detect the VPC cidr block where cluster is running and select another service cluster IP range like 172.16.0.0/12 and DNS cluster IP to be 172.16.0.10

There are cases when someone might loose existing config files on filesyste, running the bootstrap command should first sync these files from S3 to filesystem

• Dataplane spec is mandating to have allocationStrategy in the spec. If not provided it will run into the following issue:

message: "creating auto scaling group for s-9a7d13d2-e635-4418-8852-7d528e66fa69,
     ValidationError: 1 validation error detected: Value '' at 'mixedInstancesPolicy.launchStrategy.onDemandAllocationStrategy'
     failed to satisfy constraint: Member must have length greater than or equal
     to 1\n\tstatus code: 400, request id: 270dd3de-8aa9-4ff2-baeb-81ac9cfd1fd0"
   status: "False"
   type: Active

We need to look into why defaulting doesn't work. If we plan to remove defaulting, we need to update the example spec here] with allocation strategy

[ ] Install Flux or leverage Tekton to sync these configs/tests

We need a way to be able to export and retrieve the metrics from various test runs

https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets

https://karpenter.sh/v0.6.0/tasks/deprovisioning/#disruption-budgets

KIT Limits to have only LB per tested/hostcluster, which means indirectly its restricting us to have only one KIT cluster per testbed/hostcluster because creating multiple KIT clusters require multiple KIT CP LB and it won’t succeed because of the AWS-EKS-LB tags issue (while discovering subnets, it requires only one security group to have kubernetes.io/<cluster-name : owned tag.

We can access Grafana dashboard using the kube-proxy but in scenarios where API server is under load, this becomes a challenge.

• Configure load balancer service to expose Grafana dashboard for the cluster