When a part of the deployed Pods is in the pending state, the resources of this part of the deployed Pods are considered to be unavailable resources. When calculating the available resources, the available resources may be less than 0 if the resource request of pending pod is very large.

The error message for failed controller health checks should be more explicit, instead of just "controller not ready".

Currently, there's no Reason field in the plugin's response structs. We have added reason reporting in #231 and the reasons reported by built-in filter plugins would get reported in an event. We should allow webhook plugins to return reasons too.

Description

Run make local-up

What happended：

Fails with

[+] Building 925.5s (9/13)                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 798B                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/debian:buster                                                                              95.4s
 => [internal] load metadata for docker.io/library/golang:1.19                                                                               127.1s
 => [internal] load build context                                                                                                              0.1s
 => => transferring context: 40.03kB                                                                                                           0.0s
 => ERROR [builder 1/4] FROM docker.io/library/golang:1.19@sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a           798.4s
 => => resolve docker.io/library/golang:1.19@sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a                           0.0s
 => => sha256:1611d4f97c5ab666d3c123d72b3bb646dd12bbf7577dd1388fdb191d54cdf440 1.58kB / 1.58kB                                                 0.0s
 => => sha256:80b76a6c918cba6d8e68fb2b80a7afdd7ce3af457d87e413d985434ae7897533 6.87kB / 6.87kB                                                 0.0s
 => => sha256:012c0b3e998c1a0c0bedcf712eaaafb188580529dd026a04aa1ce13fdb39e42b 49.56MB / 49.56MB                                              37.1s
 => => sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a 2.36kB / 2.36kB                                                 0.0s
 => => sha256:00046d1e755ea94fa55a700ca9a10597e4fac7c47be19d970a359b0267a51fbf 24.03MB / 24.03MB                                              34.5s
 => => sha256:9f13f5a53d118643c1f1ff294867c09f224d00edca21f56caa71c2321f8ca004 0B / 64.11MB                                                  798.3s
 => => sha256:190fa1651026077cee00b53a754acbe0dc734b99255c6b274e798f6cd877ae18 92.27MB / 92.27MB                                             175.2s
 => => extracting sha256:012c0b3e998c1a0c0bedcf712eaaafb188580529dd026a04aa1ce13fdb39e42b                                                      0.5s
 => => sha256:0808c64687902329ac64331848304f6f0bc14b86d6be46bccf79292b564f6587 149.14MB / 149.14MB                                           252.0s
 => => extracting sha256:00046d1e755ea94fa55a700ca9a10597e4fac7c47be19d970a359b0267a51fbf                                                      0.2s
 => => sha256:5ec11cb68eac452710eadb46df5e6cf6ead699755303758bf1e262e47b013417 155B / 155B                                                   205.6s
 => CACHED [stage-1 1/4] FROM docker.io/library/debian:buster@sha256:46ca02d33c65ab188d6e56f26c323bf1aa9a99074f2f54176fdc3884304f58b8          0.0s
 => [stage-1 2/4] RUN if [ "cn" = "cn" ]; then sed -i 's#http://deb.debian.org#http://mirrors.tuna.tsinghua.edu.cn#g' /etc/apt/sources.list;   0.3s
 => [stage-1 3/4] RUN apt-get update -y && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*                                   5.0s
------
 > [builder 1/4] FROM docker.io/library/golang:1.19@sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a:
------
WARNING: buildx: failed to read current commit information with git rev-parse --is-inside-work-tree
Dockerfile:2
--------------------
   1 |     # multi-arch image building for kubeadmiral-controller-manager
   2 | >>> FROM --platform=${BUILDPLATFORM} golang:1.19 as builder
   3 |     ARG TARGETPLATFORM GOPROXY BUILD_FLAGS
   4 |     ADD . /build
--------------------
ERROR: failed to solve: failed to read expected number of bytes: unexpected EOF
make[1]: *** [Makefile:113: images] Error 1
make[1]: Leaving directory '/home/larry/code/kubeadmiral'
make: *** [Makefile:83: local-up] Error 2
root:kubeadmiral/ # client_loop: send disconnect: Connection reset                                                                       [13:33:25]

Then

And when I replace golang:1.20 with golang:1.19 in https://github.com/kubewharf/kubeadmiral/blob/main/hack/dockerfiles/Dockerfile#L2
It works and successfully deploy KubeAdmiral.
What happend?

Hi, the official documentation here states that it currently supports running with K8s version 1.24. Considering, the Kubernetes ecosystem moves very fast with at-least 3 releases per year, are there any plans to support higher versions?

I find kubeadmiral transfer native k8s api to fedrate type, now support no native k8s crd, like argo workflow or volcano job?

PR #46 introduces a custom pod informer that transform pod objects before caching.

With support for pod federation, we need to ensure that this custom pod informer is disabled for certain pod subcontrollers to work properly, e.g. sync, status

The unused functions are legacy we inherited from KubeFed. We should verify if we still need them and remove them as needed.

InformerManager and FederatedInformerManager manage informers for the host and member clusters, respectively, based on FederatedTypeConfigs. For each FederatedTypeConfig, the managers start an informer for the corresponding type.

If the informer for a type cannot be started or its cache cannot be synced (be it for host cluster or member clusters), we should expose this information for visibility. The current mechanism for doing so is by including the information in the response returned by the controller manager's readyz endpoint. Refer to existing controller's readyz implementation for how one can implement cache sync check for InformerManager and FederaterdInformerManager.

I followed the quick start doc, in step 3("Bootstrap Kubernetes clusters") of the prerequisites session, I run make kind command, everything went well, but no "$HOME/.kube/kubeadmiral/kubeconfig.yaml" file generated as expected, as bellows:

% KUBECONFIG=$HOME/.kube/kubeadmiral/kubeconfig.yaml kubectl config get-clusters

W0504 15:22:12.931407   22785 loader.go:223] Config not found: /Users/bytedance/.kube/kubeadmiral/kubeconfig.yaml
NAME

% ls -l $HOME/.kube/kubeadmiral/kubeconfig.yaml
ls: /Users/bytedance/.kube/kubeadmiral/kubeconfig.yaml: No such file or directory


% ls -l $HOME/.kube/kubeadmiral/

total 64
-rw-------  1 bytedance  staff  5586  5  4 11:44 kubeadmiral-host.yaml
-rw-------  1 bytedance  staff  5610  5  4 11:43 kubeadmiral-member-1.yaml
-rw-------  1 bytedance  staff  5606  5  4 11:44 kubeadmiral-member-2.yaml
-rw-------  1 bytedance  staff  5606  5  4 11:44 kubeadmiral-member-3.yaml

The APIs use a few deprecated kubebuilder marker comments, which are confusing and verbose. We should remove/replace them with currently supported alternatives.

Ref: https://book.kubebuilder.io/migration/legacy/migration_guide_v1tov2.html

At the time of creating this issue, we only support status aggregation for Deployments.

• StatefulSet
• Job
• Pod
• CronJob
• DaemonSet
• Service
• Ingress
• PV
• PVC

FederatedObject.spec.template is currently of type k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.JSON which stores the raw json bytes. This type was used because controller-gen refuses to implement support for map[string]interface{} fields for CRD generation as discussed in kubernetes-sigs/controller-tools#636. However, map[string]interface{} would be more efficient and ergonomic as it avoids the unmarshaling whenever we need to access the template. We'd like to use map[string]interface{} if we can find a way to hack controller-gen to generate CRDs nonetheless.

We could start by looking at post-gen patches (which is already done in config/crds/patches). Maybe we could ask controller-gen to ignore the template field and let the post-gen patches add it?

• Scheduler should distinguish between an Unschedulable result and an Error result
• Scheduler should pass all clusters to filter and score webhooks instead of passing one by one
• Each unschedulable cluster should be rejected with a message, and scheduler should aggregate these messages to provide meaningful information to users

I found many simular concepts shared with karmada, anyone can describe the advantage than karmada?

k8s.io/apimachinery/pkg/runtime.Object does not have GetAnnotations and SetAnnotations methods. As a result the annotation util functions in pkg/controllers/util/annotation/annotation.go resort to calling meta.Accessor(obj) to convert the object. But meta.Accessor(obj) returns an error that must be propagated upwards.

In fact, all current usages of the annotation util functions pass in objects that implement metav1.Object interface, which allows calling SetAnnotations and GetAnnotations without returning errors.

The former has not been maintained since 2021.
The latter is not a complete replacement of the former, because the former adds stack traces to errors. However, we are not making use of stack traces in errors.

Should we replace the former with the latter?

This is required to reconcile federated objects that reference a profile which did not exist previously, and the user creates the corresponding new profile to fix it. With the current behavior of the scheduler, it would not automatically reconcile these objects on profile add, and would require a manual trigger.

Context

Presently, our services are uniformly deployed via Helm across two separate Kubernetes clusters catering to distinct business requirements. However, this approach has led to a proliferation of redundant code, specifically concerning the deployment of services into different clusters.

Therefore, I'd like to pose a question: Are we equipped to support Helm's deployment mode effectively? 😉

Because certain controllers (e.g. status aggregation controller) run by the FTC controller wait for cache sync in the main thread, a wrongly configured FTC can prevent the whole FTC controller from progressing.

The image above shows the logs from the FTC controller indicating that it got blocked after starting the status controller.

Steps to reproduce:

1. Run the KubeAdmiral controller manager against a 1.23 Kubernetes host cluster using the default FTCs in config/sample/host/01-ftc.yaml.
2. Because the yaml file contains an older version of CronJob, informers for CronJob will fail to list and their informer cache will never be synced, resulting in the above behavior

link: https://github.com/kubewharf/kubeadmiral/actions/runs/6307418487/job/17124013179?pr=226

The error case is t.Run("ClusterEventHandlers should receive correct old and new clusters", func(t *testing.T).
I ran it locally multiple times and found no similar errors.

I feel that there is a small time difference between the function entering assertionCh and calling callback, which is captured by the test case. I think it would be better to put the callBackCount.Add(1) before the assertionCh operation.

The MaxClusters scheduler plugin does not have any preference for current placements. This undermines the avoidMigration field in propagation policies when maxClusters is set. We can consider adding support for the avoidMigration field.

The new behavior when avoidDisruption == true would look something like:

if len(currentClusters) == maxClusters {
    return currentClusters
} else if len(currentClusters) > maxClusters {
    // remove lowest scoring clusters
    removeLowestScoreClusters(currentClusters, clusterScores, maxClusters)
    return currentClusters
} else {
    // add highest scoring clusters that are not already present
    addHighestScoreClusters(currentClusters, clusterScores, maxClusters)
    return currentClusters
}

The code in the two packages are virtually identical except for the types they handle. Can we combine them, probably using generics?

pkg/client/generic is legacy we inherited from KubeFed. It is an unnecessary layer of abstraction and has different usage patterns from client-go's dynamic client. We'd like to standardize all clients to those provided by client-go.

In quickstart "Source the kubeconfig of the host cluster", shall I define HOST_CLUSTER_KUBECONFIG and HOST_CLUSTER_CONTEXT in my bash profile manually?

What would you like to be added?

Provide proxy API for users to access member cluster resources

Why is this needed?

Users may need to check the distribution of application resources in each member cluster. And Users do not want to log in to each cloud provider’s website or switch the kubeconfig context. It will greatly improve the convenience for users to use kubeAdmiral if we provide the proxy APIs to access member cluster resources.

What would you like to be added?

Enhance the admiralctl command-line tool to support joining or unjoining member cluster.

Why is this needed?

Currently, users need to create auth secret and FederatedCluster resource manually. when joining a member cluster. It will greatly improve the convenience for users to use kubeAdmiral if we provide the joining/unjoining command-line tool.

Sometimes we append to these global slices. If a slice has cap > len, then appending to it would mutate the underlying storage. ​https://go.dev/play/p/f5YRITlfiGj. (Golang doesn't seem to guarantee that []int{2,3} must have a capacity of 2.)

If we append to these global slices on different goroutines, it would result in a data race.

Karmada documents : https://karmada.io/docs/

Currently, ClusterResourcesFit filter plugin does not work because it cannot get resourceQuest of object.
We need to interpret the logic of calculating resourceQuest according to GVK.
We used ftc's pathDefinition to interpret replicas before, but for CRD, the process of calculating resources may be a bit complicated. We may need hook functions to allow users to customize the calculation logic of resource requests.

As the fields of policy API expand, there are more and more composite scenarios that require verification.
Though we can disable user misconfiguration at runtime, it is not as effective as blocking user input at the source.
At the same time, users configure fields in the API, but it does not take effect, which also makes users confused.

For example，
SchedulingMode is Duplicate, will not work with replicaStrategy.
ReplicaStrategy is spread, will not work with maxClusters.

Moreover, in the early stages of some features, we can prevent some unpredictable scenarios by limiting its changes.
We will not allow users to make these kinds of modifications until we have fully verified it.

For example, users change replicaStrategy=weighted to binpack.

Currently constants are defined in a couple of places:

• pkg/controllers/common/constants.go
• controller's own package
• pkg/util packages

We should organize them so they are more maintainable. When organizing, we should make sure:

• the constants are exposed to all packages in the project and not just for pkg/controllers
• the constants are categorized properly
• unused constants should be removed

The following controllers have to be refactored to follow the new logging conventions in the code style guide.

This will reduce the amount of noise produced in the controller-manager's logs and make the logs much more readable.

• FederatedCluster controller
• Scheduler
• Sync controller
• Federate controller
• Status controller
• StatusAggregator controller
• Follower controller
• PolicyRC controller