kubernetes-retired / kubefed Goto Github PK
View Code? Open in Web Editor NEWKubernetes Cluster Federation
License: Apache License 2.0
Kubernetes Cluster Federation
License: Apache License 2.0
As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.
The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".
Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)
Thanks so much, let me know if you have any questions.
(This issue was generated from a tool, apologies for any weirdness.)
[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md
So that we don't need to vendor the client. This will ease the transition to the CRD-based cluster registry when the time comes.
See also: kubernetes-retired/cluster-registry#225
Write a user guide to document setting up a fedv2 deployment. This guide should be different from the developer guide and the developer guide can reference it to avoid duplicating docs.
At a given time frame, federation might have a set of k8s resources federated. Users can further federate their CRD resources into the same federation. Additionally, with the layered approach of a lower level federated types enabling simple federation and higher level types based on the lower level controllers, some users might want all or subset of APIs and controllers eventually in their federation deployment.
This calls for a configuration mechanism, which can make some APIs (and respective controllers) optional by default, and additionally provide a mechanism to enable/disable individual API resources in the federation control plane.
The mechanism should be easy to configure (for an end user) and the implementation can ideally target the infrasctructure setup post the move to CRD based resources (if that happens).
This issue is to achieve feature parity for external/in-cluster DNS with v1. I believe in use-case form, this would be:
@shashidharatd's current PR (#59) gets us most of the way there - but we need some additional glue between FederatedService and MultiClusterDNSRecord to make this story complete.
ReplicaSchedulingPreferences
is implemented (as in #46) to support deployments using the same name <ns/name>
as the key to target the low level FederatedDeployment
resources (template, override and placement). This needs to be extended and implementation made generic, such that the same API schema can be used to target FederatedReplicaset
also using a targetRef. The following 2 items will need to be implemented for the same.
... for k8s boilerplate; as I understand it, this is a precondition of donation to a kubernetes github org.
Per resource placement, where the corresponding placement resource specifies a list of clusters to place the template is already available. It also makes a lot of sense to have a default behavior for all resources in the abscence of this per object placement specification (as alluded in #75).
There additionally is a need for a higher level placement intent that a user can specify. Although there could be more extensive placement policies that can be defined, this issue lists the need of a simple label selector based placement intent useful to the end user.
A simple spec for this resource (names could change) could be as below:
apiVersion: placement.federation.k8s.io/v1alpha1
kind: Preference
metadata:
namespace: my-ns
name: my-placement-preference
spec:
# Select resources in the namespace whose labels match
resourceSelector: app=myapp
# Select labelled clusters
clusterSelector: region=eu, zone=east
The current dynamic client in use has now been deprecated. We need to update the client-go vendored dependency and update the dynamic client to use the new version. See kubernetes/kubernetes#63446 for more details.
View Pods cross clusters.
/cc @marun
This should be overrides
when serialized, per: https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#naming-conventions
Currently PropagatedVersion
resources are named according to the scheme - (e.g. a Secret
named Foo would have a PropagatedVersion
named secret-foo). This scheme effectively limits the length of resource names to 253 (the max length allowed by k8s) - len(type name) - 1 (for the dash). A resource name longer than this limit could result in an ambiguous PropagatedVersion
name, which is likely to invalidate version-based comparison for affected resources.
Given that PropagatedVersion
is not intended to be user-facing, a potential solution could be using a hash of the resource name (e.g. -).
Federated ingress is a much sought after feature. There was a hacky implementation for it in federation v1 which supported only google environment. There is a tool(kubemci) which can setup ingress across multicluster which again works only for google environment. Also recently there was an excellent demo from @nikhiljindal and @G-Harmon in kubecon eu regarding Multicluster ingress which again works in google environment.
We need a similar feature which support other environments (cloud and on-prem).
Any idea(s), suggestions regarding this feature, please pitch in.
What happened:
pre-commit.sh failed with The command "./scripts/pre-commit.sh" exited with 1.
during PR merge that didn't change any files involved with pre-commit.
Upon closer inspection, the FederatedDeploymentOverride test was the site of failure. with E0622 19:32:33.123174 7101 runtime.go:66] Observed a panic: "close of closed channel" (close of closed channel)
=== RUN TestFederatedDeploymentOverride
Running Suite: FederatedDeploymentOverride Suite
================================================
Random Seed: 1529695950
Will run 1 of 1 specs
[::]:34070
[::]:39911
[::]:36361
I0622 19:32:31.545945 7101 serving.go:295] Generated self-signed cert (/tmp/apiserver-test619575690/.crt, /tmp/apiserver-test619575690/.key)
W0622 19:32:31.822413 7101 authorization.go:34] Authorization is disabled
W0622 19:32:31.822450 7101 authentication.go:56] Authentication is disabled
I0622 19:32:32.734425 7101 serve.go:89] Serving securely on [::]:36361
2018-06-22 19:32:32.953433 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:32.953859 I | Validating fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:32.984759 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:32.996063 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.005890 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.009944 I | Running reconcile FederatedDeploymentOverride for instance-1
2018-06-22 19:32:33.102584 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.105992 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.110035 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.110369 I | Defaulting fields for FederatedDeploymentOverride instance-1
2018-06-22 19:32:33.110771 I | Defaulting fields for FederatedDeploymentOverride instance-1
•E0622 19:32:33.123174 7101 runtime.go:66] Observed a panic: "close of closed channel" (close of closed channel)
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/asm_amd64.s:573
/home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/panic.go:502
/home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/chan.go:333
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache/shared_informer.go:529
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/asm_amd64.s:2361
panic: close of closed channel [recovered]
panic: close of closed channel
goroutine 1241 [running]:
github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x17bef60, 0x447d820)
/home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/panic.go:502 +0x229
github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc420a98d00)
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache/shared_informer.go:529 +0x29c
github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache.pop)-fm()
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388 +0x2a
github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc4214abcc8, 0xc4222a8d70)
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
/home/travis/gopath/src/github.com/kubernetes-sigs/federation-v2/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
FAIL github.com/kubernetes-sigs/federation-v2/pkg/controller/federateddeploymentoverride 3.184s
=== RUN TestFederatedDeploymentPlacement
Running Suite: FederatedDeploymentPlacement Suite
=================================================
`` `
</details>
Resource placement currently requires resource-specific placement. To be propagated to a given cluster, a template must have that cluster listed in its associated placement resource, and the containing namespace must also have a placement resource listing that cluster.
It would be desirable to make namespace-based placement the default. If the placement for a namespace specified a given cluster, all federated resources in that namespace would be propagated to that cluster if resource-specific placement was not present.
Where more fine-grained control is required, resource-specific placement could be employed. Where both namespace and resource-specific placement resources existed, placement could be determined either additively (namespace clusters + resource clusters) or exclusively (resource clusters only). Both namespace and resource-specific placement resources could indicate via a boolean whether propagation should be restricted (e.g. propagate: true|false
).
We should have an official image that we publish for this repo; should be built from master for canary build and then tagged for releases.
At step https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#deploy-the-cluster-registry , it is requested to create storage-provisioner
. Any steps can help me to set up this?
Make sure the storage provisioner is ready before deploying the Cluster Registry.
kubectl -n kube-system get pod storage-provisioner
Hi there, not much an issue but I dont know where else to post it. Where can I find a timeline for when v2 will reach beta stage, if there is one? Also, is there a setup guide for how to create a test cluster with v2?
k8s jobs has already been federated using the simple federation resources (template, placement and override) for jobs. This however does not provide higher level job scheduling logic, which can provide an automated distribution of parallelism and completions for the given job across federated clusters.
This particular use case seems quite valuable and calls for a higher level job scheduling API (aka JobSchedulingPreferences
) and an associated controller logic.
This also provides feature parity with federation V1.
The 0.0.5 version of the cluster-registry contains the CRD refactor and a new generated client (which should be very close to the old one, if not identical). We should revendor this version so that we're working with the current version of the cluster registry.
The controller manager currently runs all controllers. Ideally it would be possible to control whether each controller is started. Flags is a quick-n-dirty solution to this, and probably preferable if controllers are likely to live in separate repos post-alpha. A more maintainable solution might be storing configuration as ConfigMap (or whatever the current best practice is for Kube components).
This can make sure have an overview of all the nodes in different clusters.
/cc @marun comments for this?
cluster-registry kubebuilder has been upgraded to 0.1.12 at kubernetes-retired/cluster-registry#244 , it is better upgrade federation v2 to this version as well.
/cc @marun
We should have an easy way to implement DNS provider integration for multi-cluster service DNS.
Goals:
Note: names have been chosen carelessly to avoid wasting time, we can figure out best names later
TLDR: The work common to every implementation of multicluster DNS can be done by a controller that watches Kubernetes Service
resources in multiple clusters and maintains a resource (for sake of discussion, let's call it MultiClusterServiceDNS
) where each instance of the resource is DNS information for a single service. Vendors implement controllers that watch MultiClusterServiceDNS
and program their DNS solution.
The MultiClusterServiceDNS
programs the dns controller to monitor a set of services. Here's a very simple example with toy semantics to illustrate the idea.
Resources of this type are aligned with K8s services via name/namespace association:
metadata.namespace
is the namespace to monitor in each specified clustermetadata.name
is the name of the service in the namespace to monitor in each specified clusterThe spec contains information about the clusters to monitor and how to program DNS:
spec.dnsName
is the DNS name to programspec.clusters
is a list of clusters to monitorThese are just toy semantics for illustration purposes; actual semantics TBD.
kind: MultiClusterServiceDNS
apiVersion: dns.multicluster.k8s.io
metadata:
name: my-service
namespace: my-ns
spec:
dnsName: my-service.myorg.org
clusters:
- X
- Y
- Z
status:
dns:
- cluster: us-central-1
ip: 123.4.5.0
- cluster: us-east-1
ip: 4.5.6.7
- cluster: us-west-1
ip: 10.1.254.5
Vendors integrate with this API by writing controllers that list/watch MultiClusterServiceDNS
and program their DNS solutions.
Note: in the above example, a controller maintains the list of (cluster name, IP) pairs in the status of the MultiClusterServiceDNS
resource. It is likely that status
is not the best place for this information. When we discussed this idea in the April 18 2018 meeting of the Federation WG, we discussed a number of possibilities:
MultiClusterServiceDNS
resource's status would contain information on what resourceVersion the controller had seen and whether their was a problem communicating with one of the named clustersAdditionally, we discussed emulating the behavior of the v1 federation dns solution and programming additional DNS names with cluster/zone information, examples:
my-service.my-ns.myfederation.svc.myorg.org
my-service.my-ns.myfederation.svc.us-central-1.myorg.org
my-service.my-ns.myfederation.svc.us-east-1.myorg.org
my-service.my-ns.myfederation.svc.us-west-1.myorg.org
Federation v2 follows a layered approach such that the higher level resources depend on the lower level federated types for propogation and rconciliation of k8s objects into federated clusters.
To create, for example an automated distribution of deployment replicas, an user can only create a FederatedDeployment
and ReplicaSchedulingPreference
(as defined in #46). The higher level RSP resource controller will observe this and create an override (FederatedDeploymentOverride)
and a placement (FederatedDeploymentPlacement)
to reconcile this user intent into clusters. The RSP controller can further update the corresponding override and template to reconcile cluster capacity and/or clusters leaving or joining this federation.
A peculiar case is when the higher level resource is deleted; should the controller also delete the corresponding override and placement resources or leave them behind? This becomes peculiar, because the lower level resources (override and placement) might have existed prior to creation of the higher level resource, in which case the higher level resource, adopted and updated them, if needed.
The answer probably is to use a cascading deleting mechanism and/or a parentRef (ObjectMeta.OwnerReferences) in such a scenario.
The user can specify --cascade=true/false (with some default) to control this behaviour.
Currently namespace placement is cluster-scoped, but this complicates the application of fine-grained authz. If only the 'tenant' for a given namespace should be allowed to affect its placement, then an explicit rbac rule for just that placement resource would be required. Changing the namespace placement type to be namespaced would allow for namespace-based authz instead.
Created after discussion in Jun 13 2018 WG meeting. This issue is to explore ways to allow people to deploy workloads expressed in 'normal' k8s resources to federation with minimal or no modification. There's a very wide spectrum on things that we could do here, and there may be more than one thing that makes sense to do to provide different types of facilitation (ie, transforming YAML resources to having direct integration in tools like helm, to any number of other things).
The constants used here are (or can be used) in multiple controllers. It makes sense to move them, and any such similar code to a common location, to reduce duplication.
A goal in transitioning to CRDs (depends on #74) is to maintain the same quality of validation for the resources embedded in templates as is currently provided by first-class API types. Potential strategies:
FederatedSecret
), find the FederatedTypeConfig
that specifies the template type.FederatedTypeConfig
, retrieve the OpenAPI spec.imagePullPolicy
being either Always
, Never
, or IfNotPresent
. This may require updating the OpenAPI generation utility.It should be possible to extend the basic mechanism of federation of a new type via a set of CRDs:
It would be cool if kubefnord would generate CRD templates for you.
Currently, the scripts/deploy_federation.sh
runs apiserver-boot run in-cluster
to deploy the federation bits into a cluster. We should instead have a set of resources (statically defined is fine at first, we might consider a helm chart or other form of packaging in the future) that can be deployed without requiring the user to have a dependency installed.
This likely depends on #82 to be usable.
Going through instructions at https://github.com/font/k8s-example-apps/tree/master/sample-fed-v2-demo, particularly step https://github.com/font/k8s-example-apps/tree/master/sample-fed-v2-demo#update-federatednamespaceplacement, the test-namespace
is successfully removed from us-east
after updating the namespace placement to remove us-east
from the list of clusters.
But when the cluster us-east
is added back in, the test-namespace
does not get recreated in the us-east
cluster causing all the other resources to no get added back due to for example:
E0426 19:44:46.756125 1 controller.go:617] Failed to execute updates for FederatedReplicaSet "test-namespace/test-replicaset": [Failed to create FederatedReplicaSet "test-namespace/test-replicaset" in cluster us-east: namespaces "test-namespace" not found]
Waiting long enough will eventually see the test-namespace
created, possibly because the watch operation expires and the controller retries a list operation.
If you restart the federation pod, the test-namespace
will quickly reappear in the us-east
cluster after the controller receives the list of namespaces and namespace placements. The rest of the resources (configmap, secret, replicaset) then reappear as well once the test-namespace
exists.
Follow-on to #101; this is to add (where it makes sense) additional configuration parameters beyond basic feature gating. For example: the push reconciler is enabled/disabled by feature gate, but the resync duration for the push reconciler would be configured by a distinct component configuration API field mapped to a CLI flag.
There are two flavors of policy that we’ve discussed in Federstion WG:
This issue deals with flavor (1)
We could have a very generic validation webhook that could apply label-based policy decisions to any resource. As per #34, you should be able to apply policy to new types without doing a lot of work.
kubefed
can join clusters but not unjoin them, complicating roundtripping of federation deployment.
root@gyliu-dev1:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefnord join cluster138.k8s.local --host-cluster-context cluster138.k8s.local --add-to-registry --v=2
ERROR: logging before flag.Parse: I0705 06:17:36.583065 3075 join.go:125] Defaulting cluster context to joining cluster name cluster138.k8s.local
ERROR: logging before flag.Parse: I0705 06:17:36.583150 3075 join.go:129] Args and flags: name cluster138.k8s.local, host: cluster138.k8s.local, host-system-namespace: federation-system, kubeconfig: , cluster-context: cluster138.k8s.local, secret-name: , dry-run: false
ERROR: logging before flag.Parse: I0705 06:17:36.589103 3075 join.go:183] Performing preflight checks.
ERROR: logging before flag.Parse: I0705 06:17:36.611982 3075 join.go:270] Registering cluster with the cluster registry.
ERROR: logging before flag.Parse: I0705 06:17:36.621322 3075 join.go:278] Registered cluster with the cluster registry.
ERROR: logging before flag.Parse: I0705 06:17:36.621351 3075 join.go:204] Creating federation-system namespace in joining cluster
ERROR: logging before flag.Parse: I0705 06:17:36.636999 3075 join.go:212] Created federation-system namespace in joining cluster
ERROR: logging before flag.Parse: I0705 06:17:36.637048 3075 join.go:215] Creating cluster credentials secret
ERROR: logging before flag.Parse: I0705 06:17:36.637059 3075 join.go:404] Creating service account in joining cluster
ERROR: logging before flag.Parse: I0705 06:17:36.652299 3075 join.go:413] Created service account in joining cluster
ERROR: logging before flag.Parse: I0705 06:17:36.652333 3075 join.go:415] Creating role binding for service account in joining cluster
ERROR: logging before flag.Parse: I0705 06:17:36.676963 3075 join.go:525] Could not create cluster role binding for service account in joining cluster: ClusterRoleBinding.rbac.authorization.k8s.io "federation-controller-manager:cluster138.k8s.local-cluster138.k8s.local" is invalid: subjects[0].namespace: Required value
ERROR: logging before flag.Parse: I0705 06:17:36.677000 3075 join.go:420] Error creating role binding for service account in joining cluster: ClusterRoleBinding.rbac.authorization.k8s.io "federation-controller-manager:cluster138.k8s.local-cluster138.k8s.local" is invalid: subjects[0].namespace: Required value
ERROR: logging before flag.Parse: I0705 06:17:36.677011 3075 join.go:221] Could not create cluster credentials secret: ClusterRoleBinding.rbac.authorization.k8s.io "federation-controller-manager:cluster138.k8s.local-cluster138.k8s.local" is invalid: subjects[0].namespace: Required value
ERROR: logging before flag.Parse: F0705 06:17:36.677025 3075 join.go:105] error: ClusterRoleBinding.rbac.authorization.k8s.io "federation-controller-manager:cluster138.k8s.local-cluster138.k8s.local" is invalid: subjects[0].namespace: Required value
Deployments like other resources could be federated using the template, placement and overrides. The capability of distributing non-equal number of replicas per deployment per cluster as an user intent can be specified using overrides. However a higher level automated control loop which watches the clusters and pods health per deployement per cluster, and distrbutes (and further redistributes if need be) the replicas has its own usages and benefits.
An API spec for a resource where an user can specify his/her intent, in terms of capping the replicas (min/max) per cluster and weights per cluster is certainly useful.
This resource also provides functional parity wrt federation v1 as deployment scheduling type.
The spec for this resource is already specified in 4c96be3.
@marun @font @onyiny-ang
@pmorie FYI
While I was trying to add Completions
to FederatedJobClusterOverride
type I realized that with today code it's impossible to override an immutable field.
I see this as a gap. My understanding is that overridden resource should be directly created while looks like they're updated.
Modified code is in this branch: https://github.com/sdminonne/federation-v2/tree/bug_fix
I'm investigating this issue, but it would be great if you may comment.
Thanks
Related to #34; assuming that we have a CRD-based way to extend federation by creating some new CRD resources for:
...then there should be an API resource to have these CRDs created automatically. Example:
kind: FederatedAPI
metadata:
name: statefulset
spec:
basis:
group: apps
version: v1
kind: StatefulSet
target:
group: federation.k8s.io
version: v1alpha1
A controller would list/watch this API resource and ensure that the correct CRDs were created for the named basis
in the API group/version described by `target. In this example those would be:
FederatedStatefulSet
(possibly creating the OpenAPI schema and validations based on the OpenAPI for the basis type)FederatedStatefulSetOverride
FederatedStatefulSetPlacement
In order to have automatic federation of all the registered types in an API server, there could also be a controller that examines the discovery endpoint for a cluster and ensures a FederatedAPI
exists for the right API services. There are a number of ways a controller might be configured, like:
APIService
and CustomResourceDefinition
resourcesidea formulated with @marun
Work Items
Seems like we can do a lot better than fnord w.r.t. understandability
https://en.wikipedia.org/wiki/Fnord
kubefed ?
fedctl ?
Some higher level resources need e2e tests for actual end to end functionality validation. Below is the list of current such resources which certainly need additional e2e tests:
The implementation of these tests is dependent on #89 (or some other strategy, for example self validation external to CI using managed clusters or BYOC).
How to trigger a CI build if one fails due to temporary issue in network? Here is one such failure and i am unaware of the method to re-trigger the build. This could be a frequent issue to a developer.
/cc @kubernetes-sigs/federation-wg
Federation v2 is currently based on apiserver-builder, which automates the creation of API types and deployment of an aggregated API server to expose created types. An aggregated API server imposes considerable operational cost and complexity, and it is desirable to remove this requirement in preparation for an alpha release. kubebuilder provides a CRD-based alternative to apiserver-builder to transition to. The following strategy is suggested:
FederatedCluster
, PropagatedVersion
, FederatedTypeConfig
) to CRDA goal in transitioning to CRDs is to maintain the same quality of validation for the resources embedded in templates as is currently provided by first-class API types. The following strategy is suggested:
FederatedSecret
), find the FederatedTypeConfig
that specifies the template typeFederatedTypeConfig
, retrieve the OpenAPI specCurrently, the setup for integration and e2e is quite similar, as in only the k8s API server is run to emulate an actual cluster. This is probably suitable for majority cases, but results in limited testing scope especially in cases where feedback from actual resources run time is needed for functionality (eg, higher level scheduling type, service dns type, .. ). Also at some point the overall CI will need to involve some infrastructure which needs actual clusters for e2e validation.
This point was already discussed in federation workgroup sync and captured in notes here.
This issue is to have a trackeable location for the same.
Its easy to federate a given k8s resource with the current federation v2 infrastructure. Federated HPA is also no exception. Default propagation of hpa with possible overrides per cluster (as with using template, overrides and placements) has some, but limited usage. A higher level HPA scheduling API and controller with extended logic makes for a feature set with more usage and value.
A suitable resource name for this could be HPASchedulingPreferences
(similar to ReplicaSchedulingPreferences
) under group federatedscheduling
.
This also provides feature parity with federation V1.
Integration test seems to be flaky and randomly failing. I have noticed this issue quite a few times. Here is one such example, where the code in PR have no changes to integration tests. https://travis-ci.org/kubernetes-sigs/federation-v2/builds/381710195
--- FAIL: TestCrud (64.61s)
logger.go:52: Starting a federation of 2 clusters...
logger.go:52: Added cluster test-cluster-j9hwz to the federation
logger.go:52: Added cluster test-cluster-8wp7f to the federation
logger.go:52: Starting cluster controller
logger.go:48: Federation started.
--- FAIL: TestCrud/FederatedConfigMap (10.53s)
logger.go:52: Creating new FederatedConfigMap in namespace "1ac64e3e-300a-4e9e-b7da-0f528cad036f"
logger.go:52: Created new FederatedConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:52: Creating new FederatedConfigMapPlacement in namespace "1ac64e3e-300a-4e9e-b7da-0f528cad036f"
logger.go:52: Created new FederatedConfigMapPlacement "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:52: Creating new FederatedConfigMapOverride in namespace "1ac64e3e-300a-4e9e-b7da-0f528cad036f"
logger.go:52: Created new FederatedConfigMapOverride "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:48: Resource versions for 1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l: template "79", placement "81", override "83"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" in cluster "test-cluster-8wp7f"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" in cluster "test-cluster-j9hwz"
logger.go:52: Updating FederatedConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedConfigMapPlacement "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for ConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l" to be deleted from cluster "test-cluster-8wp7f"
logger.go:52: Deleting FederatedConfigMap "1ac64e3e-300a-4e9e-b7da-0f528cad036f/test-crud-p762l"
logger.go:44: Expecting PropagatedVersion configmap-test-crud-p762l to be deleted
--- PASS: TestCrud/FederatedDeployment (3.06s)
logger.go:52: Creating new FederatedDeployment in namespace "cf155248-e7ec-484b-a294-1fcbd370315c"
logger.go:52: Created new FederatedDeployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
logger.go:52: Creating new FederatedDeploymentPlacement in namespace "cf155248-e7ec-484b-a294-1fcbd370315c"
logger.go:52: Created new FederatedDeploymentPlacement "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
logger.go:52: Creating new FederatedDeploymentOverride in namespace "cf155248-e7ec-484b-a294-1fcbd370315c"
logger.go:52: Created new FederatedDeploymentOverride "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
logger.go:48: Resource versions for cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4: template "106", placement "108", override "109"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedDeployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedDeploymentPlacement "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for Deployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4" to be deleted from cluster "test-cluster-8wp7f"
logger.go:52: Deleting FederatedDeployment "cf155248-e7ec-484b-a294-1fcbd370315c/test-crud-r5kn4"
--- PASS: TestCrud/FederatedJob (2.88s)
logger.go:52: Creating new FederatedJob in namespace "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a"
logger.go:52: Created new FederatedJob "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
logger.go:52: Creating new FederatedJobPlacement in namespace "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a"
logger.go:52: Created new FederatedJobPlacement "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
logger.go:52: Creating new FederatedJobOverride in namespace "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a"
logger.go:52: Created new FederatedJobOverride "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
logger.go:48: Resource versions for a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g: template "142", placement "144", override "145"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedJob "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" in cluster "test-cluster-8wp7f"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" in cluster "test-cluster-j9hwz"
logger.go:52: Updating FederatedJobPlacement "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for Job "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g" to be deleted from cluster "test-cluster-8wp7f"
logger.go:52: Deleting FederatedJob "a83a87f5-f4f0-4f1e-90e9-35e2b0c79e2a/test-crud-fmw4g"
--- FAIL: TestCrud/FederatedReplicaSet (4.03s)
logger.go:52: Creating new FederatedReplicaSet in namespace "cbb25e54-72ec-4719-8858-a9d5f50cbb0f"
logger.go:52: Created new FederatedReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:52: Creating new FederatedReplicaSetPlacement in namespace "cbb25e54-72ec-4719-8858-a9d5f50cbb0f"
logger.go:52: Created new FederatedReplicaSetPlacement "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:52: Creating new FederatedReplicaSetOverride in namespace "cbb25e54-72ec-4719-8858-a9d5f50cbb0f"
logger.go:52: Created new FederatedReplicaSetOverride "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:48: Resource versions for cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24: template "170", placement "172", override "174"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" in cluster "test-cluster-j9hwz"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" in cluster "test-cluster-8wp7f"
logger.go:52: Updating FederatedReplicaSetPlacement "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" to be deleted from cluster "test-cluster-8wp7f"
logger.go:52: Waiting for ReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24" in cluster "test-cluster-j9hwz"
logger.go:52: Deleting FederatedReplicaSet "cbb25e54-72ec-4719-8858-a9d5f50cbb0f/test-crud-9wq24"
logger.go:44: Expecting PropagatedVersion replicaset-test-crud-9wq24 to be deleted
If a cluster join failed, then there will be some garbage data, such as servieAccount, cluster etc, we should make join
delete those garbage resources when join
failed.
There should be a relatively simple walkthrough that guides users through the experience of kicking the tires on federation-v2 by deploying a very simple application. Other attributes of this:
In 0.0.5, the cluster-registry becomes namespaced; currently, our FederatedCluster
resource is cluster-scoped. We should align FederatedCluster
with Cluster
by making it namespaced. As part of this, we should decide how federation should use the 'globally readable' namespace in the cluster-registry.
In step https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#automated-deployment ,
If you would like to have the deployment of the cluster registry and federation-v2 control plane automated, then invoke the deployment script by running:
./scripts/deploy-federation.sh cluster2
Where is in the form //: e.g. docker.io//federation-v2:test.
Where to get the image of federation-v2:test
?
Any steps for me to build this image?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.