tolerations & nodeselectors about gatekeeper-library HOT 7 CLOSED

They let you per namespace:
set a default set of tolerations / nodeSelectors to always add to pods if not already there. This lets you easily target pods automatically to pools of nodes. (Mutating webhook style)
Set an allowed set of tolerations / nodeSelectors that can be used. If others are used, they are denied. This prevents a user from using pools of nodes they don't have permissions to. (validating webhook style)

Both are listed here:
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

from gatekeeper-library.

What alpha admission controller are you referencing? Can you expand on what PodTolerationRestriction and PodNodeSelector do/did?

from gatekeeper-library.

The mutating webhooks in gatekeeper don't seem to have the nice librarization that the validating ones do. So for each pool, you'd need a complicated set of rules to make it work.

Probably need to add some kind of liberalization for mutating webhooks as well.

from gatekeeper-library.

As a workaround, until librarization of mutating webhooks is possible, using a RuntimeClass for that seems to work ok:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: foo
handler: default
scheduling:
  nodeSelector:
    pool: foo
  tolerations:
  - key: "pool"
    operator: "Equal"
    value: "foo"
    effect: "NoSchedule"
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: foo-pool-assignment
spec:
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["*"]
      kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
      - key: pool
        operator: In
        values: [foo]
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.runtimeClassName"
  parameters:
    assign:
      value: userv1

Then you label nodes as pool=foo and namespaces as pool=foo

Still need a way to limit which nodeSelectors and tolerations are valid.

from gatekeeper-library.

We are trying to achieve something that can remove the nodeSelector itself to have more scope for the pod placement based on the existing nodes(resources)

from gatekeeper-library.

Sorry, been a while since I've responded, so this will be in response to a few posts...

Set an allowed set of tolerations / nodeSelectors that can be used. If others are used, they are denied. This prevents a user from using pools of nodes they don't have permissions to. (validating webhook style)

Still need a way to limit which nodeSelectors and tolerations are valid.

Anything where there is a "deny if this happens" behavior should be done via validation/constraints, per:

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#guaranteeing-the-final-state-of-the-object-is-seen

so a constraint template would need to be written for that.

set a default set of tolerations / nodeSelectors to always add to pods if not already there. This lets you easily target pods automatically to pools of nodes. (Mutating webhook style)

Setting node selectors should be possible. Defaulting can be done via pathTests to run the check "only apply the nodeSelector mutator if nodeSelector is undefined". Or, you can set individual selectors within the node selector itself.

tolerations is harder because it is a non-keyed list. The ModifySet mutator, which is added in open-policy-agent/gatekeeper#1508 should be able to handle the addition/removal of tolerations.

We are trying to achieve something that can remove the nodeSelector itself to have more scope for the pod placement based on the existing nodes(resources)

What happens if you use the Assign mutator to assign an empty object {} to spec.nodeSelector?

from gatekeeper-library.

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

from gatekeeper-library.