Question: Gatekeeper PSP vs Native PSP about gatekeeper-library HOT 15 CLOSED

I'm a maintainer, so I don't want to give too much weight to my personal opinion.

I did find this AWS blog post comparing Gatekeeper and Pod Security Policy:

https://aws.amazon.com/blogs/containers/using-gatekeeper-as-a-drop-in-pod-security-policy-replacement-in-amazon-eks/

Definitely curious to hear others' thoughts on the topic!

from gatekeeper-library.

Having written some PSP's and now looking at migration, the tricky thing I don't really see off the bat is how Gatekeeper plays with RBAC.

While it was tricky to wrap my head around at first, the native PSP implementation allows you to pass permissions to be able to use PSP's to others by RBAC semantics. So if user Joe has foo PSP support, Joe can create a service account and a role/rolebinding for the service account to be able to use those permissions too. This can allow finer grained permissions and distributed management that may be very difficult with Gatekeeper PSP? Perhaps that really isn't needed though. Not sure...

from gatekeeper-library.

Note that PSPs are to be deprecated from 1.21 kubernetes/kubernetes#97171

from gatekeeper-library.

PSP's use of RBAC was one of the issues people had with it.

There were actually two ways to satisfy the RBAC for PSPs... either via the requesting user, or via the service account running the pod. As long as one of those two users has permissions to "use" a PSP, then the policy is satisfied. IIRC this leads to situations where there are enforcement gaps simply due to these two avenues for satisfying PSP's RBAC binding.

Another issue with user-based admission checks is indirect resource creation. If I create a deployment, it's that deployment's service account that is considered the creator of the pod during admission, not the original creating user. It's also not currently possible to audit ongoing compliance with user-based policies, as the requesting user is only known at request time.

The multi-binding issue is unique to PSP, but the limits on user-based admission policies are fairly universal.

If possible, it's best to write policies against a "known good" or "known bad" state, as those are the most enforceable and auditable.

from gatekeeper-library.

How do you then write a policy that restricts what features a pod can have, since like you said, a lot of k8s objects are indirectly created by users. Do you have to write a policy that checks all possible pod template types in all types that have them?

from gatekeeper-library.

Currently Gatekeeper would reject the pod. Such a state would show up as increased rejections via the Prometheus metrics or as an audit violation if a Pod creation were to sneak through.

Gatekeeper issue 298 proposed making constraints/templates smarter so they can recognize generator resources, but there are a number of problems with that approach. Not the least of which is that kubernetes is extensible... anyone can write their own controller, so we cannot know whether we have full coverage.

I think the most user-friendly solution here would be to perform these kinds of checks as part of a pre-apply pipeline. You could inflate generator resources to create "dummy pods" that would then be subject to checks against Gatekeeper policies.

This would decouple policy from knowledge of how the cluster works, allow users to catch violating generated resources early, and because such a pipeline could be extended to accommodate any sort of "generator" resource, it would play nicely with K8s as an extensible platform.

From my view the gatekeeper-validate kpt function is a step in this direction, as it allows policies to be evaluated as part of a pipeline. However, having a generator pipeline design and a set of implemented generators is work that has yet to be done. I don't know if Gatekeeper is where it should necessarily live, as Gatekeeper seems like one piece of a larger puzzle in this instance.

from gatekeeper-library.

This can allow finer grained permissions and distributed management that may be very difficult with Gatekeeper PSP? Perhaps that really isn't needed though.

I would be interested in understanding if this is a requirement for folks and what are the use cases.

from gatekeeper-library.

+1, there is likely some happy medium between designing the infrastructure around the feature-set and vice versa.

Namespaces are a pretty natural permissions boundary, but they don't necessarily work for everything (e.g. istio sidecars).

from gatekeeper-library.

@maxsmythe @ritazh Is there a way we can tweak the https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/users/template.yaml to exclude istio-init containers from the rule evaluation?

I've been breaking my head around this as istio-init containers need to run as Root and I am not willing to allow running as Root in PSP. After a lot of digging around, I came by Gatekeeper and upon seeing the template file, I wanted to check with you.

from gatekeeper-library.

@maxsmythe @ritazh Is there a way we can tweak the https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/pod-security-policy/users/template.yaml to exclude istio-init containers from the rule evaluation?

I've been breaking my head around this as istio-init containers need to run as Root and I am not willing to allow running as Root in PSP. After a lot of digging around, I came by Gatekeeper and upon seeing the template file, I wanted to check with you.

I don't think that's a good idea. Then your users can just name their containers istio-init and work around the policy.

from gatekeeper-library.

@kfox1111 true. I did not think of that. Can you suggest me something else to get around the Istio issue? I tried applying two different PSPs:

• the one which is more liberal has been tied to the istio service account
• the one which is restricted to every user group

But this did not work either as the init container was using the pod permissions instead of its service account.

I was hoping OPA/Gatekeeper would have some workaround for this.

from gatekeeper-library.

Does it still run as root if your using the cni driver? I know it at least runs unprivileged that way.

from gatekeeper-library.

@kfox1111 I tried installing Istio CNI but that messes up my k8s installation. I am using AWS EKS. And the worker nodes get marked as not ready when I enable Istio CNI on version 1.8.6.

I read that Istio versions 1.6 and above dont need root privs but I cant seem to enable that. It runs with the UID 0. When I set anything but 0, the istio-init container fails stating it cannot get iptables.

from gatekeeper-library.

I don't know much about eks. The istio instructions mention needing cni enabled in eks. Maybe this doc might help: https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html

If you use cni, then it doesn't need privilege, but I think your right. recent versions still maybe use root to run iptables to check if the cni driver successfully injected the iptables rules.

There may not be a way to get istio to run rootless yet. Maybe someone should file an issue with them for wanting rootless istio.

It may be a big stretch to try, but I think the cni driver only looks for an init-container exists with the right name. You may be able to add a mutating webhook that overrides the command to just return true skipping iptables to get istio cni to do the injection without actually running the root/iptables command. No idea if that is a good idea/would be reliable.

from gatekeeper-library.

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

from gatekeeper-library.