sighupio / fury-kubernetes-opa Goto Github PK

Release version 1.7.1 of the OPA module including:

• bugfixes
• official support for Kubernetes 1.24

Hi, because of sighupio/fury-kubernetes-monitoring#40, now is possible to decouple dashboards from Gatekeeper, using configmaps with proper annotation.
Because of is Gatekeeper is an optional module of Fury, should deploy dashboards indipendently from Gatekeeper itself

Upstream has released OPA Gatekeeper v3.8.1. We should include it in the next version of the module

Hello!
since we have introduced the protect namespace rule that now forces each DELETE event through the gatekeeper, now became mandatory to exclude them from the other check in order to avoid misbehavior.
To do that, each violation block must have the following 3 lines :

operation := input.review.operation
any([ operation == "CREATE", operation == "UPDATE" ])
operation != "DELETE"

For this reason, should be a very smart usage of conftest, to add a pipeline step that enforces the presence of those lines.
Probably some rules need to be refactored in order to make the conftest rules easier,anyway, let's see them together!

I am not able to deploy the gpm module in an environment with constrained Internet access since we're pulling the Kustomize base directly from GitHub.

I know that it will lead to some duplications, but can we consider to just add the manifests to this repository as well instead of pulling the base from GitHub?

Thank you.

Upstream has released Gatekeeper 3.9.0
https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.9.0

Currently the module folder structure has only 1 package gatekeeper with 3 subpackages, it would be better to have 3 different packages instead. Dropping one nesting level.

We should have test cases that check that when there are violations of a rule in the cluster the audit phase detects them properly.

See #59 and #47 , when the audit pod was healthy but the audit didn't show violations that existed in the cluster.

You might find in Gatekeeper Controller Manager logs messages like the following:

http: TLS handshake error from 172.16.0.3:42672: EOF.

This is a known issue considered to be harmless. Creating the issue here for visibility and tracking the upstream fix.

Upstream Ref: open-policy-agent/gatekeeper#2142

It is desirable to have an alert that gets triggered if the communication with webhook is failing. It should have more severity if the Gatekeeper VWH is running in fail mode.

Related: #38

From upstream's official documentation, we can see that in v3.6 --audit-chunk-size was set to 0:

https://open-policy-agent.github.io/gatekeeper/website/docs/v3.6.x/audit#configuring-audit

but starting from v3.7 it has been set to 500 by default:
https://open-policy-agent.github.io/gatekeeper/website/docs/v3.7.x/audit#configuring-audit

We should drop our customization, since we are targeting v3.8.x.

Upstream has released a cli tool for testing constraints and constraint templates called gator.

We should evaluate including it in our flow.

The name for this core module is not correct.

The features that this module provides are policy definition and enforcement. OPA is just an implementation detail, and it is not used directly (Gatekeeper uses it under the hood)

A better name for the module would be Policy -> kubernetes-fury-policy

We still have some references to extensions/v1beta1 and networking.k8s.io/v1beta1 for Ingress that have been deprecated in Kubernetes 1.22.

For example:

This makes errors appear in the logs.

Delete all the references to v1beta1 and use v1 instead.

With @ralgozino we found out that we can't run opa module on a private GKE Cluster (e.g. the one created by the furyctl provisoner).

"By default, firewall rules restrict the cluster master communication to nodes only on ports 443 (HTTPS) and 10250 (kubelet). Although Gatekeeper exposes its service on port 443, GKE by default enables --enable-aggregator-routing option, which makes the master to bypass the service and communicate straight to the POD on port 8443."

From: https://open-policy-agent.github.io/gatekeeper/website/docs/vendor-specific/#running-on-private-gke-cluster-nodes

This results in the following timeout:

$kubectl create ns fury

Error from server (InternalError): Internal error occurred: failed calling webhook "check-ignore-label.gatekeeper.sh": Post https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel?timeout=3s: dial tcp 10.2.0.3:8443: i/o timeout

Solutions (again from the Gatekeeper documentation):

• create a new firewall rule from master to private nodes to open port 8443 (or any other custom port)
• make the pod run on privileged port 443 (need to run the pod as root)

Since we added the check for the review.operation to the provided ConstraintTemplates, like this:

the audit process doesn't trigger a violation for the constraints created from the template, because the operation is not set when the audit process runs.

Related: open-policy-agent/gatekeeper#333

IMPORTANT: the Admission is not affected, this is only an audit issue. For example, pods that didn't comply with a policy get properly rejected because the review.operation is set.

The unique ingress rule breaks cert-manager, it tries to create a new ingress with a different path that would not be a problem, but is rejected nonetheless because it is an "ingress with the same host".

Here the template: https://github.com/sighupio/fury-kubernetes-opa/blob/master/katalog/gatekeeper/rules/templates/unique_ingress_template.yml

The rule should be able to check hosts+path collisions, at least as much as possible, otherwise we should drop it.

The default rules package hasn't been updated in a while.

Check that we didn't miss any updates from upstream and that are all still relevant.

The list of excluded namespaced by default needs to be updated

Add E2E for K8s v1.24.0 in CI

Memory usage of gatekeeper-audit depends on how much resources and constraints there are in the cluster. So if #resources to check increase, memory usage also increase which then leads to OOMKilled.

Recently we had this issue on a cluster and configuring "--audit-chunk-size" parameter resolved the issue. I think we can add this parameter as default to proactively avoid this issue on clusters with potentially growing number of resources.

You can see here a discussion about this problem and the suggestion of the parameter:

open-policy-agent/gatekeeper#1279

The Grafana dashboard for Gatekeeper is using metrics that have been changed in the latest versions of Gatekeeper, in particular:

gatekeeper_request_duration_seconds_bucket has been renamed to gatekeeper_validation_request_duration_seconds_bucket and same other metrics for the mutating operations have been added, like gatekeeper_mutator_ingestion_count

the same for the gatekeeper_request_count metric.

The official metrics list is here: https://open-policy-agent.github.io/gatekeeper/website/docs/metrics

If in the cluster is present PodSecurityPolicy it is necessary to remove the following annotation from Deployment manifest.

Trying to install this module with PodSecurityPolicy running leaad to this error message in status section of ReplicaSet:

Warning  FailedCreate  11s (x19 over 22m)  replicaset-controller  Error creating: pods "gatekeeper-controller-manager-686c494bcd-" is forbidden: unable to validate against any pod security policy: [pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/manager]: Forbidden: seccomp may not be set]