Kubernetes and Secrets Management In The Cloud
The kube-secrets-init is a Kubernetes mutating admission webhook, that mutates any K8s Pod that is using specially prefixed environment variables, directly or from Kubernetes as Secret or ConfigMap.
The kube-secrets-init injects a copy-secrets-init initContainer into a target Pod, mounts /helper/bin (default; can be changed with the volume-path flag) and copies the secrets-init tool into the mounted volume. It also modifies Pod entrypoint to secrets-init init system, following original command and arguments, extracted either from Pod specification or from Docker image.
The kube-secrets-init can be configured to skip injection for all Pods in the specific Namespace by adding the admission.secrets-init/ignore label to the Namespace.
secrets-init runs as PID 1, acting like a simple init system. It launches a single process and then proxies all received signals to a session rooted at that child process.
secrets-init also passes almost all environment variables without modification, replacing secret variables with values from secret management services.
User can put AWS secret ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to referenced secret value.
# environment variable passed to `secrets-init`
MY_DB_PASSWORD=arn:aws:secretsmanager:$AWS_REGION:$AWS_ACCOUNT_ID:secret:mydbpassword-cdma3
# environment variable passed to child process, resolved by `secrets-init`
MY_DB_PASSWORD=very-secret-passwordIt is possible to use AWS Systems Manager Parameter Store to store application parameters and secrets.
User can put AWS Parameter Store ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to referenced parameter value.
# environment variable passed to `secrets-init`
MY_API_KEY=arn:aws:ssm:$AWS_REGION:$AWS_ACCOUNT_ID:parameter/api/key
# environment variable passed to child process, resolved by `secrets-init`
MY_API_KEY=key-123456789User can put Google secret name (prefixed with gcp:secretmanager:) as environment variable value. The secrets-init will resolve any environment value, using specified name, to referenced secret value.
# environment variable passed to `secrets-init`
MY_DB_PASSWORD=gcp:secretmanager:projects/$PROJECT_ID/secrets/mydbpassword
# OR versioned secret (with version or 'latest')
MY_DB_PASSWORD=gcp:secretmanager:projects/$PROJECT_ID/secrets/mydbpassword/versions/2
# environment variable passed to child process, resolved by `secrets-init`
MY_DB_PASSWORD=very-secret-passwordIn order to resolve AWS secrets from AWS Secrets Manager and Parameter Store, secrets-init should run under IAM role that has permission to access desired secrets.
This can be achieved by assigning IAM Role to Kubernetes Pod. It's possible to assign IAM Role to EC2 instance, where container is running, but this option is less secure.
In order to resolve Google secrets from Google Secret Manager, secrets-init should run under IAM role that has permission to access desired secrets. For example, you can assign the following 2 predefined Google IAM roles to a Google Service Account: Secret Manager Viewer and Secret Manager Secret Accessor role.
This can be achieved by assigning IAM Role to Kubernetes Pod with Workload Identity. It's possible to assign IAM Role to GCE instance, where container is running, but this option is less secure.
Uncomment --provider=google flag in the deployment.yaml file.
Consider using the kube-secrets-init Helm Chart, authored and managed by Márk Sági-Kazár.
helm repo add skm https://charts.sagikazarmark.dev
helm install --generate-name --wait skm/kube-secrets-initCheck chart GitHub repository
- To deploy the
kube-secrets-initserver, we need to create a webhook service and a deployment in our Kubernetes cluster. It’s pretty straightforward, except one thing, which is the server’s TLS configuration. If you’d care to examine the deployment.yaml file, you’ll find that the certificate and corresponding private key files are read from command line arguments, and that the path to these files comes from a volume mount that points to a Kubernetes secret:
[...]
args:
[...]
- --tls-cert-file=/etc/webhook/certs/cert.pem
- --tls-private-key-file=/etc/webhook/certs/key.pem
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
[...]
volumes:
- name: webhook-certs
secret:
secretName: secrets-init-webhook-certsThe most important thing to remember is to set the corresponding CA certificate later in the webhook configuration, so the apiserver will know that it should be accepted. For now, we’ll reuse the script originally written by the Istio team to generate a certificate signing request. Then we’ll send the request to the Kubernetes API, fetch the certificate, and create the required secret from the result.
First, run webhook-create-signed-cert.sh script and check if the secret holding the certificate and key has been created:
./deployment/webhook-create-signed-cert.sh
creating certs in tmpdir /var/folders/vl/gxsw2kf13jsf7s8xrqzcybb00000gp/T/tmp.xsatrckI71
Generating RSA private key, 2048 bit long modulus
.........................+++
....................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/secrets-init-webhook-svc.default created
NAME AGE REQUESTOR CONDITION
secrets-init-webhook-svc.default 1s [email protected] Pending
certificatesigningrequest.certificates.k8s.io/secrets-init-webhook-svc.default approved
secret/secrets-init-webhook-certs configured
Note For the GKE Autopilot, run the webhook-create-self-signed-cert.sh script to generate a self-signed certificate.
Export the CA Bundle as a new environment variable CA_BUNDLE:
export CA_BUNDLE=[output value of the previous script "Encoded CA:"]Once the secret is created, we can create deployment and service. These are standard Kubernetes deployment and service resources. Up until this point we’ve produced nothing but an HTTP server that’s accepting requests through a service on port 443:
kubectl create -f deployment/deployment.yaml
kubectl create -f deployment/service.yamlNow that our webhook server is running, it can accept requests from the apiserver. However, we should create some configuration resources in Kubernetes first. Let’s start with our validating webhook, then we’ll configure the mutating webhook later. If you take a look at the webhook configuration, you’ll notice that it contains a placeholder for CA_BUNDLE:
[...]
service:
name: secrets-init-webhook-svc
namespace: default
path: "/pods"
caBundle: ${CA_BUNDLE}
[...]There is a small script that substitutes the CA_BUNDLE placeholder in the configuration with this CA. Run this command before creating the validating webhook configuration:
cat ./deployment/mutatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/mutatingwebhook-bundle.yamlCreate mutating webhook configuration:
kubectl create -f deployment/mutatingwebhook-bundle.yamlCreate Kubernetes Service Account to be used with secrets-init-webhook:
kubectl create -f deployment/service-account.yamlDefine RBAC permission for webhook service account:
# create a cluster role
kubectl create -f deployment/clusterrole.yaml
# define a cluster role binding
kubectl create -f deployment/clusterrolebinding.yaml
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent