cert-manager / aws-privateca-issuer Goto Github PK
View Code? Open in Web Editor NEWAddon for cert-manager that issues certificates using AWS ACM PCA.
License: Apache License 2.0
Addon for cert-manager that issues certificates using AWS ACM PCA.
License: Apache License 2.0
Save me from creating this with a local helm chart
An alternative is to install the current helm chart, and then kubectl apply -f
this file, or helm install
a local chart with this.
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: demo-test-root-ca
spec:
arn: arn:aws:acm-pca:us-east-2:snip:certificate-authority/snip
region: us-east-2
Original source of the above yaml https://aws.amazon.com/blogs/security/tls-enabled-kubernetes-clusters-with-acm-private-ca-and-amazon-eks-2/
It is not clear from the Readme what kind of minimal AWS policy we need to have in order to make this operator fully functional
Just a test.
During the release of v1.0.0 in Nov 2021, the plugin would add 30 days as the validity period for every certificate as a default for certificates with no validity duration specified, with this a bug was introduced which would add 30 days to the validity period on top of the specified validity duration; this was carried forward to v1.1.0 as well. This issue has been rectified in v1.2.0 (For more information please look in to PR. If the incorrect validity is a breaking issue then revoke the old certificates, move to version v1.2.0, and issue new certs.
it seems thats, AWSPCAIssuer can use any secrets in any namespaces. it may cause security problems.
make cluster runs with no errors locally on MacOS running Bash 5.x
make cluster
RUNNING=$(docker inspect -f '{{.State.Running}}' "kind-registry" 2>/dev/null || true)
if [ "$RUNNING" != 'true' ]; then
bash: -c: line 2: syntax error: unexpected end of file
MacOS 11.6
$ bash --version
bash --version
GNU bash, version 5.1.8(1)-release (x86_64-apple-darwin20.3.0)
No response
0.3.1 (Latest)
Broken Testing Infrastructure
Severity 4
With this PR #55 we are introducing the ability to issue subCA certificate when the isCA
flag is toggled. This is a feature request to add the ability to modify the subCA template ARN to use something besides a path length of zero.
This could possibly be done as a flag set on the issuer, that when set, is evaluated and used to set the appropriate template arn when issuing the subCA certificate.
No response
RootCACertificate/V1
SubordinateCACertificate_PathLen3/V1
pathlen:3
constraint i.e. X509v3 Basic Constraints: critical
CA:TRUE, pathlen:3
Certificate
of cert-manager.io/v1
). The generated CA certificate has the following constraints- X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
Expected a CA certificate with pathlen:2
The generated CA certificate had a pathlen:0
constraint, instead of the expected pathlen:2
constraint.
Is this happening due to this section in
pca.go
?
if spec.IsCA {
return prefix + "acm-pca:::template/SubordinateCACertificate_PathLen0/V1"
}
Create an EKS Cluster.
Install Cert Manager v1.10.0
# https://cert-manager.io/docs/installation/helm/
CERT_MANAGER_VERSION=v1.10.0
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version ${CERT_MANAGER_VERSION} \
--set installCRDs=true \
--wait
With the output from the script, save the Intermediate CA ARN in an environment variable-
export CA_ARN=arn:aws:acm-pca:REDACTED:REDACTED-AC:certificate-authority/REDACTED
cat <<EOF > AWSPCAIssuerPolicyTest.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "awspcaissuer",
"Action": [
"acm-pca:DescribeCertificateAuthority",
"acm-pca:GetCertificate",
"acm-pca:IssueCertificate"
],
"Effect": "Allow",
"Resource": "${CA_ARN}"
}
]
}
EOF
POLICY_ARN=$(aws iam create-policy \
--policy-name AWSPCAIssuerPolicyTest \
--policy-document file://AWSPCAIssuerPolicyTest.json \
--output json | jq -r '.Policy.Arn')
echo "POLICY_ARN = ${POLICY_ARN}"
CURRENT_CLUSTER="Please put your cluster name here"
echo "${POLICY_ARN}"
echo "${REGION}"
echo "${CURRENT_CLUSTER}"
# Currently, we are installing the plugin in the same namespace as cert-manager
export PCA_NAMESPACE=cert-manager
# latest version https://github.com/cert-manager/aws-privateca-issuer/releases
export AWSPCA_ISSUER_TAG=v1.2.2
# Enable the IAM OIDC Provider for the cluster
eksctl utils associate-iam-oidc-provider \
--cluster=${CURRENT_CLUSTER} \
--region=${REGION} \
--approve;
# Create IAM role bound to a service account
eksctl create iamserviceaccount --cluster=${CURRENT_CLUSTER} \
--region=${REGION} \
--namespace=${PCA_NAMESPACE} \
--attach-policy-arn=${POLICY_ARN} \
--override-existing-serviceaccounts \
--tags "created-by=${USER},team=${TEAM},purpose=customer-support" \
--name=aws-pca-issuer \
--role-name "ServiceAccountRolePrivateCA-${CURRENT_CLUSTER}" \
--approve;
# Install AWS Private CA Issuer Plugin
# https://github.com/cert-manager/aws-privateca-issuer/#setup
helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
helm repo update
helm install aws-pca-issuer awspca/aws-privateca-issuer \
--namespace ${PCA_NAMESPACE} \
--set image.tag=${AWSPCA_ISSUER_TAG} \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-pca-issuer \
--kube-context ${CURRENT_CLUSTER} \
--wait;
# Verify deployment status
kubectl --context ${CURRENT_CLUSTER} -n ${PCA_NAMESPACE} \
rollout status deploy/aws-pca-issuer-aws-privateca-issuer;
# edit the var if your CA is in a different region
export CA_REGION="YOUR CA REGION"
export CA_ARN="arn:aws:acm-pca:redacted:redacted:certificate-authority/redacted"
cat << EOF | kubectl apply -f -
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: my-cluster-issuer
spec:
arn: ${CA_ARN}
region: ${CA_REGION}
EOF
cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: my-internal-ca
spec:
commonName: my-internal-ca
dnsNames:
- "*.internal.my-org.ca"
isCA: true
duration: 2160h #90d
secretName: my-internal-ca-cert-manager
subject:
organizations:
- cluster.local
- cert-manager
issuerRef:
group: awspca.cert-manager.io
kind: AWSPCAClusterIssuer
name: my-cluster-issuer
EOF
Verify:
kubectl get Certificate my-internal-ca
Expected output:
NAME READY SECRET AGE
my-internal-ca True my-internal-ca-cert-manager 4s
kubectl get secret my-internal-ca-cert-manager -o yaml | yq -r '.data."tls.crt"' | base64 -d > my-internal-ca-cert-manager-tls-crt.pem
kubectl get secret my-internal-ca-cert-manager -o yaml | yq -r '.data."ca.crt"' | base64 -d > my-internal-ca-cert-manager-ca-crt.pem
my-internal-ca-cert-manager-tls-crt.pem
has the CA cert chained with the Issuer. Extract the top section from the pem file and copy it to a different file. I named it generated-ca-cert.pem
openssl x509 -in generated-ca-cert.pem -noout -text | grep -A3 Constraint
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Authority Key Identifier:
5C:F9:5F:9D:CF:86:DD:56:94:64:36:C4:REDACTED
ROOT_CAARN="arn:aws:acm-pca:REDACTED:REDACTED:certificate-authority/REDACTED"
SUBORDINATE_CERTARN="arn:aws:acm-pca:REDACTED:REDACTED:certificate-authority/REDACTED/certificate/REDACTED"
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${SUBORDINATE_CERTARN}" \
--output json | jq -r '.Certificate' > "intermediate-cert-common-purpose.pem"
openssl x509 -in intermediate-cert-common-purpose.pem -noout -text | grep -A3 Constraint
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:3
X509v3 Authority Key Identifier:
0B:97:66:22:D3:3A:FF:7D:51:10:2F:46:D1:F8:E8:E9:1D:4E:64:CA
Subordinate CA cert has pathlen:3
but the generated CA cert from that CA cert doesn't have pathlen:2
, instead it has pathlen:0
N/A.
Have already attached the expected output along with the commands in the above section.
Cert Manager -> v1.10.0
aws-privateca-issuer-> v1.2.2
Kubernetes -> 1.22
Amazon EKS platform version -> eks.6
Supported Workflow Broken
Severity 3
In private EKS clusters, VPC endpoints are typically created for regional endpoints.
The AWS sdk in v1 defaults to using the global sts endpoint so the issuer doesn't work without setting an additional env var: AWS_STS_REGIONAL_ENDPOINTS: 'regional'
. If using the v2 sdk it uses regional by default as documented here.
It appears that the AWS PCA Issuer integration with cert-manager version 1.10 does not quite work. After upgrading to cert-manager 1.10 and attempting to use the latest version of AWS PCA Cluster Issuer resources, these are the messages I see (this works with cert-manager 1.9.1):
E1121 15:03:27.073943 1 checks.go:55] cert-manager/controller/certificaterequests-issuer-selfsigned/handleSecretReference "msg"="failed to determine affected certificate requests" "error"="invalid value \"AWSPCAClusterIssuer\" for issuerRef.kind. Must be empty, \"Issuer\" or \"ClusterIssuer\"" "resource_kind"="Secret" "resource_name"="external-secrets-webhook" "resource_namespace"="my-platform" "resource_version"="v1"
Key thing to point out is "error"="invalid value \"AWSPCAClusterIssuer\" for issuerRef.kind. Must be empty, \"Issuer\" or \"ClusterIssuer\""
.
Here is my certificate definition:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: external-secrets-cert
namespace: my-platform
spec:
commonName: external-secrets-webhook.my-platform.svc
dnsNames:
- external-secrets-webhook.my-platform.svc
- external-secrets-webhook
- external-secrets-webhook.svc.cluster.local
- external-secrets-webhook.cluster.local
- localhost
duration: 2160h0m0s
ipAddresses:
- 127.0.0.1
issuerRef:
kind: ClusterIssuer
name: intermediate-ca
renewBefore: 360h0m0s
secretName: external-secrets-webhook
And my ClusterIssuer/AWS PCA ClusterIssuer:
---
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: root-ca
spec:
arn: REDACTED
region: us-gov-west-1
---
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
name: intermediate-ca
namespace: my-platform
spec:
commonName: intermediate-ca
duration: 2160h0m0s
isCA: true
privateKey:
algorithm: RSA
size: 4096
issuerRef:
group: awspca.cert-manager.io
kind: AWSPCAClusterIssuer
name: root-ca
renewBefore: 360h0m0s
secretName: intermediate-ca
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: intermediate-ca
spec:
ca:
secretName: intermediate-ca
No response
Looking for some guidance to see if this has been tested with 1.10.
As the code says at:
https://github.com/jniebuhr/aws-pca-issuer/blob/e5e5d80ede895ecc079ffa3bb3f2e0821b8d6c3b/pkg/controllers/certificaterequest_controller.go#L136
Is it an intentional limitation? Or is it still in progress? If yes than do you have any timeline for it?
We would like to issue a sub-ordinate CA signed by ACMPCA. We can do that using aws
or the web console but it would be nice if aws-pca-issuer
supports this too.
When using the chart without specifying a version number, the latest version should be deployed.
The chart attempts to pull a malformed image version and the deployment fails.
The image it's trying to pull is: cert-manager-aws-privateca-issuer:vv1.2.3
There's code here to specify the default version should be prefixed with a v
The helm chart downloads with a v in the version number by default:
Chart.yaml:
appVersion: v1.2.3
Hence 2 v's.
The workaround is to specify the chart version explicitly
aws-privateca-issuer
.No response
1.2.3
Supported Workflow Broken
Severity 2
Installing the Helm chart should use the image matching the appVersion
.
This will allow a reproducible deployment.
The latest image get always used, which will lead to unexpected upgrade in case the pod gets restarted.
helm template pca-issuer awspca/aws-privateca-issuer --version 1.2.0
Output:
...
containers:
- name: aws-privateca-issuer
image: "public.ecr.aws/k1n1h4h4/cert-manager-aws-privateca-issuer:latest"
...
Note:
I'm aware of the possibility to set image.tag to "" in the values file to fallback to the Chart.AppVersion.
No response
= 1.0.0
Other
Severity 4
There is currently a known issue with the plugin that is preventing certificate issuance due to STS GetCallerIdentity failing because of a region not specified bug, regardless of whether a region was specified or not (#54). There is an existing pull request to fix this (#53), but we are holding off on accepting any pull requests until our testing is redesigned. To fix this issue until then, please checkout the cleanup branch by running
git fetch -a
git checkout cleanup
Also, please be sure you are using the plugin with an IAM user, as that is the most reliable workflow https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey
This user must have minimum permissions listed here: https://github.com/cert-manager/aws-privateca-issuer#configuration
export AWS_SECRET_ACCESS_KEY=<Secret Access Key you generated>
export AWS_ACCESS_KEY_ID=<Access Key you generated>
Followed the below guide:
https://aws.amazon.com/blogs/containers/setting-up-end-to-end-tls-encryption-on-amazon-eks-with-the-new-aws-load-balancer-controller/
Logs:
I0923 17:41:51.050435 1 request.go:655] Throttling request took 1.019788438s, request: GET:https://172.20.0.1:443/apis/policy/v1beta1?timeout=32s
{"level":"info","ts":1632418911.1048965,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1632418911.1053565,"logger":"setup","msg":"starting manager"}
I0923 17:41:51.105568 1 leaderelection.go:243] attempting to acquire leader lease aws-pca-issuer/b858308c.awspca.cert-manager.io...
{"level":"info","ts":1632418911.105594,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0923 17:42:08.605338 1 leaderelection.go:253] successfully acquired lease aws-pca-issuer/b858308c.awspca.cert-manager.io
{"level":"info","ts":1632418928.6055086,"logger":"controller-runtime.manager.controller.certificaterequest","msg":"Starting EventSource","reconciler group":"cert-manager.io","reconciler kind":"CertificateRequest","source":"kind source: /, Kind="}
{"level":"info","ts":1632418928.6055412,"logger":"controller-runtime.manager.controller.awspcaclusterissuer","msg":"Starting EventSource","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAClusterIssuer","source":"kind source: /, Kind="}
{"level":"info","ts":1632418928.6055186,"logger":"controller-runtime.manager.controller.awspcaissuer","msg":"Starting EventSource","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAIssuer","source":"kind source: /, Kind="}
{"level":"info","ts":1632418928.7065413,"logger":"controller-runtime.manager.controller.certificaterequest","msg":"Starting Controller","reconciler group":"cert-manager.io","reconciler kind":"CertificateRequest"}
I0923 17:42:09.656073 1 request.go:655] Throttling request took 1.045965413s, request: GET:https://172.20.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
{"level":"error","ts":1632418929.7086253,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWSPCAClusterIssuer.awspca.cert-manager.io","error":"no matches for kind \"AWSPCAClusterIssuer\" in version \"awspca.cert-manager.io/v1beta1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:144\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:117\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:167\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681"}
{"level":"info","ts":1632418929.708644,"logger":"controller-runtime.manager.controller.certificaterequest","msg":"Starting workers","reconciler group":"cert-manager.io","reconciler kind":"CertificateRequest","worker count":1}
{"level":"info","ts":1632418929.7087255,"logger":"controller-runtime.manager.controller.certificaterequest","msg":"Stopping workers","reconciler group":"cert-manager.io","reconciler kind":"CertificateRequest"}
{"level":"error","ts":1632418932.3092573,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWSPCAIssuer.awspca.cert-manager.io","error":"no matches for kind \"AWSPCAIssuer\" in version \"awspca.cert-manager.io/v1beta1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:144\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:117\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:167\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681"}
{"level":"error","ts":1632418932.3093493,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"no matches for kind \"AWSPCAIssuer\" in version \"awspca.cert-manager.io/v1beta1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:530"}
{"level":"error","ts":1632418932.3094308,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:530"}
{"level":"error","ts":1632418932.3094597,"logger":"setup","msg":"problem running manager","error":"no matches for kind \"AWSPCAClusterIssuer\" in version \"awspca.cert-manager.io/v1beta1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:144\nmain.main\n\t/workspace/main.go:139\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}
Currently you can not set ENV parameters for the helm chart.
Allowing such setting would enable customers to set required settings such as corporate proxy settings via https_proxy/no_proxy, so they can reach ACM API endpoints.
I tried to contribute the below method a few months ago, but you needed tests writing and I don't have IAM access within our corperation to create PCA resources.
values.yaml
env:
#var1: myvar1
#var2: myvar2
deployment.yaml
env:
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
value: "{{ $value }}"
{{- end }}
example values
env:
http_proxy: http://corpproxy:3128
https_proxy: http://corpproxy:3128
no_proxy: "localhost.127.0.0.1,instance-data,10.0.0.0/8,169.254.169.254,100.64.0.0/10,.corptld.plc"
### Is there anything else you would like to add?
We've been using this method for months now without issues and the application appears to honour the proxy env parameters.
As discussed in #10 the ca.crt
contained the intermediate CA certificate that was used to issue the requested certificate. This was updated in #11 to have the requested certificate in tls.crt
and intermediate + root CAs in ca.crt
.
Based on cert-manager certificate documentation the expected format of the secret should be
When a certificate is issued by intermediates of the CA and the Issuer knows the intermediates, the content of tls.crt will be a resulting certificate followed by a certificate chain. The certificate chain doesnโt include a root CA certificate, as it is stored in ca.crt.
I've created #14 as an attempt to address the above.
As an added bonus pem.Encode
will add a \n
after each certificate in ca.crt
which fixes applications that expect a concatenated PEM file (tls.crt
+ ca.crt
+ tls.key
). I'm running into this with the default OpenShift ingress controller (HAProxy) for example.
When deploying the pod using a helm chart and manually creating the ServiceAccount, the pod should inherit the permissions assigned to the IAM role specified in the manually created ServiceAccount and be able to requested certificates.
The Pod appears to not inherit the correct permissions and fails to get the certificate.
cert-manager The certificate request has failed to complete and will be retried: failed to request certificate from
PCA: operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: e1e844fb-1af5-4d63-9872-bfe9e627537e, api error
AccessDeniedException: User: arn:aws:sts::xxxxxx:assumed-role/xxxxxxxx/xxxxxx is not authorized to perform: acm-pca:IssueCertificate
on resource: arn:aws:acm-pca:eu-west-2:xxxx:certificate-authority/xxxxx because no resource-based policy
allows the acm-pca:IssueCertificate action
It appears that the pod has the correct ServiceAccount role as it is getting the token?
Deployment of the PCA Issuer Helm Chart using terraform
resource "helm_release" "eks_aws_pca" {
name = "${var.site_name}-aws-pca"
chart = "aws-privateca-issuer"
repository = "https://cert-manager.github.io/aws-privateca-issuer"
version = var.aws_pca_helm_version
namespace = local.aws_pca_namespace
set {
name = "installCRDs"
value = "true"
}
set {
name = "image.repository"
value = var.aws_privateca_issuer_image
}
set {
name = "image.tag"
value = var.aws_privateca_issuer_tag
}
set {
name = "serviceAccount.create"
value = false
}
set {
name = "serviceAccount.name"
value = module.eks_aws_pca_irsa.service_account_name
}
set {
name = "approverRole.namespace"
value = local.cert_manager_namespace
}
set {
name = "approverRole.serviceAccountName"
value = local.cert_manager_service_account_name
}
}
Deploying version 1.0.0 of the Helm Chart and Version 1.0.0 of the aws-privateca container
No response
v1.0.0
Authentication Issue
Severity 2
Kubernetes: 1.19 (v1.19.6-eks-49a6c0)
aws-pca-issuer: Helm release 0.1.0
certmanager: 1.3.1
I have a working setup of this addon but currently unable to use it to auto-generate certificate using annotations such as below:
apiVersion: extensions/v1beta1
# apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: myprivate-ca
name: podinfo
namespace: podinfo
spec:
rules:
- host: podinfo.infrastructure.com
http:
paths:
- path: /
backend:
serviceName: podinfo
servicePort: 80
tls:
- hosts:
- podinfo.infrastructure.com
secretName: podinfo10
Creating certificate using the CRD works, e.g with:
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
name: podinfo-cert
namespace: podinfo
spec:
commonName: podinfo.infrastructure.com
dnsNames:
- podinfo.infrastructure.com
issuerRef:
group: awspca.cert-manager.io
kind: AWSPCAClusterIssuer
name: myprivate-ca
renewBefore: 360h0m0s
secretName: podinfo-cert
The error I get when using the ingress approach to autogenerate the certificate is:
2021-04-28T18:42:12.883939412Z {"level":"error","ts":1619635332.883713,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"podinfo/podinfo3-rxrmp","error":"MalformedCSRException: CSR must mark the SAN extension critical when it has an empty subject.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/jniebuhr/aws-pca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile\n\t/workspace/pkg/controllers/certificaterequest_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
The approach of using ingress works with other issuers as we currently use vault issuer but we want to migrate to AWS ACM.
Does this work for anyone?
Pods in the cluster should be able to talk mTLS with each other and have the same root cert dispatched by cert-manager through aws pca.
kubectl -n foo exec -it deploy/sleep -c sleep -- curl http://httpbin.foo:8000/headers -v
GET /headers HTTP/1.1
Host: httpbin.foo:8000
User-Agent: curl/7.87.0-DEV
Accept: /
Trying to move from self signed cert in istio mesh to ACM PCA certs for istio workloads and with everything seemingly successful (setup), the test apps (sleep, httbin) give a 503 while talking to each other in the same namespace. Im using cert-manager, istio-csr, awspcaissuer add on and istio istio mesh in EKS.
The cert manager, istio csr, awspca/aws-privateca-issuer are all installed using helm and successfully running
awspca issuer (cluster issuer) is created, root cert also created with the following:
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
name: istio-ca
namespace: istio-system
spec:
isCA: true
commonName: istio-ca
duration: 2160h0m0s
issuerRef:
group: awspca.cert-manager.io
kind: AWSPCAClusterIssuer
name: awspcaclusterissuer
secretName: istio-ca
subject:
organizations:
- cluster.local
- cert-manager
privateKey:
algorithm: "RSA"
size: 2048
istio is installed with overlay, everything looks good in sample apps, it comes up with new certs, the CSR are approved but the actual request fails
kubectl -n foo exec -it deploy/sleep -c sleep -- curl http://httpbin.foo:8000/headers -v
* Trying 192.20.179.155:8000...
* Connected to httpbin.foo (172.20.179.155) port 8000 (#0)
> GET /headers HTTP/1.1
> Host: httpbin.foo:8000
> User-Agent: curl/7.87.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 218
< content-type: text/plain
< date: Fri, 20 Jan 2023 02:55:07 GMT
< server: envoy
<
* Connection #0 to host httpbin.foo left intact
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
No response
istio - 1.14.1
latest for cert-manager, istio-csr, aws-privateca-issuer
Supported Workflow Broken
Severity 3
Currently working on adding unit testing, will update this issue when I complete.
I tried searching online for this error but I am unable to locate any, what exactly is the issue here and what should I do to resolve it?
"error"="admission webhook \"webhook.cert-manager.io\" denied the request: spec.issuerRef: Forbidden: referenced signer resource does not exist: {pca-issuer AWSPCAClusterIssuer awspca.cert-manager.io/v1beta1}"
AWSPCAClusterIssuer should be able to provision certificate
Our private CA authority is hosted in the networking account and has been shared with other accounts(let's call it dev-1) here. We can manually generate the certs by logging into the dev-1 account.
Now when I create an AWSPCAClusterIssuer, the CA arn I am specifying contains the network account's id (which is obvious), and when I try to create the cert using this certificate it errors out with the below error,
"level":"error","ts":1657793954.0080278,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"default/example-com-f9v6w","error":"operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: b6cd6e9b-9e6a-42b9-a96f-38806a3ffffb, api error AccessDeniedException: User: arn:aws:sts::XXXXXXXX:assumed-role/shared-dev-1-cert-manager/1657793931500012209 is not authorized to perform: acm-pca:IssueCertificate on resource: arn:aws:acm-pca:eu-west-2:YYYYYYY:certificate-authority/68275bf7-8147-4620-ba7d-4fecf3f234e0 because no resource-based policy allows the acm-pca:IssueCertificate action","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
Install certmanager on eks
Try to use a shared private CA for cert creation
No response
v1.2.2
Authentication Issue
Severity 2
As you requested in #24 I compiled the latest code (jniebuhr/aws-pca-issuer@07e5bed) and deployed it to our test cluster. The original issue was resolved but switching the AWS SDK 2 broke the functionality when KIAM or some similar is in use which is quite common in the AWS world. I think this can be caused the AWS SDK 2 by default enforces to use IMDSv2. The 0.2.1
release doesn't suffer from this nor any of our other "KIAM-handled" services. The KIAM is configured to allow access to every metadata endpoints.
The logs from the aws-pca-issuer
:
2021-05-27T09:07:37.731866000Z {"level":"error","ts":1622106457.731583,"logger":"controllers.GenericIssuer","msg":"failed to sts.GetCallerIdentity","genericissuer":"/acmpca","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 7a2a4f4b-3c04-4207-9601-e5986b0140dc, api error MissingAuthenticationToken: Request is missing Authentication Token","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/jniebuhr/aws-pca-issuer/pkg/controllers.(*GenericIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/genericissuer_controller.go:122\ngithub.com/jniebuhr/aws-pca-issuer/pkg/controllers.(*AWSPCAClusterIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/awspcaclusterissuer_controller.go:57\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
2021-05-27T09:07:37.732050000Z {"level":"error","ts":1622106457.7316792,"logger":"controller-runtime.manager.controller.awspcaclusterissuer","msg":"Reconciler error","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAClusterIssuer","name":"acmpca","namespace":"","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 7a2a4f4b-3c04-4207-9601-e5986b0140dc, api error MissingAuthenticationToken: Request is missing Authentication Token","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
The uswitch/kiam#359 could be useful to understand the issue better.
The KIAM logs says that the requested IAM role was fetched successfully when the aws-pca-issuer
pod was in the Pending
phase - KIAM fetches roles in advance for performance reasons.
time="2021-05-27T09:04:54Z" level=info msg="fetched credentials" credentials.access.key=ASIATTQ4YQR7GCJ5EJLJ credentials.expiration="2021-05-27T10:01:01Z" credentials.role="<Role Name>" generation.metadata=0 pod.iam.role="<Role Name>" pod.name=cert-manager-acmpca-5f99df9b5c-x7tq5 pod.namespace=cert-manager pod.status.ip= pod.status.phase=Pending resource.version=18948857
As there isn't any shell in the container I can't easily exec into it and check what curl
/aws
says.
The region field on AWSPCA Issuers is redundent and only allows breaking the issuer.
I might have it wrong but it seems that the region must always match the region of the specified PCA arn.
Any region value that is not the region from the PCA arn, will always break.
Therefore it seems that having the parameter at all is a bug, as it's only function is to allow the user to break the issuer.
Controller should derive the API client region from the CA arn itself.
Thank you for the very usefull tooling!!
Trying to run local unit tests but seeing this error in aws-privateca-issuer-controller-manager pod
{"level":"error","ts":1631654584.4164567,"logger":"controllers.GenericIssuer","msg":"failed to sts.GetCallerIdentity","genericissuer":"/pca-cluster-issuer-ec","error":"operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*GenericIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/genericissuer_controller.go:92\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*AWSPCAClusterIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/awspcaclusterissuer_controller.go:57\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
When we tried to deploy aws-pca-issuer
and we hit by the following issue:
2021-05-25T11:33:33.261830000Z {"level":"error","ts":1621942413.261467,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"opa/akunszt-test-9jl58","error":"InvalidArgsException: Signing algorithm SHA384WITHECDSA cannot be used with key type RSA_2048. Valid signing algorithms are: SHA256WITHRSA, SHA384WITHRSA, SHA512WITHRSA.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/jniebuhr/aws-pca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile\n\t/workspace/pkg/controllers/certificaterequest_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
This was caused that the signatureAlgorithm
function determines the signature algorithm from the certificate request, not from the signing CA.
https://github.com/jniebuhr/aws-pca-issuer/blob/f2e355d766e625d3c55b287d2a260bbdff5d61ff/pkg/aws/pca.go#L161
We have a CA in ACMPCA with an RSA_2048 key and we wanted to issue an ECDSA certificate. This doesn't work in aws-pca-issuer
but it works if you properly set the signing algorithm.
We're using this to issue a certificate using ACMPCA from the command line - it's a snippet from a far bigger script, this is just the bare logic.
# the algorithm for the local key
LOCAL_CURVE_NAME="secp384r1"
# the algorithm used by the PCA to sign the certificate
# - if the PCA has an EC key then this should be SHA512WITHECDSA
# - it the PCA has an RSA key then this should be SHA512WITHRSA
PCA_SIGNING_ALGORITHM="SHA512WITHRSA"
### zillion lines of code to create a private key and a CSR, check a lot of things, etc...
aws acm-pca issue-certificate \
--certificate-authority-arn "$PCA_ARN" \
--csr file:///dev/stdin \
--region "$AWS_REGION" \
--signing-algorithm "$PCA_SIGNING_ALGORITHM" \
--template-arn "arn:aws:acm-pca:::template/$TEMPLATE" \
--validity "Value=$EXPIRES_IN,Type=DAYS" \
--query CertificateArn \
--output text < "csr.pem"
We use the same algorithm everywhere but for a general solution you can fetch the signing CA private key easily with:
aws acm-pca describe-certificate-authority --region "$REGION" --certificate-authority-arn "$ARN" --query "CertificateAuthority.CertificateAuthorityConfiguration.KeyAlgorithm" --output text
The aws-pca-issuer
should fetch that and then use that information to construct the signing algorithm. Another solution could be to set that in the AWSPCAClusterIssuer
/AWSPCAIssuer
resources. It's rarely changes on it's own as it requires to recreate the CA on the AWS side.
A little background information why we wanted to do this:
You can use the ACMPCA to sign certificates in ACM only if the CA uses an RSA-2048 key. It's a known limitation of ACM at the moment. We have one regional CA - as they're quite costly - which we can use to issue certificates from the AWS web console in ACM or from Kubernetes using Certificate resources. This is why we have to use the RSA-2048 for the CA but we would like to use ECDSA for the certificates issued by the CA to make them a bit more future proof.
When performing linting using helm lint
on the Helm Chart, linting should succeed. Additionally, to ensure issues do not occur with changes to the Helm Chart, linting should be included as part of the testing actions.
When performing linting using helm lint
on the Helm Chart, the command fails due to an invalid YAML document separator. After debugging this further, this is caused by incorrect template directives causing issues with whitespace, particularly on line 15 in templates/rbac.yaml
. The dash at the end of that line moves that line to the line above. Removing the right dash in the directive, fixes the issue.
This can be reproduced by either using helm
or helmfile
binaries to performing the linting. If linting using the helmfile
binary, the below steps are basically automated.
$ helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
$ helm repo update
$ helm pull awspca/aws-privateca-issuer -d ./awspca --untar
$ helm lint awspca/aws-privateca-issuer
$ helm lint awspca/aws-privateca-issuer
==> Linting awspca/aws-privateca-issuer
[INFO] Chart.yaml: icon is recommended
[ERROR] templates/rbac.yaml: unable to parse YAML: invalid Yaml document separator: apiVersion: rbac.authorization.k8s.io/v1
Error: 1 chart(s) linted, 1 chart(s) failed
Helm Chart Version: 1.2.2
Helm Application Version: 1.2.2
Build Issues
Severity 3
Currently the aws-privateca-issuer cannot mange to use AWS PCA shared from another account via RAM because when you share a PCA, acm-pca:IssueCertficate
requests a condition: StringEquals acm-pca:TemplateArn: "arn:aws:acm-pca:::template/EndEntityCertificate/V1"
and there is no way to have that permission without condition.
we are migrating to arm platform, can we support for arm platform use multi-arch docker image?
No response
No response
Good Morning,
I am not sure reason behind the current solution i.e. append ca bundle ( chainPem ) with actual certificate ( certPem
) issued to FQDN .
currently, i am not proposing any changes as it may be it's kubernetes focused ? just want to show my observation.
the openshift route has three separate keys:
key: |-
-----BEGIN PRIVATE KEY-----
[...]
-----END PRIVATE KEY-----
certificate: |-
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
caCertificate: |-
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
ex. Assume, below is our chain looks like )
D ( root )
C ( issuer of B)
B ( issuer of A )
A ( actual certificate issued to requested FQDN)
Current version:
certPem := []byte(*getOutput.Certificate + "\n")
chainPem := []byte(*getOutput.CertificateChain)
certPem = append(certPem, chainPem...)
caParams := acmpca.GetCertificateAuthorityCertificateInput{
CertificateAuthorityArn: aws.String(p.arn),
}
caOutput, err := svc.GetCertificateAuthorityCertificate(&caParams)
if err != nil {
return nil, nil, err
}
caPem := []byte(*caOutput.Certificate)
return certPem, caPem, nil
in above case, cert-manager creating secret like below: i.e. tks.crt contains the CA bundle.
ca.crt: contains certificate of B
tls.crt: contains certificates of A, B, C, D
tls.key: contains private key of A
Modified version:
certPem := []byte(*getOutput.Certificate + "\n")
chainPem := []byte(*getOutput.CertificateChain)
caPem := chainPem
//certPem = append(certPem, chainPem...)
//caParams := acmpca.GetCertificateAuthorityCertificateInput{
// CertificateAuthorityArn: aws.String(p.arn),
//}
//caOutput, err := svc.GetCertificateAuthorityCertificate(&caParams)
//if err != nil {
// return nil, nil, err
//}
//caPem := []byte(*caOutput.Certificate)
return certPem, caPem, nil
in above case, cert-manager creating secret like below: completely separating CA bundle with tls.crt. and adding whole bundle to ca.crt
ca.crt: contains certificate of B, C, D
tls.crt: contains certificate of A
tls.key: contains private key of A
aws-privateca-issuer
shold work with csi-cert-manager
for mTLS without any issuse.
aws-privateca-issuer
is not working with csi-cert-manager
for mTLS and showing following error:
Issuer.yaml
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: demo-test-root-ca
spec:
arn: <CA_ARN>
region: us-east-1
node_server.yaml
---
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls-aws-pca}
name: hello-mtls-aws-pca
namespace: default
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls-aws-pca}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-aws-pca
labels: {app: hello-mtls-aws-pca}
namespace: default
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-aws-pca}}
template:
metadata:
labels: {app: hello-mtls-aws-pca}
spec:
containers:
- name: hello-mtls-aws-pca
image: test/cert-manager-server-node:latest
imagePullPolicy: IfNotPresent
resources: {requests: {cpu: 10m, memory: 20Mi}}
volumeMounts:
- mountPath: "/tls"
name: tls
volumes:
- name: tls
csi:
readOnly: true
driver: csi.cert-manager.io
volumeAttributes:
csi.cert-manager.io/issuer-kind: AWSPCAClusterIssuer
csi.cert-manager.io/issuer-name: demo-test-root-ca
csi.cert-manager.io/issuer-group: awspca.cert-manager.io
csi.cert-manager.io/dns-names: hello-mtls-aws-pca.default.svc.cluster.local
csi.cert-manager.io/duration: "20h"
csi.cert-manager.io/renew-before: "8h"
Create AWSPCAClusterIssuer
and Deployment
using above files. Wait for deployment pod coming up. It is waiting for mounting volume and volume mount failing due to no validity mentioned in CertificateRequest.
{"level":"error","ts":1636429708.7008724,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"default/db9e779d-10f6-467b-acc2-341bcd30dd74","error":"operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: 07a63afe-f788-4047-8dbe-3857c1557c61, api error ValidationException: 1 validation error detected: Value '0' at 'validity.value' failed to satisfy constraint: Member must have value greater than or equal to 1","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile\n\t/workspace/pkg/controllers/certificaterequest_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
E1109 04:22:06.405118 1 manager.go:506] manager "msg"="Failed to issue certificate, retrying after applying exponential backoff" "error"="waiting for request: request \"8961b4c0-3f33-4b30-bbdb-43dedae0a51a\" has failed: failed to request certificate from PCA: operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: c6c1fdb3-9a12-424f-bef4-245b7af6c249, api error ValidationException: 1 validation error detected: Value '0' at 'validity.value' failed to satisfy constraint: Member must have value greater than or equal to 1" "volume_id"="csi-fbd8716d6f985bae4fefa59ee13848f66cc8608bb4e4563ae69394424940f6b4"
### Version
0.3.1 (Latest)
### Have you tried the following?
- [X] Check the [Troubleshooting section](../#troubleshooting)
- [X] Search open [issues](https://github.com/cert-manager/aws-privateca-issuer/issues)
### Category
Supported Workflow Broken
### Severity
Severity 1
It would be nice to have topologySpreadConstraints support in the official helm chart
/kind feature
No response
No response
Use case:
I would like to issue a certificate where the combined number of characters of the namespace and certificate name is more than 35 characters.
For example:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: my-very-long-certificate-name-that-has-quite-a-few-characters
namespace: example
Expected behavior:
A certificate should be issued.
Actual behavior:
A certificate is not issued, and I receive an the following error in the logs for aws-privateca-issuer:
{"level":"error","ts":1624317003.908817,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"example/my-very-long-certificate-name-that-has-quite-a-few-c-8qbnl","error":"operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: c73883c0-2ae8-42f1-9369-ca1705b36119, api error ValidationException: 1 validation error detected: Value example/my-very-long-certificate-name-that-has-quite-a-few-c-8qbnl at idempotencyToken failed to satisfy constraint: Member must have length less than or equal to 36","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile
/workspace/pkg/controllers/certificaterequest_controller.go:172
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
References:
aws-privateca-issuer/pkg/aws/pca.go
Line 103 in b5f3ba4
I don't know if it's an issue in cert-manager
or in aws-pca-issuer
. If it's the former then I'll open an issue there. I just experienced this only with aws-pca-issuer
.
When aws-pca-issuer
is running then it tries to handle Certificate resources even if the issuerRef points to a "plain" Issuer resource.
In the opa
namespace I wanted to create a self-signed CA and a Certificate signed by that CA. The self-signed CA is created but aws-pca-issuer
complains about that it doesn't support CA certificates. There aren't any reference to PCA in the YAML, so it shouldn't be even aware of it or it should be skipped silently.
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: ca
namespace: opa
spec:
commonName: OpenPolicyAgent CA
duration: 87600h
isCA: true
issuerRef:
kind: ClusterIssuer
name: self-signed
privateKey:
algorithm: ECDSA
encoding: PKCS8
rotationPolicy: Always
size: 384
renewBefore: 168h
secretName: tls-ca
subject:
organizations:
- blah
The log I saw in the aws-pca-issuer
.
2021-05-25T14:59:48.961908000Z {"level":"info","ts":1621954788.961414,"logger":"controllers.CertificateRequest","msg":"AWSPCA does not support CA certificates","certificaterequest":"opa/ca-qn2gk"}
The Certificate couldn't be created, it complains about that it can't find the issuer. Also I saw that aws-pca-issuer
tried to look for an AWSPCAClusterIssuer which is - of course - doesn't exist as I created an Issuer only. Again there isn't anything PCA related in the certificate at all.
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: server
namespace: opa
spec:
commonName: OpenPolicyAgent Server
dnsNames:
- opa
- opa.opa.svc
- localhost
duration: 43800h
ipAddresses:
- "::1"
- 127.0.0.1
isCA: false
issuerRef:
kind: Issuer
name: opa-ca
privateKey:
algorithm: ECDSA
encoding: PKCS8
rotationPolicy: Always
size: 384
renewBefore: 168h
secretName: tls-server
subject:
organizations:
- blah
usages:
- server auth
The log I saw in the aws-pca-issuer
pod:
2021-05-25T14:59:50.471630000Z {"level":"error","ts":1621954790.471369,"logger":"controllers.CertificateRequest","msg":"failed to retrieve Issuer resource","certificaterequest":"opa/server-x952x","error":"AWSPCAClusterIssuer.awspca.cert-manager.io \"opa-ca\" not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/jniebuhr/aws-pca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile\n\t/workspace/pkg/controllers/certificaterequest_controller.go:150\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
2021-05-25T14:59:50.485633000Z {"level":"error","ts":1621954790.4854755,"logger":"controller-runtime.manager.controller.certificaterequest","msg":"Reconciler error","reconciler group":"cert-manager.io","reconciler kind":"CertificateRequest","name":"server-x952x","namespace":"opa","error":"AWSPCAClusterIssuer.awspca.cert-manager.io \"opa-ca\" not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
When I scale down the aws-pca-issuer
deployment to zero and re-create the certificate it works.
We're using v0.2.1 with disableApprovedCheck: true
as we have cert-manager:1.2.0
.
Currently the only method to install the ca issuer is using helm and we would like to only use native kubectl to install it.
run helm with --dry-run -o yaml but this requires the installation of helm
No response
no matches for kind "Certificate" in version "cert-manager.io/v1"
We're deploying aws-privateca-issuer via Helm (Image: ghcr.io/jniebuhr/aws-pca-issuer:v0.3.0) on an EKS cluster (k8s v1.20).
I'm trying to pass credentials to the aws-privateca-issuer by using an IAM role for its service account, following this AWS guide.
When I describe the pod, I can see that the role has been assigned successfully and the credentials are mounted on the pod:
Environment:
AWS_DEFAULT_REGION: ap-southeast-1
AWS_REGION: ap-southeast-1
AWS_ROLE_ARN: arn:aws:iam::123456789012:role/correct-role
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from aws-pca-issuer-1628064934-aws-privateca-issuer-token-mmrrp (ro)
aws-privateca-issuer does not seem to support this setup, as in the logs, it's complaining about a missing token:
{"level":"error","ts":1628242137.8225927,"logger":"controller-runtime.manager.controller.awspcaclusterissuer","msg":"Reconciler error","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAClusterIssuer","name":"poc-ca","namespace":"","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: ae89e38d-60c9-4b8c-9280-f81052238788, api error MissingAuthenticationToken: Request is missing Authentication Token","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
key part:
operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: ae89e38d-60c9-4b8c-9280-f81052238788, api error MissingAuthenticationToken: Request is missing Authentication Token
Is this supported in general? The readme states that:
Access to AWS can also be configured using an EC2 instance role.
How would we go about doing that?
aws-privateca-issuer pods are able to pull correct image at correct image tag, without having to specify an image tag.
public.ecr.aws/k1n1h4h4/cert-manager-aws-privateca-issuer:v1.2.4
vs public.ecr.aws/k1n1h4h4/cert-manager-aws-privateca-issuer:1.2.4
aws-privateca-issuer pods attempt to pull image public.ecr.aws/k1n1h4h4/cert-manager-aws-privateca-issuer:1.2.4
, which fails.
No response
No response
1.2.4
Other
Severity 3
Usage case:
I would like to use aws-pca-issuer to issue certificates that can only be used for TLS clients or servers.
Expected behavior:
Specify KeyUsage: [ "server auth" ]
or KeyUsage: [ "client auth" ]
on a certificate resource and be issued a certificate that is limited to those usages.
Actual behavior:
KeyUsage on the certificate resource is ignored and a certificate is issued with both client and server key usages.
This is because no TemplateArn
is specified to IssueCertificate()
so all certificates use the default EndEntityCertificate
template which includes both client and server key usages.
References:
We're deploying aws-privateca-issuer using Helm on EKS 1.18 with the following config but somehow it is unable to read the region from AWSPCAClusterIssuer spec
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
awspca cert-manager 1 2021-09-22 23:39:06.1956528 +0530 IST deployed aws-privateca-issuer-0.1.2 v0.3.1
Deploying the issuer using the following config
Note: We are using secrets and not KIAM
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: awspca
namespace: cert-manager
spec:
region: us-east-1
arn: arn:aws:acm-pca:us-east-1:xxxxxxxxxx:certificate-authority/xxxxxxx-xxxx-xxxx-9526-xxxxxxxxxx
secretRef:
namespace: cert-manager
name: pca-access
ERROR
{"level":"error","ts":1632339059.6446667,"logger":"controller-runtime.manager.controller.awspcaclusterissuer","msg":"Reconciler error","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAClusterIssuer","name":"awspca","namespace":"","error":"operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
Everything starts working when we pass AWS_REGION into the deployment container env variables
spec:
containers:
- args:
- --leader-elect
command:
- /manager
env:
- name: AWS_REGION
value: us-east-1
image: ghcr.io/jniebuhr/aws-pca-issuer:v0.3.1
imagePullPolicy: IfNotPresent
Any idea why the issuer would never validate?
Here are my manifests:
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAIssuer
metadata:
name: awspcaissuer
namespace: cert-manager
spec:
arn: {{ .Values.pcaArn }}
region: {{ .Values.region }}
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Chart.Name }}
labels:
name: {{ .Chart.Name }}
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: {{ .Values.roleArn }}
name: {{ .Chart.Name }}
namespace: {{ .Chart.Name }}
Here are the values I passed to the aws-pricateca-issuer helm chart and cert-manager helm chart:
cert-manager:
installCRDs: true
serviceAccount:
create: false
name: cert-manager
aws-privateca-issuer:
serviceAccount:
create: false
name: cert-manager
Here's the output when I try to describe the issuer:
$ kubectl describe -n cert-manager awspcaissuers.awspca.cert-manager.io
Name: awspcaissuer
Namespace: cert-manager
Labels: argocd.argoproj.io/instance=cert-manager
Annotations: <none>
API Version: awspca.cert-manager.io/v1beta1
Kind: AWSPCAIssuer
Metadata:
Creation Timestamp: 2021-08-03T14:23:45Z
Generation: 1
Managed Fields:
API Version: awspca.cert-manager.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:argocd.argoproj.io/instance:
f:spec:
.:
f:arn:
f:region:
Manager: argocd-application-controller
Operation: Update
Time: 2021-08-03T14:23:45Z
Resource Version: 45566948
Self Link: /apis/awspca.cert-manager.io/v1beta1/namespaces/cert-manager/awspcaissuers/awspcaissuer
UID: d5a200ab-0ff9-4f77-858d-1d6e89c2b0f7
Spec:
Arn: arn:aws:acm-pca:us-east-1:734116470960:certificate-authority/3c01fb76-eced-49a2-8076-3bf2110807a0
Region: us-east-1
Events: <none>
You'll see no events ever fire, including the one where the issuer is validated.
Here's my certificate:
$ kubectl describe -A certificate
Name: cms-devops
Namespace: default
Labels: argocd.argoproj.io/instance=cms-devops
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2021-08-03T13:09:57Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:argocd.argoproj.io/instance:
f:spec:
.:
f:commonName:
f:dnsNames:
f:duration:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:privateKey:
.:
f:algorithm:
f:size:
f:renewBefore:
f:secretName:
f:usages:
Manager: argocd-application-controller
Operation: Update
Time: 2021-08-03T13:09:57Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:lastFailureTime:
Manager: controller
Operation: Update
Time: 2021-08-03T14:09:59Z
Resource Version: 45559101
Self Link: /apis/cert-manager.io/v1/namespaces/default/certificates/cms-devops
UID: 884b894b-acf8-4136-8075-d0719934055b
Spec:
Common Name: cms-devops.william-sandbox.internal
Dns Names:
cms-devops.william-sandbox.internal
Duration: 2160h0m0s
Issuer Ref:
Group: awspca.cert-manager.io
Kind: AWSPCAClusterIssuer
Name: root-ca
Private Key:
Algorithm: RSA
Size: 2048
Renew Before: 360h0m0s
Secret Name: cms-devops-tls-app-secret
Usages:
server auth
client auth
Status:
Conditions:
Last Transition Time: 2021-08-03T14:09:59Z
Message: The certificate request has failed to complete and will be retried: issuer is not ready
Observed Generation: 1
Reason: Failed
Status: False
Type: Issuing
Last Transition Time: 2021-08-03T13:09:57Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Last Failure Time: 2021-08-03T14:09:59Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 36m (x2 over 96m) cert-manager Issuing certificate as Secret does not exist
Warning Failed 36m (x2 over 96m) cert-manager The certificate request has failed to complete and will be retried: issuer is not ready
Normal Generated 36m cert-manager Stored new private key in temporary Secret resource "cms-devops-pj7gp"
Normal Requested 36m cert-manager Created new CertificateRequest resource "cms-devops-2dfh5"
You can see that the the certificate never completes because the issuer is not ready. Is something misconfigured?
There is currently an issue where certificate requests with validity duration < 24h will fail to issue because of a division by 24 and cast to int64, resulting in 0 for the validity field. This results in an api error
from PCA because PCA requires validity is not 0.
Please see #69 for more details.
While we work to cut a release to fix this change (#70), please use validity durations greater than 24h.
In an Istio enabled environment when egress filtering is enabled, Istio uses the hostname / sni to do egress hostname matching.
If there is no tls server name / sni then Istio can't match the oubound tcp port 443 connection so it would block it.
Istio sidecar resource can allow all but that defeats the purpose of having Istio perform egress filtering.
No response
This issue provides visibility into Renovate updates and their statuses. Learn more
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
I wanted to upgrade to the latest v0.3.0 but when I tried I noticed I started seeing exceptions in the logs.
We are using AWS IAM roles as annotations on the serviceAccount to handle the authorization. This works in v0.2.1 and I've looked through the code but I see no differences between 0.2.1 and 0.3.0 that would cause this.
{"level":"error","ts":1626803861.6337998,"logger":"controllers.GenericIssuer","msg":"failed to sts.GetCallerIdentity","genericissuer":"/aws-issuer","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: dd011a20-5b5f-4191-9996-cd709af162f9, api error MissingAuthenticationToken: Request is missing Authentication Token","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*GenericIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/genericissuer_controller.go:122\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*AWSPCAClusterIssuerReconciler).Reconcile\n\t/workspace/pkg/controllers/awspcaclusterissuer_controller.go:57\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
{"level":"error","ts":1626803861.6339357,"logger":"controller-runtime.manager.controller.awspcaclusterissuer","msg":"Reconciler error","reconciler group":"awspca.cert-manager.io","reconciler kind":"AWSPCAClusterIssuer","name":"aws-issuer","namespace":"","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: dd011a20-5b5f-4191-9996-cd709af162f9, api error MissingAuthenticationToken: Request is missing Authentication Token","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
I installed aws-privateca-issuer as per documentation.
I created a new AWSPCAIssuer, everything seems to be fine, all resources are ready, no issues reported.
Status:
Conditions:
Last Transition Time: 2021-06-27T22:06:19Z
Message: Issuer verified
Reason: Verified
Status: True
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Verified 38m awspcaissuer-controller Issuer verified
Normal Verified 11m awspcaissuer-controller Issuer verified
When I create a Certificate using my AWSPCAIssuer, the CertificateRequest is generated, but there are no events logged and it is stuck in unknown state.
Any hints how to start to debug/resolve this?
Currently awspca configuration needs creating a dedicated IAM user with proper access and define a secret with access key - but since it is used in EKS it would be better to use dedicated service account with assigned IAM policy. It would be more natural way I think to integrate it. How do you think? (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)
No response
No response
My team finds it helpful to be able to search for helm charts on artifacthub.io, which also displays version information and security reports for the images used by a chart. The main chart for cert-manager is already listed, however I believe that published via a separate repository. Could you submit the repository for this plugin to artifacthub.io also?
We can continue to just directly look at this repository for chart information and updates, so this is a nice-to-have, not essential.
No response
Most probably this is caused by the version of apiextensions.k8s.io
because it looks like the go code cannot identify CRDs:
2021-07-30T14:31:32.031Z INFO controllers.GenericIssuer sts.GetCallerIdentity {"genericissuer": "/test", "arn": "arn:aws:sts::410807025456:assumed-role/k8s_certissuer_awsprivateca/1627655491949926463", "account": "410807025456", "user_id": "AROAV7JQBX4YAPTSZ6F75:1627655491949926463"}
2021-07-30T14:31:32.031Z INFO controllers.GenericIssuer Calling StoreProvisioner {"genericissuer": "/test"}
2021-07-30T14:31:32.032Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AWSPCAClusterIssuer","name":"test","uid":"3c9fd077-f103-11eb-b462-067082f90d8f","apiVersion":"awspca.cert-manager.io/v1beta1","resourceVersion":"89368672"}, "reason": "Verified", "message": "Issuer verified"}
2021-07-30T14:31:32.033Z ERROR controller-runtime.manager.controller.awspcaclusterissuer Reconciler error {"reconciler group": "awspca.cert-manager.io", "reconciler kind": "AWSPCAClusterIssuer", "name": "test", "namespace": "", "error": "the server could not find the requested resource (put awspcaclusterissuers.awspca.cert-manager.io test)"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.