MissingAuthenticationToken: Request is missing Authentication Token about aws-privateca-issuer HOT 15 CLOSED

@divyansh-gupta I have been working on a fix for it. To get it to work I had to redo how the aws config is loaded. I put together a branch and draft PR here but I am very new to Go so I am not sure if I have something wrong. Just wanted to get something for you to see and/or use to get a fix in place. I tested this and this code works for me when using the steps above. I was even able to get istio-csr and istiod working with it.

#41

from aws-privateca-issuer.

I believe the issue that I'm having is #50 . I will follow up there. Thanks!

from aws-privateca-issuer.

I'm seeing the same issue myself, although in this case, it's a brand new installation, not an existing upgrade. Like yourself, I'm also using AWS IAM Role annotations on the ServiceAccount to provide the permissions to the Pod.

from aws-privateca-issuer.

Same issue here. Tried 2.1.0 looks to work fine, guess the AWS SDK was touched between these versions breaking it.

Could not get 3.0.0 to work with WebTokens IAM or EC2 machine roles.

from aws-privateca-issuer.

I am taking a look at this issue and assigning it to myself.

from aws-privateca-issuer.

Thanks. I can confirm that downgrading to 0.2.1 worked for me too.

from aws-privateca-issuer.

@bpotaczek or @jonathanio can you please provide yours steps to reproduce the issue?

from aws-privateca-issuer.

To reproduce I either start with a fresh cluster or remove istio, istio-csr, aws-pca-issuer charts plus delete all Certificates, CertificateRequests and AWSPCAClusterIssuers.

Then I install the aws-pca-issuer with my values.yaml file which has the serviceAccount create and custom annotation for the IRSA authentication.

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXXX:role/aws-pca-issuer-k8s-aws-pca-issuer-dev

I start following the pca issuer logs.

After the pod is fully running and listening I create an AWSPCAClusterIssuer.

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
  name: aws-issuer
spec:
  region: us-east-1
  arn: arn:aws:acm-pca:us-east-1:XXXXXXXXXXXX:certificate-authority/00000000-0000-0000-0000-000000000000

I have tried using an ACM in the same account as well as a RAM share ACM and both fail.

from aws-privateca-issuer.

@bpotaczek Thanks for the steps to reproduce - I was able to reproduce the issue using an instance profile, running on an EC2 instance.

I am also testing your PR, thank you putting that out.

from aws-privateca-issuer.

Hey @bpotaczek, thanks for your work on that fix. We have merged it in: #41

We will be cutting a new release soon. Closing, please re-open if there are any other comments.

from aws-privateca-issuer.

Hey 👋 Something which seems related to this issue. With version 0.3.1 when I try to use IAM and eks service account I get access denied to the resource. Downgrading the version to v0.2.1 works fine and I'm able to issue certificates

from aws-privateca-issuer.

@joaosilva15 Thanks for reaching out.

With 0.3.1 release, you need updated IAM policy as called out in the README file:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "awspcaissuer",
      "Action": [
        "acm-pca:DescribeCertificateAuthority",
        "acm-pca:GetCertificate",
        "acm-pca:IssueCertificate"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:acm-pca:<region>:<account_id>:certificate-authority/<resource_id>"
    }
  ]
}

Can you please verify that IAM principal has the right policy attached for certificate issuance? Thanks!

from aws-privateca-issuer.

Hey. Yes the policies were attached to the role and I could use the role with the cli. Downgrading the app to 0.2.1 fixed the issue and I'm able to issue the certificate.
A thing that also was strange was that the role did not changed it's last used time in the console while the issuer was trying to issue the cert

Regarding our setup (just to clear all possibilities), we have a subordinate CA shared with the account of the issuer and we want to issue certs for that account.

Thank you :)

from aws-privateca-issuer.

Hi @joaosilva15, could you please provide steps to reproduce? I was not able to reproduce. Please check out #48 as well, is your issue related to fsGroup?

from aws-privateca-issuer.

Hi. So our setup is

We have an aws account with only acm pca. There we have a root CA and we create a subordinate CA from the root. This subordinate CA is then shared with multiple accounts.

We have then in each account where the subordinate was shared an eks cluster with certmanager and aws pca issuer running. The containers for pca issuer are running with a service account that is associate with a role with the policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "awspcaissuer",
      "Action": [
        "acm-pca:GetCertificate",
        "acm-pca:GetCertificateAuthorityCertificate",
        "acm-pca:IssueCertificate",
        "acm-pca:DescribeCertificateAuthority"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:acm-pca:eu-west-1:<account>:certificate-authority/*"
    }
  ]
}

I was running around with version 0.3.1 and getting error like {"level":"error","ts":1629727036.659929,"logger":"controllers.CertificateRequest","msg":"failed to request certificate from PCA","certificaterequest":"default/test-xhzft","error":"operation error ACM PCA: IssueCertificate, https response error StatusCode: 400, RequestID: fd99a19a-011a-4301-a426-6676fd59ca42, api error AccessDeniedException: User: arn:aws:sts::<test:account>:assumed-role/<test-role> is not authorized to perform: acm-pca:IssueCertificate on resource: arn:aws:acm-pca:eu-west-1:<ca-account>:certificate-authority/<ca-shared>","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/cert-manager/aws-privateca-issuer/pkg/controllers.(*CertificateRequestReconciler).Reconcile\n\t/workspace/pkg/controllers/certificaterequest_controller.go:171\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}

I downgraded the version to 0.2.1 and everything works so I think it should be something related with this fix? Not sure though, if you want I can open a new issue.

Regarding your comment, the issue is not that the container cannot fetch the token, it knows what is his role and region, but to clear doubts I added the fsGroup to the container and the issue still occurs.

from aws-privateca-issuer.