Jenkins Startup fails on AWS ECS due to secrets-manager-credentials-provider-plugin about aws-secrets-manager-credentials-provider-plugin HOT 20 CLOSED

Hi I had the same problem @thedevopsguyblog note because that my exact stack: casc + ECS + secrets-manager-credentials-provider-plugin.
I have two experiences to share:

• It was not happening to me on Fargate, only when I switched to EC2 cluster
• Whereas @chriskilding solution adding AWS_DEFAULT_REGION to the task definition didnt work. It works with AWS_REGION! 💯

from aws-secrets-manager-credentials-provider-plugin.

First thing to check, what IAM permissions does the Jenkins principal (or if on a cluster, the nearest relevant principal) have? If Jenkins doesn't have the right permissions it will not be able to access secrets.

from aws-secrets-manager-credentials-provider-plugin.

I SSH'd onto my ECS server, git cloned my Dockerfile, built and ran the image and was able to replicate the error.
In fact, I got a more specific error message from Jenkins during startup, this is was very helpful.

2021-07-08 12:10:56.861+0000 [id=28]    WARNING c.a.util.EC2MetadataUtils#getItems: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Failed to connect to service endpoint: 
java.net.SocketTimeoutException: connect timed out

@chriskilding I also suspected IAM perms were the root cause, but at the time was unsure.

You may or may not know this, but where do I attach the IAM Policy?

• The Ec2 autoscale group?
• Or the ECS Task?

Any contribution is greatly appreciated.

from aws-secrets-manager-credentials-provider-plugin.

I've not used Jenkins on ECS myself, but I imagine you could start by putting it on the ECS task definition. Either directly, or you might need to make a Jenkins IAM role with the policy, and set the role ARN on the task definition.

from aws-secrets-manager-credentials-provider-plugin.

Upon further investigation, the secrets-manager plugin want's access to instance-identity-documents This metadata is only available on EC2 instances.

So if someone tries to use JCasC + ECS + secrets-manager-credentials-provider-plugin they are going to run into this issue, cause the container application can't natively access the hosts metadata file.

I'll see if I can get Jenkins on ECS to access the hosts metadata file.

This will also be an issue on EKS.

from aws-secrets-manager-credentials-provider-plugin.

I can confirm that IAM is not the root cause, If the plugin fails to pull secrets due to insufficent permissions in the IAM policy, the error message will look something like this...

2021-07-09 06:23:55.581+0000 [id=29]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
com.amazonaws.services.secretsmanager.model.AWSSecretsManagerException: User: arn:aws:sts::xxxx:assumed-role/SERVERNAME-InstanceRoleID-number is not authorized to perform: secretsmanager:GetSecretValue on resource: sampleSecretVaule (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; 

Instead the Error is this...

2021-07-08 12:10:56.861+0000 [id=28]    WARNING c.a.util.EC2MetadataUtils#getItems: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Failed to connect to service endpoint: 
java.net.SocketTimeoutException: connect timed out

In an attempt to mimic AWS ECS I did the following...

1. Created a new server using the latest amazonLinux2 AMI
2. Installed awscli
3. Authenticated with AWS ECR (aws ecr get-login-password) and pulled down my custom Jenkins image
4. Jenkins is fully up and running

The container started cause it could access, 169.254.169.254/latest/dynamic/instance-identity/document, this returns a json object containing the region and availabilityZone, I'm assuming the plugin needs this to figure out where the secrets are.

When starting Jenkins in AWS ECS, that endpoint is not available.

Not really sure how to procede now.

from aws-secrets-manager-credentials-provider-plugin.

I believe we have teams in our workplace that do use the plugin on ECS or EKS, so I'll ask around and see what they suggest

from aws-secrets-manager-credentials-provider-plugin.

As a short term suggestion, you could try setting the AWS_DEFAULT_REGION on the Jenkins container in ECS and see if that helps?

from aws-secrets-manager-credentials-provider-plugin.

It's also possible that there is simply a bug to do with this in the AWS Java SDK, that may since have been fixed. Could you post which version of the Jenkins AWS Java SDK plugin you have?

from aws-secrets-manager-credentials-provider-plugin.

It's also possible that there is simply a bug to do with this in the AWS Java SDK, that may since have been fixed. Could you post which version of the Jenkins AWS Java SDK plugin you have?

I'm running Jenkins in docker using the jenkins/jenkins:lts image.

jenkins@12bf88526df6:/$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

AWS SDK Plugin

Amazon Web Services SDK - 1.11.995

from aws-secrets-manager-credentials-provider-plugin.

I'm looking again at the original NullPointerException stack trace you posted. Most likely the client in AwsSecretSource (that's the AWS SDK object) is null when the GetSecretValue happens.

The client is created in an init() method when Jenkins starts up. If this fails the client won't get made and an exception is logged.

Could you scan back in your logs and find a message around the time of that stack trace that starts with:

"Could not set up AWS Secrets Manager client. Reason:"

And post it?

from aws-secrets-manager-credentials-provider-plugin.

(That message should be logged at WARNING level btw)

from aws-secrets-manager-credentials-provider-plugin.

Hey @chriskilding , i'll have a look at this soon.
Just in the middle of migrating Jenkins to ECS and we've switched to using Parameter Store for now.

Also Parameter Store is free but Scrts Mgr is .40c a request/per month.
Financially, P.Store seems cheaper.
Why might one use Scrts Mgr over P.Store (apart from automated rotation)?

from aws-secrets-manager-credentials-provider-plugin.

It is a good question, AWS themselves seem to have created 2 services that sort-of overlap but not quite. I've never seen an explanation for why they did this.

I compared them a while back in #72. From that table, you'd use Secrets Manager if you need:

• Binary secrets
• A larger max secret size

Beyond that, we'd have to know about the design intent of each service.

from aws-secrets-manager-credentials-provider-plugin.

I was also running into this issue with other plugins that rely on the AWS SDK.
In my Dockerfile I added this line, the creds file contains region=ap-southeast-2

COPY --chown=jenkins:jenkins credentials /var/jenkins_home/.aws/credentials

It's annoying cause now I'm locked into 1 region, I could see how this might cause issues with larger organisations.
Seems like AWS plugins aren't looking at ECS Metadata.

I wonder how other people are solving this problem? @chriskilding

from aws-secrets-manager-credentials-provider-plugin.

To authenticate with AWS, the plugin is merely creating a standard version of the Secrets Manager client, which uses the DefaultAWSCredentialsProviderChain under the hood.

This is what any other user of the AWS Java SDK (V1) would do - it's not specific to Jenkins plugins.

Per AWS docs the chain looks for credentials in this order:

• Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
• Java System Properties - aws.accessKeyId and aws.secretKey
• Web Identity Token credentials from the environment or container
• Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
• Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
• Instance profile credentials delivered through the Amazon EC2 metadata service

The last 2 approaches on the list (any form of Amazon container lookup, be that EC2 or ECS) are handled in EC2ContainerCredentialsProviderWrapper. The docs for that say:

AWSCredentialsProvider that loads credentials from an Amazon Container (e.g. EC2) Credentials are solved in the following order:
If environment variable "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" is set (typically on EC2) it is used to hit the metadata service at the following endpoint: http://169.254.170.2
If environment variable "AWS_CONTAINER_CREDENTIALS_FULL_URI" is set it is used to hit a metadata service at that URI. Optionally an authorization token can be included in the "Authorization" header of the request by setting the "AWS_CONTAINER_AUTHORIZATION_TOKEN" environment variable.
If neither of the above environment variables are specified credentials are attempted to be loaded from Amazon EC2 Instance Metadata Service using the InstanceProfileCredentialsProvider.

The fact that in your case it's falling through to the EC2 metadata service suggests the ECS code branches didn't supply a credential. Could you check if one of those AWS_CONTAINER environment variables are set (or if not set it anyway for Jenkins as an override) and see if that changes things?

from aws-secrets-manager-credentials-provider-plugin.

Not sure if this was mentioned: Several online guides for Jenkins + ECS include a guideline to block access from the container to the instance metadata. Here are the instructions to do so https://aws.amazon.com/premiumsupport/knowledge-center/ecs-container-ec2-metadata/ .

I would recommend that you check your ECS stack and if there are similar instructions in the USER_DATA of the Launch Configuration or Launch Template used for launching ECS Container Instances.

from aws-secrets-manager-credentials-provider-plugin.

Great! If setting AWS_REGION is a solution that consistently works for ECS, we can add that to the README.

(It's a little strange that a common AWS_ environment variable wouldn't be set by default in an AWS environment though.)

from aws-secrets-manager-credentials-provider-plugin.

Hi @chriskilding, I will close this issue.

As you mentioned it's not the plugin thats at fault but its the AWS SDK behaving weirdly. It also seems that you will only run into this issue if your are using an EC2 cluster with ECS.

@jairov4 has confirmed the workaround.

from aws-secrets-manager-credentials-provider-plugin.

Great, I've also added the workaround (the note about setting AWS_REGION manually) to the authentication guide.

from aws-secrets-manager-credentials-provider-plugin.