Aws::Errors::MissingCredentialsError exception in ECS when using TaskRoleArn about aws-sdk-ruby HOT 10 CLOSED

I'll contact the docs examples team to have that rewritten. Thanks for pointing that out.

from aws-sdk-ruby.

Path forward is to do Kernel.warn on various credential sources (EC2 instance, ECS, Process, etc) when they aren't loaded. This should be fine to do because those credential sources are only initialized when certain hint checks are passed (i.e. existence of that relative URI ENV variable).

from aws-sdk-ruby.

Sorry you're running into this - you are right, credentials from ECS TaskRoleArn should just work.

What is AWS_CONTAINER_CREDENTIALS_RELATIVE_URI set to and are you setting it manually? Are there any other ENV vars set?

from aws-sdk-ruby.

I'm not setting it, it's being set by ECS. It's being set to the following, redacting the UUID as I assume that's something I shouldn't share:

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/[UUID]

There's no other ENV vars that appear auth related. This is a complete set of container ENV vars, minus the ones specific to my app which have a unique, app-specific prefix:

AWS_EXECUTION_ENV=AWS_ECS_EC2
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/[UUID]
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/[UUID]
ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/[UUID]
ECS_AGENT_URI=http://169.254.170.2/api/[UUID]
PATH=/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
RUBY_VERSION=3.2.3
RUBY_DOWNLOAD_URL=https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.3.tar.xz
RUBY_DOWNLOAD_SHA256=cfb231954b8c241043a538a4c682a1cca0b2016d835fee0b9e4a0be3ceba476b
GEM_HOME=/usr/local/bundle
BUNDLE_SILENCE_ROOT_WARNING=1
BUNDLE_APP_CONFIG=/usr/local/bundle

from aws-sdk-ruby.

From your host, can you try pinging http://169.254.170.2/v2/credentials/[UUID] to see if this endpoint exists?

We can get more insight if you try initializing these credentials manually:

Aws::ECSCredentials.new(http_debug_output: $stdout) # or your logger

from aws-sdk-ruby.

Also please move away from Resource models (use Client), they are not recommended because they are hand maintained and do not get updates.

from aws-sdk-ruby.

Thanks for the quick response and good troubleshooting steps. Once I realized how this is supposed to work I was able to root cause the issue: The ECS agent metadata endpoint wasn't responding. Once I resolved that Aws::ECSCredentials.new(http_debug_output: $stdout) was able to gather credentials as expected. For anyone else here from Google (there were exactly 0 hits for "ecs taskRoleArn Aws::Errors::MissingCredentialsError" last night) the solution was a host reboot.

To try and reproduce the issue I ran a container without network and instantiated a Aws::ECSCredentials object:

irb(main):002:0> Aws::ECSCredentials.new(http_debug_output: $stdout)
opening connection to 169.254.170.2:80...
opening connection to 169.254.170.2:80...                                     
opening connection to 169.254.170.2:80...                                     
opening connection to 169.254.170.2:80...                                     
opening connection to 169.254.170.2:80...                                     
opening connection to 169.254.170.2:80...                                     
=>                                                                            
#<Aws::ECSCredentials:0x00007f0b8cbc05b8                                      
 @async_refresh=false,                                                        
 @backoff=#<Proc:0x00007f0b8c26f4a0 /usr/local/bundle/gems/aws-sdk-core-3.191.0/lib/aws-sdk-core/ecs_credentials.rb:177 (lambda)>,
 @before_refresh=nil,                                                         
 @credential_path="/v2/credentials/1ffa0a10-9270-4233-985b-3bdd542f4b88",     
 @credentials=#<Aws::Credentials access_key_id=nil>,                          
 @expiration=nil,
 @host="169.254.170.2",
 @http_debug_output=#<IO:<STDOUT>>,
 @http_open_timeout=5,
 @http_read_timeout=5,
 @mutex=#<Thread::Mutex:0x00007f0b8c26f400>,
 @port=80,
 @retries=5,
 @scheme="http">

And I also reproduced the original exception as well. So this isn't a bug, as it failed due to the ecs-agent not responding, but I do think Aws::ECSCredentials should throw a warning if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set but 169.254.170.2:80 is unresponsive. That's a ENV var I doubt will ever be encountered outside ECS, so if 169.254.170.2 isn't responding that's an error worth bubbling up to the user. If I had realized last night that the http://169.254.170.2 ENV vars did matter I probably would have been able to figure it out.

So I think we can remove the bug label, but would you be open to leaving this open as a feature request to add a warning level log to the Aws::ECSCredentials class?

from aws-sdk-ruby.

Also please move away from Resource models (use Client), they are not recommended because they are hand maintained and do not get updates.

FYI https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/sns-example-send-message.html still documents using Aws::SNS::Resource.

from aws-sdk-ruby.

Related to: #2823

from aws-sdk-ruby.

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

from aws-sdk-ruby.