Are there plans to add the v1 endpoint?

I know that it's very different in the way that it's expose to the tasks since it's the agent directly, but I have a use case for it and was wondering if there are plans to add it.

Thx

Updating your example Docker Compose file link in docs/setup-networking.md from examples/docker-compose.yml to ../examples/docker-compose.yml should fix it.

I didn't want to make PR for something so trivial -- best to leave that for actual code related stuff. Anyway this tool is pretty cool and helping me as I learn ECS, so thank you!

I've very modified the example to use a profile from my local config:

    environment:
      # define the home folder; credentials will be read from $HOME/.aws
      HOME: "/home"
      # You can change which AWS CLI Profile is used
      AWS_PROFILE: "assumed-profile"

To authenticate as assumed profile at the command line I would need to provide an MFA code. My config looks something like:

[profile assumed_profile]
role_arn = arn:aws:iam::etc
mfa_serial = arn:aws:iam::etc
source_profile = default
region = eu-west-2
output = json

Running my docker compose file now I get the error:

Failed to create Credentials Service: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

How do i tell the local endpoint the MFA code so that it can assume the role?

Is there a schedule or plan to release new builds with recent changes to Docker Hub?

Hi there, can someone please help me understand what I'm doing wrong.
I'm following the docs to make local env setup working.

Here is my override file

version: "2"
networks:
  credentials_network:
    driver: bridge
    ipam:
      config:
        - subnet: "169.254.170.0/24"
          gateway: 169.254.170.1
services:
  ecs-local-endpoints:
    image: amazon/amazon-ecs-local-container-endpoints
    volumes:
      - /var/run:/var/run
      - "~/.aws:/home/.aws:ro"
    # env_file:
    #   - ~/.clokta/fusion.env
    environment:
      HOME:
      AWS_PROFILE: "fusion"
    networks:
      credentials_network:
        ipv4_address: "169.254.170.2"
  proxy:
    depends_on:
      - ecs-local-endpoints
    networks:
      credentials_network:
        ipv4_address: "169.254.170.3"
    environment:
      AWS_DEFAULT_REGION: "us-east-1"
      AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/blah-role"
      ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

I've logged in through SSO and have STS issued credentials.
With this set up I'm making the following request:

GET 169.254.170.2/role/blah-role HTTP/1.1
Host:169.254.170.2
Connection: close

As a result, I'm getting the following response:

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 25 Feb 2020 19:06:08 GMT
Content-Length: 435
Connection: close

Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, fusion.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused

ecs local container logs has this line:

ecs-local-endpoints_1  | time="2020-02-25T19:06:07Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, fusion.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused"

However, if I uncomment

    # env_file:
    #   - ~/.clokta/fusion.env

and rerun docker-compose, everything seems fine.

~/.clokta/fusion.env contains AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Those values match the [fusion] section of the ~/.aws/credentials which I'm mapping to the container, except aws_access_key_id key and friends are in the lower case.

Is there something I'm missing in the setup?

Hey,

For our particular setup, which requires the use of MFA, I get

ecs-local-endpoints_1 | time="2019-10-24T14:23:00Z" level=fatal msg="Failed to create Credentials Service: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set."

Is there an alternative way to do this?

Cheers!

My goal is to simulate in a local docker-compose (or docker compose) cluster the same credential acquisition that occurs when running my containers in Fargate (as deployed with the compose-cli).

I am able to start the amazon-ecs-local-container-endpoints in my cluster using docker-compose and I am following the example provided. However, because I want to simulate the Fargate experience, I use the /creds/{role name} model. My application container therefore has the following env vars set:

      AWS_REGION: "us-east-1"
      AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds/pipeline-PlineTaskRole-XC885YZXQBDN"
      ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

In the amazon-ecs-local-container-endpoints container, essentially copied the example:

  ecs-local-endpoints:
    image: amazon/amazon-ecs-local-container-endpoints
    volumes:
      - /var/run:/var/run
      - $HOME/.aws/:/home/.aws/
    environment:
      HOME: "/home"
      AWS_PROFILE: "default"
    networks:
      credentials_network:
        ipv4_address: "169.254.170.2"

When I start the cluster and attempt to retrieve credentials using DefaultCredentialsProvider from the AWS SDK for Java (version 2.15.57) I get an exception:

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): The requested metadata is not found at http://169.254.170.2/creds/pipeline-PlineTaskRole-XC885YZXQBDN, InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]

The last entry in that list is telling: it appears the SDK is making the right call, but the endpoints container is not answering.

Addendum: On the assumptions that my default profile was somehow deficient, I attempted to add the (inline) trust policy per the documentation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::264945611335:user/Administrator"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

However, the AWS IAM console consistently fails to accept the policy and returns a vague This policy contains the following error: Has prohibited field Principal For more information about the IAM policy grammar, see AWS IAM Policies

Hello!

Currently the dockerhub image is amd64 only. Would it be possible to push a multi-arch (amd64 + arm64) manifest so as to support Graviton/Graviton2-based instances?

Thanks!

When vending credentials using the AWS_CREDENTIALS_RELATIVE_URI=role/<role_name> environment variable, it is only possible to assume a role in the same account as the credentials in ~/.aws/credentials.

In line with a recommended account structure (Identity Account Structure) it is often the case that credentials on a developer machine are only for an identity account, but give permissions to assume roles in other accounts. This workflow does not work with this container, as the container can only assume roles in said identity account.

It would be good if this container could accept an optional environment variable for the account number in which the target role exists, and maybe even a corresponding role name that can be used for role discovery (via the iam.GetRole() SDK call)

👋 Hey all, thanks for making this image available.

Is there any planned support for V4 metadata endpoints? Do we have an ETA on this?

Thanks

Same issue as here #23

This is path 1b (grabbing a temporary credential and stashing it in ~/.aws/credentials) from issue #23 -- it makes Boto unhappy.

Retrieving the credentials, as @PettitWesley alludes to, doesn't include an expiration date because this isn't a thing in ~/.aws/credentials:

% curl http://169.254.170.2/creds
{"AccessKeyId":"ASIA----------EGKVOM","Expiration":"","RoleArn":"","SecretAccessKey":"4su0....","Token":"FQoG...."}

Boto barfs on the empty Expiration field:

# export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
# python3                       
Python 3.7.4 (default, Jul 30 2019, 19:56:38) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.9.246'
>>> boto3.client("ssm")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 827, in create_client
    credentials = self.get_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 426, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1934, in load_credentials
    creds = provider.load()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1797, in load
    return self._retrieve_or_fail()
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1812, in _retrieve_or_fail
    expiry_time=_parse_if_needed(creds['expiry_time']),
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 196, in _parse_if_needed
    return parse(value)
  File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1358, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 652, in parse
    raise ValueError("String does not contain a date:", timestr)
ValueError: ('String does not contain a date:', '')

My suggestion is to include expiration as a valid field in the ~/.aws/credentials file -- it really ought to be there for temporary credentials like this.

I'm fairly certain the session token does include an expiration date in there, but it's not for consumption outside of the AWS Auth service. (I'm fairly certain it's encrypted, and if we could decrypt it anyone could also go about generating secret keys and session tokens, which would be... bad. The Auth team would know for sure, though.)

I am unable to successfully get task metadata from the v3 metadata uri due to a failure to connect to the docker.sock file.

In order to avoid any issues stemming from my application itself, I opened a terminal in the ecs-local-endpoints container, and tested some curl commands. The output matches what I see in the logs when I call from my application.

sh-4.2# curl http://169.254.170.2/v3 
Internal Server Error: Failed to list running containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

sh-4.2# curl http://169.254.170.2/v3/task
Internal Server Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The creds endpoint works just fine:

sh-4.2# curl http://169.254.170.2/creds
{"AccessKeyId":"XXXXXXXX","Expiration":"2022-12-06T16:58:13Z","RoleArn":"","SecretAccessKey":"XXXXXX","Token":"XXXXX"}

The docker.sock file is properly mapped into the container. And just in case the symlink was an issue, I also tried just mapping the direct path, but it doesn't seem to make a difference.

sh-4.2# ls -l /var/run/docker.sock
lrwxr-xr-x 1 root root 43 Nov 28 20:23 /var/run/docker.sock -> /Users/my.username/.docker/run/docker.sock

## and with direct path mapped ##

sh-4.2# ls -l /var/run/docker.sock
srwxr-xr-x 1 root root 0 Dec  6 16:08 docker.sock

Here is my docker-compose overrides file:

version: "2.4"

networks:
  credentials_network:
    driver: bridge
    ipam:
      config:
        - subnet: "169.254.170.0/24"
          gateway: 169.254.170.1
services:
  ecs-local-endpoints:
    image: amazon/amazon-ecs-local-container-endpoints:latest-arm64
    volumes:
      - /var/run:/var/run
      - $HOME/.aws/:/home/.aws/
    environment:
      HOME: "/home"
      AWS_PROFILE: "default"
    networks:
      credentials_network:
        ipv4_address: "169.254.170.2"

  app:
    depends_on:
      - ecs-local-endpoints
    networks:
      credentials_network:
        ipv4_address: "169.254.170.3"
    environment:
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_REGION=us-east-1
      - AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
      - ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v3
      - ECS_CONTAINER_METADATA_URI=http://169.254.170.2

I am using the latest version of Docker Desktop -- 4.15.0 (93002) on an M1 Macbook running MacOS 13.0

Is there something obviously wrong here, or is there a known issue either with running on M1 Macs or newer versions of docker?

I'm trying to run ecs-local-container-endpoints using the recommended approach ( Bridge Network )

I've set up everything as the guide suggest but the ecs-local-container do not work and prints this message evey time I send a request to http://169.254.170.2/creds:

Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, sm-sdx.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: no route to host

Although if I set it up using the ip tables approach eveything works fine.

I'm running on Ubuntu 20.4 with Docker version 20.10.9 and Docker compose version 1.29.2

It seems this exporter can't collect MSK metrics ?

Use case:

I am working on an application that gets deployed to ECS (Fargate). Our team has had good success using Docker Compose and localstack to simulate an environment with DynamoDB, S3, and SQS while developing/testing locally. I just started working on a reusable library that will extract container metrics from the ECS Task Metadata Endpoint. I was excited after finding this project, and eager to plug it into our existing Docker Compose script to better simulate the ECS environment.

It appears the current version only works with legit AWS credentials. We have some automated build agents, test scripts, and team members that don't have access to AWS. Instead of having to provision and manage this additional access, it would be great if this container were configurable to pipe the IAM and STS requests to the localstack services running in our Docker Compose network.

I'm not very familiar with the inner-workings of this container. I'm not sure if any additional service-specific endpoints should be open to customization beyond IAM and STS.

It would be great to publish this to the ECR Public Gallery: https://gallery.ecr.aws/

This would especially useful for anyone using Docker Hub anonymously or from a free account, in light of the new pull rate limits imposed.

Hi,
I am working with compose based on scenario 1 (https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/), everything work well till I configured my environment to MFA authentication.
Before MFA i just used the profile name and mounting the .aws directory, I change the docker-compose.override to use the current session, the problem is that I get (from Boto) ResourceNotFoundException when I am trying to work with Dynamo DB. any idea what am I missing?

in order to use the authentication session I did the following changes (see env vars below):

services:
# This container vends credentials to your containers
ecs-local-endpoints:
  # The Amazon ECS Local Container Endpoints Docker Image
  image: amazon/amazon-ecs-local-container-endpoints
  volumes:
    # Mount /var/run so we can access docker.sock and talk to Docker
    - /var/run:/var/run
    # Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
    # On Windows, this directory can be found at "%UserProfile%\.aws"
    - $HOME/.aws/:/home/.aws/
  environment:
    # define the home folder; credentials will be read from $HOME/.aws
    HOME: "/home"
    # You can change which AWS CLI Profile is used
    AWS_PROFILE: "DevAdmin"
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}" ---- **Get the KEY_ID from the local machine after MFA authentication**
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}" ---- **Get the ACCESS_KEY from the local machine after MFA** authentication
    AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}" ---- **Get the SESSION_TOKEN from the local machine after MFA** authentication
    AWS_REGION: "eu-west-1"

10X in advance.
Nir

The "Generic Metadata Injection" section within the "features" document contains the wrong url to the generic example

Broken Link (last line)

See example for overriding task metadata response [here](../example/generic).

Should be set to:

See example for overriding task metadata response [here](../examples/generic).

In docker hub, the tag latest etc is duplicated as latest-arm64 with each having one arch. This causes docker to select the incorrect build and run it by emulation on Docker for Mac on ARM. The latest tag should provide both architectures.

So different arch can pull the image of their own arch with pointing to "latest". https://hub.docker.com/layers/amazon/amazon-ecs-local-container-endpoints/latest/images/sha256-2f1be750a040f71540a1bca3c9fe3295030a022595a2fc8d3966623bd3f75bd2?context=explore

I am running this trying to use sms, but I get the following when I run docker-compose up
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
app_1 | aws_session_token=aws_session_token, config=config)
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 826, in create_client
app_1 | credentials = self.get_credentials()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 425, in get_credentials
app_1 | 'credential_provider').load_credentials()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1688, in load_credentials
app_1 | creds = provider.load()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1551, in load
app_1 | return self._retrieve_or_fail()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1566, in _retrieve_or_fail
app_1 | expiry_time=_parse_if_needed(creds['expiry_time']),
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 137, in _parse_if_needed
app_1 | return parse(value)
app_1 | File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1358, in parse
app_1 | return DEFAULTPARSER.parse(timestr, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 652, in parse
app_1 | raise ValueError("String does not contain a date:", timestr)
app_1 | ValueError: ('String does not contain a date:', '')
app_1 | unable to load app 0 (mountpoint='') (callable not found or import error)

Within the example Local Stack docker-compose file, custom iam and sts endpoint environment variables are passed as IAM_CUSTOM_ENDPOINT and STS_CUSTOM_ENDPOINT although there are not supported within the image which results in the image to route requests to AWS. Looking at the config.go file within the mainline and v1.4.0 the correct environment variables should be IAM_ENDPOINT and STS_ENDPOINT to enable custom endpoints.

if you using multiple sso profiles and use the sso_session config in .aws/config it currently fails to correctly use a shared sso session.

version 1.44.298 of the AWS go sdk introduced support for sso_session https://github.com/aws/aws-sdk-go/releases/tag/v1.44.298

There is already a pr to bump the aws sdk, #275

Is a new release planned or is there something i can help with to prepare a new release?

This is a great project!

I'm just curious if there is a functional reason why the tutorial hasn't been updated to specify version 3 of the docker-compose files. I was able to remove the 'gateway' configuration of the credentials network (no longer honored in v3 of docker-compose) and the local container endpoint continued to vend my ECS task credentials perfectly. I was curious if other use cases aren't supported by the new docker-compose configuration schema or if the documentation is simply behind.

Thanks!

'sh' executable is required by aws-sdk-go to invoke credential_process.

See related PR for more context.

I cannot perform iam:* commands (I tried create-role, delete-role, or list-groups) when the request is signed by a token issued by my ecs-local-container-endpoints via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds

I first encountered this using AWS SDK for Java 1.11.551 and then replicated it on aws-cli/1.16.148. I'm using endpoints 1.0.1 but 1.0.0 didn't work either.

Steps to reproduce using aws-cli/1.16.158 Python/3.6.8 Linux/5.0.13-arch1-1-ARCH botocore/1.12.148 are as follows:

# setup ~/.aws
aws configure

# verify role creation works with standard policy
 aws iam create-role --role-name joe-test --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'

# break local .aws and enable ecs endpoints
mv ~/.aws ~/.aws-nope
export AWS_DEFAULT_REGION=us-east-1
export AWS_REGION=us-east-1
export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
docker run -d -p 51679:51679 -v /var/run:/var/run -v $HOME/.aws-nope/:/home/.aws/ -e "ECS_LOCAL_METADATA_PORT=51679" --name ecs-local-endpoints amazon/amazon-ecs-local-container-endpoints:latest

# verifiy token service is working
aws s3 ls

# create a new role but it's busted
aws iam create-role --role-name joe-test-broken --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'

An error occurred (InvalidClientTokenId) when calling the CreateRole operation: The security token included in the request is invalid

# cannot delete other role
aws iam delete-role --role-name joe-test

An error occurred (InvalidClientTokenId) when calling the DeleteRole operation: The security token included in the request is invalid

# fix the cli
mv ~/.aws-nope ~/.aws

# delete the example role
aws iam delete-role --role-name joe-test

Running it in debug the response was:

2019-05-14 21:47:11,248 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '5b2f97bc-76b3-11e9-95f1-97b56e28630b', 'Content-Type': 'text/xml', 'Content-Length': '305', 'Date': 'Wed, 15 May 2019 01:47:11 GMT'}
2019-05-14 21:47:11,248 - MainThread - botocore.parsers - DEBUG - Response body:
b'\n \n Sender\n InvalidClientTokenId\n The security token included in the request is invalid\n \n 5b2f97bc-76b3-11e9-95f1-97b56e28630b\n\n'

I found this because my local CodeBuild was failing. I pulled a wire debug from a local build and compared it to the wire on CodeBuild and the requests look good.

Attempting to run the ecs-local-endpoints container on a system behind a corporate proxy that does SSL interception wtih self-signed certificates results in a sequence of errors of the form (excluding timestamp):

level=error msg="HTTP 500 - RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/: x509: certificate signed by unknown authority"

Other AWS tools (e.g., the AWS CLI and Python SDK) provide mechanisms to disable certificate verification, but no such mechanism appears to be included for the ecs-local-countainers-endpoints.

Just wondering if there's any plans to support v4 of the ecs task metadata endpoint in the near future. Thanks

I believe I'm having a similar issue to #89 but it's different enough that I'm opening a new issue. (Somewhat similar to #118 and maybe #35)

As snehal-kolte mentioned, I'm following the same document as testing containers: https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/

And also referencing the docs for how to vend credentials that are in this repo: https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/docs/features.md#vend-credentials-to-containers

Most of the time there are no issues but on occasion the following error appears in the logs:

{"level":"error","message":"Unhandled promise rejection: CredentialsProviderError: Could not load credentials from any providers","meta":{},"timestamp":"2022-03-22T21:26:36.174Z"}
You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
CredentialsProviderError: Could not load credentials from any providers
    at providers (/<project-dir>/<project-dir>/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js:25:19)
    at /<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async coalesceProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
    at async SignatureV4.credentialProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:32:24)
    at async SignatureV4.signRequest (/<project-dir>/<project-dir>/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:84:29)
    at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:18
    at async StandardRetryStrategy.retry (/<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22

The workaround has been running the same command again to hit our endpoint and it will always work (we never see the error back-to-back).

My docker-compose.override.yml:

version: "2"
networks:
    credentials_network:
        driver: bridge
        ipam:
            config:
                - subnet: "169.254.170.0/24"
                  gateway: 169.254.170.1
services:
    ecs-local-endpoints:
        image: amazon/amazon-ecs-local-container-endpoints
        volumes:
            - /var/run:/var/run
            - $HOME/.aws/:/home/.aws/
        environment:
            AWS_PROFILE: "$AWS_PROFILE"
            HOME: "/home"
        networks:
            credentials_network:
                ipv4_address: "169.254.170.2"

    app:
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.3"
        environment:
            AWS_DEFAULT_REGION: "us-east-1"
            AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/$APP_AWS_TASK_ROLE"
            ECS_CONTAINER_METADATA_URI: "http://169.254.170.2"
            ECS_CONTAINER_METADATA_URI_V4: "http://169.254.170.2/v3"

• I don't seem to be able to consistently duplicate this error because the credentials work most of the time. But pretty regularly I see the above error.
• I'm running 1.4.0 of the amazon-ecs-local-container-endpoints.
• No modifications have been made.
• Nothing unusual or noteworthy of my environment or deployment.

Team,
We are referring the https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/ and set up docker-compose.yml and docker-compose.override.yml like below

docker-compose.yml
version: "2"
services:
app:
build: my_image
ports:
- 8080:80
environment:
PORT: "80"

docker-compose.override.yml
version: "2"
networks:

credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1

services:

volumes:
  - %UserProfile%\.aws
environment:
  AWS_PROFILE: "default"
networks:
  credentials_network:
    # This special IP address is recognized by the AWS SDKs and AWS CLI
    ipv4_address: "169.254.170.2"

app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_REGION: "us-east-1"
ABBREVIATED_REGION: "us01"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"

Error:
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@29c5ee1d: profile file cannot be null, WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@59429fac: Failed to connect to service endpoint: ]

@hencrice Is there something missing in my configuration set up?
credentials file look like below
[default]
aws_access_key_id= xyz
aws_secret_access_key= xyz
aws_security_token="xyz"
aws_session_token="xyz"

I have some issues with setting it up correctly. Credentials are in place but Expiration is always empty and exceptions are thrown because of that.
I'm trying to use this image for the most basic scenario to debug application which uses dynamodb.

I've tried to add manually expiration to credentials file but didn't work.

Example response from /creds:
{ "AccessKeyId": "not empty", "Expiration": "", "RoleArn": "", "SecretAccessKey": "not empty", "Token": "not empty" }

Endpoint /creds works just fine, but when I try to use /role/<role name> like it says in this guide, the ecs-local-endpoints container keeps responding with the error:

ecs-local-endpoints_1  | time="2021-12-02T21:21:07Z" level=error msg="HTTP 500 - InvalidClientTokenId: The security token included in the request is invalid\n\tstatus code: 403, request id: 080b81eb-c67d-4b47-932f-276bb9c1d480"

See my docker-compose.override.yml:

version: "2"
networks:
    # This special network is configured so that the local metadata
    # service can bind to the specific IP address that ECS uses
    # in production
    credentials_network:
        driver: bridge
        ipam:
            config:
                - subnet: "169.254.170.0/24"
                  gateway: 169.254.170.1
services:
    # This container vends credentials to your containers
    ecs-local-endpoints:
        # The Amazon ECS Local Container Endpoints Docker Image
        image: amazon/amazon-ecs-local-container-endpoints:1.4.0-amd64
        volumes:
          # Mount /var/run so we can access docker.sock and talk to Docker
          - /var/run:/var/run
          # Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
          # On Windows, this directory can be found at "%UserProfile%\.aws"
          - $HOME/.aws/:/home/.aws/
        environment:
          # define the home folder; credentials will be read from $HOME/.aws
          HOME: "/home"
          # You can change which AWS CLI Profile is used
          AWS_PROFILE: "default"
        networks:
            credentials_network:
                # This special IP address is recognized by the AWS SDKs and AWS CLI 
                ipv4_address: "169.254.170.2"
                
    # Here we reference the application container that we are testing
    # You can test multiple containers at a time, simply duplicate this section
    # and customize it for each container, and give it a unique IP in 'credentials_network'.
    app:
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.3"
        environment:
          AWS_DEFAULT_REGION: "us-gov-west-1"
          AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/test-role"

Extra info: I'm operating with an assumed role session principal. I.e., when I get my caller identity on the host, I get:

$ aws sts get-caller-identity
{
    "UserId": "XXXXXXXXXX",
    "Account": "XXXXXXXXXX",
    "Arn": "arn:aws-us-gov:sts::XXXXXXXXXX:assumed-role/PowerUsers/XXXXXXXXXX"
}

The docs recommend mounting ~/.aws into the container, but this may not work for a variety of reasons (e.g., a custom credential process that's not accessible to the container). An alternative I've discovered is to provide a local IMDSv2 server on the host and use the AWS_EC2_METADATA_SERVICE_ENDPOINT environment variable on the container.

I've included this capability in aws-export-credentials. It looks like this:

# in a terminal (choosing 8081 as the port for this example)
$ aws-export-credentials --imds 8081

then in your docker-compose.override.yml from the tutorial, the relevant section would look something like this:

services:
    # This container vends credentials to your containers
    ecs-local-endpoints:
        # The Amazon ECS Local Container Endpoints Docker Image
        image: amazon/amazon-ecs-local-container-endpoints
        volumes:
          # Mount /var/run so we can access docker.sock and talk to Docker
          - /var/run:/var/run
        ports:
          # Map the IMDS server's port from the host
          - "8081:8081"
        environment:
          # use credentials from the IMDS server on the host
          AWS_EC2_METADATA_SERVICE_ENDPOINT=http://host.docker.internal:8081/
        networks:
            credentials_network:
                # This special IP address is recognized by the AWS SDKs and AWS CLI 
                ipv4_address: "169.254.170.2"

Hi,

I'm trying to use the /role-arn/{role arn} option for the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. If I try using a role with a path in it e.g. arn:aws:iam::111122223333:role/division_abc/subdivision_xyz/custom_role it does not appear to work. I believe it is due to the Gorilla Mux path pattern used at https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/local-container-endpoints/config/config.go#L73, it doesn't allow expansion to multiple /.

I checked out the code and added a simple expansion RoleArnCredentialsPath = "/role-arn/{roleArn}/{roleName:.+}" and this fixed the problem. I did need to update the STS role session name to exclude any / as this is not allowed in the role session name.

Am I understanding this correctly or am I missing something? I can create a PR with a fix if needed.

This does not support "aws configure sso". If I copy the credentials into the ~/.aws/credentials file it works. Can support for sso be added