The Link in this README goes to a 404 server not found in the consul ecs mesh-task readme.

The issue

When a task_definition passed to mesh-task contains the property healthCheck = null, the healt check side-car will always report a failiure.

The why

In https://github.com/hashicorp/terraform-aws-consul-ecs/blob/v0.5.2/modules/mesh-
task/main.tf#L71-L74 only the keys existence is tested, but specially for healthCheck the value should also be tested, mainly because the task_def passed in var.container_definitions might contain healthCheck = null due to optional flags in task_def creation.

This will pass the task_def to

Current implementation assumes that AWS SecretManager instances to store Consul secrets are located in the same AWS account as the ECS services (acl-controller and mesh-task) and using the default AWS KMS keys.

It would be helpful to have the option to use CMK/external AWS Secrets instance. The only change needed would be to pass the KMS key and adjust the service execution policies.

I've created a small PR with this approach. Yet it's not clear to me how to proceed with this secret consul_https_ca_cert_arn, but we can review it in the PR: TODO.

Module: acl-controller

Regarding secrets, currently the module has the following input variables:

• consul_bootstrap_token_secret_arn
• consul_server_ca_cert_arn

The module then creates IAM permissions to the GetSecretValue action. Normally this works fine with secret ARNs that only contain a single value such as:

arn:aws:secretsmanager:<region>:<aws_account_id>:secret:<secret_name>

But when a secret with a JSON structure is used, then the ARN in the ECS task definition becomes:

arn:aws:secretsmanager:region:aws_account_id:secret:secret-name:json-key:version-stage:version-id

Unfortunately the IAM permission fails (does not apply) when this type of secret ARN is used and the acl-controller fails to start because its unable to retrieve the secret value.

I was able to confirm this by manually updating the IAM policy by removing the :json-key:version-stage:version-id suffix and the acl-controller was then able to start successfully.

How would it be possible to use secrets with ARNs that specify a JSON key?

I've noticied that bootstrap times of consul side-cars in v0.7.x are taking more than I expected, let's say 1m40s-2m00 with very simple services which need ~10secs to bootstrap.

I've started playing the with the health check configurations of consul-dataplane and consul-ecs-control-plane.

See v-rosa@e196aad

I've managed to reduce consul bootstrap to ~50-60 seconds.

Do you think these health check values are adequate? This was tested with a service which have only 1 upstream and 1 down stream. Do the number of configured up/downstreams affect heavily the bootstrap time?

I also understand the that the v0.8.0 is being cooked which will change the side cars structure. Do you think is valuable to hotfix v0.7.x? Personally at my company we won't know when we'll be able to jump to v0.8.0 and meanwhile these bootstrap times are kinda unconfortable.

I'm trying to test out the arm64 cpu architecture on AWS Fargate but it seem that the mesh-task module does not allow us to configure the runtime_platform on the aws_ecs_dask_definition terraform resource.

Any plans to support this?

The the roles variables state they will create a role if none is provided, it seems roles for cloudwatch "create log group" are still being created.

In many areas and business there is governance on IAM role creation and as such, no roles should be created at all if iam role ARNs are already provided.

Currently it's not possible to run the ecs execute-command action if the task-def sets readonlyRootFilesystem to true. E.g.

aws ecs execute-command  \
    --region us-east-1 \
    --cluster stg-internal-pet \
    --task 355ef4d394294fdd91acb5af1876806f \
    --container consul-ecs-controller \
    --command "/bin/bash" \
    --interactive

Given by default ECS Controller enables both:

• https://github.com/hashicorp/terraform-aws-consul-ecs/blob/main/modules/controller/main.tf#L65
• https://github.com/hashicorp/terraform-aws-consul-ecs/blob/main/modules/controller/main.tf#L20

Execute command won't work if some work arounds are implemented, like:

• aws-containers/amazon-ecs-exec-checker#21 (comment)
• https://toris.io/2021/06/using-ecs-exec-with-readonlyrootfilesystem-enabled-containers/

Or if we disable readonlyRootFilesystem when we decide to enable enable_execute_command at the ECS service level.

Btw this issue was introduced by me here: 8a8b9b0

In the current Terraform configuration the tag value of consul.hashicorp.com/mesh is hard coded. In order to append to this value the Terraform template must be changed to accept tags provided through the tags variable.

  tags = {
    "consul.hashicorp.com/mesh" = "true"
  }

Executing "Consul With Dev Server on ECS EC2 Example" in a region other than "us-east-1" leads to an InvalidAMIID.NotFound error b/c the AMI ID is hardcoded.

Suggested fix: The correct AMI for the region can be retrieved programmatically from the dedicated public parameter for the ECS optimized AMI.

Hey, we're testing the Consul ECS v0.8.0 terraform module, more specifically the mesh-task.

While testing we've found that after applying the changes, subsequent plans will keep displaying perpetual diffs in the container definitions about default values not being present. This doesn't happens in the v0.7.1 at least.

Terraform version: v1.3.7
AWS Provider version: v5.50.0
ECS deployment model: Fargate (meaning variable enable_transparent_proxy is set to false, this is relevant as per my findings).

Perpetual drift:

# module.ecs_services["consul-test-module-clt-test-task"].module.consul_mesh_template[0].aws_ecs_task_definition.this must be replaced
-/+ resource "aws_ecs_task_definition" "this" {
      ~ arn                      = "arn:aws:ecs:us-east-1:redacted:task-definition/stg-consul-test-module-clt-test-task-template:5" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:redacted:task-definition/stg-consul-test-module-clt-test-task-template" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                  - cpu              = 0 -> null
                    name             = "some-container-injected"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (6 unchanged elements hidden)
                } # forces replacement,
              ~ {
                    name             = "consul-dataplane"
                  - systemControls   = [] -> null
                    # (14 unchanged elements hidden)
                } # forces replacement,
              ~ {
                    name             = "consul-ecs-health-sync"
                  - systemControls   = [] -> null
                    # (13 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  ~ linuxParameters  = {
                      ~ capabilities       = {
                          - add  = [] -> null
                          - drop = [] -> null
                        }
                        # (1 unchanged element hidden)
                    }
                    name             = "consul-ecs-mesh-init"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                    # (9 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  - cpu              = 0 -> null
                    name             = "consul-test-module-clt-test-task"
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (8 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  - cpu              = 0 -> null
                    name             = "some-container-injected"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (7 unchanged elements hidden)
                } # forces replacement,
            ]
        )
      ~ id                       = "stg-consul-test-module-clt-test-task-template" -> (known after apply)
      ~ revision                 = 5 -> (known after apply)
        tags                     = {
            "Module"                              = "consul-test-module-clt"
            "Name"                                = "consul-test-module-clt-test-task"
            "consul.hashicorp.com/mesh"           = "true"
            "consul.hashicorp.com/module"         = "terraform-aws-consul-ecs"
            "consul.hashicorp.com/module-version" = "0.8.0"
            "consul.hashicorp.com/namespace"      = "default"
            "consul.hashicorp.com/partition"      = "stg"
            "consul.hashicorp.com/service-name"   = "consul-test-module-clt-test-task"
        }
        # (10 unchanged attributes hidden)
        # (3 unchanged blocks hidden)
    }

From my investigation, basically the issue seems related to the empty dictionary capabilities inside linuxParameters.

The initial apply creates this an empty capabilities dict inside linuxParameters dict:

           ~ {
                  ~ linuxParameters  = {
                      + capabilities       = {}
                        # (1 unchanged element hidden)
                    }
                    name             = "consul-ecs-mesh-init"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                    # (9 unchanged elements hidden)
                } # forces replacement,

The subsequent plan tries to change it to null:

# module.ecs_services["consul-test-module-clt-test-task"].module.consul_mesh_template[0].aws_ecs_task_definition.this must be replaced
-/+ resource "aws_ecs_task_definition" "this" {
      ~ arn                      = "arn:aws:ecs:us-east-1:redacted:task-definition/stg-consul-test-module-clt-test-task-template:5" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-east-1:redacted:task-definition/stg-consul-test-module-clt-test-task-template" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                  - cpu              = 0 -> null
                    name             = "some-container-injected"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (6 unchanged elements hidden)
                } # forces replacement,
              ~ {
                    name             = "consul-dataplane"
                  - systemControls   = [] -> null
                    # (14 unchanged elements hidden)
                } # forces replacement,
              ~ {
                    name             = "consul-ecs-health-sync"
                  - systemControls   = [] -> null
                    # (13 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  ~ linuxParameters  = {
                      ~ capabilities       = {
                          - add  = [] -> null
                          - drop = [] -> null
                        }
                        # (1 unchanged element hidden)
                    }
                    name             = "consul-ecs-mesh-init"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                    # (9 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  - cpu              = 0 -> null
                    name             = "consul-test-module-clt-test-task"
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (8 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  - cpu              = 0 -> null
                    name             = "some-container-injected"
                  - portMappings     = [] -> null
                  - systemControls   = [] -> null
                  - volumesFrom      = [] -> null
                    # (7 unchanged elements hidden)
                } # forces replacement,
            ]
        )
      ~ id                       = "stg-consul-test-module-clt-test-task-template" -> (known after apply)
      ~ revision                 = 5 -> (known after apply)
        tags                     = {
            "Module"                              = "consul-test-module-clt"
            "Name"                                = "consul-test-module-clt-test-task"
            "consul.hashicorp.com/mesh"           = "true"
            "consul.hashicorp.com/module"         = "terraform-aws-consul-ecs"
            "consul.hashicorp.com/module-version" = "0.8.0"
            "consul.hashicorp.com/namespace"      = "default"
            "consul.hashicorp.com/partition"      = "stg"
            "consul.hashicorp.com/service-name"   = "consul-test-module-clt-test-task"
        }
        # (10 unchanged attributes hidden)
        # (3 unchanged blocks hidden)
    }

To avoid this, I need to slighty adjust the linuxParameters: #319

I'm trying to connect the example mesh-task to a previously set up Consul server hosted in an EC2 instance, using the retry_join input in the mesh-task definition:

retry_join = "provider=aws tag_key=Name tag_value=consul"

However, when the task starts only the mesh-init and the consul-client enter the Running status, while the sidecar-proxy and example-client-app remain in Pending status:

If I switch back to the configuration using the consul_server_service_name input, every container starts successfully.

Things I've already checked that might be causing the issue:

• Is Server accepting connections? -> Consul server is up and already connected to another sidecar service hosted in an EC2 instance
• Are Security Groups correct? -> Both Server and ECS Tasks have all ports/protocols open to "0.0.0.0/0" connections
• Is the app starting in Docker? -> I'm using the provided app from the examples

I'm not quite sure how I can debug the issue, as there are no log files for pending containers on ECS (or at least I believe there aren't any).

Could there be any config missing on my Consul server, or is there any issue when using retry_join with the AWS provider?

I wanted to use this module to deploy a terminating Gateway in ECS, however since I'm unable to modify the consul connect command in any way, I can't use this module to do so.

I believe some type of optional variable for this would be very helpful to take full advantage of envoy

Consul supports defining multiple services for different ports, with their own healthchecks and everything. This is something my company needs to support sending data between containers but also sending out metrics via an admin panel from our applications. Is there a chance this could be implemented in the near future?

Hey Folks! Hope this issue finds you all well! 😄

I found this repo at Consul's Docs Page.

I became interested in the consul because of its Service Mesh capability.

In the Service Mesh overview documentation page and in this another example repository aws-samples/amazon-ecs-fargate-consul-connect-example, they talk about "Consult Connect", as if "Connect" were a specific service/functionality.

But I couldn't find any mention of "Connect" here in this repository, although I believe the purpose is the same.

Just to be sure: does this repository implement a "Consul Connect" example? Is "Connect" still a thing?

I was testing the following demo: https://github.com/hashicorp/terraform-aws-hcp-consul/blob/main/examples/hcp-ecs-demo/main.tf

It relies on the mesk-task module of this repository to spin up an ECS service with all the needed side-cars.

Using Consul entreprise v1.13.3, its impossible to start the consul-client side car given the following error:

==> failed to parse /consul/agent-defaults.hcl: 1 error occurred:
* invalid config key telemetry.disable_compat_1.9

Looking at the release notes, this paramter should be removed from v1.13 onwards.

Given this mesk-task module is a dependency of the HCP feature which allows to bootstrap a Consul cluster with auto-generated terraform code, I would say it should fix by removing the parameter so users won't complain about a broken demo.

I can quickly open a PR to address it.

The gateway-task submodule does not provide a way to disable IAM role creation and thus providing my own IAM Role ARN.

The gateway-task submodule also does not provide a way to supply it with my own security group IDs for the ECS service/task, only allowing for creating an LB security group and nothing more (which seems mesh gateway specific, but is not very helpful if your kind is terminating-gateway).

Hello!

The mesh-task submodule intakes a variable called log_configuration, to customize/configure logs for an ECS task built by this module. The variable is added to the merged container_definitions object for the generated ecs_task_definition resource.

Variable definition:

Appended value:

The AWS Developer guide includes an options block for logConfiguration that includes a setting for awslogs-create-group with values of true/false, to determine whether the task definition creates the Cloudwatch logs group, or references the defined values instead. Snippet below:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-west-2",
                    "awslogs-create-group": "true",
                    "awslogs-stream-prefix": "firelens"

In my code, I have created a mesh-task:

variable "awslogs_create_group" {
  type    = bool
  default = true
}

# Object to add to log_configuration's "options" sub-block.
locals {
  logs_options = {

. . . 

    awslogs-create-group  = var.awslogs_create_group

. . . 

  }
}


module "mesh_task" {

. . .

  log_configuration = {
    logDriver = "awslogs"
    options   = local.logs_options
  }

. . . 
}

This module invocation returns a go struct error during terraform plan

Error: ECS Task Definition container_definitions is invalid: Error decoding JSON: json: cannot unmarshal bool into Go struct field LogConfiguration.LogConfiguration.Options of type string

│   with module..module.mesh_task["task"].aws_ecs_task_definition.this,
│   on .terraform/modules//modules/mesh-task/main.tf line 180, in resource "aws_ecs_task_definition" "this":
│  180:   container_definitions = jsonencode(
│  181:     flatten(
│  182:       concat(
│  183:         local.container_defs_with_depends_on,
│  184:         [

. . . 

│  220:             logConfiguration = var.log_configuration

Changing awslogs-create-group's value from bool(true), to string(true) (Or, true -> "true") fixes this error to let the task definition create the cloudwatch group.

Looking at the log_configuration declaration, it seems like there isn't type checking or validation on the variable. There are validations for other variables: For example, the upstreams variable in variables.tf

Could it make sense to create an object definition for log_configuration, with a validation? Example below (Note: This is a rough suggestion, using existing validations in the module as reference)

variable "log_configuration" {
  type = object({
    logDriver = string
    options = object({
      awslogs-group         = string
      awslogs-region        = string
      awslogs-stream-prefix = string
      awslogs-create-group  = string
    })
  })
  validation {
    error_message = "awslogs-create-group must be a string 'true'"
    condition = allfalse(flatten([
      for log_var in var.log_configuration : [
        type(can(lookup(log_var.options, "awslogs-create-group"), string))
      ]
    ]))
  }
}

I couldn't find information on the available task definition values to configure in the mesh-task module, and relied on the ecs_task_definition resource, and this module's variables file to understand how to build the ECS task definition. Since some of the larger variables (log_configuration, container_definitions, volumes) are set to a type of any, it required some context switching to fully understand what was needed to get the mesh-task created, in addition to this specific error.

Thank you! 😺

The module is configured to take the default datacenter name "dc1". We should give the consumer the ability to specify the datacenter.

By default the task definition readonlyRootFilesystem is set to false. If not set to true, it will grant write permissions to the root file system. Could you clarify if write permission is needed for the acl-controller?

This triggers the following AWS Security Hub alert:

[ECS.5] This control checks if ECS containers are limited to read-only access to mounted root filesystems. This control fails if the ReadonlyRootFilesystem parameter in the container definition of ECS task definitions is set to ‘false’. Remediation instructions

Hello all, we are seeing the following tls communication issue when deploying the ecs controller with tls enabled.

Consul server version
Consul v1.17.0
Revision 4e3f428b
Build Date 2023-11-03T14:56:56Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

ecs controller image
variable "consul_ecs_image" {
description = "Consul ECS image to use in all tasks."
type = string
default = "hashicorp/consul-ecs:0.8.0"
}

The ecs controller is running as a fargate task and we see the following error in the logs

[ERROR] connection error: error="fetching supported dataplane features: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""

When running consul montior with trace logging on the server we see the corresponding server side error

2024-03-20T20:37:12.469Z [TRACE] agent: [core][Server #2] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "ServerHandshake(\"10.255.186.202:46110\") failed: tls: client didn't provide a certificate"

We are passing the following to the controller

  source = "hashicorp/consul-ecs/aws//modules/controller"
  version = "0.8.0"

  name_prefix         = var.name
  ecs_cluster_arn     = var.ecs_cluster_arn
  region              = var.region
  subnets             = var.private_subnets
  consul_server_hosts = var.consul_server_hosts
  consul_ca_cert_arn  = var.consul_ca_cert_arn
  launch_type         = "FARGATE"

  consul_bootstrap_token_secret_arn = var.consul_server_bootstrap_token_arn

  log_configuration = {
    logDriver = "awslogs"
    options = {
      awslogs-group         = var.log_group_name
      awslogs-region        = var.region
      awslogs-stream-prefix = "consul-controller"
    }
  }

  consul_ecs_image = var.consul_ecs_image
  tls              = true
}

Are we missing something on the client side configuration?

How are we supposed to specify consul_https_ca_cert_arn when Consul's certificate is from ACM? In that use-case, we don't have access to the CA.

Seems that variable outbound_only is used only to do some kind of asserts here :

I'm trying to understand if this affects the consul side cars bootstrap in any way.

Am I missing something?

SSM Encrypted Parameter ARNs should also be a viable source, and shouldn't require the use of the acl controller as this forces users to commit to using another resource in their AWS account just to use the module securely.

The gateway task submodule does not allow for adding in one's own container definitions to the task, I believe this should be added due to someone adding sidecars for things like:

• log ingestion
• metric collection
• custom metric creation in cloudwatch (I have a sidecar that posts the active connections of envoy to cloudwatch for fargate scaling purposes)

It would be helpful to have a container definition to facilitate this. I'm going to work on a PR for this

Currently when a service-a needs to communicate with a upstream, let's say service-b:1234 the following mesh-task is needed:

module "service-a" {
  source = "../../modules/mesh-task"
  upstreams = [
    {
      destinationName = "service-b"
      localBindPort   = 1234
    }
  ]

  container_definitions = [{
    environment = [
      {
        name  = "UPSTREAM_URI_B"
        value = "http://localhost:1234"
      }
    ]

Now if service-a needs to have a new upstream (service-c) using the same port, wouldn't this create a conflict?

module "service-a" {
  source = "../../modules/mesh-task"
  upstreams = [
    {
      destinationName = "service-b"
      localBindPort   = 1234
    },
    {
      destinationName = "service-c"
      localBindPort   = 1234
    }
  ],
  container_definitions = [{
    environment = [
      {
        name  = "UPSTREAM_URI_B"
        value = "http://localhost:1234"
      },
      {
        name  = "UPSTREAM_URI_C"
        value = "http://localhost:1234" ## this doesn't make sense for me
      },
    ]
...
}

Would be possible to allow to set the upstreams like http://<consul-service-name>:<binding-port>, I tried this but apparently it doesn't work.

This is for the dev-server module of this project.

By default, the security group attached to the ECS service (in this case the consul server), is the default security group of the VPC used (reference).

Currently, this option is not configurable in this module (check main.tf line 66).

If the default security group of the VPC used does not permit the IP addresses used by the load balancer (as what happened in my case) then you won't be able to connect to the Consul UI via the load balancer.

The acl-controller submodule creates an ECS service without specifying a security group (reference).

This results in the default security group for the VPC being used and if no inbound or outbound rules are defined, then the ECS service fails to start. AWS also seems to say that the security group is a required argument in the ECS dashboard.

This was verified by creating the acl-controller ECS service manually in the AWS console and specify a security group that does contains inbound and outbound rules.

• Terraform version: 1.2.3
• Mesh module version: 0.4.1

https://github.com/hashicorp/terraform-aws-consul-ecs/blob/v0.4.1/modules/mesh-task/iam.tf#L2

The workaround still produces the following error:

│ The "count" value depends on resource attributes that cannot be determined
│ until apply, so Terraform cannot predict how many instances will be
│ created. To work around this, use the -target argument to first apply only
│ the resources that the count depends on.

I also tried specifying depends_on so that the IAM roles are created before the mesh module and that didn't work either.

Currently:

The Consul Client Command template does a curl against ECS_CONTAINER_METADATA_URI, Fargate now uses ECS_CONTAINER_METADATA_URI_V4 which the consul-ecs binary in the ecs container looks for, but the client command does not.

This makes the module incompatible with fargate.