Giter Club home page Giter Club logo

terraform-aws-eks-cluster's Introduction

terraform-aws-eks-cluster

Last UpdateLast UpdatedLast UpdatedLast UpdatedLatest ReleaseLast UpdatedSlack Community

Terraform module to provision an EKS cluster on AWS.

This Terraform module provisions a fully-configured AWS EKS (Elastic Kubernetes Service) cluster. It's engineered to integrate smoothly with Karpenter and EKS addons, forming a critical part of Cloud Posse's reference architecture. Ideal for teams looking to deploy scalable and manageable Kubernetes clusters on AWS with minimal fuss.

Tip

๐Ÿ‘ฝ Use Atmos with Terraform

Cloud Posse uses atmos to easily orchestrate multiple environments using Terraform.
Works with Github Actions, Atlantis, or Spacelift.

Watch demo of using Atmos with Terraform
Example of running atmos to manage infrastructure from our Quick Start tutorial.

Introduction

The module provisions the following resources:

  • EKS cluster of master nodes that can be used together with the terraform-aws-eks-node-group and terraform-aws-eks-fargate-profile modules to create a full-blown EKS/Kubernetes cluster. You can also use the terraform-aws-eks-workers module to provision worker nodes for the cluster, but it is now rare for that to be a better choice than to use terraform-aws-eks-node-group.
  • IAM Role to allow the cluster to access other AWS services
  • EKS access entries to allow IAM users to access and administer the cluster

Usage

For a complete example, see examples/complete.

For automated tests of the complete example using bats and Terratest (which tests and deploys the example on AWS), see test/src.

Other examples:

  provider "aws" {
    region = var.region
  }

  # Note: This example creates an explicit access entry for the current user,
  # but in practice, you should use a static map of IAM users or roles that should have access to the cluster.
  # Granting access to the current user in this way is not recommended for production use.
  data "aws_caller_identity" "current" {}
  
  # IAM session context converts an assumed role ARN into an IAM Role ARN.
  # Again, this is primarily to simplify the example, and in practice, you should use a static map of IAM users or roles.
  data "aws_iam_session_context" "current" {
    arn = data.aws_caller_identity.current.arn
  }
  
  locals {
    # The usage of the specific kubernetes.io/cluster/* resource tags below are required
    # for EKS and Kubernetes to discover and manage networking resources
    # https://aws.amazon.com/premiumsupport/knowledge-center/eks-vpc-subnet-discovery/
    # https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/deploy/subnet_discovery.md
    tags = { "kubernetes.io/cluster/${module.label.id}" = "shared" }
  
    # required tags to make ALB ingress work https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html
    public_subnets_additional_tags = {
      "kubernetes.io/role/elb" : 1
    }
    private_subnets_additional_tags = {
      "kubernetes.io/role/internal-elb" : 1
    }
  
    # Enable the IAM user creating the cluster to administer it,
    # without using the bootstrap_cluster_creator_admin_permissions option,
    # as an example of how to use the access_entry_map feature.
    # In practice, this should be replaced with a static map of IAM users or roles
    # that should have access to the cluster, but we use the current user
    # to simplify the example.
    access_entry_map = {
      (data.aws_iam_session_context.current.issuer_arn) = {
        access_policy_associations = {
          ClusterAdmin = {}
        }
      }
    }
  }

  module "label" {
    source = "cloudposse/label/null"
    # Cloud Posse recommends pinning every module to a specific version
    # version  = "x.x.x"

    namespace  = var.namespace
    name       = var.name
    stage      = var.stage
    delimiter  = var.delimiter
    tags       = var.tags
  }

  module "vpc" {
    source = "cloudposse/vpc/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    ipv4_primary_cidr_block = "172.16.0.0/16"

    tags    = local.tags
    context = module.label.context
  }

  module "subnets" {
    source = "cloudposse/dynamic-subnets/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    availability_zones   = var.availability_zones
    vpc_id               = module.vpc.vpc_id
    igw_id               = [module.vpc.igw_id]
    ipv4_cidr_block      = [module.vpc.vpc_cidr_block]
    nat_gateway_enabled  = true
    nat_instance_enabled = false

    public_subnets_additional_tags  = local.public_subnets_additional_tags
    private_subnets_additional_tags = local.private_subnets_additional_tags

    tags    = local.tags
    context = module.label.context
  }

  module "eks_node_group" {
    source = "cloudposse/eks-node-group/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    instance_types    = [var.instance_type]
    subnet_ids        = module.subnets.private_subnet_ids
    health_check_type = var.health_check_type
    min_size          = var.min_size
    max_size          = var.max_size
    cluster_name      = module.eks_cluster.eks_cluster_id

    # Enable the Kubernetes cluster auto-scaler to find the auto-scaling group
    cluster_autoscaler_enabled = var.autoscaling_policies_enabled

    context = module.label.context
  }

  module "eks_cluster" {
    source = "cloudposse/eks-cluster/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version = "x.x.x"

    subnet_ids            = concat(module.subnets.private_subnet_ids, module.subnets.public_subnet_ids)
    kubernetes_version    = var.kubernetes_version
    oidc_provider_enabled = true

    addons = [
      # https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#vpc-cni-latest-available-version
      {
        addon_name                  = "vpc-cni"
        addon_version               = var.vpc_cni_version
        resolve_conflicts_on_create = "OVERWRITE"
        resolve_conflicts_on_update = "OVERWRITE"
        service_account_role_arn    = var.vpc_cni_service_account_role_arn # Creating this role is outside the scope of this example
      },
      # https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
      {
        addon_name                  = "kube-proxy"
        addon_version               = var.kube_proxy_version
        resolve_conflicts_on_create = "OVERWRITE"
        resolve_conflicts_on_update = "OVERWRITE"
        service_account_role_arn    = null
      },
      # https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html
      {
        addon_name                  = "coredns"
        addon_version               = var.coredns_version
        resolve_conflicts_on_create = "OVERWRITE"
        resolve_conflicts_on_update = "OVERWRITE"
        service_account_role_arn    = null
      },
    ]
    addons_depends_on = [module.eks_node_group]

    context = module.label.context

    cluster_depends_on = [module.subnets]
  }

Module usage with two unmanaged worker groups:

  locals {
    # Unfortunately, the `aws_ami` data source attribute `most_recent` (https://github.com/cloudposse/terraform-aws-eks-workers/blob/34a43c25624a6efb3ba5d2770a601d7cb3c0d391/main.tf#L141)
    # does not work as you might expect. If you are not going to use a custom AMI you should
    # use the `eks_worker_ami_name_filter` variable to set the right kubernetes version for EKS workers,
    # otherwise the first version of Kubernetes supported by AWS (v1.11) for EKS workers will be selected, but the
    # EKS control plane will ignore it to use one that matches the version specified by the `kubernetes_version` variable.
    eks_worker_ami_name_filter = "amazon-eks-node-${var.kubernetes_version}*"
  }

  module "eks_workers" {
    source = "cloudposse/eks-workers/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    attributes                         = ["small"]
    instance_type                      = "t3.small"
    eks_worker_ami_name_filter         = local.eks_worker_ami_name_filter
    vpc_id                             = module.vpc.vpc_id
    subnet_ids                         = module.subnets.public_subnet_ids
    health_check_type                  = var.health_check_type
    min_size                           = var.min_size
    max_size                           = var.max_size
    wait_for_capacity_timeout          = var.wait_for_capacity_timeout
    cluster_name                       = module.label.id
    cluster_endpoint                   = module.eks_cluster.eks_cluster_endpoint
    cluster_certificate_authority_data = module.eks_cluster.eks_cluster_certificate_authority_data
    cluster_security_group_id          = module.eks_cluster.eks_cluster_managed_security_group_id

    # Auto-scaling policies and CloudWatch metric alarms
    autoscaling_policies_enabled           = var.autoscaling_policies_enabled
    cpu_utilization_high_threshold_percent = var.cpu_utilization_high_threshold_percent
    cpu_utilization_low_threshold_percent  = var.cpu_utilization_low_threshold_percent

    context = module.label.context
  }

  module "eks_workers_2" {
    source = "cloudposse/eks-workers/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    attributes                         = ["medium"]
    instance_type                      = "t3.medium"
    eks_worker_ami_name_filter         = local.eks_worker_ami_name_filter
    vpc_id                             = module.vpc.vpc_id
    subnet_ids                         = module.subnets.public_subnet_ids
    health_check_type                  = var.health_check_type
    min_size                           = var.min_size
    max_size                           = var.max_size
    wait_for_capacity_timeout          = var.wait_for_capacity_timeout
    cluster_name                       = module.label.id
    cluster_endpoint                   = module.eks_cluster.eks_cluster_endpoint
    cluster_certificate_authority_data = module.eks_cluster.eks_cluster_certificate_authority_data
    cluster_security_group_id          = module.eks_cluster.eks_cluster_managed_security_group_id

    # Auto-scaling policies and CloudWatch metric alarms
    autoscaling_policies_enabled           = var.autoscaling_policies_enabled
    cpu_utilization_high_threshold_percent = var.cpu_utilization_high_threshold_percent
    cpu_utilization_low_threshold_percent  = var.cpu_utilization_low_threshold_percent

    context = module.label.context
  }

  module "eks_cluster" {
    source = "cloudposse/eks-cluster/aws"
    # Cloud Posse recommends pinning every module to a specific version
    # version     = "x.x.x"

    subnet_ids            = concat(module.subnets.private_subnet_ids, module.subnets.public_subnet_ids)
    kubernetes_version    = var.kubernetes_version
    oidc_provider_enabled = true # needed for VPC CNI

    access_entries_for_nodes = {
      EC2_LINUX = [module.eks_workers.workers_role_arn, module.eks_workers_2.workers_role_arn]
    }

    context = module.label.context
  }

Warning

Release 4.0.0 contains major breaking changes that will require you to update your existing EKS cluster and configuration to use this module. Please see the v3 to v4 migration path for more information. Release 2.0.0 (previously released as version 0.45.0) contains some changes that, if applied to a cluster created with an earlier version of this module, could result in your existing EKS cluster being replaced (destroyed and recreated). To prevent this, follow the instructions in the v1 to v2 migration path.

Note

Prior to v4 of this module, AWS did not provide an API to manage access to the EKS cluster, causing numerous challenges. With v4 of this module, it exclusively uses the AWS API, resolving many issues you may read about that had affected prior versions. See the version 2 README and release notes for more information on the challenges and workarounds that were required prior to v3.

Important

In Cloud Posse's examples, we avoid pinning modules to specific versions to prevent discrepancies between the documentation and the latest released versions. However, for your own projects, we strongly advise pinning each module to the exact version you're using. This practice ensures the stability of your infrastructure. Additionally, we recommend implementing a systematic approach for updating versions to avoid unexpected changes.

Makefile Targets

Available targets:

  help                                Help screen
  help/all                            Display help for all targets
  help/short                          This help short screen
  lint                                Lint terraform code

Requirements

Name Version
terraform >= 1.3.0
aws >= 5.34.0
tls >= 3.1.0, != 4.0.0

Providers

Name Version
aws >= 5.34.0
tls >= 3.1.0, != 4.0.0

Modules

Name Source Version
label cloudposse/label/null 0.25.0
this cloudposse/label/null 0.25.0

Resources

Name Type
aws_cloudwatch_log_group.default resource
aws_eks_access_entry.linux resource
aws_eks_access_entry.map resource
aws_eks_access_entry.standard resource
aws_eks_access_entry.windows resource
aws_eks_access_policy_association.list resource
aws_eks_access_policy_association.map resource
aws_eks_addon.cluster resource
aws_eks_cluster.default resource
aws_iam_openid_connect_provider.default resource
aws_iam_policy.cluster_elb_service_role resource
aws_iam_role.default resource
aws_iam_role_policy_attachment.amazon_eks_cluster_policy resource
aws_iam_role_policy_attachment.amazon_eks_service_policy resource
aws_iam_role_policy_attachment.cluster_elb_service_role resource
aws_kms_alias.cluster resource
aws_kms_key.cluster resource
aws_vpc_security_group_ingress_rule.custom_ingress_rules resource
aws_vpc_security_group_ingress_rule.managed_ingress_cidr_blocks resource
aws_vpc_security_group_ingress_rule.managed_ingress_security_groups resource
aws_iam_policy_document.assume_role data source
aws_iam_policy_document.cluster_elb_service_role data source
aws_partition.current data source
tls_certificate.cluster data source

Inputs

Name Description Type Default Required
access_config Access configuration for the EKS cluster.
object({
authentication_mode = optional(string, "API")
bootstrap_cluster_creator_admin_permissions = optional(bool, false)
})
{} no
access_entries List of IAM principles to allow to access the EKS cluster.
It is recommended to use the default user_name because the default includes
the IAM role or user name and the session name for assumed roles.
Use when Principal ARN is not known at plan time.
list(object({
principal_arn = string
user_name = optional(string, null)
kubernetes_groups = optional(list(string), null)
}))
[] no
access_entries_for_nodes Map of list of IAM roles for the EKS non-managed worker nodes.
The map key is the node type, either EC2_LINUX or EC2_WINDOWS,
and the list contains the IAM roles of the nodes of that type.
There is no need for or utility in creating Fargate access entries, as those
are always created automatically by AWS, just as with managed nodes.
Use when Principal ARN is not known at plan time.
map(list(string)) {} no
access_entry_map Map of IAM Principal ARNs to access configuration.
Preferred over other inputs as this configuration remains stable
when elements are added or removed, but it requires that the Principal ARNs
and Policy ARNs are known at plan time.
Can be used along with other access_* inputs, but do not duplicate entries.
Map access_policy_associations keys are policy ARNs, policy
full name (AmazonEKSViewPolicy), or short name (View).
It is recommended to use the default user_name because the default includes
IAM role or user name and the session name for assumed roles.
As a special case in support of backwards compatibility, membership in the
system:masters group is is translated to an association with the ClusterAdmin policy.
In all other cases, including any system:* group in kubernetes_groups is prohibited.
map(object({
# key is principal_arn
user_name = optional(string)
# Cannot assign "system:*" groups to IAM users, use ClusterAdmin and Admin instead
kubernetes_groups = optional(list(string), [])
type = optional(string, "STANDARD")
access_policy_associations = optional(map(object({
# key is policy_arn or policy_name
access_scope = optional(object({
type = optional(string, "cluster")
namespaces = optional(list(string))
}), {}) # access_scope
})), {}) # access_policy_associations
}))
{} no
access_policy_associations List of AWS managed EKS access policies to associate with IAM principles.
Use when Principal ARN or Policy ARN is not known at plan time.
policy_arn can be the full ARN, the full name (AmazonEKSViewPolicy) or short name (View).
list(object({
principal_arn = string
policy_arn = string
access_scope = object({
type = optional(string, "cluster")
namespaces = optional(list(string))
})
}))
[] no
additional_tag_map Additional key-value pairs to add to each map in tags_as_list_of_maps. Not added to tags or id.
This is for some rare cases where resources want additional configuration of tags
and therefore take a list of maps with tag key, value, and additional configuration.
map(string) {} no
addons Manages aws_eks_addon resources.
Note: resolve_conflicts is deprecated. If resolve_conflicts is set and
resolve_conflicts_on_create or resolve_conflicts_on_update is not set,
resolve_conflicts will be used instead. If resolve_conflicts_on_create is
not set and resolve_conflicts is PRESERVE, resolve_conflicts_on_create
will be set to NONE.
list(object({
addon_name = string
addon_version = optional(string, null)
configuration_values = optional(string, null)
# resolve_conflicts is deprecated, but we keep it for backwards compatibility
# and because if not declared, Terraform will silently ignore it.
resolve_conflicts = optional(string, null)
resolve_conflicts_on_create = optional(string, null)
resolve_conflicts_on_update = optional(string, null)
service_account_role_arn = optional(string, null)
create_timeout = optional(string, null)
update_timeout = optional(string, null)
delete_timeout = optional(string, null)
}))
[] no
addons_depends_on If provided, all addons will depend on this object, and therefore not be installed until this object is finalized.
This is useful if you want to ensure that addons are not applied before some other condition is met, e.g. node groups are created.
See issue #170 for more details.
any null no
allowed_cidr_blocks A list of IPv4 CIDRs to allow access to the cluster.
The length of this list must be known at "plan" time.
list(string) [] no
allowed_security_group_ids A list of IDs of Security Groups to allow access to the cluster. list(string) [] no
associated_security_group_ids A list of IDs of Security Groups to associate the cluster with.
These security groups will not be modified.
list(string) [] no
attributes ID element. Additional attributes (e.g. workers or cluster) to add to id,
in the order they appear in the list. New attributes are appended to the
end of the list. The elements of the list are joined by the delimiter
and treated as a single ID element.
list(string) [] no
cloudwatch_log_group_kms_key_id If provided, the KMS Key ID to use to encrypt AWS CloudWatch logs string null no
cluster_attributes Override label module default cluster attributes list(string)
[
"cluster"
]
no
cluster_depends_on If provided, the EKS will depend on this object, and therefore not be created until this object is finalized.
This is useful if you want to ensure that the cluster is not created before some other condition is met, e.g. VPNs into the subnet are created.
any null no
cluster_encryption_config_enabled Set to true to enable Cluster Encryption Configuration bool true no
cluster_encryption_config_kms_key_deletion_window_in_days Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction number 10 no
cluster_encryption_config_kms_key_enable_key_rotation Cluster Encryption Config KMS Key Resource argument - enable kms key rotation bool true no
cluster_encryption_config_kms_key_id KMS Key ID to use for cluster encryption config string "" no
cluster_encryption_config_kms_key_policy Cluster Encryption Config KMS Key Resource argument - key policy string null no
cluster_encryption_config_resources Cluster Encryption Config Resources to encrypt, e.g. ['secrets'] list(any)
[
"secrets"
]
no
cluster_log_retention_period Number of days to retain cluster logs. Requires enabled_cluster_log_types to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. number 0 no
context Single object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as null to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.
any
{
"additional_tag_map": {},
"attributes": [],
"delimiter": null,
"descriptor_formats": {},
"enabled": true,
"environment": null,
"id_length_limit": null,
"label_key_case": null,
"label_order": [],
"label_value_case": null,
"labels_as_tags": [
"unset"
],
"name": null,
"namespace": null,
"regex_replace_chars": null,
"stage": null,
"tags": {},
"tenant": null
}
no
create_eks_service_role Set false to use existing eks_cluster_service_role_arn instead of creating one bool true no
custom_ingress_rules A List of Objects, which are custom security group rules that
list(object({
description = string
from_port = number
to_port = number
protocol = string
source_security_group_id = string
}))
[] no
delimiter Delimiter to be used between ID elements.
Defaults to - (hyphen). Set to "" to use no delimiter at all.
string null no
descriptor_formats Describe additional descriptors to be output in the descriptors output map.
Map of maps. Keys are names of descriptors. Values are maps of the form
{<br> format = string<br> labels = list(string)<br>}
(Type is any so the map values can later be enhanced to provide additional options.)
format is a Terraform format string to be passed to the format() function.
labels is a list of labels, in order, to pass to format() function.
Label values will be normalized before being passed to format() so they will be
identical to how they appear in id.
Default is {} (descriptors output will be empty).
any {} no
eks_cluster_service_role_arn The ARN of an IAM role for the EKS cluster to use that provides permissions
for the Kubernetes control plane to perform needed AWS API operations.
Required if create_eks_service_role is false, ignored otherwise.
string null no
enabled Set to false to prevent the module from creating any resources bool null no
enabled_cluster_log_types A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [api, audit, authenticator, controllerManager, scheduler] list(string) [] no
endpoint_private_access Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false bool false no
endpoint_public_access Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true bool true no
environment ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT' string null no
id_length_limit Limit id to this many characters (minimum 6).
Set to 0 for unlimited length.
Set to null for keep the existing setting, which defaults to 0.
Does not affect id_full.
number null no
kubernetes_network_ipv6_enabled Set true to use IPv6 addresses for Kubernetes pods and services bool false no
kubernetes_version Desired Kubernetes master version. If you do not specify a value, the latest available version is used string "1.21" no
label_key_case Controls the letter case of the tags keys (label names) for tags generated by this module.
Does not affect keys of tags passed in via the tags input.
Possible values: lower, title, upper.
Default value: title.
string null no
label_order The order in which the labels (ID elements) appear in the id.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.
list(string) null no
label_value_case Controls the letter case of ID elements (labels) as included in id,
set as tag values, and output by this module individually.
Does not affect values of tags passed in via the tags input.
Possible values: lower, title, upper and none (no transformation).
Set this to title and set delimiter to "" to yield Pascal Case IDs.
Default value: lower.
string null no
labels_as_tags Set of labels (ID elements) to include as tags in the tags output.
Default is to include all labels.
Tags with empty values will not be included in the tags output.
Set to [] to suppress all generated tags.
Notes:
The value of the name tag, if included, will be the id, not the name.
Unlike other null-label inputs, the initial setting of labels_as_tags cannot be
changed in later chained modules. Attempts to change it will be silently ignored.
set(string)
[
"default"
]
no
managed_security_group_rules_enabled Flag to enable/disable the ingress and egress rules for the EKS managed Security Group bool true no
name ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.
This is the only ID element not also included as a tag.
The "name" tag is set to the full id string. There is no tag with the value of the name input.
string null no
namespace ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique string null no
oidc_provider_enabled Create an IAM OIDC identity provider for the cluster, then you can create IAM roles to associate with a
service account in the cluster, instead of using kiam or kube2iam. For more information,
see EKS User Guide.
bool false no
permissions_boundary If provided, all IAM roles will be created with this permissions boundary attached string null no
public_access_cidrs Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0. list(string)
[
"0.0.0.0/0"
]
no
regex_replace_chars Terraform regular expression (regex) string.
Characters matching the regex will be removed from the ID elements.
If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits.
string null no
region OBSOLETE (not needed): AWS Region string null no
service_ipv4_cidr The CIDR block to assign Kubernetes service IP addresses from.
You can only specify a custom CIDR block when you create a cluster, changing this value will force a new cluster to be created.
string null no
stage ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release' string null no
subnet_ids A list of subnet IDs to launch the cluster in list(string) n/a yes
tags Additional tags (e.g. {'BusinessUnit': 'XYZ'}).
Neither the tag keys nor the tag values will be modified by this module.
map(string) {} no
tenant ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for string null no

Outputs

Name Description
cloudwatch_log_group_kms_key_id KMS Key ID to encrypt AWS CloudWatch logs
cloudwatch_log_group_name The name of the log group created in cloudwatch where cluster logs are forwarded to if enabled
cluster_encryption_config_enabled If true, Cluster Encryption Configuration is enabled
cluster_encryption_config_provider_key_alias Cluster Encryption Config KMS Key Alias ARN
cluster_encryption_config_provider_key_arn Cluster Encryption Config KMS Key ARN
cluster_encryption_config_resources Cluster Encryption Config Resources
eks_addons_versions Map of enabled EKS Addons names and versions
eks_cluster_arn The Amazon Resource Name (ARN) of the cluster
eks_cluster_certificate_authority_data The Kubernetes cluster certificate authority data
eks_cluster_endpoint The endpoint for the Kubernetes API server
eks_cluster_id The name of the cluster
eks_cluster_identity_oidc_issuer The OIDC Identity issuer for the cluster
eks_cluster_identity_oidc_issuer_arn The OIDC Identity issuer ARN for the cluster that can be used to associate IAM roles with a service account
eks_cluster_ipv6_service_cidr The IPv6 CIDR block that Kubernetes pod and service IP addresses are assigned from
if kubernetes_network_ipv6_enabled is set to true. If set to false this output will be null.
eks_cluster_managed_security_group_id Security Group ID that was created by EKS for the cluster.
EKS creates a Security Group and applies it to the ENI that are attached to EKS Control Plane master nodes and to any managed workloads.
eks_cluster_role_arn ARN of the EKS cluster IAM role
eks_cluster_version The Kubernetes server version of the cluster

Related Projects

Check out these related projects.

Tip

Use Terraform Reference Architectures for AWS

Use Cloud Posse's ready-to-go terraform architecture blueprints for AWS to get up and running quickly.

โœ… We build it with you.
โœ… You own everything.
โœ… Your team wins.

Request Quote

๐Ÿ“š Learn More

Cloud Posse is the leading DevOps Accelerator for funded startups and enterprises.

Your team can operate like a pro today.

Ensure that your team succeeds by using Cloud Posse's proven process and turnkey blueprints. Plus, we stick around until you succeed.

Day-0: Your Foundation for Success

  • Reference Architecture. You'll get everything you need from the ground up built using 100% infrastructure as code.
  • Deployment Strategy. Adopt a proven deployment strategy with GitHub Actions, enabling automated, repeatable, and reliable software releases.
  • Site Reliability Engineering. Gain total visibility into your applications and services with Datadog, ensuring high availability and performance.
  • Security Baseline. Establish a secure environment from the start, with built-in governance, accountability, and comprehensive audit logs, safeguarding your operations.
  • GitOps. Empower your team to manage infrastructure changes confidently and efficiently through Pull Requests, leveraging the full power of GitHub Actions.

Request Quote

Day-2: Your Operational Mastery

  • Training. Equip your team with the knowledge and skills to confidently manage the infrastructure, ensuring long-term success and self-sufficiency.
  • Support. Benefit from a seamless communication over Slack with our experts, ensuring you have the support you need, whenever you need it.
  • Troubleshooting. Access expert assistance to quickly resolve any operational challenges, minimizing downtime and maintaining business continuity.
  • Code Reviews. Enhance your teamโ€™s code quality with our expert feedback, fostering continuous improvement and collaboration.
  • Bug Fixes. Rely on our team to troubleshoot and resolve any issues, ensuring your systems run smoothly.
  • Migration Assistance. Accelerate your migration process with our dedicated support, minimizing disruption and speeding up time-to-value.
  • Customer Workshops. Engage with our team in weekly workshops, gaining insights and strategies to continuously improve and innovate.

Request Quote

โœจ Contributing

This project is under active development, and we encourage contributions from our community.

Many thanks to our outstanding contributors:

For ๐Ÿ› bug reports & feature requests, please use the issue tracker.

In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.

  1. Review our Code of Conduct and Contributor Guidelines.
  2. Fork the repo on GitHub
  3. Clone the project to your own machine
  4. Commit changes to your own branch
  5. Push your work back up to your fork
  6. Submit a Pull Request so that we can review your changes

NOTE: Be sure to merge the latest changes from "upstream" before making a pull request!

๐ŸŒŽ Slack Community

Join our Open Source Community on Slack. It's FREE for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally sweet infrastructure.

๐Ÿ“ฐ Newsletter

Sign up for our newsletter and join 3,000+ DevOps engineers, CTOs, and founders who get insider access to the latest DevOps trends, so you can always stay in the know. Dropped straight into your Inbox every week โ€” and usually a 5-minute read.

๐Ÿ“† Office Hours

Join us every Wednesday via Zoom for your weekly dose of insider DevOps trends, AWS news and Terraform insights, all sourced from our SweetOps community, plus a live Q&A that you canโ€™t find anywhere else. It's FREE for everyone!

License

License

Preamble to the Apache License, Version 2.0

Complete license is available in the LICENSE file.

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.

Trademarks

All other trademarks referenced herein are the property of their respective owners.


Copyright ยฉ 2017-2024 Cloud Posse, LLC

README footer

Beacon

terraform-aws-eks-cluster's People

Contributors

adamcrews avatar aknysh avatar alezkv avatar benclapp avatar bogdanbarna avatar brunochauvet avatar cloudpossebot avatar dependabot[bot] avatar dmitriy-lukyanchikov avatar dotcipher avatar gowiem avatar jasimmk avatar johncblandii avatar karlskewes avatar max-lobur avatar maximmi avatar nitrocode avatar nuru avatar osterman avatar osulli avatar renovate[bot] avatar robusgauli avatar shaunc avatar sweetops avatar tyu0912 avatar vkhatri avatar vymarkov avatar wintergren avatar woz5999 avatar yujunz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terraform-aws-eks-cluster's Issues

Include provision to restrict EKS public api access

Use Case:

As a multi cloud/multi account user, I should be able to secure/restrict access to my kubernetes public api to specific CIDRs. In my case, I should restrict access to my CI server ip blocks and accountvpc nat gateways.

Solution:
Using public_access_cidrs in vpc_config, we will be able to restrict this.

vpc_config:
    public_access_cidrs - (Optional) List of CIDR blocks. Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0. 

destroy does not remove cluster from kubeconfig

Destroy complete! Resources: 42 destroyed.
shauncutts@Shauns-MacBook-Pro complete % kubectl config get-clusters                   
NAME
...
arn:aws:eks:us-east-1:721727348533:cluster/eg-test-eks-cluster
...

Presumably "on delete provisioner" with kubectl config delete-cluster ... would do the job?

--- PS ---

For kubeconfig_path, I'm using:

# get path to kubectl config
data "external" "kube-config" { 
  program = [
    "/usr/bin/env", "bash", "-c", 
    "echo -n '{'\\\"config\\\": \\\"$${KUBECONFIG-$HOME/.kube/config}\\\"'}'"]
}
locals {
  kubeconfig_path = length(var.kubeconfig_path) == 0 ? data.external.kube-config.config : var.kubeconfig_path
}

Perhaps that would be useful as a default?

Error creating EKS Node Group

Seems like a minor issue, but when applying for the first time, I get an error:

module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Applying Auth ConfigMap with kubectl...
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Added new context arn:aws:eks:us-east-1:XXXXXXXXXXXX:cluster/org-dev-development-cluster to ~/.kube/config
module.development-cluster.null_resource.apply_configmap_auth[0]: Still creating... [10s elapsed]
module.development-cluster.null_resource.apply_configmap_auth[0]: Still creating... [21s elapsed]
module.development-cluster.null_resource.apply_configmap_auth[0]: Still creating... [31s elapsed]
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Unable to connect to the server: dial tcp 18.212.47.97:443: i/o timeout
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-15T15:50:38Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): configmap/aws-auth created
module.development-cluster.null_resource.apply_configmap_auth[0] (local-exec): Applied Auth ConfigMap with kubectl
module.development-cluster.null_resource.apply_configmap_auth[0]: Creation complete after 40s [id=275934236970031856]
Error: error creating EKS Node Group (org-development-cluster:org-dev-development-workers): ResourceNotFoundException: No cluster found for name: org-dev-development-cluster.
{
  Message_: "No cluster found for name: org-dev-development-cluster."
}
  on .terraform/modules/development-node-group/main.tf line 72, in resource "aws_eks_node_group" "default":
  72: resource "aws_eks_node_group" "default" {

After terraform apply the second time, it works. Is this a timing issue or something? Expected?

worker nodes not joining even after I run kubectl -f apply config-map-aws-auth-xxx.yaml

Hello gents,

I used your awesome EKS module to install kubernetes on eks earlier today, installation went well and dandy. A first kubectl get nodes yield the No resources found. I applied the aws-auth.yaml config file that should provide the necessary privilege for adding nodes but no nodes were added. My nodes and liveliness check all passed. Any idea what may be missing here?

Thanks.

Option to create IPv6 cluster

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

Allow creating EKS cluster with IPv6. https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.html

Expected Behavior

Add variable which will work as a toggle between IPv4 and IPv6.

Use Case

Being able to deploy EKS cluster with IPv6.

Describe Ideal Solution

As in expected behavior.

Alternatives Considered

Switching to terraform-aws-modules/terraform-aws-eks#1759

Additional Context

Need to wait for terraform-provider-aws 3.72.0 release, which will include hashicorp/terraform-provider-aws#22485. This will happen during current or following week.

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth"

Found a bug? Maybe our Slack Community can help.

Slack Community

Describe the Bug

I noticed that if the eks cluster is switching subnets, particularly public + private, to only private, the eks cluster will return an endpoint of localhost (for aws_eks_cluster.default[0].endpoint).

โ•ท
โ”‚ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused
โ”‚
โ”‚   with module.eks_cluster.kubernetes_config_map.aws_auth[0],
โ”‚   on .terraform-mdev/modules/eks_cluster/auth.tf line 135, in resource "kubernetes_config_map" "aws_auth":
โ”‚  135: resource "kubernetes_config_map" "aws_auth" {

Related code

provider "kubernetes" {
# Without a dummy API server configured, the provider will throw an error and prevent a "plan" from succeeding
# in situations where Terraform does not provide it with the cluster endpoint before triggering an API call.
# Since those situations are limited to ones where we do not care about the failure, such as fetching the
# ConfigMap before the cluster has been created or in preparation for deleting it, and the worst that will
# happen is that the aws-auth ConfigMap will be unnecessarily updated, it is just better to ignore the error
# so we can proceed with the task of creating or destroying the cluster.
#
# If this solution bothers you, you can disable it by setting var.dummy_kubeapi_server = null
host = local.enabled ? coalesce(aws_eks_cluster.default[0].endpoint, var.dummy_kubeapi_server) : var.dummy_kubeapi_server

Workaround 1

To get around this issue, I have to delete the kubeconfig map resource and then the module can be tricked to redeploying the eks cluster (due to the change in subnets).

cd components/terraform/eks/eks

terraform state rm 'module.eks_cluster.kubernetes_config_map.aws_auth[0]'

or in atmos

atmos terraform state eks/eks --stack dev-use2-qa rm 'module.eks_cluster.kubernetes_config_map.aws_auth[0]'

if this workaround was done by mistake, you can re-import the deleted config

atmos terraform import eks/eks --stack dev-use2-qa 'module.eks_cluster.kubernetes_config_map.aws_auth[0]' kube-system/aws-auth

Workaround 2

I've also ran into this issue when importing an existing cluster into the terraform module. My workaround for the import is to do a terraform init and modify the downloaded module eks_cluster's auth.tf to set the host arg of the kubernetes provider to the dummy url.

vim .terraform/modules/eks_cluster/auth.tf
--- a/auth.tf
+++ b/auth.tf
@@ -94,7 +94,7 @@ provider "kubernetes" {
   # so we can proceed with the task of creating or destroying the cluster.
   #
   # If this solution bothers you, you can disable it by setting var.dummy_kubeapi_server = null
-  host                   = local.enabled ? coalesce(aws_eks_cluster.default[0].endpoint, var.dummy_kubeapi_server) : var.dummy_kubeapi_server
+  host                   = var.dummy_kubeapi_server

Proposal

Instead, it would be nice if we could either detect that the endpoint returns localhost and use something else that won't fail the kubeconfig, or disable the kubernetes provider completely when the endpoint is localhost.

Template provider deprecated

Hello,

Is there any plans to remove "template provider" from this module and migrate to templatefile?

https://registry.terraform.io/providers/hashicorp/template/latest/docs

The template provider is deprecated and the provider has been archived in accordance with HashiCorp's provider archiving process. While released versions of the provider will remain available, we recommend that you replace usages of this provider as follows.

The problem I have is: template provider is not available for darwin_arm64

Error: configmaps "aws-auth" already exists

Describe the Bug

When setting kubernetes_config_map_ignore_role_changes = false to make changes to aws-auth configmap, build fails with
Error: configmaps "aws-auth" already exists

Expected Behavior

Build should succeed in updating/replacing the aws-auth configmap on the cluster.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Create cluster
  2. Set kubernetes_config_map_ignore_role_changes = false
  3. Terraform apply

Environment (please complete the following information):

git::https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=tags/0.35.0

Full example breaks with kubernetes version 1.20

Describe the Bug

Using the example but changing the kubernetes version to 1.20 breaks. The example works fine with version 1.19.

It gives the following error:

โ”‚ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
โ”‚ 
โ”‚   with module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0],
โ”‚   on .terraform/modules/eks_cluster/auth.tf line 83, in resource "kubernetes_config_map" "aws_auth_ignore_changes":
โ”‚   83: resource "kubernetes_config_map" "aws_auth_ignore_changes" {

which seems to indicate a dependency on a kubernetes provider that is trying to update the config map

Expected Behavior

Example should work with latest version of kubernetes support by eks

Steps to Reproduce

Steps to reproduce the behavior:

  1. clone working example dir
  2. Terraform init
    3.T erraform plan
  3. See error

Screenshots

If applicable, add screenshots or logs to help explain your problem.

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

  • OS: [e.g. Linux, OSX, WSL, etc] Linux
uname -a
Linux pop-os 5.11.0-7614-generic #15~1622578982~20.10~383c0a9-Ubuntu SMP Wed Jun 2 00:54:41 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

terraform
examples/complete/main.tf
  • cloudposse/eks-node-group/aws 2.12.0
  • cloudposse/label/null 0.25.0
  • cloudposse/dynamic-subnets/aws 2.4.1
  • cloudposse/vpc/aws 2.1.1
examples/complete/versions.tf
  • aws >= 5.34
  • kubernetes >= 2.7.1
  • null >= 2.0
  • tls >= 3.1.0
  • hashicorp/terraform >= 1.3.0
examples/complete/vpc-cni.tf
  • cloudposse/eks-iam-role/aws 2.1.1
main.tf
  • cloudposse/label/null 0.25.0
versions.tf
  • aws >= 5.34.0
  • hashicorp/terraform >= 1.3.0

  • Check this box to trigger a request for Renovate to run again on this repository

map_additional_iam_users does not work

Describe the Bug

First Created a eks cluster without any map_additional_iam_users variable
then added the following lines into terraform.tfvars and run

map_additional_iam_users = [
    {
    userarn = "arn:aws:iam::xyz:user/myuser"
    username = "myuser"
    groups   =  ["system:masters"]
 }
 ]

then

$ terraform plan

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

does not show any change

Terraform Apply times out when endpoint_public_access = false

Found a bug? Maybe our Slack Community can help.

Slack Community

Describe the Bug

Terraform Apply times out when when endpoint_public_access = false.

Expected Behavior

Terraform Commands should work from a computer on a on-prem network connected to EKS VPC

Steps to Reproduce

Steps to reproduce the behavior:

  1. Create an EKS Cluster with endpoint_public_access = false
  2. Run Terraform Apply command

Screenshots

module.eks.module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]: Refreshing state... [id=kube-system/aws-auth]

Error: Get "https://*************************.sk1.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 10.46.162.87:443: i/o timeout

Environment (please complete the following information):

Local machine on a on-prem network connected to EKS VPC

Additional Context

Default EKS Security Groups attached to the EKS does not allow traffic from outside

Support KUBERNETES_EXEC_INFO for kube_exec_auth

Describe the Feature

aws-cli is planning to stop using apiVersion: client.authentication.k8s.io/v1alpha1 and migrate to v1beta1 for eks get-token commands, see aws/aws-cli#6476. This time it is planned change and apiVersion can be configured via KUBERNETES_EXEC_INFO env var. This module should follow convention or provide way to specify expected apiVersion returned by exec auth. See #123.

Expected Behavior

Support upcoming aws eks get-token apiVersion when using kube_exec_auth_enabled = true.

Use Case

To my findings kube_exec_auth_enabled = true is the only option, that allows Terraform properly plan and apply when upgrading Kubernetes version on deployed clusters.

Describe Ideal Solution

Ideally end user shouldn't worry about what apiVersion is used by exec auth, but so far there is no easy way to autodetect this, as this version should be known prior to execution.

Alternatives Considered

Documenting expected KUBERNETES_EXEC_INFO value, so module continues working when aws-cli makes a switch.

Additional Context

https://kubernetes.io/docs/reference/access-authn-authz/_print/#input-and-output-formats

add permissions_boundary variable to aws_iam_role

Describe the Feature

add permissions_boundary variable to aws_iam_role

Expected Behavior

module should have a way to take permission_boundary as input

Use Case

Describe Ideal Solution

a new variable is added

Alternatives Considered

none

Additional Context

none

Can't tear down cluster after provisioning.

Describe the Bug

After provisioning an EKS Cluster with this module, you cannot then tear it down by commenting out the module.

Expected Behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

  1. Create module
  2. Plan and Apply
  3. Comment out the module
  4. Plan
  5. Error: Provider configuration not present

Screenshots

image

Environment (please complete the following information):

Terraform Cloud

intermittent timeout on create of `null_resource.apply_configmap_auth[0]`

I'm getting an error about 1/2 time processing "apply_configmap_auth" when creating a cluster w/ two worker groups. I have configured not to install aws cli or kubectl, so I go straight on to interacting with the cluster, which seems not to be ready yet.

Perhaps there should be a retry loop (perhaps around an initial attempt to get some info via kubectl)? NB -- re-terraform apply-ing always works, so retry loop should be effective.

module.eks-cluster.module.eks_cluster.null_resource.apply_configmap_auth[0]: Creating...

Error: Error running command '      set -e

      install_aws_cli=false
      if [[ "$install_aws_cli" = true ]] ; then
          echo 'Installing AWS CLI...'
          mkdir -p .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          cd .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          curl -LO https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
          unzip ./awscli-bundle.zip
          ./awscli-bundle/install -i .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          export PATH=$PATH:.terraform/modules/eks-cluster.eks_cluster/.terraform/bin:.terraform/modules/eks-cluster.eks_cluster/.terraform/bin/bin
          echo 'Installed AWS CLI'
          which aws
          aws --version
      fi

      install_kubectl=false
      if [[ "$install_kubectl" = true ]] ; then
          echo 'Installing kubectl...'
          mkdir -p .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          cd .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
          chmod +x ./kubectl
          export PATH=$PATH:.terraform/modules/eks-cluster.eks_cluster/.terraform/bin
          echo 'Installed kubectl'
          which kubectl
      fi

      aws_cli_assume_role_arn=
      aws_cli_assume_role_session_name=
      if [[ -n "$aws_cli_assume_role_arn" && -n "$aws_cli_assume_role_session_name" ]] ; then
        echo 'Assuming role  ...'
        mkdir -p .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
        cd .terraform/modules/eks-cluster.eks_cluster/.terraform/bin
        curl -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 -o jq
        chmod +x ./jq
        source <(aws --output json sts assume-role --role-arn "$aws_cli_assume_role_arn" --role-session-name "$aws_cli_assume_role_session_name"  | jq -r  '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')
        echo 'Assumed role '
      fi

      echo 'Applying Auth ConfigMap with kubectl...'
      aws eks update-kubeconfig --name=ff-test-k8test-cluster --region=us-east-1 --kubeconfig=/Users/shauncutts/.kube/config 
      kubectl version --kubeconfig /Users/shauncutts/.kube/config
      kubectl apply -f .terraform/modules/eks-cluster.eks_cluster/configmap-auth.yaml --kubeconfig /Users/shauncutts/.kube/config
      echo 'Applied Auth ConfigMap with kubectl'
': exit status 1. Output: Applying Auth ConfigMap with kubectl...
Updated context arn:aws:eks:us-east-1:721727348533:cluster/ff-test-k8test-cluster in /Users/shauncutts/.kube/config
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: dial tcp 54.209.75.156:443: i/o timeout

Conflicting `aws_iam_role_policy` resources

Describe the Bug

After #132 there are two aws_iam_role_policy resources, that reference same role. This results into permanent diff in terraform plan and only one inline policy being applied.

% terraform plan
...
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.eks.aws_iam_role.default[0] has been changed
  ~ resource "aws_iam_role" "default" {
        id                    = "poc-01-cluster"
        name                  = "poc-01-cluster"
        tags                  = {
            "Attributes" = "cluster"
            "Name"       = "poc-01-cluster"
        }
        # (9 unchanged attributes hidden)

      - inline_policy {
          - name   = "poc-01-cluster" -> null
          - policy = jsonencode(
                {
                  - Statement = [
                      - {
                          - Action   = [
                              - "elasticloadbalancing:SetSubnets",
                              - "elasticloadbalancing:SetIpAddressType",
                              - "ec2:DescribeInternetGateways",
                              - "ec2:DescribeAddresses",
                              - "ec2:DescribeAccountAttributes",
                            ]
                          - Effect   = "Allow"
                          - Resource = "*"
                          - Sid      = ""
                        },
                    ]
                  - Version   = "2012-10-17"
                }
            ) -> null
        }
      + inline_policy {
          + name   = "poc-01-cluster"
          + policy = jsonencode(
                {
                  + Statement = [
                      + {
                          + Action   = "logs:CreateLogGroup"
                          + Effect   = "Deny"
                          + Resource = "*"
                          + Sid      = ""
                        },
                    ]
                  + Version   = "2012-10-17"
                }
            )
        }
    }
  # module.eks.aws_iam_role_policy.cluster_elb_service_role[0] has been changed
  ~ resource "aws_iam_role_policy" "cluster_elb_service_role" {
        id     = "poc-01-cluster:poc-01-cluster"
        name   = "poc-01-cluster"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action   = [
                          - "elasticloadbalancing:SetSubnets",
                          - "elasticloadbalancing:SetIpAddressType",
                          - "ec2:DescribeInternetGateways",
                          - "ec2:DescribeAddresses",
                          - "ec2:DescribeAccountAttributes",
                        ] -> "logs:CreateLogGroup"
                      ~ Effect   = "Allow" -> "Deny"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.eks.aws_iam_role_policy.cluster_elb_service_role[0] will be updated in-place
  ~ resource "aws_iam_role_policy" "cluster_elb_service_role" {
        id     = "poc-01-cluster:poc-01-cluster"
        name   = "poc-01-cluster"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action   = "logs:CreateLogGroup" -> [
                          + "elasticloadbalancing:SetSubnets",
                          + "elasticloadbalancing:SetIpAddressType",
                          + "ec2:DescribeInternetGateways",
                          + "ec2:DescribeAddresses",
                          + "ec2:DescribeAccountAttributes",
                        ]
                      ~ Effect   = "Deny" -> "Allow"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Expected Behavior

There shouldn't be diff after successful apply.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Apply any of repo examples
  2. Run 'terraform apply' again
  3. See diff that cannot be properly applied

Screenshots

N/A

Environment (please complete the following information):

% terraform version
Terraform v1.0.11
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.66.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.6.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0
+ provider registry.terraform.io/hashicorp/vault v2.24.1

Additional Context

N/A

Don't create a new Security Group for EKS cluster if it's only used with Managed Node Groups

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

  • Don't create a new Security Group for EKS cluster if it's used with Managed Node Groups

Describe Ideal Solution

We need to create a new Security Group only if the EKS cluster is used with unmanaged worker nodes.
EKS creates a security group for the cluster, places the control plane and managed nodes into the security group, and allows all communications between the control plane and the managed worker nodes.
If only Managed Node Groups are used, we don't need to create a separate Security Group; otherwise we place the cluster in two SG - one that is created by EKS, the other that the module creates.

See https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html for more details.

Kubernetes provider 2.0 breaks terraform-aws-eks-cluster v0.29.1 - v0.30.2

Describe the Bug

Hashicorp released Kubernetes provider 2.0.0 which has breaking changes and breaks terraform-aws-eks-cluster v0.29.1 through and including v0.30.2. (While earlier versions are not compatible either, they have appropriate version pinning for the Kubernetes provider.)

Steps to Reproduce

$ terraform plan
...
Error: Unsupported argument

  on .terraform/modules/eks_cluster/auth.tf line 81, in provider "kubernetes":
  81:   load_config_file       = var.kubernetes_load_config_file

An argument named "load_config_file" is not expected here.

Creating CloudWatch Log Group failed: ResourceAlreadyExistsException:

what

Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists:  The CloudWatch Log Group '/aws/eks/eg-test-eks-cluster/cluster' already exists

why

See: hashicorp/terraform#14750, terraform-aws-modules/terraform-aws-eks#920

This is happening because EKS Cluster gets destroyed after Terraform delete the Cloudwatch Log Group. The AmazonEKSServicePolicy IAM policy (that is assigned to EKS Cluster role by default within this module) has permissions to CreateLogGroup and anything else needed to continue to logging correctly. When the Terraform destroys the Cloudwatch Log Group, the EKS Cluster that is running create it again. Then, when you run Terraform Apply again, the Cloudwatch Log Group doesn't exist in your state anymore (because the Terraform actually destroyed it) and the Terraform doesn't know this resource created outside him. terraform-aws-modules/terraform-aws-eks/issues/920

fix for tests

  • add random attribute to tests

Allow k8s config file load

Describe the Feature

Expose a kubernetes_load_config_file variable that allows configuring the kubernetes provider flag for load_config_file

Expected Behavior

Setting a bool type variable that would be passed through to https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L81

Use Case

I ran into issues when the cluster I have provisioned was provisioned with a different user than I upgraded with. Since introduction of the native k8s config map resource in https://github.com/cloudposse/terraform-aws-eks-cluster/releases/tag/0.21.0, migration has been an issue since some clusters were provisioned with different user accounts than those I have on CI. It became needed to migrate to the native config map resource by using the local config file once during the upgrade then turning the load_config_file flag back off when done.

Fail with I/O timeout due to bad configuration of the Kubernetes provider

Describe the Bug

Creating or modifying an EKS cluster fails (sometimes) with errors suggesting the Kubernetes provider is not properly configured. This appears to be more of a problem with Terraform 0.14 than with Terraform 0.13.

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: i/o timeout

Steps to Reproduce

Steps to reproduce the behavior:

  1. Run terraform plan
  2. See the error

Environment (please complete the following information):

  • Terraform 0.14.6
  • terraform-aws-eks-cluster 0.34.1

Update

This appears to be a bug in the Kubernetes v2 Terraform provider (see comments below). Until v2 works the way we want, we recommend using terraform-aws-eks-cluster v0.37.0 or later and pinning the Kubernetes provider to ~> 1.13

aws_iam_role tags not defined

I need to define tags linked to aws_iam_role resources. Could you add tags to aws_iam_role default please?
In main.tf

Ability to install and manage EKS addons

Describe the Feature

EKS Addons is GA feature and they are installed by default when creating EKS cluster manually via AWS web console.

Expected Behavior

Module would provide parameters to specify addons to be installed with cluster. Ability to manage addon versions and associated service account roles is a must, as they have their own lifecycle.

Use Case

Keeping up with current EKS behavior. Also EKS upgrade path between versions is documented in way where it's expected that cluster has addons installed.

Describe Ideal Solution

Ideally module should maintain default list of addons and versions that used by selected kubernetes version, with ability of update addons when needed.

Alternatives Considered

Manage aws_eks_addon resources outside of the module.

Additional Context

None.

Unable to add cluster_log_types

On an existing EKS cluster that was created with this module, I'm am unable to add cluster_log_types to the cluster.

module "eks_cluster" {
  source                       = "cloudposse/eks-cluster/aws"
  enabled                      = data.aws_ssm_parameter.enabled_eks_cluster.value
  version                      = "0.39.0"
  name                         = "eks-${var.env}"
  region                       = var.region
  tags                         = merge(data.terraform_remote_state.tags.outputs.tags,local.eks_tags)
  vpc_id                       = data.terraform_remote_state.vpc.outputs.vpc_id
  subnet_ids                   = data.terraform_remote_state.vpc.outputs.app_subnets
  endpoint_private_access      = true
  endpoint_public_access       = false
  allowed_cidr_blocks          = local.cidr_access_blocks
  kubernetes_version           = data.aws_ssm_parameter.kubernetes_version.value
  oidc_provider_enabled        = true
  workers_role_arns            = [module.eks_node_group.eks_node_group_role_arn]
  map_additional_iam_roles     = local.map_additional_iam_roles
  enabled_cluster_log_types    = ["api","audit","authenticator","controllerManager","scheduler"]
}

Expected Behavior

A log group should be created and the EKS resource should be updated to collect control plane logs.

The Error

Error: error associating EKS Cluster (cluster-name) encryption config: InvalidParameterException: Encryption is already enabled on the cluster
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "0244cb16-107c-49fb-95b8-0a48783d5722"
  },
  ClusterName: <cluster-name>,
  Message_: "Encryption is already enabled on the cluster"
}

  on .terraform/modules/eks_cluster/main.tf line 45, in resource "aws_eks_cluster" "default":
  45: resource "aws_eks_cluster" "default" {

Additional Context

It is worth noting that once I remove the "enabled_cluster_log_types" from the module, everything works fine.

Error downloading the module at init time

Initializing modules...

  • module.eks_cluster
  • module.eks_workers
    Getting source "../../"
  • module.eks_cluster.label
    Error downloading modules: Error loading modules: module label: Error parsing .terraform/modules/84d35de50b912d05df660b215d5fb775/main.tf: At 2:22: Unknown token: 2:22 IDENT join

pods in pending state

Hi after applying terraform files in examples/complete repo, when i apply a deployment file for nginx, i get following:

kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-16-105-17.ec2.internal Ready 43m v1.14.7-eks-1861c5
ip-172-16-143-67.ec2.internal Ready 43m v1.14.7-eks-1861c5

kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-784b7cc96d-dkh7f 0/1 Pending 0 39m
nginx-deployment-784b7cc96d-l44kx 0/1 Pending 0 39m
nginx-deployment-784b7cc96d-l8dxq 0/1 Pending 0 39m

kubectl cluster-info dump | grep schedule
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"message": "no nodes available to schedule pods",
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"schedulerName": "default-scheduler",
"numberMisscheduled": 0,
"schedulerName": "default-scheduler",
"numberMisscheduled": 0,
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
"schedulerName": "default-scheduler",
I0109 14:35:30.812439 1 flags.go:33] FLAG: --ipvs-scheduler=
I0109 14:35:29.309333 1 flags.go:33] FLAG: --ipvs-scheduler=
"schedulerName": "default-scheduler"
"schedulerName": "default-scheduler"
"schedulerName": "fargate-scheduler",
"schedulerName": "fargate-scheduler",
"schedulerName": "fargate-scheduler", "component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"component": "default-scheduler"
"message": "no nodes available to schedule pods",

Can you check why these pods don't get scheduled on available worker nodes. I used t3.medium size for worker nodes. Thanks

This doesn't create a kube config?

If I don't enable oidc, it errors out:

Error: Unauthorized

  on ../../auth.tf line 85, in resource "kubernetes_config_map" "aws_auth_ignore_changes":
  85: resource "kubernetes_config_map" "aws_auth_ignore_changes" {


cat: /Users/rlewkowicz/.kube/config: No such file or directory

How am I supposed to provide the file in advance if it doesn't exist. Isn't this supposed to make one?

edit: Now it just seems to fail in general, I did delete the config, I'm hoping the last run didn't apply a config map to a random cluster

Migrate kube_exec_auth to `client.authentication.k8s.io/v1beta1` api version

Describe the Bug

Running EKS 1.21. Module configured as

  kube_exec_auth_enabled             = true
  kube_exec_auth_aws_profile_enabled = true
  kube_exec_auth_aws_profile         = local.aws_profile

  kubernetes_config_map_ignore_role_changes = false

with latest AWS CLI v1.

terraform plan on existing provisioned cluster produces:

โ•ท
โ”‚ Error: Get "https://1B1210D97134DAE6C8D072075758CAE9.gr7.eu-central-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps/aws-auth": getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1, plugin returned version client.authentication.k8s.io/v1beta1
โ”‚ 
โ”‚   with module.eks.kubernetes_config_map.aws_auth[0],
โ”‚   on .terraform/modules/eks/auth.tf line 132, in resource "kubernetes_config_map" "aws_auth":
โ”‚  132: resource "kubernetes_config_map" "aws_auth" {
โ”‚ 
โ•ต

Expected Behavior

Plan should succeed.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Install latest AWS CLI v1
  2. Provision EKS cluster with mentioned parameters

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

% terraform version
Terraform v1.0.3
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.50.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.3.2
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0
+ provider registry.terraform.io/hashicorp/vault v2.21.0

Additional Context

aws/aws-cli@3569d49

Error : module.eks-cluster.null_resource.apply_configmap_auth[0] (local-exec): * es

I've setup vars in the template and ran terraform apply then at the end of the build - I get this error msg:
Any ideas?
module.eks-cluster.null_resource.apply_configmap_auth[0] (local-exec): Invalid choice: 'eks', maybe you meant:

module.eks-cluster.null_resource.apply_configmap_auth[0] (local-exec): * es

Error: Error running command ' while [[ ! -e .terraform/modules/eks-cluster/cloudposse-terraform-aws-eks-cluster-e4b29b1/configmap-auth.yaml ]] ; do sleep 1; done &&
aws eks update-kubeconfig --name=k8s-cluster --region=eu-west-1 --kubeconfig=~/.kube/config &&
kubectl apply -f .terraform/modules/eks-cluster/cloudposse-terraform-aws-eks-cluster-e4b29b1/configmap-auth.yaml --kubeconfig ~/.kube/config
': exit status 2. Output: usage: aws [options] [ ...] [parameters]
To see help text, you can run:

Add Example Usage

what

  • Add example invocation

why

  • We need this so we can soon enable automated continuous integration testing of module

attributes should not be extended without condition

Found a bug? Maybe our Slack Community can help.

Slack Community

Describe the Bug

The main.tf has this:

module "label" {
  ...
  attributes  = compact(concat(var.attributes, ["cluster"]))
  ...

and this:

resource "aws_eks_cluster" "default" {
  count                     = var.enabled ? 1 : 0
  name                      = module.label.id

This means that even if I set var.attributes to empty, clearly stating "I don't want any attributes", the module goes ahead and adds a "cluster" attribute. So my cluster's name will always end in "-cluster". Is this concat necessary? I'm thinking it is a bug not a feature.

Expected Behavior

If I set attributes to [] in eks_cluster, then that's what it should be throughout. Eg I should end up with a cluster name that does not end in "-cluster".

Enable encyption_config in EKS Cluster Creation (secrets using KMS at rest - on etcd -)

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

aws_eks_cluster now support Encryption Config to use KMS on Secrets on rest (etcd)

Expected Behavior

To have:

  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks.arn
      resources        = ["secrets"]
    }
  ]

Use Case

https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/

Describe Ideal Solution

A clear and concise description of what you want to happen. If you don't know, that's okay.

Alternatives Considered

Explain what alternative solutions or features you've considered.

Additional Context

Add any other context or screenshots about the feature request here.

multi-worker EKS not working

Describe the Bug

Trying to setup a multi-worker EKS cluster.

Following the example from here.
And the "Module usage with two worker groups" section from the documentation.

Modified the main.tf:

provider "aws" {
  region = var.region
}

module "label" {
  source     = "git::https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.16.0"
  namespace  = var.namespace
  name       = var.name
  stage      = var.stage
  delimiter  = var.delimiter
  attributes = compact(concat(var.attributes, list("cluster")))
  tags       = var.tags
}

locals {
  # The usage of the specific kubernetes.io/cluster/* resource tags below are required
  # for EKS and Kubernetes to discover and manage networking resources
  # https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html#base-vpc-networking
  tags = merge(module.label.tags, map("kubernetes.io/cluster/${module.label.id}", "shared"))

  # Unfortunately, most_recent (https://github.com/cloudposse/terraform-aws-eks-workers/blob/34a43c25624a6efb3ba5d2770a601d7cb3c0d391/main.tf#L141)
  # variable does not work as expected, if you are not going to use custom ami you should
  # enforce usage of eks_worker_ami_name_filter variable to set the right kubernetes version for EKS workers,
  # otherwise will be used the first version of Kubernetes supported by AWS (v1.11) for EKS workers but
  # EKS control plane will use the version specified by kubernetes_version variable.
  eks_worker_ami_name_filter = "amazon-eks-node-${var.kubernetes_version}*"
}

module "vpc" {
  source     = "git::https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.8.1"
  namespace  = var.namespace
  stage      = var.stage
  name       = var.name
  attributes = var.attributes
  cidr_block = "10.20.0.0/16"
  tags       = local.tags
}

module "subnets" {
  source               = "git::https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.19.0"
  availability_zones   = var.availability_zones
  namespace            = var.namespace
  stage                = var.stage
  name                 = var.name
  attributes           = var.attributes
  vpc_id               = module.vpc.vpc_id
  igw_id               = module.vpc.igw_id
  cidr_block           = module.vpc.vpc_cidr_block
  nat_gateway_enabled  = false
  nat_instance_enabled = false
  tags                 = local.tags
}

module "eks_workers" {
  source                             = "git::https://github.com/cloudposse/terraform-aws-eks-workers.git?ref=tags/0.13.0"
  namespace                          = var.namespace
  stage                              = var.stage
  name                               = "small"
  attributes                         = var.attributes
  tags                               = var.tags
  instance_type                      = "t3.small"
  vpc_id                             = module.vpc.vpc_id
  subnet_ids                         = module.subnets.public_subnet_ids
  health_check_type                  = var.health_check_type
  min_size                           = var.min_size
  max_size                           = var.max_size
  wait_for_capacity_timeout          = var.wait_for_capacity_timeout
  cluster_name                       = module.eks_cluster.eks_cluster_id
  cluster_endpoint                   = module.eks_cluster.eks_cluster_endpoint
  cluster_certificate_authority_data = module.eks_cluster.eks_cluster_certificate_authority_data
  cluster_security_group_id          = module.eks_cluster.security_group_id

  # Auto-scaling policies and CloudWatch metric alarms
  autoscaling_policies_enabled           = var.autoscaling_policies_enabled
  cpu_utilization_high_threshold_percent = var.cpu_utilization_high_threshold_percent
  cpu_utilization_low_threshold_percent  = var.cpu_utilization_low_threshold_percent
}

module "eks_workers_2" {
  source                             = "git::https://github.com/cloudposse/terraform-aws-eks-workers.git?ref=tags/0.13.0"
  namespace                          = var.namespace
  stage                              = var.stage
  name                               = "medium"
  attributes                         = var.attributes
  tags                               = var.tags
  instance_type                      = "t3.medium"
  vpc_id                             = module.vpc.vpc_id
  subnet_ids                         = module.subnets.public_subnet_ids
  health_check_type                  = var.health_check_type
  min_size                           = var.min_size
  max_size                           = var.max_size
  wait_for_capacity_timeout          = var.wait_for_capacity_timeout
  cluster_name                       = module.eks_cluster.eks_cluster_id
  cluster_endpoint                   = module.eks_cluster.eks_cluster_endpoint
  cluster_certificate_authority_data = module.eks_cluster.eks_cluster_certificate_authority_data
  cluster_security_group_id          = module.eks_cluster.security_group_id

  # Auto-scaling policies and CloudWatch metric alarms
  autoscaling_policies_enabled           = var.autoscaling_policies_enabled
  cpu_utilization_high_threshold_percent = var.cpu_utilization_high_threshold_percent
  cpu_utilization_low_threshold_percent  = var.cpu_utilization_low_threshold_percent
}

module "eks_cluster" {
  source     = "git::https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=tags/0.20.0"
  namespace  = var.namespace
  stage      = var.stage
  name       = var.name
  attributes = var.attributes
  tags       = var.tags
  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.subnets.public_subnet_ids

  region     = var.region

  kubernetes_version = var.kubernetes_version
  kubeconfig_path    = var.kubeconfig_path

  oidc_provider_enabled = false

  workers_role_arns          = [module.eks_workers.workers_role_arn, module.eks_workers_2.workers_role_arn]
  workers_security_group_ids = [module.eks_workers.security_group_id, module.eks_workers_2.security_group_id]
}

and terraform.tfvars (renamed from fixtures.us-east-2.tfvars):

region = "eu-central-1"

availability_zones = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]

namespace = "eg"

stage = "test"

name = "eks-debugging"

instance_type = "t2.small"

health_check_type = "EC2"

wait_for_capacity_timeout = "10m"

max_size = 3

min_size = 2

autoscaling_policies_enabled = true

cpu_utilization_high_threshold_percent = 80

cpu_utilization_low_threshold_percent = 20

associate_public_ip_address = true

kubernetes_version = "1.14"

kubeconfig_path = "~/.kube/config-eks-debug"

oidc_provider_enabled = true

enabled_cluster_log_types = ["audit"]

cluster_log_retention_period = 7

The issue which I get is that the workers do NOT join the cluster:

$ kubectl --kubeconfig ~/.kube/config-eks-debug get nodes
No resources found in default namespace.

Expected Behavior

Worker nodes join the cluster

Steps to Reproduce

Steps to reproduce the behavior:

  1. Clone:
git clone https://gist.github.com/38a1467a53805aaa2acee359804a3072.git .
  1. Apply
terraform apply
  1. Verify no workers are in the cluster:
$ kubectl --kubeconfig ~/.kube/config-eks-debug get nodes
No resources found in default namespace.

Screenshots

If applicable, add screenshots or logs to help explain your problem.

Environment (please complete the following information):

Anything that will help us triage the bug will help. Here are some ideas:

  • OS:
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
  • Version
$ terraform -v
Terraform v0.12.20
Module Version
terraform-null-label 0.16.0
terraform-aws-vpc 0.8.1
terraform-aws-dynamic-subnets 0.19.0
terraform-aws-eks-workers 0.13.0
terraform-aws-eks-workers 0.13.0
terraform-aws-eks-cluster 0.20.0

Additional Context

Add any other context about the problem here.

Feature Request: support `service_ipv4_cidr`

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

service_ipv4_cidr should be configurable.

Expected Behavior

service_ipv4_cidr should be null by default, and can be overridden to support another CIDR block range.

Use Case

Historically, EKS would set the Service ClusterIP CIDR block to either 10.100.0.0/16 or 172.20.0.0/16, and the user would not be able to override it. This would force the user to avoid using these ranges in the VPC and any peered VPCs.

Being able to override this CIDR block would allow the user to tailor the EKS cluster to their infrastructure's networking requirements, rather than the other way around.

Describe Ideal Solution

The aws_eks_cluster resource should have a dynamic kubernetes_network_config configuration block and set service_ipv4_cidr if a variable of the same name is supplied to the module.

Alternatives Considered

  • None (the only alternative is to do nothing and not support this).

Additional Context

Missing attribute?

I'm having an issue using the example provided, which result in the following after a terraform plan

Error: Error running plan: 5 error(s) occurred:

* module.eks_cluster.output.eks_cluster_id: Resource 'aws_eks_cluster.default' does not have attribute 'id' for variable 'aws_eks_cluster.default.*.id'
* module.eks_cluster.output.eks_cluster_version: Resource 'aws_eks_cluster.default' does not have attribute 'version' for variable 'aws_eks_cluster.default.*.version'
* module.eks_cluster.local.certificate_authority_data_list: local.certificate_authority_data_list: Resource 'aws_eks_cluster.default' does not have attribute 'certificate_authority' for variable 'aws_eks_cluster.default.*.certificate_authority'
* module.eks_cluster.output.eks_cluster_arn: Resource 'aws_eks_cluster.default' does not have attribute 'arn' for variable 'aws_eks_cluster.default.*.arn'
* module.eks_cluster.output.eks_cluster_endpoint: Resource 'aws_eks_cluster.default' does not have attribute 'endpoint' for variable 'aws_eks_cluster.default.*.endpoint'

How can I fix this?

Private cluster sg rules destroyed before cluster can be deleted

Describe the Bug

When running terraform destroy on a private cluster, ingress rules for allowed cidr blocks are destroyed before the aws-auth configmap, leading to a timeout error.

Expected Behavior

Successful terraform destroy

Steps to Reproduce

Steps to reproduce the behavior:

  1. Create a private only cluster with allowed_cidr_blocks set to include your build location
  2. Run terraform destroy from within the cidr

Logs

module.eks_cluster.aws_security_group_rule.ingress_cidr_blocks[0]: Destroying... 
...
module.eks_cluster.aws_security_group_rule.ingress_cidr_blocks[0]: Destruction complete after 1s
...
Error: Delete "https://private-endpoint.ca-central-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 10.x.x.x:443: i/o timeout

Environment (please complete the following information):

Using v0.29.0 because of open issues with tf 0.14 and kubernetes provider > 2.0

Enable AWS_VPC_K8S_CNI_EXTERNALSNAT=true

In order to use a Network Load Balancer with "Preserve Source IP" as Ingress to a Kubernetes service, you need to put the EKS cluster in a private subnet and set AWS_VPC_K8S_CNI_EXTERNALSNAT=true in the aws-node Daemonset. According to AWS, the way to do this is

kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true

We need some way to set this via Terraform, since it is a required setting.

References

Is associate_public_ip_address used?

Nodes in the cluster I spun up are getting public IPs that aren't within the subnet's public cidr block, using the combination of the modules:

I wasn't able to find the source of the problem until I found the flag in this repo outlining associate_public_ip_address, which seemed like it was the source of the problem considering the README.md specifies that it's default is true. Based on the AWS EC2 Instance Addressing docs, the public subnet IPs might be getting clobbered by the auto-assignment of the ec2 instance IP:

Enabling or disabling the public IP addressing feature during launch, which overrides the subnet's public IP addressing attribute. For more information, see Assigning a Public IPv4 Address During Instance Launch.

  1. Is associate_public_ip_address used anywhere? I tried switching it and noticed that I had no diff in state to apply?
  2. More of a meta-question: but is that the right flag I should be looking to change?

Allow customization of cluster assume role policy

Have a question? Please checkout our Slack Community or visit our Slack Archive.

Slack Community

Describe the Feature

Add a variable that allows to provide a custom assume role policy or add additional stuff to the statement. Or just your own IAM role.

Use Case

I have a managed AWS account where all roles MUST be able to be assumed by a monitoring role.

node not joining cluster

Hi,

during the deploymnet everithing went well, but when i tryed to query the cluster i get

kubectl get no
No resources found.

this are the logs from the node.

Mar 19 14:01:01 ip-172-18-14-119 kubelet: E0319 14:01:01.309269 3780 kubelet_node_status.go:103] Unable to register node "ip-172-18-14-119.eu-north-1.compute.internal" with API server: Unauthorized Mar 19 14:01:01 ip-172-18-14-119 kubelet: E0319 14:01:01.958798 3780 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-18-14-119.eu-north-1.compute.internal" not found Mar 19 14:01:02 ip-172-18-14-119 kubelet: E0319 14:01:02.055118 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:02 ip-172-18-14-119 kubelet: E0319 14:01:02.055257 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:02 ip-172-18-14-119 kubelet: E0319 14:01:02.055622 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:03 ip-172-18-14-119 kubelet: W0319 14:01:03.269962 3780 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d Mar 19 14:01:03 ip-172-18-14-119 kubelet: E0319 14:01:03.270349 3780 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Mar 19 14:01:03 ip-172-18-14-119 kubelet: E0319 14:01:03.560006 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:03 ip-172-18-14-119 kubelet: E0319 14:01:03.560011 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:03 ip-172-18-14-119 kubelet: E0319 14:01:03.560053 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:05 ip-172-18-14-119 kubelet: E0319 14:01:05.058712 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:05 ip-172-18-14-119 kubelet: E0319 14:01:05.058712 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:05 ip-172-18-14-119 kubelet: E0319 14:01:05.059272 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:06 ip-172-18-14-119 kubelet: E0319 14:01:06.557641 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:06 ip-172-18-14-119 kubelet: E0319 14:01:06.557641 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:06 ip-172-18-14-119 kubelet: E0319 14:01:06.558190 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:07 ip-172-18-14-119 kubelet: E0319 14:01:07.361823 3780 certificate_manager.go:299] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Unauthorized Mar 19 14:01:08 ip-172-18-14-119 kubelet: E0319 14:01:08.058109 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:08 ip-172-18-14-119 kubelet: E0319 14:01:08.058109 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:08 ip-172-18-14-119 kubelet: E0319 14:01:08.058151 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:08 ip-172-18-14-119 kubelet: W0319 14:01:08.271248 3780 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d Mar 19 14:01:08 ip-172-18-14-119 kubelet: E0319 14:01:08.271387 3780 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Mar 19 14:01:08 ip-172-18-14-119 kubelet: I0319 14:01:08.309459 3780 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach Mar 19 14:01:08 ip-172-18-14-119 kubelet: I0319 14:01:08.309494 3780 kubelet_node_status.go:317] Adding node label from cloud provider: beta.kubernetes.io/instance-type=t3.medium Mar 19 14:01:08 ip-172-18-14-119 kubelet: I0319 14:01:08.309505 3780 kubelet_node_status.go:328] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=eu-north-1a Mar 19 14:01:08 ip-172-18-14-119 kubelet: I0319 14:01:08.309511 3780 kubelet_node_status.go:332] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=eu-north-1 Mar 19 14:01:08 ip-172-18-14-119 kubelet: I0319 14:01:08.323430 3780 kubelet_node_status.go:79] Attempting to register node ip-172-18-14-119.eu-north-1.compute.internal Mar 19 14:01:08 ip-172-18-14-119 kubelet: E0319 14:01:08.912198 3780 kubelet_node_status.go:103] Unable to register node "ip-172-18-14-119.eu-north-1.compute.internal" with API server: Unauthorized Mar 19 14:01:09 ip-172-18-14-119 kubelet: E0319 14:01:09.656549 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:09 ip-172-18-14-119 kubelet: E0319 14:01:09.656709 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:09 ip-172-18-14-119 kubelet: E0319 14:01:09.657268 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:11 ip-172-18-14-119 kubelet: E0319 14:01:11.179279 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Unauthorized Mar 19 14:01:11 ip-172-18-14-119 kubelet: E0319 14:01:11.179279 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Unauthorized Mar 19 14:01:11 ip-172-18-14-119 kubelet: E0319 14:01:11.181153 3780 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized Mar 19 14:01:11 ip-172-18-14-119 kubelet: E0319 14:01:11.959015 3780 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-18-14-119.eu-north-1.compute.internal" not found

Add an ability to setup initial deployments during cluster setup as an addons.

Description

The initial cluster setup usually requires few initial deployments of following services inside eks cluster such as:

  1. Kubernetes dashboard
  2. Storage class for provisioning of persistent volumes
  3. External DNS
  4. ALB Ingress Controller
  5. Pod autoscalar,

Expected terraform module for eks cluster

module "eks_cluster" {
  source             = "../../"
  namespace          = var.namespace
  stage              = var.stage
  name               = var.name
  attributes         = var.attributes
  tags               = var.tags
  region             = var.region
  vpc_id             = module.vpc.vpc_id
  subnet_ids         = module.subnets.public_subnet_ids
  kubernetes_version = var.kubernetes_version
  kubeconfig_path    = var.kubeconfig_path

  workers_role_arns          = [module.eks_workers.workers_role_arn]
  workers_security_group_ids = [module.eks_workers.security_group_id]

addons  = [
    "https://raw.githubusercontent.com/cloudpoose/terraform-aws-eks-cluster/master/addons/storage-class.yaml.tpl",
    "https://raw.githubusercontent.com/cloudpoose/terraform-aws-eks-cluster/master/addons/dashboard.yaml.tpl"
  ]
}

Consequences

This would create k8s deployments during runtime by rendering these templates using template_file data resource and apply those deployments using local-exec. These addons would be optional and would only be applied if we pass it to module using addons variable which would be of type list(string).

Asks for var.cluster_certificate_authority_data when referencing module.cluster as in examples

Hi,

I've made a setup similar to your example where I have both eks_cluster and eks_workers modules.

in eks_cluster we have this:
cluster_certificate_authority_data = "${module.eks_cluster.eks_cluster_certificate_authority_data}"

Now when I run a tf plan I'm asked to supply a value for:

var.cluster_certificate_authority_data
  The base64 encoded certificate data required to communicate with the cluster

  Enter a value:

That variable is required by your workers module but should get it's value from the cluster to be created. How is the dependencies handled here as per your example? Is the workers module resources created before the EKS cluster? As the cluster also references workers module here:
workers_security_group_ids = ["${module.eks_workers.security_group_id}"]
I guess there will be some dependency issues.

allowed_cidr_blocks and allowed_security_groups cannot access workers

Describe the Bug

Setting allowed_cidr_blocks and allowed_security_groups does not enable access to workers

Expected Behavior

Clients attempting to access workers from allowed_cidr_blocks and allowed_security_groups should be allowed

Additional Context

This module creates aws_security_group.default (output as security_group_id etc.) but only the control plane is in the created security group. The workers are not.

Recommendation

  1. Remove allowed_cidr_blocks, allowed_security_groups, and aws_security_group.default.
  2. Add control_plane_allowed_security_groups and pass it into EKS cluster VPC config to enable custom security group access to the control plane.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.