Terraform AWS EKS

Terraform module will be created Autoscaling, Workers, EKS, Node Groups.

We eat, drink, sleep and most importantly love DevOps. We are working towards strategies for standardizing architecture while ensuring security for the infrastructure. We are strong believer of the philosophy Bigger problems are always solved by breaking them into smaller manageable problems. Resonating with microservices architecture, it is considered best-practice to run database, cluster, storage in smaller connected yet manageable pieces within the infrastructure.

This module is basically combination of Terraform open source and includes automatation tests and examples. It also helps to create and improve your infrastructure with minimalistic code instead of maintaining the whole infrastructure code yourself.

We have fifty plus terraform modules. A few of them are comepleted and are available for open source usage while a few others are in progress.

Prerequisites

This module has a few dependencies:

Examples

IMPORTANT: Since the master branch used in source varies based on new modifications, we suggest that you use the release versions here.

Sample example

Here is an example of how you can use this module in your inventory structure:

module "eks-cluster" {
     source      = "clouddrove/eks/aws"
     version     = "0.15.0"

     ## Tags
     name        = "eks"
     environment = "test"
     label_order = ["environment", "application", "name"]
     enabled     = true

     ## Network
     vpc_id                              = module.vpc.vpc_id
     eks_subnet_ids                      = module.subnets.public_subnet_id
     worker_subnet_ids                   = module.subnets.private_subnet_id
     allowed_security_groups_cluster     = []
     allowed_security_groups_workers     = []
     additional_security_group_ids       = [module.ssh.security_group_ids]
     endpoint_private_access             = false
     endpoint_public_access              = true
     public_access_cidrs                 = ["0.0.0.0/0"]
     cluster_encryption_config_resources = ["secrets"]
     associate_public_ip_address         = false
     key_name                            = module.keypair.name

     ## volume_size
     volume_size = 20

     ## ondemand
     ondemand_enabled          = true
     ondemand_instance_type    = ["t3.small", "t3.medium", "t3.small"]
     ondemand_max_size         = [1, 0, 0]
     ondemand_min_size         = [1, 0, 0]
     ondemand_desired_capacity = [1, 0, 0]

     ondemand_schedule_enabled            = true
     ondemand_schedule_max_size_scaleup   = [0, 0, 0]
     ondemand_schedule_desired_scaleup    = [0, 0, 0]
     ondemand_schedule_min_size_scaleup   = [0, 0, 0]
     ondemand_schedule_min_size_scaledown = [0, 0, 0]
     ondemand_schedule_max_size_scaledown = [0, 0, 0]
     ondemand_schedule_desired_scale_down = [0, 0, 0]


     ## Spot
     spot_enabled          = true
     spot_instance_type    = ["t3.small", "t3.medium", "t3.small"]
     spot_max_size         = [1, 0, 0]
     spot_min_size         = [1, 0, 0]
     spot_desired_capacity = [1, 0, 0]
     max_price             = ["0.20", "0.20", "0.20"]

     spot_schedule_enabled            = true
     spot_schedule_min_size_scaledown = [0, 0, 0]
     spot_schedule_max_size_scaledown = [0, 0, 0]
     spot_schedule_desired_scale_down = [0, 0, 0]
     spot_schedule_desired_scaleup    = [0, 0, 0]
     spot_schedule_max_size_scaleup   = [0, 0, 0]
     spot_schedule_min_size_scaleup   = [0, 0, 0]

     ## Schedule time
     scheduler_down = "0 19 * * MON-FRI" #diffrent
     scheduler_up   = "0 6 * * MON-FRI"

     #node_group
     node_group_enabled              = true
     node_group_name                 = ["tools", "api"]
     node_group_instance_types       = ["t3.small", "t3.medium"]
     node_group_min_size             = [1, 1]
     node_group_desired_size         = [1, 1]
     node_group_max_size             = [2, 2]
     node_group_volume_size          = 20
     before_cluster_joining_userdata = ""
     node_group_capacity_type        = "ON_DEMAND"
     node_groups = {
      tools = {
        node_group_name           = "autoscale"
        subnet_ids                = module.subnets.private_subnet_id
        ami_type                  = "AL2_x86_64"
        node_group_volume_size    = 100
        node_group_instance_types = ["t3.large"]
        kubernetes_labels         = {}
        kubernetes_version        = "1.20"
        node_group_desired_size   = 1
        node_group_max_size       = 1
        node_group_min_size       = 1
        node_group_capacity_type  = "ON_DEMAND"
        node_group_volume_type    = "gp2"
      }
     }

     ## Cluster
     wait_for_capacity_timeout = "15m"
     apply_config_map_aws_auth = true
     kubernetes_version        = "1.18"
     map_additional_iam_users = [
       {
         userarn  = "arn:aws:iam::924144197303:user/[email protected]"
         username = "[email protected]"
         groups   = ["system:masters"]
       },
       {
         userarn  = "arn:aws:iam::924144197303:user/[email protected]"
         username = "[email protected]"
         groups   = ["system:masters"]
       }

     ]


     ## Health Checks
     cpu_utilization_high_threshold_percent = 80
     cpu_utilization_low_threshold_percent  = 20
     health_check_type                      = "EC2"

     ## EBS Encryption
     ebs_encryption = true

     ## logs
     enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
    }

Inputs

Name	Description	Type	Default	Required
additional_security_group_ids	Additional list of security groups that will be attached to the autoscaling group.	`list(string)`	`[]`	no
allowed_cidr_blocks_cluster	List of CIDR blocks to be allowed to connect to the EKS cluster.	`list(string)`	`[]`	no
allowed_cidr_blocks_workers	List of CIDR blocks to be allowed to connect to the worker nodes.	`list(string)`	`[]`	no
allowed_security_groups_cluster	List of Security Group IDs to be allowed to connect to the EKS cluster.	`list(string)`	`[]`	no
allowed_security_groups_workers	List of Security Group IDs to be allowed to connect to the worker nodes.	`list(string)`	`[]`	no
ami_release_version	AMI version of the EKS Node Group. Defaults to latest version for Kubernetes version	`string`	`""`	no
ami_type	Type of Amazon Machine Image (AMI) associated with the EKS Node Group. Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64`, `AL2_x86_64_GPU`. Terraform will only perform drift detection if a configuration value is provided	`string`	`"AL2_x86_64"`	no
apply_config_map_aws_auth	Whether to generate local files from `kubeconfig` and `config_map_aws_auth` and perform `kubectl apply` to apply the ConfigMap to allow the worker nodes to join the EKS cluster.	`bool`	`true`	no
associate_public_ip_address	Associate a public IP address with the worker nodes in the VPC.	`bool`	`true`	no
attributes	Additional attributes (e.g. `1`).	`list(any)`	`[]`	no
before_cluster_joining_userdata	Additional commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production	`string`	`""`	no
cluster_encryption_config_enabled	Set to `true` to enable Cluster Encryption Configuration	`bool`	`false`	no
cluster_encryption_config_kms_key_deletion_window_in_days	Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction	`number`	`10`	no
cluster_encryption_config_kms_key_enable_key_rotation	Cluster Encryption Config KMS Key Resource argument - enable kms key rotation	`bool`	`true`	no
cluster_encryption_config_kms_key_policy	Cluster Encryption Config KMS Key Resource argument - key policy	`string`	`null`	no
cluster_encryption_config_resources	Cluster Encryption Config Resources to encrypt, e.g. ['secrets']	`list(any)`	[ "secrets" ]	no
cluster_log_retention_period	Number of days to retain cluster logs. Requires `enabled_cluster_log_types` to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html.	`number`	`30`	no
cluster_namespace	Kubernetes namespace for selection	`string`	`""`	no
cpu_utilization_high_threshold_percent	Worker nodes AutoScaling Group CPU utilization high threshold percent.	`number`	`80`	no
cpu_utilization_low_threshold_percent	Worker nodes AutoScaling Group CPU utilization low threshold percent.	`number`	`20`	no
delimiter	Delimiter to be used between `organization`, `environment`, `name` and `attributes`.	`string`	`"-"`	no
disable_api_termination	If `true`, enables EC2 Instance Termination Protection.	`bool`	`false`	no
ebs_encryption	Enables EBS encryption on the volume (Default: false). Cannot be used with snapshot_id.	`bool`	`true`	no
eks_subnet_ids	A list of subnet IDs to launch resources in EKS.	`list(string)`	`[]`	no
enabled	Whether to create the resources. Set to `false` to prevent the module from creating any resources.	`bool`	`true`	no
enabled_cluster_log_types	A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [`api`, `audit`, `authenticator`, `controllerManager`, `scheduler`].	`list(string)`	`[]`	no
endpoint_private_access	Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false.	`bool`	`false`	no
endpoint_public_access	Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true.	`bool`	`true`	no
environment	Environment (e.g. `prod`, `dev`, `staging`).	`string`	`""`	no
fargate_enabled	Whether fargate profile is enabled or not	`bool`	`false`	no
health_check_type	Controls how health checking is done. Valid values are `EC2` or `ELB`.	`string`	`"EC2"`	no
image_id	EC2 image ID to launch. If not provided, the module will lookup the most recent EKS AMI. See https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html for more details on EKS-optimized images.	`string`	`""`	no
instance_interruption_behavior	The behavior when a Spot Instance is interrupted. Can be hibernate, stop, or terminate. (Default: terminate).	`string`	`"terminate"`	no
key_name	SSH key name that should be used for the instance.	`string`	`""`	no
kms_key_arn	The ARN of the KMS Key	`string`	`""`	no
kubernetes_config_map_ignore_role_changes	Set to `true` to ignore IAM role changes in the Kubernetes Auth ConfigMap	`bool`	`true`	no
kubernetes_labels	Key-value mapping of Kubernetes labels. Only labels that are applied with the EKS API are managed by this argument. Other Kubernetes labels applied to the EKS Node Group will not be managed	`map(any)`	`{}`	no
kubernetes_version	Desired Kubernetes master version. If you do not specify a value, the latest available version is used.	`string`	`""`	no
label_order	Label order, e.g. `name`,`application`.	`list(any)`	`[]`	no
local_exec_interpreter	shell to use for local_exec	`list(string)`	[ "/bin/sh", "-c" ]	no
managedby	ManagedBy, eg 'CloudDrove' or 'AnmolNagpal'.	`string`	`"[email protected]"`	no
map_additional_aws_accounts	Additional AWS account numbers to add to `config-map-aws-auth` ConfigMap	`list(string)`	`[]`	no
map_additional_iam_roles	Additional IAM roles to add to `config-map-aws-auth` ConfigMap	list(object({ rolearn = string username = string groups = list(string) }))	`[]`	no
map_additional_iam_users	Additional IAM users to add to `config-map-aws-auth` ConfigMap	list(object({ userarn = string username = string groups = list(string) }))	`[]`	no
max_price	The maximum hourly price you're willing to pay for the Spot Instances.	`list(any)`	`[]`	no
name	Name (e.g. `app` or `cluster`).	`string`	`""`	no
node_group_enabled	Enabling or disabling the node group.	`bool`	`false`	no
node_group_taint_enabled	Whether to `enable` or `disable` node group taints	`bool`	`false`	no
node_groups	Node group configurations	map(object({ node_group_name = string subnet_ids = list(string) ami_type = string node_group_volume_size = number node_group_instance_types = list(string) kubernetes_labels = map(string) kubernetes_version = string node_group_desired_size = number node_group_max_size = number node_group_min_size = number node_group_capacity_type = string node_group_volume_type = string node_group_taint_key = string node_group_taint_value = string node_group_taint_effect = string }))	{ "tools": { "ami_type": "AL2_x86_64", "kubernetes_labels": {}, "kubernetes_version": "1.18", "node_group_capacity_type": "ON_DEMAND", "node_group_desired_size": 1, "node_group_instance_types": [ "t3.small" ], "node_group_max_size": 2, "node_group_min_size": 1, "node_group_name": "tools", "node_group_taint_effect": "", "node_group_taint_key": "", "node_group_taint_value": "", "node_group_volume_size": 20, "node_group_volume_type": "gp2", "subnet_ids": [ "subnet-0314766e56d1eff14", "subnet-051b8c18ce7c0c8ea", "subnet-0a3ba212912cb4263" ] } }	no
node_security_group_ids	Set of EC2 Security Group IDs to allow SSH access (port 22) from on the worker nodes.	`list(string)`	`[]`	no
oidc_provider_enabled	Create an IAM OIDC identity provider for the cluster, then you can create IAM roles to associate with a service account in the cluster, instead of using kiam or kube2iam. For more information, see https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html	`bool`	`false`	no
ondemand_desired_capacity	The desired size of the autoscale group.	`list(any)`	`[]`	no
ondemand_desired_size	Desired number of worker nodes	`number`	`2`	no
ondemand_enabled	Whether to create `aws_autoscaling_policy` and `aws_cloudwatch_metric_alarm` resources to control Auto Scaling.	`bool`	`false`	no
ondemand_instance_type	Instance type to launch.	`list(any)`	`[]`	no
ondemand_max_size	The maximum size of the autoscale group.	`list(any)`	`[]`	no
ondemand_min_size	The minimum size of the autoscale group.	`list(any)`	`[]`	no
ondemand_scale_up_desired	The number of Amazon EC2 instances that should be running in the group.	`number`	`1`	no
ondemand_schedule_desired_scale_down	The number of Amazon EC2 instances that should be running in the group.	`list(any)`	`[]`	no
ondemand_schedule_desired_scaleup	The schedule desired size of the autoscale group.	`list(any)`	`[]`	no
ondemand_schedule_enabled	AutoScaling Schedule resource	`bool`	`false`	no
ondemand_schedule_max_size_scaledown	The maximum size for the Auto Scaling group. Default 0. Set to -1 if you don't want to change the minimum size at the scheduled time.	`list(any)`	`[]`	no
ondemand_schedule_max_size_scaleup	The schedule maximum size of the autoscale group.	`list(any)`	`[]`	no
ondemand_schedule_min_size_scaledown	The minimum size for the Auto Scaling group. Default 0. Set to -1 if you don't want to change the minimum size at the scheduled time.	`list(any)`	`[]`	no
ondemand_schedule_min_size_scaleup	The schedule minimum size of the autoscale group.	`list(any)`	`[]`	no
public_access_cidrs	The list of cidr blocks to access AWS EKS cluster endpoint. Default [`0.0.0.0/0`]	`list(string)`	`[]`	no
repository	Terraform current module repo	`string`	`"https://github.com/clouddrove/terraform-aws-eks"`	no
resources_to_tag	List of auto-launched resource types to tag. Valid types are "instance", "volume", "elastic-gpu", "spot-instances-request".	`list(string)`	`[]`	no
scheduler_down	What is the recurrency for scaling up operations ?	`string`	`"0 19 * * MON-FRI"`	no
scheduler_up	What is the recurrency for scaling down operations ?	`string`	`"0 6 * * MON-FRI"`	no
spot_desired_capacity	The number of Amazon EC2 instances that should be running in the group.	`list(any)`	`[]`	no
spot_enabled	Whether to create the spot instance. Set to `false` to prevent the module from creating any spot instances.	`bool`	`false`	no
spot_instance_type	Sport instance type to launch.	`list(any)`	`[]`	no
spot_max_size	The maximum size of the spot autoscale group.	`list(any)`	`[]`	no
spot_min_size	The minimum size of the spot autoscale group.	`list(any)`	`[]`	no
spot_scale_up_desired	The number of Amazon EC2 instances that should be running in the group.	`list(any)`	`[]`	no
spot_schedule_desired_scale_down	The number of Amazon EC2 instances that should be running in the group.	`list(any)`	`[]`	no
spot_schedule_desired_scaleup	The schedule desired size of the autoscale group.	`list(any)`	`[]`	no
spot_schedule_enabled	AutoScaling Schedule resource for spot	`bool`	`false`	no
spot_schedule_max_size_scaledown	The maximum size for the Auto Scaling group of spot instances. Default 0. Set to -1 if you don't want to change the minimum size at the scheduled time.	`list(any)`	`[]`	no
spot_schedule_max_size_scaleup	The schedule maximum size of the autoscale group.	`list(any)`	`[]`	no
spot_schedule_min_size_scaledown	The minimum size for the Auto Scaling group of spot instances. Default 0. Set to -1 if you don't want to change the minimum size at the scheduled time.	`list(any)`	`[]`	no
spot_schedule_min_size_scaleup	The schedule minimum size of the autoscale group.	`list(any)`	`[]`	no
tags	Additional tags (e.g. map(`BusinessUnit`,`XYZ`).	`map(any)`	`{}`	no
use_existing_security_group	If set to `true`, will use variable `workers_security_group_id` to run EKS workers using an existing security group that was created outside of this module, workaround for errors like `count cannot be computed`.	`bool`	`false`	no
volume_size	The size of ebs volume.	`number`	`20`	no
volume_type	The type of volume. Can be `standard`, `gp2`, or `io1`. (Default: `standard`).	`string`	`"standard"`	no
vpc_id	VPC ID for the EKS cluster.	`string`	`""`	no
wait_for_capacity_timeout	A maximum duration that Terraform should wait for ASG instances to be healthy before timing out. Setting this to '0' causes Terraform to skip all Capacity Waiting behavior.	`string`	`"15m"`	no
wait_for_cluster_command	`local-exec` command to execute to determine if the EKS cluster is healthy. Cluster endpoint are available as environment variable `ENDPOINT`	`string`	`"curl --silent --fail --retry 60 --retry-delay 5 --retry-connrefused --insecure --output /dev/null $ENDPOINT/healthz"`	no
worker_subnet_ids	A list of subnet IDs to launch resources in workers.	`list(string)`	`[]`	no
workers_security_group_id	The name of the existing security group that will be used in autoscaling group for EKS workers. If empty, a new security group will be created.	`string`	`""`	no

Outputs

Name	Description
eks_cluster_arn	The Amazon Resource Name (ARN) of the cluster.
eks_cluster_certificate_authority_data	The base64 encoded certificate data required to communicate with the cluster.
eks_cluster_endpoint	The endpoint for the Kubernetes API server.
eks_cluster_id	The name of the cluster.
eks_cluster_security_group_arn	ARN of the EKS cluster Security Group.
eks_cluster_security_group_id	ID of the EKS cluster Security Group.
eks_cluster_security_group_name	Name of the EKS cluster Security Group.
eks_cluster_version	The Kubernetes server version of the cluster.
eks_fargate_arn	Amazon Resource Name (ARN) of the EKS Fargate Profile.
eks_fargate_id	EKS Cluster name and EKS Fargate Profile name separated by a colon (:).
eks_node_group_arn	Amazon Resource Name (ARN) of the EKS Node Group
eks_node_group_id	EKS Cluster name and EKS Node Group name separated by a colon
eks_node_group_resources	List of objects containing information about underlying resources of the EKS Node Group
eks_node_group_status	Status of the EKS Node Group
iam_role_arn	ARN of the worker nodes IAM role.
kubernetes_config_map_id	ID of `aws-auth` Kubernetes ConfigMap
tags	A mapping of tags to assign to the resource.
workers_autoscaling_group_arn	ARN of the AutoScaling Group.
workers_autoscaling_group_default_cooldown	Time between a scaling activity and the succeeding scaling activity.
workers_autoscaling_group_desired_capacity	The number of Amazon EC2 instances that should be running in the group.
workers_autoscaling_group_health_check_grace_period	Time after instance comes into service before checking health.
workers_autoscaling_group_health_check_type	`EC2` or `ELB`. Controls how health checking is done.
workers_autoscaling_group_id	The AutoScaling Group ID.
workers_autoscaling_group_max_size	The maximum size of the AutoScaling Group.
workers_autoscaling_group_min_size	The minimum size of the AutoScaling Group.
workers_autoscaling_group_name	The AutoScaling Group name.
workers_launch_template_arn	ARN of the launch template.
workers_launch_template_id	ID of the launch template.
workers_security_group_arn	ARN of the worker nodes Security Group.
workers_security_group_id	ID of the worker nodes Security Group.
workers_security_group_name	Name of the worker nodes Security Group.

Testing

In this module testing is performed with terratest and it creates a small piece of infrastructure, matches the output like ARN, ID and Tags name etc and destroy infrastructure in your AWS account. This testing is written in GO, so you need a GO environment in your system.

You need to run the following command in the testing folder:

  go test -run Test

Feedback

If you come accross a bug or have any feedback, please log it in our issue tracker, or feel free to drop us an email at [email protected].

If you have found it worth your time, go ahead and give us a ★ on our GitHub!

About us

At CloudDrove, we offer expert guidance, implementation support and services to help organisations accelerate their journey to the cloud. Our services include docker and container orchestration, cloud migration and adoption, infrastructure automation, application modernisation and remediation, and performance engineering.

We are The Cloud Experts!

We ❤️ Open Source and you can check out our other modules to get help with your new Cloud ideas.

sohanyadav / terraform-aws-eks Goto Github PK