aws-ia / terraform-aws-eks-blueprints Goto Github PK

View Code? Open in Web Editor NEW

2.5K 61.0 1.4K 31.14 MB

Configure and deploy complete EKS clusters.

Home Page: https://aws-ia.github.io/terraform-aws-eks-blueprints/

License: Apache License 2.0

Shell 3.43% HCL 95.21% Python 1.36%

eks-clusters kubernetes fargate eks eks-fargate eks-addons bottlerocket terraform aws-eks-cluster

terraform-aws-eks-blueprints's Introduction

Amazon EKS Blueprints for Terraform

Welcome to Amazon EKS Blueprints for Terraform!

This project contains a collection of Amazon EKS cluster patterns implemented in Terraform that demonstrate how fast and easy it is for customers to adopt Amazon EKS. The patterns can be used by AWS customers, partners, and internal AWS teams to configure and manage complete EKS clusters that are fully bootstrapped with the operational software that is needed to deploy and operate workloads.

Motivation

Kubernetes is a powerful and extensible container orchestration technology that allows you to deploy and manage containerized applications at scale. The extensible nature of Kubernetes also allows you to use a wide range of popular open-source tools in Kubernetes clusters. However, With the wide array of tooling and design choices available, configuring an EKS cluster that meets your organization’s specific needs can take a significant amount of time. It involves integrating a wide range of open-source tools and AWS services as well as expertise in AWS and Kubernetes.

AWS customers have asked for patterns that demonstrate how to integrate the landscape of Kubernetes tools and make it easy for them to provision complete, opinionated EKS clusters that meet specific application requirements. Customers can utilize EKS Blueprints to configure and deploy purpose built EKS clusters, and start onboarding workloads in days, rather than months.

Consumption

EKS Blueprints for Terraform has been designed to be consumed in the following manners:

Reference: Users can refer to the patterns and snippets provided to help guide them to their desired solution. Users will typically view how the pattern or snippet is configured to achieve the desired end result and then replicate that in their environment.
Copy & Paste: Users can copy and paste the patterns and snippets into their own environment, using EKS Blueprints as the starting point for their implementation. Users can then adapt the initial pattern to customize it to their specific needs.

EKS Blueprints for Terraform are not intended to be consumed as-is directly from this project. In "Terraform speak" - the patterns and snippets provided in this repository are not designed to be consumed as a Terraform module. Therefore, the patterns provided only contain variables when certain information is required to deploy the pattern (i.e. - a Route53 hosted zone ID, or ACM certificate ARN) and generally use local variables. If you wish to deploy the patterns into a different region or with other changes, it is recommended that you make those modifications locally before applying the pattern. EKS Blueprints for Terraform will not expose variables and outputs in the same manner that Terraform modules follow in order to avoid confusion around the consumption model.

However, we do have a number of Terraform modules that were created to support EKS Blueprints in addition to the community hosted modules. Please see the respective projects for more details on the modules constructed to support EKS Blueprints for Terraform; those projects are listed below.

terraform-aws-eks-blueprint-addon - (Note the singular form) Terraform module which can provision an addon using the Terraform helm_release resource in addition to an IAM role for service account (IRSA).
terraform-aws-eks-blueprint-addons - (Note the plural form) Terraform module which can provision multiple addons; both EKS addons using the aws_eks_addon resource as well as Helm chart based addons using the terraform-aws-eks-blueprint-addon module.
terraform-aws-eks-blueprints-teams - Terraform module that creates Kubernetes multi-tenancy resources and configurations, allowing both administrators and application developers to access only the resources which they are responsible for.

Related Projects

In addition to the supporting EKS Blueprints Terraform modules listed above, there are a number of related projects that users should be aware of:

GitOps
- terraform-aws-eks-ack-addons - Terraform module to deploy ACK controllers onto EKS clusters
- crossplane-on-eks - Crossplane Blueprints is an open source repo to bootstrap Amazon EKS clusters and provision AWS resources using a library of Crossplane Compositions (XRs) with Composite Resource Definitions (XRDs).
Data on EKS
- data-on-eks - A collection of blueprints intended for data workloads on Amazon EKS.
- terraform-aws-eks-data-addons - Terraform module to deploy multiple addons that are specific to data workloads on EKS clusters.
Observability Accelerator
- terraform-aws-observability-accelerator - A set of opinionated modules to help you set up observability for your AWS environments with AWS-managed observability services such as Amazon Managed Service for Prometheus, Amazon Managed Grafana, AWS Distro for OpenTelemetry (ADOT) and Amazon CloudWatch
Karpenter Blueprints
- karpenter-blueprints - includes a list of common workload scenarios, some of them go in depth with the explanation of why configuring Karpenter and Kubernetes objects in such a way is important.

Terraform Caveats

EKS Blueprints for Terraform does not intend to teach users the recommended practices for Terraform nor does it offer guidance on how users should structure their Terraform projects. The patterns provided are intended to show users how they can achieve a defined architecture or configuration in a way that they can quickly and easily get up and running to start interacting with that pattern. Therefore, there are a few caveats users should be aware of when using EKS Blueprints for Terraform:

We recognize that most users will already have an existing VPC in a separate Terraform workspace. However, the patterns provided come complete with a VPC to ensure a stable, deployable example that has been tested and validated.
Hashicorp does not recommend providing computed values in provider blocks , which means that the cluster configuration should be defined in a workspace separate from the resources deployed onto the cluster (i.e. - addons). However, to simplify the pattern experience, we have defined everything in one workspace and provided instructions to provision the patterns using a targeted apply approach. Users are encouraged to investigate a Terraform project structure that suites their needs; EKS Blueprints for Terraform does not have an opinion in this matter and will defer to Hashicorp's guidance.
Patterns are not intended to be consumed in-place in the same manner that one would consume a module. Therefore, we do not provide variables and outputs to expose various levels of configuration for the examples. Users can modify the pattern locally after cloning to suite their requirements.
Please see the FAQ section on authenticating Kubernetes based providers (kubernetes, helm, kubectl) to Amazon EKS clusters regarding the use of static tokens versus dynamic tokens using the awscli.

Support & Feedback

EKS Blueprints for Terraform is maintained by AWS Solution Architects. It is not part of an AWS service and support is provided as a best-effort by the EKS Blueprints community. To provide feedback, please use the issues templates provided. If you are interested in contributing to EKS Blueprints, see the Contribution guide.

Security

See CONTRIBUTING for more information.

License

Apache-2.0 Licensed. See LICENSE.

terraform-aws-eks-blueprints's People

Contributors

Stargazers

Watchers

Forkers

kenerwin88 rsgaikwad stevekim-git sachin-net farislinux joypersonal ramnamy qpc-database xntric78 karimarous sre-redrocket mbeacom chtompki hcampbel ulaganathannamachivayam vara-bonthu davar-playgrounds jasonumiker lefthander kpena027 jaymccon raghavendrakambhampati ale-conekta cloudologia fwan2000 aymensegni polliard-jmfe leeaandrob emanuelr93 devopsguru2020 obiomap rothgar marianod92 kcoleman731 sadique26 awsitcloudpro shyam-amazon tenhishadow sungmen999 nilesh7345kaka jessesanford narendra311777 santiagovj22 eduardbulai soostdijck raghavmaxxsure milosh3411 aws-devops-projects homie-du schwichti dkaliberda jbadru73 jasonskidmore-terraform prakashja akanam mitchac cga-work rahimigit pbg2302 sirlegendary deepesh-jais tranvanloc412 klapcsik gbulanov wsarwari-amzn ivallhon dhillonfarms normalfaults bigfootbiomedical luuthanhminh jeraldinek felipemartinez10 sagrdo ppscon devopsuser479 eugenestarchenko mhemanth55 ricardosouzamorais onlydole julianoslackbr sam-gordon-f torggg zen-python cvlc zawwinhtun78 prakashkancham nuatu devops-corner colbysadams awsvikram jomcy-amzn rex4god thomas-mundt dtfh81 jdwilly2001 puru-aws maheshr-amzn tetratelabs gkudikala mmarinm

terraform-aws-eks-blueprints's Issues

Feature: Multi-tenant (Ingress per tenant)

Since the awesome accelerator already supports multi-tenant scenarios, it would be great to also allow a tenant to have a dedicated ingress, like NGNIX.

The controller would include the corresponding cert, DNS etc.

Example
Finance namespaces can have one Ingress Controller and Marketing division can have their own Ingress Controller.

This would also translate into cost savings on our implementations.

Fluentbit is unable to log to cloudwatch

I am trying to use aws-for-fluentbit module to forward container logs to cloud watch. I am getting the following error.

[2021/09/08 02:51:19] [ warn] [engine] failed to flush chunk '1-1631069464.479851328.flb', retry in 99 seconds: task_id=55, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:19] [ warn] [engine] failed to flush chunk '1-1631069474.477401863.flb', retry in 8 seconds: task_id=57, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:24] [ warn] [engine] failed to flush chunk '1-1631069479.476392887.flb', retry in 11 seconds: task_id=58, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:27] [ warn] [engine] failed to flush chunk '1-1631069384.474304953.flb', retry in 473 seconds: task_id=39, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:29] [ warn] [engine] failed to flush chunk '1-1631069484.779773075.flb', retry in 6 seconds: task_id=59, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:34] [ warn] [engine] failed to flush chunk '1-1631069489.478379979.flb', retry in 9 seconds: task_id=60, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:35] [ warn] [engine] failed to flush chunk '1-1631069479.476392887.flb', retry in 95 seconds: task_id=58, input=tail.0 > output=es.1 (out_id=1)
[2021/09/08 02:51:37] [error] [upstream] connection #49 to 127.0.0.1:443 timed out after 10 seconds

[QUESTION] How to pass additional security groups to managed nodes?

Please describe your quesiton here

I want to set additional inbound security rules for my managed nodes.
My first approach was to pass additional security groups via the variable worker_additional_security_group_ids.
Second approach was to get a reference to the auto-generated security group via module.eks-ssp.managed_node_groups[0]["default"].managed_nodegroup_sec_group_id[0] and add the security rule to it.
I also tried to set create_launch_template = true and create_worker_security_group = true .
It seems to me that the eks-ssp module and the internal eks module both are creating security groups, but that one that is returned as output is not the one being actually used.

What is the right way to pass additional security groups to managed nodes?

Yes, I have checked the repo for existing issues before raising this question

"kubernetes_config_map" "aws_auth" + "kubernetes_config_map" "amazon_vpc_cni" errors - need help please

When I follow your example code I got this error:

module "eks-ssp" {
    source = "github.com/aws-samples/aws-eks-accelerator-for-terraform"

    # EKS CLUSTER
    vpc_id             = module.aws_vpc.vpc_id
    private_subnet_ids = module.aws_vpc.private_subnets

  # EKS CONTROL PLANE VARIABLES
    create_eks         = true
    kubernetes_version = local.kubernetes_version

  # EKS MANAGED NODE GROUPS
  managed_node_groups = {
    mg_4 = {
      node_group_name = "managed-ondemand"
      instance_types  = ["m4.large"]
      subnet_ids      = module.aws_vpc.private_subnets
    }
  }
}

# Deploy Kubernetes Add-ons with sub module
module "eks-ssp-kubernetes-addons" {
    source = "github.com/aws-samples/aws-eks-accelerator-for-terraform//modules/kubernetes-addons"

    eks_cluster_id                        = module.eks-ssp.eks_cluster_id
    eks_oidc_issuer_url                   = module.eks-ssp.eks_oidc_issuer_url
    eks_oidc_provider_arn                 = module.eks-ssp.eks_oidc_provider_arn

    # EKS Addons
    enable_amazon_eks_vpc_cni             = true
    enable_amazon_eks_coredns             = true
    enable_amazon_eks_kube_proxy          = true
    enable_amazon_eks_aws_ebs_csi_driver  = true

    #K8s Add-ons
    enable_aws_load_balancer_controller   = true
    enable_metrics_server                 = true
    enable_cluster_autoscaler             = true
    enable_aws_for_fluentbit              = true
    enable_argocd                         = true
    enable_ingress_nginx                  = true

    depends_on = [module.eks-ssp.managed_node_groups]
}

And this is the error:

module.eks-ssp.module.aws_eks_teams.data.aws_eks_cluster.eks_cluster: Reading...
module.eks-ssp.module.aws_eks_teams.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.eks-ssp.module.aws_eks_teams.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks-ssp.module.aws_eks_teams.data.aws_eks_cluster.eks_cluster: Read complete after 0s [id=aws-preprod-dev-eks]
module.eks-ssp.module.aws_eks_teams.aws_iam_policy.platform_team_eks_access: Creating...
module.eks-ssp.module.aws_eks_teams.data.aws_caller_identity.current: Read complete after 0s [id=567917856707]
module.eks-ssp.kubernetes_config_map.aws_auth[0]: Creating...
module.eks-ssp.module.aws_eks_teams.aws_iam_policy.platform_team_eks_access: Creation complete after 1s [id=arn:aws:iam::567917856707:policy/aws-preprod-dev-PlatformTeamEKSAccess]
╷
│ Error: Post "https://AEFF2E34F967C93890E9AF5728D4F61D.gr7.eu-central-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps": dial tcp: lookup AEFF2E34F967C93890E9AF5728D4F61D.gr7.eu-central-1.eks.amazonaws.com on 127.0.0.53:53: no such host
│ 
│   with module.eks-ssp.kubernetes_config_map.aws_auth[0],
│   on .terraform/modules/eks-ssp/aws-auth-configmap.tf line 19, in resource "kubernetes_config_map" "aws_auth":
│   19: resource "kubernetes_config_map" "aws_auth" {
│ 
╵
╷
│ Error: Post "https://AEFF2E34F967C93890E9AF5728D4F61D.gr7.eu-central-1.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps": dial tcp: lookup AEFF2E34F967C93890E9AF5728D4F61D.gr7.eu-central-1.eks.amazonaws.com on 127.0.0.53:53: no such host
│ 
│   with module.eks-ssp.kubernetes_config_map.amazon_vpc_cni,
│   on .terraform/modules/eks-ssp/main.tf line 112, in resource "kubernetes_config_map" "amazon_vpc_cni":
│  112: resource "kubernetes_config_map" "amazon_vpc_cni" {

I can fix it, when I run after the deployment:

aws eks --region eu-west-1 update-kubeconfig --name <cluster-name>
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc

But I think it should not be like that. How can I get a clean run without the fix?

KMS Key

Hi Team,

Is there an option to change/update deletion_window_in_days for KMS key.

[Bug]: Namespace is hard coded to argocd in all argocd applications

Welcome to Amazon SSP EKS Accelerator!

Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version

v3.2.0

What is your environment, configuration and the example used?

Terraform version v1.1.4
on darwin_amd64
Used the argocd example with EKS

What did you do and What did you see instead?

I tried installing the Argo CD apps and ran into the namespace parameter not being honored

This is because the namespace is hard coded to "argocd" in the TF.
https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/1ca8d8de850674caa05162b3cb69153fad5f0815/modules/kubernetes-addons/argocd/main.tf#L88 and in https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/9c38fd09b64b313796dc27b621b7a842d2fdba5b/modules/kubernetes-addons/argocd/argocd-application/templates/application.yaml#L18

Additional Information

No response

eks-cluster-with-new-vpc doesn't work

As title says: You cannot actually use this deployment, to the best of my ability to discern. That specific file assumes that you already have an s3 backend configured, and the readme for it references a totally different repository (ie, is completely wrong and cannot be followed).

Feature request: Pass workers_additional_policies to nodes

Hi,

I want to pass additional policies to EKS nodes / workers. The EKS accelerator uses the EKS module under the hood which has the workers_additional_policies argument for that. However, the argument is not exposed by the accelerator.

Is there a workaround for this? I want to use the accelerator as a module. One solution might be to get the IAM role for the EKS nodes from the accelerator and attach a policy via the aws_iam_role_policy_attachment resouce from the aws provider. However, I do not see how I can get the IAM role from the accelerator.

One question out of curiosity

I have looked in to this project and noticed a configuration that I want to use aswell, but couldnt get it to work.

For some reason you seem to be able to initialise a provider with module output:

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

While the Terraform Documentation states the following:

You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

I dont understand how that is possible? My use case is that I deploy a Vault Instance and then in the next module, configure that Vault with the IP and root token returned be the module before. But during planphase terraform fails because it cannot access that Vault...since it doesnt exist. Am I missing something?

Appreciate any help

Documentation: Add example for ArgoCD way of managing add-ons

As per the docs there are 2 ways to manage addons

Via Terraform by leveraging the Terraform Helm provider.
Via GitOps with ArgoCD.

The examples/ directory has examples that make use of Terraform Helm provider. It would be great to have a working example with ArgoCD method as well.

Example in eks-cluster-with-new-vpc doesn't work

It looks like the example in the deploy/eks-cluster-with-new-vpc doesn't work:

git clone https://github.com/aws-samples/aws-eks-accelerator-for-terraform
cd aws-eks-accelerator-for-terraform/deploy/eks-cluster-with-new-vpc
terraform init
terraform apply
...
╷
│ Error: Invalid function argument
│
│   on ../../kubernetes-addons/keda/locals.tf line 75, in locals:
│   75:   default_keda_helm_values = [templatefile("${path.module}/keda-values.yaml", {
│   76:     keda-sa-name = local.keda_service_account_name
│   77:   })]
│     ├────────────────
│     │ path.module is "../../kubernetes-addons/keda"
│
│ Invalid value for "path" parameter: no file exists at ../../kubernetes-addons/keda/keda-values.yaml; this function works only with files that are distributed as part of the
│ configuration source code, so if this file will be created by a resource in this configuration you must instead obtain this result from an attribute of that resource.
╵
╷
│ Error: Reference to undeclared local value
│
│   on ../../kubernetes-addons/prometheus/locals.tf line 71, in locals:
│   71:     ampWorkspaceUrl    = local.amp_workspace_url
│
│ A local value with the name "amp_workspace_url" has not been declared.

Thanks for looking at it...

Hardcoded awsRegion in cluster_autoscaler helm chart

Hi, I was wondering why the awsRegion of the cluster_autoscaler helm chart is hardcoded to eu-west-1. Shouldn't that be provider.aws.region?

set {
    name  = "awsRegion"
-    value = "eu-west-1"
+   value = provider.aws.region
  }

https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/66591a041bd8291c69dc82162b510ece50db9637/helm/cluster_autoscaler/main.tf#L47

Support for Agones Kubernetes controller for gaming servers

Agones is an Open source Kubernetes Controller with custom resource definitions and is used to create, run, manage and scale dedicated game server processes within Kubernetes clusters using standard Kubernetes tooling and APIs.
This model also allows any matchmaker to interact directly with Agones via the Kubernetes API to provision a dedicated game server.

Add Agones Helm Chart to this module allow the users to deploy to EKS Cluster

https://agones.dev/site/

AWS Karpenter support

Hi,

Are there any plans to include AWS Karpenter as an alternative to the cluster autoscaler?

Terraform plan

Hi Team,

I am trying to create a cluster with managed node group with a main.tf, variable.tf and tfvars file but the terraform plan is taking more than 15 minutes to execute. Please let me know if there something which I am doing wrong/different.

I am using https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/main/examples/1-eks-cluster-with-new vpc/main.tf example.

Thank you

issue with straight steps

Error: Error fetching Availability Zones: UnauthorizedOperation: You are not authorized to perform this operation.
│ status code: 403, request id: 0a328641-b053-46e7-acdb-53e80a1a3740
│
│ with data.aws_availability_zones.available,
│ on main.tf line 35, in data "aws_availability_zones" "available":
│ 35: data "aws_availability_zones" "available" {}

[FEATURE] Example showing disk + Amazon EKS encryption

Describe the solution you'd like

Would be possible to enhance (or add) one example to show disk + EKS encryption using KMS?

It should be quite easy for Amazon EKS encryption using cluster_kms_key_arn variable, but example for worker node disk encryption is probably not trivial, because it may require creation of launch template.

Anyway - any example for Amazon EKS + worker node disk encryption would be welcome...

CSI EFS Driver Add-on

Hello,

This is a feature request. I tried this repo and note that some AWS components are still missing.
Would it be possible to add CSI EBS and CSI EFS drivers ?

Kind regards
Romain

[Feature] ArgoCD Kubernetes Addon

Add ArgoCD Kubernetes Addon using Helm Chart

[QUESTION] EKS kubernetes-addons failed with "Kubernetes cluster unreachable"

Hello,

First I wanted to thank you for the great framework!

I am trying to create an EKS cluster and add several of the add-ons. I am executing this plan using Terraform 1.1.3 and running it on Terraform Cloud. Everything seems to work fine until it gets to the add-ons where it starts to fail. The errors I got are:

{"@level":"warn","@message":"Warning: Helm release \"cluster-autoscaler\" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.778747Z","diagnostic":{"severity":"warning","summary":"Helm release \"cluster-autoscaler\" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.cluster_autoscaler[0].helm_release.cluster_autoscaler[0]","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/cluster-autoscaler/main.tf","start":{"line":1,"column":46,"byte":45},"end":{"line":1,"column":47,"byte":46}},"snippet":{"context":"resource \"helm_release\" \"cluster_autoscaler\"","code":"resource \"helm_release\" \"cluster_autoscaler\" {","start_line":1,"highlight_start_offset":45,"highlight_end_offset":46,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: timed out waiting for the condition","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.780436Z","diagnostic":{"severity":"error","summary":"timed out waiting for the condition","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.cluster_autoscaler[0].helm_release.cluster_autoscaler[0]","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/cluster-autoscaler/main.tf","start":{"line":1,"column":46,"byte":45},"end":{"line":1,"column":47,"byte":46}},"snippet":{"context":"resource \"helm_release\" \"cluster_autoscaler\"","code":"resource \"helm_release\" \"cluster_autoscaler\" {","start_line":1,"highlight_start_offset":45,"highlight_end_offset":46,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.782146Z","diagnostic":{"severity":"error","summary":"Kubernetes cluster unreachable: the server has asked for the client to provide credentials","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.metrics_server[0].helm_release.metrics_server[0]","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/metrics-server/main.tf","start":{"line":19,"column":42,"byte":1049},"end":{"line":19,"column":43,"byte":1050}},"snippet":{"context":"resource \"helm_release\" \"metrics_server\"","code":"resource \"helm_release\" \"metrics_server\" {","start_line":19,"highlight_start_offset":41,"highlight_end_offset":42,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: unexpected EKS Add-On (aws-development-zone-eks:aws-ebs-csi-driver) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)\n[WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.783462Z","diagnostic":{"severity":"error","summary":"unexpected EKS Add-On (aws-development-zone-eks:aws-ebs-csi-driver) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)\n[WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.aws_ebs_csi_driver[0].aws_eks_addon.aws_ebs_csi_driver","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/aws-ebs-csi-driver/main.tf","start":{"line":19,"column":47,"byte":1054},"end":{"line":19,"column":48,"byte":1055}},"snippet":{"context":"resource \"aws_eks_addon\" \"aws_ebs_csi_driver\"","code":"resource \"aws_eks_addon\" \"aws_ebs_csi_driver\" {","start_line":19,"highlight_start_offset":46,"highlight_end_offset":47,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.789925Z","diagnostic":{"severity":"error","summary":"Kubernetes cluster unreachable: the server has asked for the client to provide credentials","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.ingress_nginx[0].helm_release.nginx[0]","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/ingress-nginx/main.tf","start":{"line":19,"column":33,"byte":1040},"end":{"line":19,"column":34,"byte":1041}},"snippet":{"context":"resource \"helm_release\" \"nginx\"","code":"resource \"helm_release\" \"nginx\" {","start_line":19,"highlight_start_offset":32,"highlight_end_offset":33,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.800154Z","diagnostic":{"severity":"error","summary":"Kubernetes cluster unreachable: the server has asked for the client to provide credentials","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.aws_load_balancer_controller[0].helm_release.lb_ingress[0]","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/aws-load-balancer-controller/main.tf","start":{"line":19,"column":38,"byte":1045},"end":{"line":19,"column":39,"byte":1046}},"snippet":{"context":"resource \"helm_release\" \"lb_ingress\"","code":"resource \"helm_release\" \"lb_ingress\" {","start_line":19,"highlight_start_offset":37,"highlight_end_offset":38,"values":[]}},"type":"diagnostic"}
{"@level":"error","@message":"Error: unexpected EKS Add-On (aws-development-zone-eks:coredns) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s)\n[WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration","@module":"terraform.ui","@timestamp":"2022-01-19T23:05:22.801372Z","diagnostic":{"severity":"error","summary":"unexpected EKS Add-On (aws-development-zone-eks:coredns) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s)\n[WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration","detail":"","address":"module.cluster.module.eks-ssp-kubernetes-addons.module.aws_coredns[0].aws_eks_addon.coredns","range":{"filename":".terraform/modules/cluster.eks-ssp-kubernetes-addons/modules/kubernetes-addons/aws-coredns/main.tf","start":{"line":19,"column":36,"byte":1043},"end":{"line":19,"column":37,"byte":1044}},"snippet":{"context":"resource \"aws_eks_addon\" \"coredns\"","code":"resource \"aws_eks_addon\" \"coredns\" {","start_line":19,"highlight_start_offset":35,"highlight_end_offset":36,"values":[]}},"type":"diagnostic"}

I'm new to Terraform, but it seems to me that the apply times out before completion and can't authenticate with the cluster. I appreciate any help you can give me.

Additional context

The main.tf I am using is the following:

variable "region" {
  type        = string
  description = "AWS region"
  default     = "us-east-2"
}


variable "instance_type" {
  type        = string
  description = "Type of EC2 instance to provision"
  default     = "t2.micro"
}

variable "tenant" {
  type        = string
  description = "AWS account name or unique id for tenant"
  default     = "aws"
}

variable "environment" {
  type        = string
  description = "Environment area"
  default     = "development"
}


terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.66.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.6.1"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.4.1"
    }
  }
  backend "remote" {
    organization = "Valienta"

    workspaces {
      name = "terraform-infrastructure"
    }
  }

  required_version = ">= 0.13.0"
}


data "aws_availability_zones" "available" {}

data "aws_eks_cluster" "cluster" {
  name = module.eks-ssp.eks_cluster_id
}

data "aws_eks_cluster_auth" "cluster_auth" {
  name = module.eks-ssp.eks_cluster_id
}

provider "aws" {
  region = var.region
}

provider "kubernetes" {
  experiments {
    manifest_resource = true
  }
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster_auth.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}


locals {
  tenant             = var.tenant      # AWS account name or unique id for tenant
  environment        = var.environment # Environment area eg., preprod or prod
  zone               = "zone"          # Environment with in one sub_tenant or business unit
  kubernetes_version = "1.21"

  vpc_cidr       = "10.0.0.0/16"
  vpc_name       = join("-", [local.tenant, local.environment, local.zone, "vpc"])
  eks_cluster_id = join("-", [local.tenant, local.environment, local.zone, "eks"])

  terraform_version = "Terraform v1.0.1"
}

module "aws_vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "v3.2.0"

  name = local.vpc_name
  cidr = local.vpc_cidr
  azs  = data.aws_availability_zones.available.names

  public_subnets  = [for k, v in data.aws_availability_zones.available.names : cidrsubnet(local.vpc_cidr, 8, k)]
  private_subnets = [for k, v in data.aws_availability_zones.available.names : cidrsubnet(local.vpc_cidr, 8, k + 10)]

  enable_nat_gateway   = true
  create_igw           = true
  enable_dns_hostnames = true
  single_nat_gateway   = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.eks_cluster_id}" = "shared"
    "kubernetes.io/role/elb"                        = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.eks_cluster_id}" = "shared"
    "kubernetes.io/role/internal-elb"               = "1"
  }
}

#---------------------------------------------------------------
# Consume aws-eks-accelerator-for-terraform module
# https://github.com/aws-samples/aws-eks-accelerator-for-terraform
#---------------------------------------------------------------
module "eks-ssp" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform"

  tenant            = local.tenant
  environment       = local.environment
  zone              = local.zone
  terraform_version = local.terraform_version

  # EKS Cluster VPC and Subnet mandatory config
  vpc_id             = module.aws_vpc.vpc_id
  private_subnet_ids = module.aws_vpc.private_subnets

  # EKS CONTROL PLANE VARIABLES
  create_eks         = true
  kubernetes_version = local.kubernetes_version

  # EKS MANAGED NODE GROUPS
  managed_node_groups = {
    mg_nodes_1 = {
      node_group_name = "managed-ondemand"
      instance_types  = [var.instance_type]
      subnet_ids      = module.aws_vpc.private_subnets
      desired_size    = 1
      max_size        = 3
      min_size        = 1
      max_unavailable = 1
    }
  }
}

# Deploy Kubernetes Add-ons with sub module
module "eks-ssp-kubernetes-addons" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform//modules/kubernetes-addons"

  eks_cluster_id = module.eks-ssp.eks_cluster_id

  # EKS Addons
  enable_amazon_eks_vpc_cni            = true
  enable_amazon_eks_coredns            = true
  enable_amazon_eks_kube_proxy         = true
  enable_amazon_eks_aws_ebs_csi_driver = true

  #K8s Add-ons
  enable_aws_load_balancer_controller = true
  enable_metrics_server               = true
  enable_cluster_autoscaler           = true
  # enable_aws_for_fluentbit            = true
  # enable_argocd                       = true
  enable_ingress_nginx = true

  depends_on = [module.eks-ssp.managed_node_groups]
}

Yes, I have checked the repo for existing issues before raising this question

Thank you!

[Bug]: It is not possible to define Node Group in tfvars

Welcome to Amazon SSP EKS Accelerator!

Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version

v3.2.2

What is your environment, configuration and the example used?

I would like to use the aws-eks-accelerator-for-terraform module with managed_node_groups variable defined "externally" like:

# variables.tfvars
managed_node_groups = {
  ng01 = {
    node_group_name = "mgmt01-ng01"

    instance_types = ["t2.small"]
    # subnet_ids = module.aws_vpc.private_subnets   <- This can not be part of "TF variable"
  }
  ng02 = {
    node_group_name = "mgmt01-ng01"

    instance_types = ["t2.small"]
    # subnet_ids = module.aws_vpc.private_subnets   <- This can not be part of "TF variable"
  }
}

# main.tf
# terraform apply -var-file="variables.tfvars"

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.74.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

variable "managed_node_groups" {
  description = "Map of maps of eks_node_groups to create"
  type        = any
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.11.3"

  name = "test-eks"
  cidr = "10.0.0.0/16"

  azs             = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway      = true
  single_nat_gateway      = true
  enable_dns_hostnames    = true
  map_public_ip_on_launch = true

  public_subnet_tags = {
    "kubernetes.io/cluster/test-eks" = "shared"
    "kubernetes.io/role/elb"         = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/test-eks"  = "shared"
    "kubernetes.io/role/internal-elb" = 1
  }
}

module "aws-eks-accelerator-for-terraform" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform?ref=main"

  tenant      = "mytenant"
  environment = "myenv"
  zone        = "eu-central-1"

  # EKS Cluster VPC and Subnet mandatory config
  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets

  # EKS CONTROL PLANE VARIABLES
  create_eks         = true
  kubernetes_version = "1.21"
  cluster_name       = "test123"

  cluster_endpoint_private_access = false
  cluster_endpoint_public_access  = false

  # EKS MANAGED NODE GROUPS
  managed_node_groups = var.managed_node_groups
}

But I'm getting the error:

│ Error: Not enough list items
│
│   with module.aws-eks-accelerator-for-terraform.module.aws_eks_managed_node_groups["ng02"].aws_eks_node_group.managed_ng,
│   on .terraform/modules/aws-eks-accelerator-for-terraform/modules/aws-eks-managed-node-groups/main.tf line 5, in resource "aws_eks_node_group" "managed_ng":
│    5:   subnet_ids             = local.managed_node_group["subnet_ids"]
│
│ Attribute requires 1 item minimum, but config has only 0 declared.

The aws-eks-managed-node-groups module expects subnet_ids variable which must be part of the "node_group" definition like it is here: https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/ee7468c52eefa52b5ecfae7a2ffc58b34156550e/examples/eks-cluster-with-new-vpc/main.tf#L122

Unfortunately Terraform will not allow you to specify subnet_ids = module.aws_vpc.private_subnets as part of variable :-(

Possible solution may be creating the eks_managed_node_group_defaults:

eks_managed_node_group_defaults = {
  subnet_ids = module.aws_vpc.private_subnets
}

# managed_node_groups variable doesn't need to have `subnet_ids` definition, because it is already part of the `eks_managed_node_group_defaults`
managed_node_groups = var.managed_node_groups

Like it is here: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/9c9ac81e3a3a91d52b5341d005de80a05a2fa0e8/node_groups.tf#L242

Anyway -> The goal is to have possibility to define node_groups as TF variable (tfvars) and not have it "hardcoded" in the module like it is in the examples: https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/ee7468c52eefa52b5ecfae7a2ffc58b34156550e/examples/eks-cluster-with-new-vpc/main.tf#L117-L124

Thank you...

What did you do and What did you see instead?

I can always hardcode the node_group definition directly into the module like it is in the examples:

But this is not a good practice - node groups should be in variables to be easily changed without changing the "main" code...

Additional Information

No response

[FEATURE] Windows support of OpenTelemetry addon

Is your feature request related to a problem? Please describe

I want to collect performance metrics for Windows nodes in EKS cluster. However, the aws-opentelemetry-eks addon currently supports only linux nodes. In general, it seems to be feasible to collect metrics from windows nodes: https://aws-otel.github.io/docs/setup/build-collector-on-windows

Describe the solution you'd like

Collect performance metrics from windows nodes.

Describe alternatives you've considered

Install OpenTelemetry Collector manually on windows nodes: https://aws-otel.github.io/docs/setup/build-collector-on-windows

[Bug]: disk_size on Windows nodes

Welcome to Amazon SSP EKS Accelerator!

Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version

latest

What is your environment, configuration and the example used?

module "eks" {
    source = "git::https://github.com/aws-samples/aws-eks-accelerator-for-terraform.git"

    tenant            = local.tenant
    environment       = local.environment
    zone              = local.zone

    # EKS CLUSTER
    kubernetes_version       = var.kubernetesVersion
    vpc_id             = module.vpc.vpc_id
    private_subnet_ids = module.vpc.private_subnets   # Enter Private Subnet IDs
    
    create_eks = true

    enable_windows_support = true
    self_managed_node_groups = {
    "windows" = {
            node_group_name = "windows"
            launch_template_os          = "windows"
            subnet_ids      = module.vpc.private_subnets
            desired_size    = var.windowsNodeCountMin
            min_size        = var.windowsNodeCountMin
            max_size        = var.windowsNodeCountMax
            disk_size       = 100
            
            k8s_labels = {
                "node.kubernetes.io/os" = "windows"
            }
        }}
}

What did you do and What did you see instead?

I am trying to enlarge the disk size of a self-managed windows node in EKS. However, the additional space is not available and I get disk pressure when my data exceed the default size of 50GB.

The launch template shows correct values:
Volume type: EBS
Device name: /dev/xvda
Size (GiB): 100
Volume type: gp2
Encrypted: yes

The EC instance get two volumes
/dev/sda1 of size 50 GiB
/dev/xvda of size 100 GiB

Additional Information

Running the following command on the instance gives:


PS C:\Windows\system32> get-volume

DriveLetter FriendlyName FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining     Size
----------- ------------ -------------- --------- ------------ ----------------- -------------     ----
                         NTFS           Fixed     Healthy      OK                     19.84 GB 19.87 GB
C                        NTFS           Fixed     Healthy      OK                     11.03 GB    50 GB

PS C:\Windows\system32> get-disk

Number Friendly Name                                                                                                                      Serial Number                    HealthStatus         OperationalStatus      Total Size Partition
                                                                                                                                                                                                                                  Style
------ -------------                                                                                                                      -------------                    ------------         -----------------      ---------- ----------
3      Msft Virtual Disk                                                                                                                                                   Healthy              Online                      20 GB GPT
1      NVMe Amazon Elastic B                                                                                                              vol0e689853a02aee15f_00000001.   Healthy              Online                     100 GB RAW
0      NVMe Amazon Elastic B                                                                                                              vol06b0ff78111f7d0e7_00000001.   Healthy              Online                      50 GB MBR

[QUESTION] does this project still use terraform-aws-eks community module ?

Please describe your quesiton here

The project mentions in the README.md This project leverages the community terraform-aws-ek) modules for deploying EKS Clusters. I can't seem to see anywhere in the modules/ folder reference to this terraform-aws-eks module.

Is it still the case that this project uses https://github.com/terraform-aws-modules/terraform-aws-eks?

Yes, I have checked the repo for existing issues before raising this question

[Bug]: Issue with Karpenter example

Welcome to Amazon SSP EKS Accelerator!

Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

│ Error: Invalid value for module argument
│
│ on main.tf line 135, in module "karpenter-launch-templates":
│ 135: launch_template_config = {
│ 136: linux = {
│ 137: ami = "ami-0adc757be1e4e11a1"
│ 138: launch_template_prefix = "karpenter"
│ 139: iam_instance_profile = module.aws-eks-accelerator-for-terraform.self_managed_node_group_iam_instance_profile_id[0]
│ 140: vpc_security_group_ids = module.aws-eks-accelerator-for-terraform.worker_security_group_id
│ 141: block_device_mappings = [
│ 142: {
│ 143: device_name = "/dev/xvda"
│ 144: volume_type = "gp2"
│ 145: volume_size = "200"
│ 146: }
│ 147: ]
│ 148: },
│ 149: bottlerocket = {
│ 150: ami = "ami-03909df9bfcc1e215"
│ 151: launch_template_os = "bottlerocket"
│ 152: launch_template_prefix = "bottle"
│ 153: iam_instance_profile = module.aws-eks-accelerator-for-terraform.self_managed_node_group_iam_instance_profile_id[0]
│ 154: vpc_security_group_ids = module.aws-eks-accelerator-for-terraform.worker_security_group_id
│ 155: block_device_mappings = [
│ 156: {
│ 157: device_name = "/dev/xvda"
│ 158: volume_type = "gp2"
│ 159: volume_size = "200"
│ 160: }
│ 161: ]
│ 162: },
│ 163: }
│
│ The given value is not suitable for child module variable "launch_template_config" defined at ../../modules/launch-templates/variables.tf:1,1-34: element "linux":
│ attribute "vpc_security_group_ids": list of string required.

What did you do and What did you see instead?

Deployment Steps
Step1: Clone the repo using the command below

git clone https://github.com/aws-samples/aws-eks-accelerator-for-terraform.git

Step2: Run Terraform INIT

to initialize a working directory with configuration files

cd examples/eks-cluster-with-karpenter/
terraform init

Step3: Run Terraform PLAN

to verify the resources created by this execution

export AWS_REGION= # Select your own region
terraform plan

Additional Information

No response

Fix code scanning alert - CloudWatch Without Retention Period Specified

Tracking issue for:

https://github.com/aws-samples/aws-eks-accelerator-for-terraform/security/code-scanning/161

Fix: Change the type from string to number to fix the issue

The official documentation site link is broken!

The official documentation site link (https://aws-ia.github.io/terraform-ssp-amazon-eks). is broken!

"dial tcp 127.0.0.1:80: connect: connection refused" on kubernetes-addons module

I am running terraform using gitlab-ci and there are some unsupported variable errors thrown today.
After chasing for a while, I observed that the main branch has been updated. The k8s addon config has been refactored as a sub-module. So I updated my terraform code from

module "eks" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform"

  tenant      = local.tenant
  environment = local.environment
  zone        = local.zone

  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets

  create_eks         = true
  kubernetes_version = local.kubernetes_version

  managed_node_groups = {
    ondemand = {
      # Node Group configuration
      node_group_name = "managed-ondemand"

      # Node Group scaling configuration
      desired_size = local.desired_size
      min_size     = local.min_size
      max_size     = local.max_size

      # Node Group compute configuration
      capacity_type  = "ON_DEMAND"
      instance_types = [local.instance_type]
      disk_size      = 50

      # Node Group network configuration
      subnet_ids = module.vpc.private_subnets
    }
  }

  #K8s Add-ons
  aws_lb_ingress_controller_enable = true
  metrics_server_enable            = true
  cluster_autoscaler_enable        = true
}

module "kubernetes-addons" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform//modules/kubernetes-addons"

  eks_cluster_id = module.eks.eks_cluster_id

  # EKS Addons
  enable_amazon_eks_vpc_cni    = true
  enable_amazon_eks_coredns    = true
  enable_amazon_eks_kube_proxy = true

  #K8s Add-ons
  enable_aws_load_balancer_controller = true
  enable_metrics_server               = true
  enable_cluster_autoscaler           = true

}

Unfortunately, the terraform apply is failed.

module.makeitfun_eks.module.kubernetes-addons.module.aws_coredns[0].aws_eks_addon.coredns: Still creating... [6m40s elapsed]
module.makeitfun_eks.module.kubernetes-addons.module.aws_coredns[0].aws_eks_addon.coredns: Still creating... [6m50s elapsed]
module.makeitfun_eks.module.kubernetes-addons.module.aws_coredns[0].aws_eks_addon.coredns: Still creating... [7m0s elapsed]
module.makeitfun_eks.module.kubernetes-addons.module.aws_coredns[0].aws_eks_addon.coredns: Creation complete after 7m4s [id=makeitfun-development-all-eks:coredns]
╷
│ Error: Post "http://localhost/api/v1/namespaces/kube-system/serviceaccounts": dial tcp 127.0.0.1:80: connect: connection refused
│ 
│   with module.makeitfun_eks.module.kubernetes-addons.module.aws_load_balancer_controller[0].kubernetes_service_account.aws_load_balancer_controller_sa,
│   on .terraform/modules/makeitfun_eks.kubernetes-addons/modules/kubernetes-addons/aws-load-balancer-controller/main.tf line 346, in resource "kubernetes_service_account" "aws_load_balancer_controller_sa":
│  346: resource "kubernetes_service_account" "aws_load_balancer_controller_sa" {
│ 
╵
╷
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│   with module.makeitfun_eks.module.kubernetes-addons.module.cluster_autoscaler[0].helm_release.cluster_autoscaler[0],
│   on .terraform/modules/makeitfun_eks.kubernetes-addons/modules/kubernetes-addons/cluster-autoscaler/main.tf line 19, in resource "helm_release" "cluster_autoscaler":
│   19: resource "helm_release" "cluster_autoscaler" {
│ 
╵
╷
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│   with module.makeitfun_eks.module.kubernetes-addons.module.metrics_server[0].helm_release.metrics_server[0],
│   on .terraform/modules/makeitfun_eks.kubernetes-addons/modules/kubernetes-addons/metrics-server/main.tf line 19, in resource "helm_release" "metrics_server":
│   19: resource "helm_release" "metrics_server" {
│ 
╵
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

https://i.ibb.co/8zfv4st/Screenshot-2021-12-28-at-5-56-55-PM.png

Does it seem the submodule cannot ready the newly created kube config?

here is my gitlab-ci:

default:
  image:
    name: hashicorp/terraform:latest
    entrypoint:
      - /usr/bin/env
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  before_script:
    - echo "Initialing..."
    - terraform init -backend-config=vars/backend.hcl.json -backend-config="access_key=${AWS_ACCESS_KEY_ID}" -backend-config="secret_key=${AWS_SECRET_ACCESS_KEY}"
    - terraform workspace select ${CI_COMMIT_REF_NAME}
  cache:
    key: terraform
    paths:
      - .terraform

stages:
  - validate
  - plan
  - apply

validate:
  stage: validate
  environment:
    name: ${CI_COMMIT_REF_NAME}
  only:
    - development
  script:
    - terraform validate

plan:
  stage: plan
  environment:
    name: ${CI_COMMIT_REF_NAME}
  only:
    - development
  script:
    - terraform plan -var-file=vars/$CI_COMMIT_REF_NAME/variables.tfvars
  dependencies:
    - validate

apply non production:
  stage: apply
  environment:
    name: ${CI_COMMIT_REF_NAME}
  only:
    - development
  script:
    - terraform apply -var-file=vars/$CI_COMMIT_REF_NAME/variables.tfvars -input=false -refresh=true -auto-approve=true
  dependencies:
    - plan

apply production:
  extends: apply non production
  when: manual
  only:
    - master

Cluster name

Hi Team,

Is there any option to pass cluster name instead of tenant, environment and zone separately. I see that cluster name is derived from eks_tags module. Is it possible to add an option to pass just the cluster name if the eks_tags values are not provided.

[Bug]: Running eks-cluster-with-argocd example, causes a crash with unsupported attribute

Welcome to Amazon SSP EKS Accelerator!

Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version

3.2.2

What is your environment, configuration and the example used?

Terraform version - 1.0.5

example used - https://github.com/aws-samples/aws-eks-accelerator-for-terraform/tree/main/examples/eks-cluster-with-argocd with no changes.

What did you do and What did you see instead?

I ran the normal terraform init and terraform plan during the plan stage, I get

~/Documents/Customers/lab/eks/aws-eks-accelerator-for-terraform-3.2.2/examples/eks-cluster-with-argocd > terraform plan
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2

╷
│ Error: Unsupported attribute
│ 
│   on ../../modules/kubernetes-addons/argocd/data.tf line 24, in data "aws_secretsmanager_secret" "ssh_key":
│   24:   for_each = { for k, v in var.applications : k => v if v.ssh_key_secret_name != null }
│ 
│ This object does not have an attribute named "ssh_key_secret_name".
╵
╷
│ Error: Unsupported attribute
│ 
│   on ../../modules/kubernetes-addons/argocd/data.tf line 24, in data "aws_secretsmanager_secret" "ssh_key":
│   24:   for_each = { for k, v in var.applications : k => v if v.ssh_key_secret_name != null }
│ 
│ This object does not have an attribute named "ssh_key_secret_name".

But expected no error messages

Additional Information

This issue appears to be related to PR #201 and the changes it makes to the argocd addon.

Launch Template

Hi Team,
I am testing the terraform code and I see that there are two launch template being created and the autoscaling group is using the default template which is not encrypting the EBS.

Support Terraform v0.15.0 / v1.0.0

This repo is currently using Terraform (TF) v0.14.x or lower. It would be nice to use the latest TF version. I ran into a few issues when trying to do that - the terraform apply command errors out. I found that use of Terraform v0.15.0+ requires the following changes:

Replace TF function map() by tomap().
Update EKS module vesion to at least v15.2.0, to replace its internal use of list() by tolist().
Update AWS provider version to at least v3.37.0, to support the Warm Pool support added by the EKS module v15.2.0.
The source folder name can no longer be passed as an argument to the terraform init, plan, and apply commands.

Add kube-state-metrics helm chart

helm repo add bitnami https://charts.bitnami.com/bitnami
https://github.com/bitnami/bitnami-docker-kube-state-metrics
https://artifacthub.io/packages/helm/bitnami/kube-state-metrics

[ebs-csi & argocd] Support Graviton instance

I want to use the repo together with AWS Graviton instances. Sadly, not all pods are running as expected.

Problem

I tried version 3.2.2 together with

terraform 1.1.4
instance type t4g.medium
ami AL2_ARM_64

Sadly, the "ebs-csi-node" and "ebs-csi-controller" will never start up. The same applies to argo-cd.

Target

I would love to have a cluster running with Graviton instances as without including EBS-CSI and ArgoCD.

My assumption is that the helm release uses an outdated version, or a version pinned to X86 and needs to be architecture agnostic.

This compatibility site states that CSI should support ARM, for ArgoCD there is still an open issue, and you have to compile from source for now.

irsa module fails to create service account for EKS managed vpc-cni addon

The error can be reproduced by running example "8-eks-cluster-with-eks-addons" from https://github.com/aws-samples/terraform-ssp-eks-patterns. Service account "aws-node" already exists when irsa_addon module tries to create it for vpc_cni addon:

Error: serviceaccounts "aws-node" already exists │ │ with module.aws-eks-accelerator-for-terraform.module.vpc_cni[0].module.irsa_addon.kubernetes_service_account_v1.irsa, │ on .terraform\modules\aws-eks-accelerator-for-terraform\modules\irsa\main.tf line 32, in resource "kubernetes_service_account_v1" "irsa": │ 32: resource "kubernetes_service_account_v1" "irsa" {

Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused with module.eks-ssp.kubernetes_config_map.aws_auth[0]

I'm trying to deploy a cluster with self managed node groups. No matter what config options I use, I always come up with the following error:

Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused
with module.eks-ssp.kubernetes_config_map.aws_auth[0]
on .terraform/modules/eks-ssp/aws-auth-configmap.tf line 19, in resource "kubernetes_config_map" "aws_auth":
resource "kubernetes_config_map" "aws_auth" {

The .tf file looks like this:

module "eks-ssp" {
    source = "github.com/aws-samples/aws-eks-accelerator-for-terraform"

    # EKS CLUSTER
    tenant            = "DevOpsLabs2"
    environment       = "dev-test"
    zone              = ""
    terraform_version = "Terraform v1.1.4"

    # EKS Cluster VPC and Subnet mandatory config
    vpc_id             = "xxx"
    private_subnet_ids = ["xxx","xxx", "xxx", "xxx"]

    # EKS CONTROL PLANE VARIABLES
    create_eks         = true
    kubernetes_version = "1.19"

  # EKS SELF MANAGED NODE GROUPS
    self_managed_node_groups = {
    self_mg = {
      node_group_name        = "DevOpsLabs2"
      subnet_ids             = ["xxx","xxx", "xxx", "xxx"]
      create_launch_template = true
      launch_template_os     = "bottlerocket"       # amazonlinux2eks  or bottlerocket or windows
      custom_ami_id          = "xxx"
      public_ip              = true                   # Enable only for public subnets
      pre_userdata           = <<-EOT
            yum install -y amazon-ssm-agent \
            systemctl enable amazon-ssm-agent && systemctl start amazon-ssm-agent \
        EOT

      disk_size     = 20
      instance_type = "t2.small"
      desired_size  = 2
      max_size      = 10
      min_size      = 2
      capacity_type = "" # Optional Use this only for SPOT capacity as  capacity_type = "spot"

      k8s_labels = {
        Environment = "dev-test"
        Zone        = ""
        WorkerType  = "SELF_MANAGED_ON_DEMAND"
      }

      additional_tags = {
        ExtraTag    = "t2x-on-demand"
        Name        = "t2x-on-demand"
        subnet_type = "public"
      }
      create_worker_security_group = false # Creates a dedicated sec group for this Node Group
    },
  }
}

module "eks-ssp-kubernetes-addons" {
    source = "github.com/aws-samples/aws-eks-accelerator-for-terraform//modules/kubernetes-addons"

    eks_cluster_id                        = module.eks-ssp.eks_cluster_id

    # EKS Addons
    enable_amazon_eks_vpc_cni             = true
    enable_amazon_eks_coredns             = true
    enable_amazon_eks_kube_proxy          = true
    enable_amazon_eks_aws_ebs_csi_driver  = true

    #K8s Add-ons
    enable_aws_load_balancer_controller   = true
    enable_metrics_server                 = true
    enable_cluster_autoscaler             = true
    enable_aws_for_fluentbit              = true
    enable_argocd                         = true
    enable_ingress_nginx                  = true

    depends_on = [module.eks-ssp.self_managed_node_groups]
}

"terraform init" fails

I cloned the repo and tried "terraform init" in various project directories for example:
deploy/eks-cluster-with-new-vpc
aws-eks-accelerator-for-terraform/deploy/advanced/live/preprod/eu-west-1/application_acct/dev

I always get:
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given
│ constraints >= 3.15.0, >= 3.40.0, >= 3.56.0, >= 3.60.0, ~> 3.60.0, >= 3.63.0
╵
I used Terraform v1.0.8 and v1.0.11.

Add OpenTelemetry to project

Provide the option to instrument the EKS cluster with OpenTelemetry helm charts out of the box.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector

References:

https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

documentation for k8s_taints

k8s_taints are not documented right now.
see https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/main/modules/aws-eks-managed-node-groups/locals.tf#L24

Here is a working example you can use:

k8s_taints = [{key= "purpose", value="execution", "effect"="NO_SCHEDULE"}]

I haven't tested it with fargate profiles:
https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/479b1a909c07fdc072181056bdcb799d90d8a877/modules/aws-eks-fargate-profiles/locals.tf#L8

NGINIX Ingress Controller feature

Provide a feature to deploy NGINIX Ingress controller with an example and a README

https://kubernetes.github.io/ingress-nginx/

self-managed windows nodes don't join EKS cluster

Hi,
I followed this example https://github.com/aws-samples/aws-eks-accelerator-for-terraform/tree/main/examples/5-eks-cluster-with-windows-support to add windows nodes to my eks cluster. In fact, the respective auto scaling group has one instance, but it is not available in my eks cluster. What could be the problem?

[QUESTION] Why my Windows containers have no internet access?

Please describe your quesiton here

I have added a Windows node to my EKS cluster which has internet access. However, when a pod is started on the windows node, I have no Internet access inside the container:

Invoke-WebRequest -URI www.github.com
Invoke-WebRequest : The remote name could not be resolved: 'www.github.com'
At line:1 char:1
+ Invoke-WebRequest -URI www.github.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

How can I troubleshoot the problem?

Yes, I have checked the repo for existing issues before raising this question

Audit docs and examples for usage of source

In all of our readmes, docs and examples we should be pinning the usage of our modules to specific versions.

Default of endpoint_private_access

Shouldn't the default for endpoint_private_access be false? At least, the description should be consistent.

variable "endpoint_private_access" {
  type        = bool
-  default     = true
+  default     = false
  description = "Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false"
}

see https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/df6e87237a47c63fc65cad68e14436503f9a1427/source/variables.tf#L124

Convert Kubernetes Addons variables to one map variable

Problem:
Currently Kubernetes Addons variables defined as individual variable(string or bool etc.) in tfvars file. This increases the complexity of managing and adding these variables across all the variables.tf files in the repo. Abstracting any new variable to TFVARS for Kubernetes Addon requires adding in all variables.tf files.

Solution:
Converting Kubernetes Addon each individual variables into one Map variable. This allows us to add more options through key/value pairs for k8s addon upgrades with minimal changes to the code. We only expose two variables per Kubernetes Add-on

aws_managed_prometheus_enable = true
aws_managed_prometheus_k8s_addon = {}

CUREENT

#---------------------------------------------------------//
ENABLE PROMETHEUS
#---------------------------------------------------------//


prometheus_enable             = true
prometheus_helm_chart_url     = "https://prometheus-community.github.io/helm-charts"
prometheus_helm_chart_name    = "prometheus"
prometheus_helm_chart_version = "14.4.0"
prometheus_image_tag          = "v2.26.0"
alert_manager_image_tag       = "v0.21.0"
configmap_reload_image_tag    = "v0.5.0"
node_exporter_image_tag       = "v1.1.2"
pushgateway_image_tag         = "v1.3.1"

TARGET

prometheus_enable             = true

aws_managed_prometheus_k8s_addon = {
  prometheus_helm_chart_url     = "https://prometheus-community.github.io/helm-charts"
  prometheus_helm_chart_name    = "prometheus"
  prometheus_helm_chart_version = "14.4.0"
  prometheus_image_tag          = "v2.26.0"
  alert_manager_image_tag       = "v0.21.0"
  configmap_reload_image_tag    = "v0.5.0"
  node_exporter_image_tag       = "v1.1.2"
  pushgateway_image_tag         = "v1.3.1"
}

Config driven Node groups and Fargate profiles creation

Currently Managed node groups, self managed node groups and fargate profiles requires TF code change to main.tf file. This new feature helps to define node groups/ fargate profiles as a map of values in TFVARS file and the TF code should handle the dynamic creation of Node groups and the fargate profiles.

As a part of this change, we are rewriting all the EKS modules(EKS Core, EKS Fargate, EKS Managed Node Groups, Self Managed etc.). This will be released soon as a big release

Enable EMR on EKS Feature

Enable EMR on EKS Feature through sub module

This feature will be released as a part of Release3. Currently the work is in progress in Feature branch

local.argocd_add_on_config.ingressNginx and local.argocd_add_on_config.nginxIngress

Why there are the keys ingressNginx and nginxIngress? Looks like a mistake.

https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/3ca142345b00d5546e0c2236b7eb60f862f995b0/locals.tf#L95

https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/3ca142345b00d5546e0c2236b7eb60f862f995b0/locals.tf#L98

Update helm chart to version 0.1.11 for aws-for-fluent-bit

k8s_labels for self managed node group doesn't work

Hi,
It seems k8s_labels for self managed node group doesn't work.
I tried it with this example.
Thanks.

aws-ia / terraform-aws-eks-blueprints Goto Github PK

terraform-aws-eks-blueprints's Introduction

Amazon EKS Blueprints for Terraform

Motivation

Consumption

Related Projects

Terraform Caveats

Support & Feedback

Security

License

terraform-aws-eks-blueprints's People

Contributors

Stargazers

Watchers

Forkers

terraform-aws-eks-blueprints's Issues

Please describe your quesiton here

Welcome to Amazon SSP EKS Accelerator!

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

What did you do and What did you see instead?

Additional Information

Describe the solution you'd like

Additional context

More

Welcome to Amazon SSP EKS Accelerator!

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

What did you do and What did you see instead?

Additional Information

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Welcome to Amazon SSP EKS Accelerator!

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

What did you do and What did you see instead?

Additional Information

Please describe your quesiton here

More

Welcome to Amazon SSP EKS Accelerator!

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

What did you do and What did you see instead?

Additional Information

Welcome to Amazon SSP EKS Accelerator!

Amazon EKS Accelerator Release version

What is your environment, configuration and the example used?

What did you do and What did you see instead?

Additional Information

Problem

Target

References:

Please describe your quesiton here

More

Recommend Projects

Recommend Topics

Recommend Org