Hi,
Could you provide and add-on to enable installing GuarDuty agent via a GitOps managed solution like ArgoCD?

Thank you

Can we use the Helm charts from this repository to implement a GitOps bridge? Additionally, do we need to have a separate namespace for GitOps, or should it be under any EKS-managed namespace?

When enableing the enable_aws_cloudwatch_metrics add-on, the DaemonSet pods were not being created, as it could not find the service account.

Feature Request

I suggest adding hashicorp vault as an add-on.

Kindly advise if the project is open for adding more valuable/commonly used applications.

here is their official helm chart repo: https://github.com/hashicorp/vault-helm

Strimzi Kafka Operator is a Kubernetes Operator used to install Apache Kafka on Kubernetes.

This PR if a Follow up to aws-ia/terraform-aws-eks-blueprints#1090

Installing kube-prometheus-stack fails with the below error:

Does some other repo need to be added to argocd to get this install working?

Hi,
can you support add-on for HAProxy Kubernetes Ingress Controller as done for nginx ?

HAProxy is a great solution and being able to use it in eks (or in general in k8s) as a GitOps solution managed by ArgoCD would be valuable.

Thank you

The chart for strimzi-kafka-operator is not configured to create a namespace, causing ArgoCD to fail to sync. For context, see aws-ia/terraform-aws-eks-blueprints#1309 (comment).

I've not put some metadatas to configure external-dns or ingress-nginx then I got some errors:

failed to execute go template controller: service: type: {{default "LoadBalancer" .metadata.annotations.ingress_nginx_service_type}} : template: :3:44: executing "" at <.metadata.annotations.ingress_nginx_service_type>: map has no entry for key "ingress_nginx_service_type"

With configurations like :

values: |
              controller:
                service:
                  type: {{default "LoadBalancer" .metadata.annotations.ingress_nginx_service_type}}

I would like the GoTemplate to select the default value without throwing an error.

Karpenter helm chart name should be karpenter, otherwise, it generates an error.

• https://charts.karpenter.sh/index.yaml
• https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/add-ons/karpenter/Chart.yaml#L18

-- name: Karpenter
-  version: 0.6.5
+- name: karpenter
+  version: 0.10.0

There are also a bunch of errors here:

https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/karpenter.yaml#L17-L24

       parameters:
-        - name: karpetner.serviceAccount.name
+        - name: karpenter.serviceAccount.name
           value: {{ .Values.karpenter.serviceAccountName }}
-        - name: karpetner.ClusterName
+        - name: karpenter.ClusterName
           value: {{ .Values.clusterName }}
-        - name: karpetner.controller.ClusterEndpoint
+        - name: karpenter.ClusterEndpoint
           value: {{ .Values.karpenter.controllerClusterEndpoint }}
-        - name: karpetner.aws.defaultInstanceProfile
+        - name: karpenter.aws.defaultInstanceProfile
           value: {{ .Values.karpenter.awsDefaultInstanceProfile }}

The add-on-ebs-csi-resources failed to start properly with the error in argued:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): charts/addons/aws-ebs-csi/resources: app path does not exist

The path for the resource is not correct

Fails to deploy gateway API with Argocd.

Seems the account-id is escaped

ailed deploy app1 due to err BadRequestException: '\"1234567890\"/eks-blueprint-workshop-blue/vpc-0c07625c57ef272de' is not a valid value for TagValue - it contains illegal characters

link to aws-ia/terraform-aws-eks-blueprints-addons#306

When installing via gitops, cluster issuers do not get installed.

https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/ingress-nginx.yaml#L17

It currently reads namespace: nginx however it should be namespace: ingress-nginx

Hello,
We have used eks-blueprints-add-ons for argo and kube_prometheus_stack and installed in 3 environments. We are using EKS 1.23.

1. In one environment, we see only namespace is created and not any resources/pods under it.

[root@ip-10-71-0-1 ~]# kubectl get ns
NAME STATUS AGE
argocd Active 26d
cert-manager Active 26d
karpenter Active 26d
kube-node-lease Active 26d
kube-prometheus-stack Active 26d

[root@ip-10-71-0-1 ~]# kubectl get pods -n kube-prometheus-stack
No resources found in kube-prometheus-stack namespace.

2. In other environment, kube-prometheus-stack-grafana,kube-state-metrics, prometheus-node-exporter pods got created and running, but not stack-operator and below is the error:

[root@ip-10-73-0-13 ~]# kubectl events kube-prometheus-stack-operator-6f4f8975fb-qwcfk -n kube-prometheus-stack
LAST SEEN TYPE REASON OBJECT MESSAGE
33m (x267 over 44h) Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-vvxtg Unable to attach or mount volumes: unmounted volumes=[tls-secret], unattached volumes=[kube-api-access-tck4q tls-secret]: timed out waiting for the condition
8m39s (x1328 over 44h) Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-vvxtg MountVolume.SetUp failed for volume "tls-secret" : secret "kube-prometheus-stack-admission" not found
3m40s (x915 over 44h) Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-vvxtg Unable to attach or mount volumes: unmounted volumes=[tls-secret], unattached volumes=[tls-secret kube-api-access-tck4q]: timed out waiting for the condition
3m8s Normal Scheduled Pod/kube-prometheus-stack-operator-6f4f8975fb-qwcfk Successfully assigned kube-prometheus-stack/kube-prometheus-stack-operator-6f4f8975fb-qwcfk to ip-10-73-2-246.ap-southeast-2.compute.internal
3m8s Normal SuccessfulCreate ReplicaSet/kube-prometheus-stack-operator-6f4f8975fb Created pod: kube-prometheus-stack-operator-6f4f8975fb-qwcfk
65s Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-qwcfk Unable to attach or mount volumes: unmounted volumes=[tls-secret], unattached volumes=[tls-secret kube-api-access-xh9wx]: timed out waiting for the condition
60s (x9 over 3m8s) Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-qwcfk MountVolume.SetUp failed for volume "tls-secret" : secret "kube-prometheus-stack-admission" not found

60s (x9 over 3m8s) Warning FailedMount Pod/kube-prometheus-stack-operator-6f4f8975fb-qwcfk MountVolume.SetUp failed for volume "tls-secret" : secret "kube-prometheus-stack-admission" not found

Could you please let me know what i am missing to resolve this issues.

When the {{metadata.annotations.addons_repo_basepath}} is set to /argocd only works for addons and not for the bootstrap app of apps

for example:

• https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/argocd/bootstrap/control-plane/exclude/bootstrap.yaml#L24
• https://github.com/gitops-bridge-dev/gitops-bridge/blob/main/argocd/iac/terraform/examples/eks/hello-world/bootstrap/addons.yaml#L24

We should have a consistent naming in the app of app templates, this allows upstream whoever consumes this repo to follow the same naming patterns for all addons. Examples:

For AWS Load Balancer Controller: 👍

# chart/templates/aws-load-balancer-controller.yaml
{{- if and (.Values.awsLoadBalancerController) (.Values.awsLoadBalancerController.enable) -}}

For Appmesh Controller: 👎

# chart/templates/appmesh-controller.yaml
{{- if and (.Values.appMesh) (.Values.appMesh.enable) -}}  ==> should be Values.appmeshController

Another minor one, but can we change enable to enabled?

{{- if and (.Values.awsLoadBalancerController) (.Values.awsLoadBalancerController.enable) -}}
==>>
{{- if and (.Values.awsLoadBalancerController) (.Values.awsLoadBalancerController.enabled) -}}

I am working on getting SSP CDK to consume this repo (aws-quickstart/cdk-eks-blueprints#300). Any feedback on this are appreciated. While waiting, I will submit a PR for this.

Setup:

I am using the following terraform repo to deploy eks cluster: https://github.com/aws-ia/terraform-aws-eks-blueprints
In this repo, I have enabled agrocd_addson and referred this repo for app of apps model deployment.

Issue:
When I enabled the tetrate-istio adds_on, argocd fails the deployment due to no namespace.
I compared other app templates in the "eks-blueprints-add-ons" and there is no difference. None of the templates has auto-creating namespace in the sync options. But others deploy namespace along with all other resources from helm chart.

Kindly advise on this.

As described in aws-ia/terraform-aws-eks-blueprints#913, the kubecost helm chart repo URL is not defined properly in the chart definition.

The "/cost-analyzer" string should not be present. The correct URL is "oci://public.ecr.aws/kubecost"

The name of datadog operator app in "https://github.com/aws-ia/terraform-aws-eks-blueprints" is "datadog-operator" which is different than the name in this repo here "datadogOperator".

If you are using terraform infra repo, it only allows its own name "datadog-operator" which does not exist here. Therefore, argocd does deploy this app if you set "argocd_manage_add_ons = true" there.

I believe, we need to adjust the name in this repo.

Regards!

Hello,

Not an issue actually. I'm just wondering if there's a reason this repo does not contain any tags. It would be very useful to be able to reference a specific tag in the eks blueprints terraform modules. Thanks.

When deploying the Kubecost Add-On with ArgoCD, it fails with the error :

Failed sync attempt to 41eef70473f63b5564ffb953d47a9a70b4463cf4: one or more objects failed to apply, reason: namespaces "kubecost" not found (retried 1 times).

The Argo Application did not create the Kubecost namespace, so id can not be deployed automatically

Currently, to customize the values of each add-on, I need to override the values in the App of App chart: https://github.com/aws-samples/ssp-eks-add-ons/blob/main/chart/values.yaml#L64 . However, the values exposed for each add-on are limited. For example, Karpenter has many other values that can be set: https://github.com/aws/karpenter/blob/main/charts/karpenter/values.yaml

Is there any way I can customize the full list of values for each add-on?

If one enables Grafana within the kubernetes-addons Terraform module as follows, the Grafana Helm Chart never gets deployed.

ISSUE
The add-on definition and chart template are missing in this repo and need to be created.

module "kubernetes-addons" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.10.0/modules/kubernetes-addons"

  eks_cluster_id = module.eks_blueprints.eks_cluster_id

  #---------------------------------------------------------------
  # ARGO CD ADD-ON
  #---------------------------------------------------------------

  enable_argocd         = true
  argocd_manage_add_ons = true # Indicates that ArgoCD is responsible for managing/deploying Add-ons.
  argocd_applications = {
    addons    = local.addon_application
    #workloads = local.workload_application
  }

  argocd_helm_config = {
    values = [templatefile("${path.module}/argocd-values.yaml", {})]
  }

  #---------------------------------------------------------------
  # ADD-ONS - You can add additional addons here
  # https://aws-ia.github.io/terraform-aws-eks-blueprints/add-ons/
  #---------------------------------------------------------------
  enable_aws_load_balancer_controller  = true
  enable_amazon_eks_aws_ebs_csi_driver = true
  enable_grafana                      = true
} 

There are some fields that will ALWAYS show a difference in ArgoCD unless ignoreDifferences is set correctly in the Application manifest for tetrate-istio

See: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration

I'll add the exact config here, if/when I work it out.

I am using eks-blueprint-add-ons along with ArgoCD App of Apps pattern via eks-blueprints and terraform. Below is my configs. I cannot get the service account created for the cluster autoscaler add on to properly populate annotations so IRSA works correctly.

ArgoCD terraform config:

module "eks_blueprints_kubernetes_addons" {
  source                            = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.0.4"
  eks_cluster_id                    = module.eks.cluster_id
  enable_argocd                     = true
  argocd_manage_add_ons             = true
  argocd_admin_password_secret_name = data.aws_secretsmanager_secret.argocd_password.arn

  argocd_helm_config = {
    name             = "argo-cd"
    chart            = "argo-cd"
    repository       = "https://argoproj.github.io/argo-helm"
    version          = "4.6.0"
    namespace        = "argocd"
    timeout          = "1200"
    create_namespace = true
    values           = [templatefile("${path.module}/argocd-values.yaml", {})]
  }

  argocd_applications = {
    workloads = {
      path                = "envs/dev"
      repo_url            = "<my private repo>"
      ssh_key_secret_name = data.aws_secretsmanager_secret.argocd_sshkey.arn
      values              = {}
    }
    addons = {
      path                = "chart"
      repo_url            = "<my private repo>"
      add_on_application  = true
      ssh_key_secret_name = data.aws_secretsmanager_secret.argocd_sshkey.arn
      values              = {}
    }
  }
  depends_on = [module.eks.eks_managed_node_groups]
}

ArgoCD helm values.yaml:

redis-ha:
  enabled: true

controller:
  enableStatefulSet: true

server:
  autoscaling:
    enabled: true
    minReplicas: 2

repoServer:
  autoscaling:
    enabled: true
    minReplicas: 2

In /add-ons/cluster-autoscaler/values.yaml:

rbac:
  create: true
  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::<my-account>:role/service-role/<my-role>"

In /add-ons/cluster-autoscaler/Chart.yaml:

apiVersion: v2
name: cluster-autoscaler
description: A Helm chart for installing cluster-autoscaler
type: application

# The chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.1.0

# Version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: "1.0"

dependencies:
  - name: cluster-autoscaler
    version: 9.15.0
    repository: https://kubernetes.github.io/autoscaler

In /chart/values.yaml:

# Cluster Autoscaler Values
clusterAutoscaler:
  enable: true
  serviceAccountName: clusterautoscaler

ArgoCD sees the values:

However, the serviceaccount created in the cluster does not have the annotation:

kubectl describe serviceaccount clusterautoscaler -n kube-system

Name:                clusterautoscaler
Namespace:           kube-system
Labels:              app.kubernetes.io/instance=cluster-autoscaler
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=aws-cluster-autoscaler
                     argocd.argoproj.io/instance=cluster-autoscaler
                     helm.sh/chart=cluster-autoscaler-9.15.0
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   clusterautoscaler-token-lrh5q
Tokens:              clusterautoscaler-token-lrh5q
Events:              <none>

Am i doing something wrong here? Appreciate the help.

EDIT: To add to this, i ended up modifying the values.yaml to such:

cluster-autoscaler:
  extraArgs:
    v: 4
    stderrthreshold: info
    cloud-provider: aws
    logtostderr: true
    skip-nodes-with-local-storage: false
    expander: least-waste
    balance-similar-node-groups: true
    skip-nodes-with-system-pods: false

  deployment:
    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

  rbac:
    create: true
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::<my-account>:role/service-role/<my-role>"
      create: true
      name: clusterautoscaler
        
  resources:
    limits:
      cpu: 200m
      memory: 512Mi
    requests:
      cpu: 200m
      memory: 512Mi

Still doesnt work via ArgoCD. However, if run a helm template on values.yaml its working correctly:

# Source: cluster-autoscaler/charts/cluster-autoscaler/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: "test"
    app.kubernetes.io/name: "aws-cluster-autoscaler"
    app.kubernetes.io/managed-by: "Helm"
    helm.sh/chart: "cluster-autoscaler-9.15.0"
  name: clusterautoscaler
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::<my-account>:role/service-role/<my-role>
automountServiceAccountToken: true

Please rename the point With the EKS Blueprints Framework to With the EKS Blueprints Framework for Terraform. Otherwise, it gives an impression that Argo CD is capable of deploying Kubernetes clusters.

Also, this
Please see the ArgoCD add-on documentation for details on how to easily bootstrap an EKS cluster

That 'bootstrap' thing. As if you are able to deploy a Kubernetes cluster with Argo CD from scratch.
Please mention that you need to create a cluster with Terraform in the first place

**Is your feature request related to a problem?
I would like to request the addition of support for the Amazon Elastic Kubernetes Service (EKS) Pod Identity addon in the EKS Anywhere Addons repository. The Amazon EKS Pod Identity addon is a valuable feature that allows applications running on Amazon EKS clusters to securely obtain AWS Identity and Access Management (IAM) credentials. This capability is essential for applications that need to interact with AWS services, such as Amazon S3, DynamoDB, or AWS Secrets Manager, while running within Kubernetes pods.

Describe the solution
Describe the solution you'd like
Adding support for the Amazon EKS Pod Identity addon would greatly benefit users who deploy applications on EKS Anywhere clusters. This feature enables applications to assume IAM roles based on annotations assigned to Kubernetes service accounts, providing fine-grained access control to AWS resources. This is particularly valuable in scenarios where applications need to access AWS resources securely without relying on static IAM credentials.

Additional context
Amazon EKS Pod Identity: A New Way for Applications on EKS to Obtain IAM Credentials

Amazon EKS Pod Identity Documentation

If found that there is a missing application when I tried to deploy kyverno using Argocd.
The chart is present in the add-ons folder but it's missing in the chart folder

This is in the upstream project, but not in this examples repos.

https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/modules/kubernetes-addons/aws-node-termination-handler

Originally reported here: aws-ia/terraform-aws-eks-blueprints#498

I had this setup in my code when I was testing it:

  enable_prometheus                   = true
  prometheus_helm_config = {
    version          = "15.8.5"
    create_namespace = true
    namespace        = "monitoring"
    values           = [templatefile("${path.module}/helm_values/prometheus-values.yaml.tftpl", {nodeSelector = local.primaryNodeSelector})]
  }

The values template looks like this:

# See:
# https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
#
# This is a template, so we can get values from terraform.
#

alertmanager:
  nodeSelector: ${jsonencode({ for ns_key, ns_value in nodeSelector : ns_key => ns_value })}

server:
  nodeSelector: ${jsonencode({ for ns_key, ns_value in nodeSelector : ns_key => ns_value })}
  retention: 1h

  resources:
    requests:
      cpu: 500m
      memory: 512Mi

  global:
    scrape_interval: 15s

pushgateway:
  nodeSelector: ${jsonencode({ for ns_key, ns_value in nodeSelector : ns_key => ns_value })}

It looks like the monitoring namespace is created, but the ArgoCD application/helm is not targeting the right namespace.

So, looking at the line here:

https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/prometheus.yaml#L33

It looks like the destination namespace is hard-coded (and maybe in other examples) and does not respect what was set in the Terraform code

Hi, looks like the enable_kube_prometheus_stack is part of the GitOps/ArgoCD addon, any plan to get Grafana/ADOT enabled as part of the addons ?

repoURL: {{ .Values.repoUrl }} hardcoded

currently hardcoded

spec:
  source:
    repoURL: https://github.com/aws-samples/eks-blueprints-add-ons

It should be

spec:
  source:
    repoURL: {{ .Values.repoUrl }}

Places where is hardcoded
https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/argo-rollouts.yaml#L12
https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/kube-state-metrics.yaml#L12
https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/chart/templates/kubernetes-dashboard.yaml#L12

even though prom node exporter is part of the kube prom stack addon . Need this as a separate addon for alternate architectures where the full prom stack is not needed.