Services like operate doesn't seem to work without the standalone gateway, If we want to make it possible to swtich bettwen embedded and standlone gateway then the broker contact point shouldn't be hard coded like it is done here https://github.com/camunda-community-hub/camunda-cloud-helm/blob/main/charts/zeebe-operate-helm/templates/configmap.yaml#L24

Possible solution in ccsm helm:

We could create a helper template and define zeebe.contactPoint. Depending on gateway enabled or not we set the contact point to the gateway or broker.

Add curator to the single chart and make it configurable.

• #159
• Add zeebe index config
• Add operate index config
• Add tasklist index config

Installing the latest version produces the following error:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]

When installing zeebe-full-helm on k8s 1.22 the nginx ingress is crash-looping. This might be due to updates in k8s:
https://kubernetes.io/blog/2021/07/26/update-with-ingress-nginx/

Add optimize chart to the single chart.

• Requires: Identity in chart
• Create a new sub chart in the ccsm-helm chart
  • Add manifest templates for optimize #287
  • Configure optimize to use identity #287
  • #286
• #289
• Add possible configurations as values to the values.yaml file
• Verify default resources with SaaS G3-s #298
• Document the configurations in values file and readme
  • #284
• #299
• Test optimize sub chart
  • Add template tests #287
  • #283
• Update architecture image, you can find the source in google drive. zeebe/ccsm-helm/CCSM Helm Architecture Image.
  • #285
• Add documentation about optimize to the camunda docs
  • Update architecture image and mention in self-managed section camunda/camunda-docs#849
  • Created follow up issue to improve Optimize docs camunda/camunda-docs#845

Depends on #127

I wish to use the value global.zeebe to name the Kubernetes resources zeebe rather than zeebe-zeebe.

Works in old version

In version 0.0.89 of the chart this works fine:

> helm install zeebe zeebe/zeebe-cluster --version 0.0.89 --set global.zeebe=zeebe

> kubectl get pods
NAME                             READY   STATUS
elasticsearch-master-0           1/1     Running
elasticsearch-master-1           1/1     Running
elasticsearch-master-2           1/1     Running
zeebe-0                          1/1     Running
zeebe-1                          1/1     Running
zeebe-2                          1/1     Running
zeebe-gateway-5f89d47657-pl8jw   1/1     Running

Breaks in current version

In the latest versions of the chart this causes the startup to fail:

> helm install zeebe zeebe/zeebe-cluster --version 0.0.99 --set global.zeebe=zeebe

> kubectl get pods
NAME                             READY   STATUS
elasticsearch-master-0           1/1     Running
elasticsearch-master-1           1/1     Running
elasticsearch-master-2           1/1     Running
zeebe-0                          0/1     CrashLoopBackOff
zeebe-1                          0/1     CrashLoopBackOff
zeebe-2                          0/1     CrashLoopBackOff
zeebe-gateway-5657db69c4-bzfvw   1/1     Running

> kubectl logs zeebe-0
[...]
***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to bind properties under 'zeebe.broker.gateway.network.port' to int:

    Property: zeebe.broker.gateway.network.port
    Value: tcp://10.0.96.44:9600
    Origin: "zeebe.broker.gateway.network.port" from property source "systemProperties"
    Reason: failed to convert java.lang.String to int

I am using Helm 3.

I tried to configure podSecurityContext/securityContext with zeebe-cluster-helm 1.2.10.

It fails with
Error: YAML parse error on zeebe-cluster-helm/templates/statefulset.yaml: error converting YAML to JSON: yaml: line 104: mapping values are not allowed in this context helm.go:88: [debug] error converting YAML to JSON: yaml: line 104: mapping values are not allowed in this context

The values file contains:

podSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - all

The generated content is:

      securityContext:
        allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all

with a bad indentation (capabilities and allowPrivilegeEscalation) should be aligned.

I tried with {{ toYaml .Values.podSecurityContext | indent 10 | trim }} instead of {{ toYaml .Values.podSecurityContext | indent 12 | trim }} the generation is OK (it is more a securityContext than a podSecurityContext in fact).

Indent should be 12 for Deployment (didn't need to update it) and 10 for statefulset ...

Description:
I have deployed zeebe cluster and zeebe operate separately, In the zeebe cluster, the elasticsearch is configured with the AWS ELK but facing issue with zeebe operate.

Configmap.yaml:
kind: ConfigMapmetadata:
name: {{ include "zeebe-operate.fullname" . }}
apiVersion: v1
data:
application.yml: |
# Operate configuration file
camunda.operate:
elasticsearch:
host: {{ .Values.global.elasticsearch.host }}
port: {{ .Values.global.elasticsearch.port }}
username: {{ .Values.global.elasticsearch.username }}
password: {{ .Values.global.elasticsearch.password }}
prefix: zeebe-record-operate

ERROR:
2020-06-29 05:35:10.397 ERROR 6 --- [ main] o.c.o.e.ElasticsearchConnector : Error occurred while connecting to Elasticsearch: clustername [elasticsearch], https://xx.xx.xx.x.x.x.x.ap-south-1.es.amazonaws.com:443. Will be retried...

java.io.IOException: https://xxxxx-xxxx-xxx-xx-xxx-x..ap-south-1.es.amazonaws.com: Name or service not known
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$670b527.CGLIB$esClient$0() ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$670b527$$FastClassBySpringCGLIB$$af2d84c1.invoke() ~[camunda-operate-common-0.23.0.jar!/:?]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$670b527.esClient() ~[camunda-o

Use the zeebe benchmark defaults

 # JavaOpts:
 # DEFAULTS
 JavaOpts: >-
   -XX:MaxRAMPercentage=25.0
   -XX:+ExitOnOutOfMemoryError
   -XX:+HeapDumpOnOutOfMemoryError
   -XX:HeapDumpPath=/usr/local/zeebe/data
   -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log
   -Xlog:gc*:file=/usr/local/zeebe/data/gc.log:time:filecount=7,filesize=8M

We should rework the Zeebe configmap, could be done similar to https://github.com/camunda-community-hub/camunda-cloud-operator/blob/main/operator/controllers/zeebe_controller.go

See related discussion #145 (comment) the gateway doesn't need this

When trying to install the zeebe-cluster-helm chart I am getting:

Error: admission webhook "validation.gatekeeper.sh" denied the request: [denied by autogke-no-write-mode-hostpath] hostPath volume proc used in container prometheus-node-exporter uses path /proc which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: ["/var/log/"]. Requesting user: <XXXX> and groups: <["system:authenticated"]>
[denied by autogke-no-write-mode-hostpath] hostPath volume sys used in container prometheus-node-exporter uses path /sys which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: ["/var/log/"]. Requesting user: <XXXXXX> and groups: <["system:authenticated"]>
[denied by autogke-no-host-port] container prometheus-node-exporter specifies a host port; disallowed in Autopilot. Requesting user: <XXXXXXX> and groups: <["system:authenticated"]>
[denied by autogke-disallow-hostnamespaces] enabling hostPID is not allowed in Autopilot. Requesting user: <XXXXX> and groups: <["system:authenticated"]>
[denied by autogke-disallow-hostnamespaces] enabling hostNetwork is not allowed in Autopilot. Requesting user: <XXXXX> and groups: <["system:authenticated"]>

I am trying to install zeebe via helm charts in a separate namespace, the same fails. I have capture the debug logs from helm command as below:

$ helm install zeebe zeebe/zeebe-full-helm -n zeebe --debug
install.go:178: [debug] Original chart version: ""
install.go:199: [debug] CHART PATH: /home/edgeprov/.cache/helm/repository/zeebe-full-helm-1.3.1.tgz

client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD alertmanagerconfigs.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD alertmanagers.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD podmonitors.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD probes.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD prometheuses.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD prometheusrules.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD servicemonitors.monitoring.coreos.com is already present. Skipping.
client.go:128: [debug] creating 1 resource(s)
install.go:151: [debug] CRD thanosrulers.monitoring.coreos.com is already present. Skipping.
W0117 18:44:32.951595 1150858 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission" ServiceAccount
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission" ClusterRole
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission" ClusterRoleBinding
client.go:328: [debug] clusterrolebindings.rbac.authorization.k8s.io "zeebe-ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission" Role
client.go:328: [debug] roles.rbac.authorization.k8s.io "zeebe-ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission" RoleBinding
client.go:328: [debug] rolebindings.rbac.authorization.k8s.io "zeebe-ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "zeebe-ingress-nginx-admission-create" Job
client.go:328: [debug] jobs.batch "zeebe-ingress-nginx-admission-create" not found
client.go:128: [debug] creating 1 resource(s)
client.go:528: [debug] Watching for changes to Job zeebe-ingress-nginx-admission-create with timeout of 5m0s
client.go:556: [debug] Add/Modify event for zeebe-ingress-nginx-admission-create: ADDED
client.go:595: [debug] zeebe-ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
helm.go:88: [debug] failed pre-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/[email protected]/command.go:902
main.main
helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371

My problem was that I wanted to setup Zeebe with operate for our benchmarks and
I saw that the throughput was significant lower when we use the zeebe-full instead of the zeebe-cluster chart. https://github.com/zeebe-io/zeebe-benchmark/issues/23

I used the same configuration for the full chart as for the cluster chart, the only difference was that I
put all properties under zeebe-cluster.

Like:

zeebe-cluster:
  image:
    repository: camunda/zeebe
    tag: SNAPSHOT
    pullPolicy: IfNotPresent

  # ZEEBE CFG

  clusterSize: 3
  partitionCount: 3
  replicationFactor: 3
  cpuThreadCount: 4
  ioThreadCount: 4

  ... etc.

I expected that this was the way to go since the README in this repo says the same https://github.com/zeebe-io/zeebe-full-helm. Furthermore I expected that this works that way because the chart is called liked that.

Luckily I found this conversation, which shows the usage of zeebe instead of zeebe-cluster.

This fixes also my benchmark setup and worked now as expected:

zeebe:
  image:
    repository: camunda/zeebe
    tag: SNAPSHOT
    pullPolicy: IfNotPresent

  # ZEEBE CFG

  clusterSize: 3
  partitionCount: 3
  replicationFactor: 3
  cpuThreadCount: 4
  ioThreadCount: 4

  ... etc.

We probably should add an example configuration or something like that how to configure this or we change the name to zeebe-cluster, this would things probably more clear.

https://docs.renovatebot.com/configuration-options/#bumpversion

One helm chart to rule them all.

Create one single chart which contains all applications as sub charts.

Todo:

• Create Parent chart #135
• Copy Zeebe cluster chart #135
  • Refactor old values file, introduce better sectioning/grouping
  • Iterate over README and properties documentation
  • First release https://github.com/camunda-community-hub/camunda-cloud-helm/releases/tag/ccsm-helm-0.0.1
• Copy Operate chart #139
  • Per default enabled
• Copy Tasklist chart #142
  • Per default enabled
• Separate Gateway as sub chart https://camunda.slack.com/archives/C02UT3T8DQR/p1643631277348409 #145
• Add Notes.txt - is show when installing helm chart https://codersociety.com/blog/articles/helm-best-practices#4-document-your-charts
• Disable ingress per default #150
• Document value files https://helm.sh/docs/chart_best_practices/values/#document-valuesyaml
  • Zeebe sub chart #141
  • Operate sub chart #143
  • Tasklist sub chart #145
• Add labels #147
  • #131 https://helm.sh/docs/chart_best_practices/labels/#standard-labels
• Add REAME.md #151
  • https://codersociety.com/blog/articles/helm-best-practices#4-document-your-charts
  • remove or replace the old ones
  • Document all values in Readme
• #149
  • #154
• Build a release https://github.com/camunda-community-hub/camunda-cloud-helm/releases/tag/ccsm-helm-0.0.10

Hello!

I would like to configure zeebe operate helm chart with an elasticsearch authentication. I configured url with this schema: https://<es_user>:<es_pass>@<es_host>:<es_port> but it does not work.

Do you have a solution to use elasticsearch auth with zeebe-operate helm chart ?

Thanks

Tried to setup zeebe-cluster using the operator. Everything seems to be working fine except gateway health check.

It's always throwing out_of_service response.

{"status":"OUT_OF_SERVICE","components":{"diskSpace":{"status":"UP","details":{"total":21462233088,"free":17460678656,"threshold":10485760,"exists":true}},"gatewayClusterAwareness":{"status":"UP"},"gatewayPartitionLeaderAwareness":{"status":"UP"},"gatewayResponsive":{"status":"UP","details":{"timeOut":"PT0.5S"}},"gatewayStarted":{"status":"UP"},"livenessDiskSpace":{"status":"UP","details":{"total":21462233088,"free":17460678656,"threshold":1048576,"exists":true}},"livenessGatewayClusterAwareness":{"status":"UP","details":{"derivedFrom":"ClusterAwarenessHealthIndicator","wasEverUp":true,"maxDowntime":"PT5M","lastSeenDelegateHealthStatus":{"status":"UP"}}},"livenessGatewayPartitionLeaderAwareness":{"status":"UP","details":{"derivedFrom":"PartitionLeaderAwarenessHealthIndicator","wasEverUp":true,"maxDowntime":"PT5M","lastSeenDelegateHealthStatus":{"status":"UP"}}},"livenessGatewayResponsive":{"status":"UP","details":{"derivedFrom":"ResponsiveHealthIndicator","wasEverUp":true,"maxDowntime":"PT10M","lastSeenDelegateHealthStatus":{"status":"UP","details":{"timeOut":"PT5S"}}}},"livenessMemory":{"status":"UP","details":{"threshold":0.01}},"livenessState":{"status":"UP"},"memory":{"status":"UP","details":{"threshold":0.1}},"readinessState":{"status":"OUT_OF_SERVICE"}},"groups":["liveness","readiness","startup"]}

Please refer to this slack thread

Sometimes on our benchmarks we also want to see metrics from elastic, for that normally the elastic-exporter is used

We configured it normally like this, in order to work with out Prometheus setup.

es:
  uri: "http://elasticsearch-master-headless:9200"
serviceMonitor:
  enabled: true
  labels:
    release: "metrics"

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Retry-After: unexpected status code 200

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

• #124
• #128
• #189
• #179
• #125
• camunda/camunda-docs#609
• Optional #127
• Optional #126 We will do this in upcoming quarter

https://github.com/camunda-cloud/zeebe-controller-k8s/pull/845#issuecomment-1035954492 should be done in the charts as well to mitigate a bug in operate

CAMUNDA_OPERATE_ARCHIVER_WAITPERIODBEFOREARCHIVING for operate 1.3.x (also alphas) to 2h.

Hello, I had a problem with default config for tasklist. The service did not work in default installation I had to change main object of config from zeebe to camunda.

From:

# Tasklist configuration file

zeebe.tasklist:
  # Set Tasklist username and password.
  # If user with <username> does not exists it will be created.
  # Default: demo/demo
---TRUNC---

To:

# Tasklist configuration file

camunda.tasklist:
  # Set Tasklist username and password.
  # If user with <username> does not exists it will be created.
  # Default: demo/demo
---TRUNC---

It seems that the current logging properties are out dated. I asked for feedback from the operate team. https://camunda.slack.com/archives/C9B5270DA/p1643803696900629

In the deployment guide it seems to reference only the log4j file, so we could do that. https://docs.camunda.io/docs/self-managed/operate-deployment/configuration/#logging

Originally posted by @Zelldon in #143 (comment)

Docs: https://docs.camunda.io/docs/self-managed/optimize-deployment/setup/
Docker Hub: https://hub.docker.com/r/camunda/optimize

Hi,
I wanted to deploy all pods related to Zeebe in a different node pools in AKS. I figured it out that I can run all the pods in one node pool except the tasklist and operate pods.
Is there anyway to set these values for Tasklist and operate pods?

I have used below values.yaml file for installing the zeebe-full-helm chart.

helm upgrade --namespace zeebe --install --wait --timeout 10m0s zb zeebe/zeebe-full-helm -f values.yaml

`
global:
zeebe: "{{ .Release.Name }}-zeebe"

zeebe-cluster-helm:
elasticsearch:
tolerations:
- key: "zeebe"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
nodeSelector:
zeebe: isolatenodepool
gateway:
tolerations:
- key: "zeebe"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
nodeSelector:
zeebe: isolatenodepool

tolerations:

• key: "zeebe"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"
  nodeSelector:
  zeebe: isolatenodepool

cloudevents:
enabled: false
tasklist:
enabled: true
zeeqs:
enabled: false

ingress-nginx:
controller:
tolerations:
- key: "zeebe"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
nodeSelector:
zeebe: isolatenodepool
service:
ports:
http: 80
https: 443
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

zeebe-tasklist-helm:
ingress:
enabled: true
annotations:
ingress.kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
path: /
host: tasklist.zeebe.weud
tls:
enabled: false
secretName:
nodeSelector:
zeebe: isolatenodepool
tolerations:

• key: "zeebe"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

zeebe-operate-helm:
ingress:
enabled: true
annotations:
ingress.kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
path: /
host: operate.zeebe.weud
tls:
enabled: false
secretName:
nodeSelector:
zeebe: isolatenodepool
tolerations:

• key: "zeebe"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"
  `

result in AKS cluster:

When using the Helm charts to demo or test something, it's sometimes useful to be able to inject sidecar containers, or extra init containers. Some operators (e.g. linkerd) do this for you through annotations, others require you to do this yourself (e.g. Jaeger - they support annotations as well, but only on deployments for whatever reason).

AT:

• add extraAnnotations value to allow specifying annotations for pods
• add extraContainers value to allow specifying sidecar containers for pods
• add extraInitContainers value to allow specifying additional init containers for pods (e.g. testing migration script, for example)
• add extraVolumes and extraVolumeMounts to allow injecting/overwriting files during deployment

At the moment I really only care to scope all of these for Zeebe, not sure if it makes sense to have these for other services; some of them might even have partial support already.

NOTE: see some examples here and here, and some discussion here

The config parameters are described in a bullet point list currently, but I personally would love to see an example file for the stupids (like me) who are not completely sure how to apply them exactly.

As you can see here I spontaneously messed it up: https://github.com/berndruecker/kafka-connect-zeebe-benchmark/pull/2/files ;-)

• create simple documentation on how to change the ingresses hosts when NGINX ingress controller is deployed
• create script that fetch the ingress external IP address and json patch all the existing ingresses

Currently we have the problem that we quite often add new features or try to fix bugs but we didn't test it automatically. It already often happens that on new releases our current benchmark didn't work anymore. I would like to introduce some tests for the helm charts.

Maybe we can make use of golden files https://medium.com/@jarifibrahim/golden-files-why-you-should-use-them-47087ec994bf

As a last chapter it would be awesome to see some kind of contribution guide/ section in the readme.
Maybe you could just link one created by @celanthe. I just found this page here. Nevertheless, Camunda Platform deals with it separately in their GitHub Repository.

Verify whether we have the standardized and recommended labels set https://helm.sh/docs/chart_best_practices/labels/#standard-labels

related to #124

Description

It's not currently possible to support non-root deployments with the Helm charts. In order to do so, we need to:

1. Allow users to configure the StatefulSet's .spec.template.securityContext so users can add runAsUser, runAsGroup, and fsGroup (and set it to 1000 as recommended value)
2. Update the volumes section and set the defaultMode of the config volume to 0777 instead of 0744.

We need to test our charts, to verify and ensure functionality and to guarantee that we not break them on adding more features or fixing bugs.

• Verify which tool fit best, ct or terratest
  • Template Test: Terratest
  • Template Test: Golden Files
  • IT: Terratest
  • IT: CT and Helm test
• Add github action to run go tests 778dfac and follow up commits
• Implement tests
  • Template tests:
    • #176
    • #188
    • #192
    • Configuration Tests for all conditionals
      • Tasklist Templates #192
      • Operate Templates #192
      • Zeebe Templates
      • Zeebe-Gateway Templates
  • Integration Test
    • Create a gha which can access the zeebe k8 cluster and run the it tests #212
      • Example for setup k8 config https://github.com/zeebe-io/zeebe-chaos/blob/master/chaos-workers/Dockerfile
      • Service account resources https://github.com/zeebe-io/zeebe-chaos/tree/master/chaos-workers/deployment/service-account
      • Kubeconfig https://github.com/zeebe-io/zeebe-chaos/blob/master/chaos-workers/kubeconfig
      • Add serviceaccount to zeebe-cluster https://github.com/zeebe-io/zeebe-chaos/tree/ccsm-helm-serviceaccount
      • Add token to github repo secrets
    • Create a test which tests the complete round trip, with default values #214
• #185
• #206

Right now Prometheus can be deployed with a Zeebe Cluster, but in order to access grafana you need to port forward to it, this can be simplified by enabling the ingress in the prometheus chart and making sure that our Ingress Controller expose the right paths. This has proved to be challenging and time-consuming but I am sure that it can be done.

Hi

I recently deployed zeebe-cluster-helm chart version 0.0.88 and as part of my testing noticed that the gateway service exposes an endpoint on port 9600, which seems to always return:

503 Service Unavailable
upstream connect error or disconnect/reset before headers. reset reason: connection failure

I know this endpoint is exposed by the Zeebe pods and used by Kubernetes to check when the pod is ready, but in the gateway deployment I don't think the 9600 endpoint is used and the gateway pod seems to be ready and working, despite the 503 error.

So, is the 9600 endpoint required on the gateway pod and if yes, what might be causing the 503 error?

I tested this endpoint by calling :9600/ready, perhaps for the gateway this endpoint is invalid?

I have checked the logs, which I can provide if it helps, but they simply report that a 503 Service Unavailable error was returned.

Worth mentioning... The reason I say the gateway pod is working is because when I request the Topology I get the following response:

{
  "brokers": [
    {
      "partitions": [],
      "nodeId": 0,
      "host": "workflow-engine-zeebe-0.workflow-engine-zeebe.ah-playground.svc.cluster.local",
      "port": 26501
    }
  ],
  "clusterSize": 1,
  "partitionsCount": 1,
  "replicationFactor": 1
}

So I have assumed the standalone gateway is talking to the Zeebe Cluster.

config:
Zeebe version 0.22.1

Zeebe Cluster:
clusterSize: 1
partitionCount: 1
replicationFactor: 1

ES
replicas: 1

Thanks
Andy

Description

Nodes in a Zeebe cluster use SWIM (i.e. gossip) to propagate information about other nodes. This is how topology is propagated, and how which node is subscribed to which notification event as well. Within SWIM, each node is identified by its memberId; if two nodes share the same memberId, then problems can arise as for the other nodes they are essentially the same. As you can imagine, that has some pretty bad consequences, such as breaking topology (e.g. incomplete topologies) or missing long polling notifications.

To fix this, we simply need to ensure every gateway replica gets a different memberId. This can be done by setting the env var ZEEBE_GATEWAY_CLUSTER_MEMBERID to the hostname or some such. I think at the moment it is unfortunately hardcoded to be zeebe-gateway, which could lead to the issues I've described above if someone uses more than one gateway replica.

Note that this does not affect embedded gateways.

/cc @salaboy @Zelldon

I tried:

zeebe-operate-helm:
  resources:
    limits:
      cpu: 14
      memory: 54Gi
    requests:
      cpu: 14
      memory: 32Gi

in zeebe-full-helm to configure Operate resources similar to how it is done for zeebe-cluster-helm. However, I saw that the deployment.yaml is having resources hard-coded. I suggest to do it similar to zeebe-cluster-helm's statefulset.yaml.

values.yml:

global:
  image:
    repository: camunda/zeebe
    tag: SNAPSHOT
    pullPolicy: Always
operate:
  enabled: true
  global:
    image:
      repository: camunda/operate
zeebe:
  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"
....

generates this deployment for Operate:

# Source: ccsm-helm/charts/operate/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: os-benchmark-1-operate
  labels:
    app: camunda-cloud-self-managed
    app.kubernetes.io/name: operate
    helm.sh/chart: operate-0.0.9
    app.kubernetes.io/instance: os-benchmark-1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.3.1
    app.kubernetes.io/part-of: camunda-cloud-self-managed
    app.kubernetes.io/component: operate
  annotations:
spec:
  replicas: 1
  selector:
    matchLabels: ...
  template:
    metadata:
      labels: ...
    spec:
      containers:
      - name: operate
        image: "camunda/zeebe:SNAPSHOT"
...

I'd expect the global.image.repository overwrite for Operate to work, but I'm not too familiar with Helm.

I've found the values.yml in the Zeebe benchmarks: https://github.com/camunda-cloud/zeebe/blob/main/benchmarks/setup/default/zeebe-values.yaml#L112-L116

We are using Zeebe with the elasticsearch exporter and the Operate dashboard. After upgrading from 1.0.1 to 1.3.3 operate does not show any processes from after the update and prints migrations errors.
I tried to start versions 1.1 and 1.2 for both the zeebe broker and operate dashboard to migrate the data, but it didn't help. It seems operate is unable to use any of the new data saved by the new broker version.
Zeebe or Elasticsearch do not show any errors and I found data in the index zeebe-record_process-instance_1.3.3_2022-02-03, so the data should be saved correctly in elasticsearch.
operate-logs.csv

Zeebe 1.3.3
Operate 1.3.3
Elasticsearch v7.14.1

The helm chart should be able to also deploy an curator job like:

CFG Map

[zell zeebe-benchmark/ ns:zell-helm]$ cat setup/curator-configmap.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: curator-config
  labels:
    app: curator
data:
  action_file.yml: |-
    ---
    # Remember, leave a key empty if there is no value.  None will be a string,
    # not a Python "NoneType"
    #
    # Also remember that all examples have 'disable_action' set to True.  If you
    # want to use this action as a template, be sure to set this to False after
    # copying it.
    actions:
      1:
        action: delete_indices
        description: "Clean up ES by deleting old indices"
        options:
          timeout_override:
          continue_if_exception: False
          disable_action: False
          ignore_empty_list: True
        filters:
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y-%m-%d'
          unit: days
          unit_count: 1
          field:
          stats_result:
          epoch:
          exclude: False
  config.yml: |-
    ---
    # Remember, leave a key empty if there is no value.  None will be a string,
    # not a Python "NoneType"
    client:
      hosts:
        - elasticsearch-master-headless
      port: 9200
      url_prefix:
      use_ssl: False
      certificate:
      client_cert:
      client_key:
      ssl_no_validate: False
      http_auth:
      timeout: 30
      master_only: False
    logging:
      loglevel: INFO
      logfile:
      logformat: default
      blacklist: ['elasticsearch', 'urllib3']

Cronjob

[zell zeebe-benchmark/ ns:zell-helm]$ cat setup/curator-cronjob.yaml 
# https://medium.com/@hagaibarel/running-curator-as-a-kubernetes-cronjob-19eaab9afd3b
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: curator
  labels:
    app: curator
spec:
  schedule: "0 3 * * *" # run every 03:00 AM daily
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - image: bobrik/curator:5.7.6
            name: curator
            args: ["--config", "/etc/config/config.yml", "/etc/config/action_file.yml"]
            volumeMounts:
            - name: config
              mountPath: /etc/config
          volumes:
          - name: config
            configMap:
              name: curator-config
          restartPolicy: OnFailure

Deploying the zeebe ful chart $ helm install test zeebe/zeebe-full-helm will depeloy a ingress controller, which should forward traffic to operate (?).

But accessing the external ip doesn't work and shows a bad gateway.

Reported by an user https://camunda-cloud.slack.com/archives/C6WGNHV2A/p1643705936901069

Workaround port forward traffic to operate service directly.

i have tried the zeebe-operate helm chart and wanted to override the resource part like it is possible in zeebe-cluster-helm. But here the resource part is hardcoded written in deployment.yaml so it is not possible. Would be good to have it defined the same way like in https://github.com/zeebe-io/zeebe-cluster-helm/blob/master/zeebe-cluster/templates/statefulset.yaml

Checkout https://github.com/camunda-community-hub/camunda-helm/tree/main/.chglog

Looks like imagePullSecrets support is missing from the helm chart.
I does work if you set the secrets in the ServiceAccount but it would be great if it is added for the pods as well.

Two use cases could be the following:

1. Creating custom zeebe images in a private image registry (eg. images with custom exporters added).
2. Use initContainers from a private image registry.

Add Identity Application to the single chart. Dependency of #126.

• Add Keycloak dependency #251
  • https://github.com/codecentric/helm-charts/tree/master/charts/keycloak
  • https://bitnami.com/stack/keycloak/helm (might be prefered)
  • Add postgress
• Add Identity as new sub chart #251
  • configure identity to access keycloak
  • Write tests for identity templates see https://github.com/camunda/camunda-cloud-helm/blob/main/CONTRIBUTING.md
• Postponed Bonus: Add TLS support #256
• Adjust README and add identity section (document all values in readme and values file) #252
• Adjust existing services, like tasklist and operate to use identity
  • Adjust operate #268
  • Adjust Tasklist #270
• Add all possible variables to the values.yaml
• Write integration tests #258
  • find a new way to login, we need to login to identity see https://camunda.slack.com/archives/CRQAL5A1M/p1648561004641359
• Update architecture image, you can find the source in google drive. zeebe/ccsm-helm/CCSM Helm Architecture Image. https://camunda.slack.com/archives/C02UT3T8DQR/p1648563744879099
  • c5fd159
• Add documentation about identity to the camunda docs camunda/camunda-docs#689
• Configuration:
  • #266
  • #267 we can do this later
  • Make server port configurable See currently no need for it, configuring this would also break the services etc.

k8s 1.21 says: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget

Examples:

• https://github.com/camunda-cloud/zeebe/blob/main/benchmarks/setup/default/curator-cronjob.yaml
• https://github.com/camunda-cloud/zeebe-controller-k8s/blob/main/templates/elasticsearch_curator_cronjob.yaml

AT:

• Default disabled
• When enabled cronjob run every day 0 0 * * * https://crontab.guru/every-day
• Configurable
• Add first config

Hi,

In camunda-community-hub/zeebe-helm@af79d0c#diff-8bc94a3006768345d17c827c0bb5840e6e4daee6de1c2b1ea672f53f162d171dL56, the url of the chart repository was changed, but that wasn't reflected in the README here, nor in other repositories (zeebe-operate, zeebe-cluster, ...).

The old URL (helm.zeebe.io) returns 404.

It's not possible to deploy the zeebe-operate-helm-chart to K8S-Clusters with a version < 1.19.0 due to an incompatibility with the ingress's apiVersion.

Is this by design because you don't want to support K8S-versions which are EOL with this chart?
Otherwise I'd like to provide a "fix" for that by adjusting the ingress.yaml as it is created per default by $ helm create chart:

# [...]
{{- if .Values.ingress.enabled -}}
# [...]
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
  {{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
  {{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
  {{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
# [...]
spec:
  {{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
  ingressClassName: {{ .Values.ingress.className }}
  {{- end }}
# [...]

$ k version
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.17", GitCommit:"68b4e26caf6ede7af577db4af62fb405b4dd47e6", GitTreeState:"clean", BuildDate:"2021-03-18T00:54:02Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ helm upgrade --install --namespace ${K8S_NAMESPACE} camundacloud/zeebe-operate-helm .
Error: UPGRADE FAILED: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1"