Is your feature request related to a problem? Please describe.
I'd like to be able to add an annotation to the Service that the EmqxBroker generates such as: coredns.io/hostname: "mqtt.example.com". This annotation specifically in my case causes coredns to add a DNS record resolving mqtt.example.com. to the ip address of the service.

Describe the solution you'd like
I think the spec.emqxTemplate.listener object on the EmqxBroker should support an annotations field which would then be used when generating the Service around here.

Describe alternatives you've considered
Alternatives would require that my load balancer ip is static and then manually adding the DNS record to core dns.
Or manually crafting a custom Service to run side by side with the one produced by the operator and effectively just leave the generated Service un-used.

What happened?

The EMQX crd has issues with CD systems like FluxCD or ArgoCD that do server-side apply dry-runs to validate changes for reconciliation.

What did you expect to happen?

All potential fields on the CRD are on the schema so that server side dry-runs succeed with zero changes (or configured to ignore said changes)

How can we reproduce it (as minimally and precisely as possible)?

Apply a broker crd:

kubectl apply --server-side -f broker.yaml

then run:

kubectl apply --server-side -f broker.yaml --dry-run=server

The response is likely to be:

Error from server: .spec.emqxTemplate.serviceTemplate.metadata.creationTimestamp: field not declared in schema

Anything else we need to know?

Warning  ReconciliationFailed     91s (x3 over 11m)  kustomize-controller  EmqxBroker/mqtt/emqx dry-run failed, error: .spec.emqxTemplate.serviceTemplate.metadata.creationTimestamp: field not declared in schema

This is the message directly from Flux, but the command in the reproduction steps is all that FluxCD is doing behind the covers.

EMQX version

bash-5.1$ emqx_ctl broker
sysdescr  : EMQ X Broker
version   : 4.4.0
uptime    : 17 minutes, 8 seconds
datetime  : 2022-07-29 00:18:05

OS version

# On Linux:
bash-5.1$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.3
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
$ uname -a
Linux emqx-0 5.15.45-1-virt #2-Alpine SMP Mon, 06 Jun 2022 10:19:37 +0000 x86_64 Linux

Note this is just a worker node in my K3s kubernetes cluster.

Log files

in v1beta sample:

spec:
  serviceAccountName: emqx
  image: emqx/emqx:4.3.11
  replicas: 3
  labels:
    cluster: emqx
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: standard
        resources:
          requests:
            storage: 20Mi
        accessModes:
        - ReadWriteOnce

in v1beta2 sample:

spec:
  serviceAccountName: "emqx"
  image: emqx/emqx:4.3.11
  replicas: 3
  labels:
    cluster: emqx
  storage:
    storageClassName: standard
    resources:
      requests:
        storage: 20Mi
    accessModes:
    - ReadWriteMany

Not sure if this is a bug but this is something I am trying to get through and not sure what I can do to get past the error.

I feel like I am close but I am not sure if I am just terminating wrong with the ingress controller and what I am trying is not possible. Or, the certificate is in the way and I need to do something to alleviate the issue.

Another idea I see is using a HAPROXY LB in front of the ingress-nginx controller.

First, I am using the ingress Nginx tcp/udp controller for Kubernetes described here with an Letsencrypt RA CA

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

This allows me through my HELM installation to create a tcp config and mapping.

 helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $NAMESPACE --set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path=/healthz,controller.service.annotations.service\.beta\.kubernetes\.io/azure-dns-label-name=$DNS_LABEL --set controller.service.loadBalancerIP=$STATIC_IP --set tcp.18083=$NAMESPACE/emqx-ee:18083,tcp.8883=$NAMESPACE/emqx-ee:8883

That does allow for an mqtts:// ssl connection to go through all the way to the 8883 backend pod. The issue is when I want to terminate at the loadbalancer and send the resulting incoming traffic to terminate the TLS and go to the TCP 1883 port.

To try this I change the port tcp definition

from this:

'8883': ingress-emqx/emqx-ee:8883

to this:

'8883': ingress-emqx/emqx-ee:1883

When I connect the client mqttx to 8883 CA I get the resulting error:

Error: Client network socket disconnected before secure TLS connection was established

What exactly does the termination for ingress-nginx? Is it only the ingress instruction rule or is it the controller TCP Proxy Protocol?

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
https://kubernetes.github.io/ingress-nginx/examples/tls-termination/

Is the only way to terminate is to utilize the ingress instruction/rewrite rule? If so, I think this is where the controller won't work because the route ingress rule only works with http and won't act upon mqtt layer-7 protocol.

Is there a way for the ingress to align with the ingress-controller service? I'm not sure it even matters to the previous issue. I simply want to take the incoming 8883 port and switch it to the resulting 1883 port and IP.

What happened?

If I need to change a setting inside bootstrapConfig using EMQX Operator I need to delete the cluster and create it again.
Apart from this really annoying behavior, if I enable persistence for core node, the cluster did not get created anymore, only core node is deployed and the operator stuck on replicant creation.
To start it again I need to delete the persistent volume loosing all the settings.

What did you expect to happen?

Change to bootstrapConfig should be allowed and StatefulSet/Deployment should be re-deployed.

How can we reproduce it (as minimally and precisely as possible)?

Create a cluster with 1 core and 3 replicant. Core should have a persistent volume enabled.
Delete the cluster (leaving the volume), change any setting, re-deploy the cluster, only core node get deployed.
Replicant and listeners service are not re-created.

Anything else we need to know?

No response

EMQX version

$ ./bin/emqx_ctl broker
sysdescr  : EMQX
version   : 5.0.13
datetime  : 2023-01-02T14:38:25.510136161+00:00
uptime    : 6 minutes, 56 seconds

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux emqx-core-0 5.4.172-90.336.amzn2.x86_64 emqx/emqx#1 SMP Wed Jan 19 23:08:01 UTC 2022 x86_64 GNU/Linux

Log files

Hello,

I am trying to get the operator running on an openshift installation but I face an TLS issue between Kube Api Server Operator and webhooks back to the EMQX Operator (or so it seems)

I installed the operator using the provided Helmchart, but disabled cert-manager support (setting it to false in values.yaml), because Openshift comes with similar functionality.
In particular, I created a TLS-Secret for the webhook-server by annotating its Service (the one that the helm chart creates) with

annotations:
    service.beta.openshift.io/serving-cert-secret-name: emqx-operator-webhook-server-cert

which triggers openshifts certificate system to create a secret of the specified name and populate it with a tls.crt (certificate chain) and a tls.key (private key).

Also on the webhook side, the ca-bundle can be injected by annotating the validating webhook resource and also the mutating webhook resource (both webhook resources that have been created by helm) with

  annotations:
    service.beta.openshift.io/inject-cabundle: 'true'

which populates the webhooks[].clientConfig.caBundle field with the ca cert that corresponds to the above service cert.

(I of course realize, that upon upgrading the helmchart, I will have to do that again, unless the helm chart could support an openshift mode instead of cert-manager)

However I keep getting errors on both sides:
On kube-apiserver-operator side I get many errors like

E1005 08:24:39.214279 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:40.222487 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:42.241517 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:43.248170 1 degraded_webhook.go:128] x509: certificate signed by unknown authority

and on the emqx operator I keep getting

2022/10/05 08:28:37 http: TLS handshake error from 10.130.5.86:40936: remote error: tls: bad certificate
2022/10/05 08:28:38 http: TLS handshake error from 10.130.5.86:40952: remote error: tls: bad certificate
2022/10/05 08:28:39 http: TLS handshake error from 10.130.5.86:40968: remote error: tls: bad certificate
2022/10/05 08:28:41 http: TLS handshake error from 10.130.5.86:39222: remote error: tls: bad certificate

where 10.130.5.86 is the kupe-apiserver-operator pod

I also tried getting the webhook-server certificate by using openssl tool to connect to it via the service and a port-forward. This seem to giving me the expected response.

Now upon creation of a EMQX ressource, the emqx operator pod starts crashlooping, without any apparent error in its log.

I1005 09:07:21.179399 1 request.go:601] Waited for 1.003849249s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/org.eclipse.che/v2?timeout=32s
2022-10-05T09:07:23Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/validate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO setup starting manager
2022-10-05T09:07:23Z INFO controller-runtime.webhook.webhooks Starting webhook server
2022-10-05T09:07:23Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2022-10-05T09:07:23Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
2022-10-05T09:07:23Z INFO controller-runtime.certwatcher Updated current TLS certificate
I1005 09:07:23.976315 1 leaderelection.go:248] attempting to acquire leader lease 5g-sc/19fd6fcc.emqx.io...
2022-10-05T09:07:23Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
2022-10-05T09:07:23Z INFO controller-runtime.certwatcher Starting certificate watcher
2022/10/05 09:07:32 http: TLS handshake error from 10.130.5.86:41042: remote error: tls: bad certificate
2022/10/05 09:07:33 http: TLS handshake error from 10.130.5.86:41050: remote error: tls: bad certificate
2022/10/05 09:07:35 http: TLS handshake error from 10.130.5.86:41060: remote error: tls: bad certificate
2022/10/05 09:07:38 http: TLS handshake error from 10.130.5.86:41068: remote error: tls: bad certificate
2022/10/05 09:07:39 http: TLS handshake error from 10.130.5.86:44646: remote error: tls: bad certificate
2022/10/05 09:07:41 http: TLS handshake error from 10.130.5.86:44650: remote error: tls: bad certificate
2022/10/05 09:07:54 http: TLS handshake error from 10.130.5.86:37628: remote error: tls: bad certificate
2022/10/05 09:07:55 http: TLS handshake error from 10.130.5.86:37634: remote error: tls: bad certificate

I am running out of ideas to what to check next.
Was emqx run on Openshift before?
Is there a way to skip ssl verification on the emqx ("insecure" flag)?
Any other ideas?

Kind regards

Discussed in #471

admission webhook "validator.apps.emqx.io" denied the request: bootstrap config cannot be updated, old bootstrap config: 

Describe the bug
When I try to deploy a v5.0.6 emqx image the resulting statefulset does not properly create emqx cluster.
It is like the operator v1.2.5 is not designed to operate emqx v5 broker images. So my question: is this a bug or it is just not designed for v5 yet ?

To Reproduce
deploy something like:

apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  name: emqx
  namespace: emqx-operator-system
spec:
  replicas: 3
  emqxTemplate:
    image: emqx/emqx:5.0.6
    serviceTemplate:
      metadata:
        name: emqx
        labels:
          app: emqx
      spec:
        selector:
          app: emqx
        type: ClusterIP
        ports:
          - name: mqtt
            port: 1883
            protocol: TCP
            targetPort: 1883
          - name: mqttssl
            port: 8883
            protocol: TCP
            targetPort: 8883
          - name: dashboard
            port: 18083
            protocol: TCP
            targetPort: 18083
    config:
      prometheus.enable: "true"
    modules:
      - name: emqx_mod_trace
        enable: true
  persistent:
    resources:
      requests:
        storage: 2Gi
    accessModes:
      - ReadWriteOnce

After the broker starts I have some warnings in logs and autocluster is not working. Every broker node has a cluster size of 1 -> they don't form a cluster.

emqx [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP",                                                                               
emqx                    "EMQX_CLUSTER__DNS__TYPE",                                                                                                                  
emqx                    "EMQX_DASHBOARD__DEFAULT_USER__LOGIN",                                                                                                      
emqx                    "EMQX_DASHBOARD__DEFAULT_USER__PASSWORD",                                                                                                   
emqx                    "EMQX_PLUGINS__ETC_DIR","EMQX_PLUGINS__LOADED_FILE"]                                                                                        
emqx prometheus.enable = EMQX_PROMETHEUS__ENABLE = true                                                                                                             
emqx rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual                                                                                                         
emqx log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false                                                                            
emqx log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true                                                                                          
emqx cluster.dns.name = EMQX_CLUSTER__DNS__NAME = emqx-headless.emqx-operator-system.svc.cluster.local                                                              
emqx node.name = EMQX_NODE__NAME = [email protected]                                                                                                                 
emqx 2022-08-28T10:28:07.426244+00:00 [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP", "EMQX_CLUSTER__DNS__TYPE", "EMQX_DASHBOARD__D
emqx Listener ssl:default on :8883 started.                                                                                                                         
emqx Listener tcp:default on :1883 started.                                                                                                                         
emqx Listener ws:default on :8083 started.                                                                                                                          
emqx Listener wss:default on :8084 started.                                                                                                                         
emqx Listener http:dashboard on :18083 started.                                                                                                                     
emqx EMQX 5.0.6 is running now! 

Expected behavior
The broker starts and the number of nodes in the cluster via emqx dashboard is 3.

Anything else we need to know?:
It's like the operator v1.2.5 is not configuring proper env variables for broker v5. That's why logs show some warnings.

Environment details::

• Kubernetes version: 1.21.10
• Cloud-provider/provisioner: self-hosted
• emqx-operator version: 1.2.5
• Install method: helm

Is your feature request related to a problem? Please describe.
There does not seem to be a way to use pre-existing certificates in kind: EmqxBroker

Describe the solution you'd like
Have an option to define an existing secret containing a TLS certificate for the secure ports

Describe alternatives you've considered
I can't find one right now in the information available on the kind: EmqxBroker. An other way of doing this would be to be able to specifiy an existing cluster issuer and dns names for automatic generation of the needed certs.

Related issue emqx/emqx#6337

Describe the bug
I installed emqx-operator with helmfile in namespace emqx-operator (instead of emqx-operator-system as described in README). I configured EMQX crd resource and while installing I got following error:

COMBINED OUTPUT:
  Release "emqx" does not exist. Installing it now.
  Error: release emqx failed, and has been uninstalled due to atomic being set: Internal error occurred: failed calling webhook "mutating.apps.emqx.io": failed to call webhook: Post "https://emqx-operator-webhook-service.emqx-operator-system.svc:443/mutate-apps-emqx-io-v2alpha1-emqx?timeout=10s": service "emqx-operator-webhook-service" not found

To Reproduce
Steps to reproduce the behavior:

1. Spin fresh k8s cluster
2. Install emqx operator to emqx-operator namespace

    $ helm install emqx-operator emqx/emqx-operator \
      --namespace emqx-operator\
      --create-namespace

3. Apply EMQX resource

cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
 name: emqx
spec:
 image: emqx/emqx:5.0.6
EOF

Expected behavior
Looks like the namespace of emqx-operator is hardcoded somewhere. Changing namespace should not break the instalation.

Anything else we need to know?:
Should get the namespace of emqx-operator within the helm chart with - .Values.Release.Namespace

Environment details::

• Kubernetes version: v1.23.10
• Cloud-provider/provisioner: Linode
• emqx-operator version: 2.0.0
• Install method: Helm + Helmfile

Describe the bug
Emqx core node fails to start on k8s after deletion and recreation with persistent volume. The goal was to deploy a broker that preserves the settings (dashboard users, mqtt clients auth & authz) even if the EMQX respource is deleted.

To Reproduce
`1. Install emqx-operator via Helm in k8s
2. Deploy a basic emqx broker with volumeClaimTemplate
4. Wait for it to start, change login via dashboard and change password.l
5. delete EMQX resource and recreate it

One of three core nodes doesn't start and is in error.

Expected behavior
All the pods start, both core and replicants nodes.

Anything else we need to know?:
emqx yaml file:

apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
  namespace: emqx-operator-system
spec:
  image: emqx/emqx:5.0.8
  coreTemplate:
    metadata:
      name: emqx-core
      labels:
        apps.emqx.io/instance: emqx
        apps.emqx.io/db-role: core
    spec:
      replicas: 3
      volumeClaimTemplates:
        storageClassName: longhorn-single-replica
        resources:
          requests:
            storage: 512Mi
        accessModes:
          - ReadWriteOnce
      livenessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 60
        periodSeconds: 30
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /status
          port: 18083
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 12
      podSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: Always
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
      lifecycle:
        preStop:
          exec:
            command: [ "/bin/sh","-c","emqx ctl cluster leave" ]

Environment details::

• Kubernetes version: v1.21.10
• Cloud-provider/provisioner: hosted
• emqx-operator version: 2.0.1
• Install method: helm

Logs:

dashboard.bootstrap_users_file = EMQX_DASHBOARD__BOOTSTRAP_USERS_FILE = "/opt/emqx/data/bootstrap_user"                                                                                            
rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual                                                                                                                                             
log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false                                                                                                                
log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true                                                                                                                              
node.db_role = EMQX_NODE__DB_ROLE = core                                                                                                                                                           
node.name = EMQX_NODE__NAME = emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local                                                                                                
2022-10-18T11:03:36.662853+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned
_network, 'emqx@emqx-core-1.emqx-headless.emqx-operator-system.svc.cluster.local'}                                                                                                                 
2022-10-18T11:03:36.663234+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned
_network, 'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local'}                                                                                                                 
2022-10-18T11:03:36.854583+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** (core dumped to file: "/opt/emqx/MnesiaCore.emqx@emqx-core-0.
emqx-headless.emqx-operator-system.svc.cluster.local_1666_91016_807092"), ** FATAL ** Failed to merge schema: Incompatible schema cookies. Please, restart from old backup.'[email protected]
eadless.emqx-operator-system.svc.cluster.local' = [{name,schema},{type,set},{ram_copies,[]},{disc_copies,['emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-1
.emqx-headless.emqx-operator-system.svc.cluster.local']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,sch
ema},{attributes,[table,cstruct]},{user_properties,[{mnesia_backend_types,[{rocksdb_copies,mnesia_rocksdb},{null_copies,mria_mnesia_null_storage}]}]},{frag_properties,[]},{storage_properties,[]},
{cookie,{{1666087950880376159,-576460752303423101,1},'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local'}},{version,{{7,0},{'[email protected]
m.svc.cluster.local',{1666,88190,221401}}}}], 'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local' = [{name,schema},{type,set},{ram_copies,[]},{disc_copies,['[email protected]
mqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-1.emqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local
']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,schema},{attributes,[table,cstruct]},{user_properties,[{
mnesia_backend_types,[{rocksdb_copies,mnesia_rocksdb},{null_copies,mria_mnesia_null_storage}]}]},{frag_properties,[]},{storage_properties,[]},{cookie,{{1666086344331374145,-576460752303423202,1},
'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'}},{version,{{8,0},{'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local',{1666,86350,531401}}}}]         
2022-10-18T11:03:46.809511+00:00 [error] Generic server mnesia_subscr terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,#Ref<0.1684304479.1029832724.
194029>}.                                                                                                                                                                                          
2022-10-18T11:03:46.809864+00:00 [error] Generic server mnesia_monitor terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,[],[],true,[],undefined,[],[
]}.                                                                                                                                                                                                
2022-10-18T11:03:46.809510+00:00 [error] Generic server mnesia_recover terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,undefined,undefined,undefine
d,0,false,true,[]}.                                                                                                                                                                                
2022-10-18T11:03:46.811065+00:00 [error] crasher: initial call: mnesia_subscr:init/1, pid: <0.1832.0>, registered_name: mnesia_subscr, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.e
rl"},{line,481}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.1826.0>], message_queue_len: 0, messages: [], links: [], dictionar
y: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 6297; neighbours:                                                                                            
2022-10-18T11:03:46.810244+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.1825.0>, registered_name: [], exit: {{normal,{mnesia_app,start,[normal,]]}},[{application_master
,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.1824.0>], message_queue_len: 1, messages: [{'EXIT',<0.1826.
0>,normal}], links: [<0.1824.0>,<0.1685.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 29, reductions: 167; neighbours:                                        
2022-10-18T11:03:46.811386+00:00 [error] crasher: initial call: gen_event:init_it/6, pid: <0.1828.0>, registered_name: mnesia_event, exit: {killed,[{gen_event,terminate_server,4,[{file,"gen_event
.erl"},{line,405}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_sup,<0.1826.0>], message_queue_len: 1, messages: [{notify,{mnesia_system_event,{mnesia_do
wn,'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'}}}], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 10958, stack_size: 29, reductions: 27853; n
eighbours:                                                                                                                                                                                         
2022-10-18T11:03:46.813591+00:00 [error] crasher: initial call: mnesia_monitor:init/1, pid: <0.1831.0>, registered_name: mnesia_monitor, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server
.erl"},{line,481}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.1826.0>], message_queue_len: 0, messages: [], links: [<53724.183
1.0>,<53725.3553.0>,<0.1873.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 29, reductions: 13705; neighbours:                                                 
2022-10-18T11:03:46.813855+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.1816.0>, registered_name: [], exit: {{bad_return,{{mria_app,start,[normal,]]}

After acl.conf is mounted to /mounted/acl/acl.conf by the operator, is the native ACL HTTP API unavailable?

At the same time, after changing the acl configuration by modifying the acl field under the spec, you need to reload the acl module before the corresponding rules take effect. is that so?

• emqx-operator version: 1.2.1

Describe the bug
I am trying to apply some pod annotations to the controller manager. My values contains

podAnnotations:
  ad.datadoghq.com/manager.tags: |
    {"env":"dev","service":"operator","version":"2.0.0"}

To Reproduce
I get an YAML parse error with version emqx-operator 2.0.0, helm v3.7.2, kubectl 1.22.0
Error: YAML parse error on emqx-operator/templates/controller-manager.yaml: error converting YAML to JSON: yaml: line 26: did not find expected key helm.go:88: [debug] error converting YAML to JSON: yaml: line 26: did not find expected key YAML parse error on emqx-operator/templates/controller-manager.yaml

It would appear that the indentation on the line https://github.com/emqx/emqx-operator/blob/main/deploy/charts/emqx-operator/templates/controller-manager.yaml#L32 is incorrect. The output produced is

  template:
    metadata:
        annotations:
        ad.datadoghq.com/haproxy.tags:
          env: dev
          service: operator
          version: 2.0.0

I would expect

  template:
    metadata:
      annotations:
        ad.datadoghq.com/haproxy.tags:
          env: dev
          service: operator
          version: 2.0.0

Describe the bug

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227                                                                                                               
1.6530916833830116e+09    ERROR    controller.emqxbroker    Reconciler error    {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "test-emqx", "namespace": "default", "error": "S
tatefulSet.apps \"test-emqx\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile                                                                                                                                 
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114                                                                                                               
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                                                                                          
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311                                                                                                               
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                                                                                                                       
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266                                                                                                               
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                                                                                                                             
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Anything else we need to know?:

Environment details::

• Kubernetes version: v1.20.8
• Cloud-provider/provisioner:
• emqx-operator version: v1.1.8
• Install method: helm

Describe the solution you'd like
We install emqx-operator by helm or yaml, but both have no method to setting the private registry for emqx-operator-reloader image.

Describe alternatives you've considered
the emqx-operator-reloader image's registry should be same to emqx-operator-controller's registry.

Describe the bug
After following the getting-started page to setup the emqx-operator I provisioned a emqx-cluster.
The pods start and are running, but the status commands return errors:

kubectl exec -n emqx -it emqx-0 -c emqx -- emqx_ctl status
Node '[email protected]' not responding to pings.
/opt/emqx/bin/emqx: line 46: die: command not found
command terminated with exit code 127

To Reproduce
Steps to reproduce the behavior:

1. Deploy the operator to a EKS cluster with Kubernetes 1.22.9
2. Deploy a simple broker (can be without persistence, I tested that.)
3. Check the output of the status commands and get errors.

Expected behavior
Not to get errors on the status commands after provisioning a simple broker with no config.

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.22.9
• Cloud-provider/provisioner: AWS EKS
• emqx-operator version: 1.2.4
• Install method: helm, emqx deployed as crd
  emqx-manifest:

---
apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  name: emqx
  labels:
    app: emqx
    environment: dev
spec:
  persistent:
    accessModes:
      - ReadWriteOnce
    storageClassName: ebs-gp3
    resources:
      requests:
        storage: 1Gi
  emqxTemplate:
    image: emqx/emqx:4.4.6

Did I do something wrong here?

Describe the bug
I'm getting the following error when attempting deploy an EmqxBroker:

1.647123650787955e+09    ERROR    controller.emqxbroker    Reconciler error    {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "emqx2", "namespace": "home-iot", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}

The service does not get created, nor does the statefulset. I do see the headless service get created, but no ports are on it.

To Reproduce
Here's the resource I am applying (note: I'm using Flux CD which replaces things like ${SECRET_CLUSTER_DOMAIN}):

---
apiVersion: apps.emqx.io/v1beta2
kind: EmqxBroker
metadata:
  name: &name emqx2
  namespace: home-iot
spec:
  image: emqx/emqx:4.4.0
  replicas: 3
  labels:
    cluster: *name
  env:
    - name: EMQX_ALLOW_ANONYMOUS
      value: 'true'
    - name: EMQX_ADMIN_PASSWORD
      value: "${SECRET_EMQX_ADMIN_PASSWORD}"
    - name: EMQX_AUTH__MNESIA__PASSWORD_HASH
      value: sha256
    - name: EMQX_AUTH__USER__1__USERNAME
      value: "${SECRET_MQTT_USERNAME}"
    - name: EMQX_AUTH__USER__1__PASSWORD
      value: "${SECRET_MQTT_PASSWORD}"
  storage:
    storageClassName: rook-ceph-block
    resources:
      requests:
        storage: 20Mi
    accessModes:
      - ReadWriteOnce
  emqxTemplate:
    listener:
      type: LoadBalancer
      loadBalancerIP: 192.168.69.61
      ports:
        mqtt: 1883
        mqtts: 8883
        ws: 8083
        wss: 8084
        dashboard: 18083
        api: 8081
      annotations:
        external-dns.alpha.kubernetes.io/hostname: "mqtt2.${SECRET_CLUSTER_DOMAIN}"
        coredns.io/hostname: "mqtt2.${SECRET_CLUSTER_DOMAIN}"
    acl:
      - permission: allow
        username: "dashboard"
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
            - "#"
      - permission: allow
        ipaddress: "127.0.0.1"
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: deny
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: allow
    plugins:
      - name: emqx_management
        enable: true
      - name: emqx_recon
        enable: true
      - name: emqx_retainer
        enable: true
      - name: emqx_dashboard
        enable: true
      - name: emqx_telemetry
        enable: true
      - name: emqx_rule_engine
        enable: true
      - name: emqx_bridge_mqtt
        enable: false
    modules:
      - name: emqx_mod_acl_internal
        enable: true
      - name: emqx_mod_presence
        enable: true
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: apps.emqx.io/instance
                  operator: In
                  values:
                    - *name
            topologyKey: kubernetes.io/hostname

Expected behavior
Broker and service gets created.

Anything else we need to know?:
Full logs:

1.6471236501591592e+09    INFO    emqx-controller    Reconciling    {"Request.Namespace": "home-iot", "Request.Name": "emqx2"}
1.6471236503776915e+09    ERROR    emqx-controller    crate resource failed    {"groupVersionKind": "/v1, Kind=Service", "namespace": "home-iot", "name": "emqx2", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
github.com/emqx/emqx-operator/controllers/apps.(*Handler).createOrUpdate
    /workspace/controllers/apps/ensure.go:54
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Ensure
    /workspace/controllers/apps/ensure.go:25
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Do
    /workspace/controllers/apps/handler.go:53
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Reconcile
    /workspace/controllers/apps/reconcile.go:99
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6471236503984983e+09    ERROR    emqx-controller    Reconcile handler    {"Request.Namespace": "home-iot", "Request.Name": "emqx2", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6471236503985639e+09    ERROR    controller.emqxbroker    Reconciler error    {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "emqx2", "namespace": "home-iot", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

Environment details::

• Kubernetes version: 1.22.4
• Cloud-provider/provisioner: Baremetal machines running on Talos OS
• emqx-operator version: 1.1.5
• Install method: e.g. helm/static manifests: Operator is installed as a Helm release. The Broker is a static manifest.

Describe the bug

The installation of emqx-operator worked fine, but on try install emqx cluster the follow error happened:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "mutating.broker.emqx.io": failed to call webhook: Post "https://emqx-operator-webhook-service.emqx-operator-system.svc:443/mutate-apps-emqx-io-v1beta4-emqxbroker?timeout=10s": context deadline exceeded

Output if emqx-operator is running:

$ kubectl get pods -l "control-plane=controller-manager" -n emqx-operator-system
NAME                                                READY   STATUS    RESTARTS   AGE
emqx-operator-controller-manager-6496bd4c9d-rg8hx   1/1     Running   0          158m

Below the codes tried to get emqx cluster running as described here: https://github.com/emqx/emqx-operator/blob/main/docs/en_US/deployment/on-gcp-gke.md

Emqx Cluster 5.0

cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
spec:
  image: "emqx:5.0"
  coreTemplate:
    spec:
      volumeClaimTemplates:
        storageClassName: standard
        resources:
          requests:
            storage: 20Mi
        accessModes:
        - ReadWriteOnce
  dashboardServiceTemplate:
    spec:
      type: LoadBalancer
  listenersServiceTemplate:
    spec:
      type: LoadBalancer
EOF

Emqx Cluster 4.4.15

cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
  name: emqx
spec:
  persistent:
    metadata:
      name: emqx
    spec:
      storageClassName: standard
      resources:
        requests:
          storage: 20Mi
      accessModes:
        - ReadWriteOnce
  template:
    spec:
      emqxContainer:
        image:
          repository: emqx/emqx
          version: 4.4.15
  serviceTemplate:
    spec:
      type: LoadBalancer
EOF

We also tried different versions of emqx-operator.

To Reproduce
Theses documentations resulted on same error:
https://docs.emqx.com/en/emqx-operator/latest/getting-started/getting-started.html
https://github.com/emqx/emqx-operator/blob/main/docs/en_US/deployment/on-gcp-gke.md

Expected behavior

Emqx cluster ready using the emqx-operator.

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.24.12-gke.500
• Cloud-provider/provisioner: GCP
• emqx-operator version: 2.1.1
• cert-manager: 1.11.1
• Install method emqx: helm

I am trying to use the operator to accomplish setting up an ingress controller so I can outbound access the mqtt broker and dashboard.

I am using Azure AKS with pretty much the default setup from the EQMX website.
https://docs.emqx.com/en/enterprise/v4.4/getting-started/install-on-k8s.html

Because ingress is setup to natively take on http traffic you have to modify the controller to handle TCP traffic such as MQTT 1883.

This stack has a pretty good overview on how you should do this.
https://stackoverflow.com/questions/61430311/exposing-multiple-tcp-udp-services-using-a-single-loadbalancer-on-k8s

The problem is, I don't really have any documentation on how I can setup and edit the custom resource. Are there specific instructions for this?

When I try to make an edit I am always getting unknown field use validate=false. So I don't think that is the right way I am trying to edit the implied intention of what it is I want to do.

I am trying to use a configmap to map the tcp setting to the container pods. or in anyway map the ports from the ingress controller.

An example of setting up a configmap is here. https://emqx.medium.com/building-k8s-cluster-of-emq-x-mqtt-broker-starting-from-scratch-fe96ad95e563

But I don't really see a way to do that on my container installation from this operator.

When I add this to the kubectl apply I get the following error.

apiVersion: apps.emqx.io/v1beta3
kind: EmqxEnterprise
metadata:
  name: emqx-ee
  labels:
    "foo": "bar"
spec:
  emqxTemplate:
    image: emqx/emqx-ee:4.4.7
    envFrom:
          - configMapRef:
              name: emqx-ee-config

error validating data: ValidationError(EmqxEnterprise.spec.emqxTemplate): unknown field "envFrom" in io.emqx.apps.v1beta3.EmqxEnterprise.spec.emqxTemplate; if you choose to ignore these errors, turn validation off with --validate=false

Admittedly, I am new to kubernetes so if you could help me with the issue it would be greatly appreciated.

TODO:

• Delete .spec.lables, should use meta.labels
• Delete .spec.annotations, should use meta.annotations
• EmqxEnterprise's .spec.license supports base64 encoding, and support external secret
• Delete .spec.serviceAccount, when use emqx 4.4.x, serviceAccount is not required

Describe the bug
The emqx operator reload the emqx_auth_mysql very frequently. When the message come to emqx when emqx_auth_mysql is reloading and the cache need to be refreshed, the message will failed because the hook is disappearing. Since the result will cache, so the client will failed until cache refreshed.

To Reproduce
Deploy the emqx by emqx operator , and auth by mysql.

Expected behavior
The message will not dropped.

Anything else we need to know?:
Broker EMQX with v4.4.3
EMQX operator with

Environment details::

• Kubernetes version:
• Cloud-provider/provisioner:
• emqx-operator version: 1.2.0
• Install method: e.g. helm/static manifests

Hello all,

I would like to hear your suggestion about EMQX users migration.
We are planning to upgrade emqx version and use it with operator.

What is the best way of migrating existing users from emqx v4 to v5. We are going to use emqx with operator.

I would like to hear your suggestions.

Thank you.

Describe the bug

emqx Service doesn't use those nodePorts specified in EmqxBroker object.

To Reproduce
Steps to reproduce the behavior:

With custom object:

apiVersion: apps.emqx.io/v1beta2
kind: EmqxBroker
metadata:
  name: emqx
spec:
  serviceAccountName: "emqx"
  image: emqx/emqx:4.3.11
  replicas: 3
  labels:
    cluster: emqx
  storage:
    storageClassName: emqx
    resources:
      requests:
        storage: 20Mi
    accessModes:
      - ReadWriteOnce
  # env:
  #   - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
  #     value: admin
  #   - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
  #     value: ?
  emqxTemplate:
    listener:
      type: NodePort
      ports:
        mqtt: 1883
        mqtts: 8883
        ws: 8083
        wss: 8084
        dashboard: 18083
        api: 8081
      nodePorts:
        mqtt: 30038
        mqtts: 30039
        ws: 30040
        wss: 30041
        dashboard: 30042
        api: 30043
    acl:
      - permission: allow
        username: "dashboard"
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
            - "#"
      - permission: allow
        ipaddress: "127.0.0.1"
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: deny
        action: subscribe
        topics:
          filter:
            - "$SYS/#"
          equal:
            - "#"
      - permission: allow
    plugins:
      - name: emqx_management
        enable: true
      - name: emqx_recon
        enable: true
      - name: emqx_retainer
        enable: true
      - name: emqx_dashboard
        enable: true
      - name: emqx_telemetry
        enable: true
      - name: emqx_rule_engine
        enable: true
      - name: emqx_bridge_mqtt
        enable: false
    modules:
      - name: emqx_mod_acl_internal
        enable: true
      - name: emqx_mod_presence
        enable: true

Expected behavior

emqx Service should use nodePorts(30038 30039 etc.) specified in EmqxBroker object, but results in random allocated:

Anything else we need to know?:

Environment details::

• Kubernetes version: v1.20.12
• Cloud-provider/provisioner: kubeadm
• emqx-operator version: 1.1.2
• Install method: helm with chart version 1.0.1

Describe the bug
There are two cases that failed when executing the make test

To Reproduce

1. switch tag 1.1.7
2. make test

Expected behavior
All cases should be passed

Anything else we need to know?:

Environment details::

• Kubernetes version: minikube version: v1.23.0
• Cloud-provider/provisioner:
• emqx-operator version:
• Install method: e.g. helm/static manifests

Discussed in #628

emqx.yaml 完整配置在哪里能找到文档？

Describe the bug
Following the deployment guide I got a mounting error by creating the pod:

  Normal   Scheduled    16s               default-scheduler  Successfully assigned infra/emqx-core-0 to aks-nodepool-36253939-vmss000001
  Warning  FailedMount  8s (x5 over 16s)  kubelet            MountVolume.MountDevice failed for volume "pvc-e529276e-79e8-4af7-8138-e40fe495adbd" : rpc error: code = Internal desc = volume(dev-rg#fb884d6e96a1f417587d381#pvcn-e529276e-79e8-4af7-8138-e40fe495adbd###infra) mount fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/c1362300e0665788580430e44151e8a12846b896d45d3f93c0847c17a67b5eee/globalmount failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o nconnect=8,vers=4,minorversion=1,sec=sys fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/c1362300e0665788580430e44151e8a12846b896d45d3f93c0847c17a67b5eee/globalmount
Output: mount.nfs: access denied by server while mounting fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd
  Warning  FailedMount  73s  kubelet  Unable to attach or mount volumes: unmounted volumes=[emqx-core-data], unattached volumes=[bootstrap-user bootstrap-config kube-api-access-9r6hs emqx-core-data]: timed out waiting for the condition

Environment details::

• Kubernetes version: 1.24.6
• Cloud-provider/provisioner: Azure (AKS)
• emqx-operator version: 2.1.1
• Install method: helm

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: azurefile-csi-nfs
    provisioner: file.csi.azure.com
    allowVolumeExpansion: true
    parameters:
      protocol: nfs
    mountOptions:
      - nconnect=8

apiVersion: apps.emqx.io/v2alpha1
    kind: EMQX
    metadata:
      name: emqx
      namespace: infra
    spec:
      image: "emqx/emqx:5.0.19"
      imagePullPolicy: IfNotPresent
      coreTemplate:
        spec:
          volumeClaimTemplates:
            storageClassName: azurefile-csi-nfs
            resources:
              requests:
                storage: 20Mi
            accessModes:
            - ReadWriteOnce
          replicas: 1
      replicantTemplate:
        spec:
          replicas: 3
      dashboardServiceTemplate:
        metadata:
          name: emqx-dashboard
        spec:
          type: ClusterIP
          selector:
            apps.emqx.io/db-role: core
          ports:
            - name: "dashboard-listeners-http-bind"
              protocol: TCP
              port: 18083
              targetPort: 18083
      listenersServiceTemplate:
        metadata:
          name: emqx-listeners
        spec:
          type: ClusterIP
          ports:
            - name: "tcp-default"
              protocol: TCP
              port: 1883
              targetPort: 1883

按照用户说明操作在 aws(亚马逊**宁夏区) eks 集群上跑不起来，
请问能不能提供一个在eks上运行的 demo

NAME     READY   STATUS             RESTARTS   AGE    IP               NODE                                                NOMINATED NODE   READINESS GATES
emqx-0   0/1     CrashLoopBackOff   3          112s   192.168.75.204   ip-192-168-88-157.cn-northwest-1.compute.internal   <none>           <none>
emqx-1   0/1     Error              4          112s   192.168.24.100   ip-192-168-21-143.cn-northwest-1.compute.internal   <none>           <none>
emqx-2   0/1     CrashLoopBackOff   3          112s   192.168.39.78    ip-192-168-50-143.cn-northwest-1.compute.internal   <none>           <none>
emqx-0   0/1     Running            4          2m1s   192.168.75.204   ip-192-168-88-157.cn-northwest-1.compute.internal   <none>           <none>
emqx-1   0/1     CrashLoopBackOff   4          2m4s   192.168.24.100   ip-192-168-21-143.cn-northwest-1.compute.internal   <none>           <none>
emqx-0   0/1     Error              4          2m5s   192.168.75.204   ip-192-168-88-157.cn-northwest-1.compute.internal   <none>           <none>
emqx-2   0/1     Running            4          2m6s   192.168.39.78    ip-192-168-50-143.cn-northwest-1.compute.internal   <none>           <none>
emqx-2   0/1     Error              4          2m11s   192.168.39.78    ip-192-168-50-143.cn-northwest-1.compute.internal   <none>           <none>
emqx-0   0/1     CrashLoopBackOff   4          2m16s   192.168.75.204   ip-192-168-88-157.cn-northwest-1.compute.internal   <none>           <none>

Events:
  Type     Reason                  Age                     From                     Message
  ----     ------                  ----                    ----                     -------
  Normal   Scheduled               7m33s                   default-scheduler        Successfully assigned default/emqx-1 to ip-192-168-21-143.cn-northwest-1.compute.internal
  Normal   SuccessfulAttachVolume  7m31s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ba0d0dbc-7251-4f37-a6de-e493eafe6a26"
  Normal   SuccessfulAttachVolume  7m31s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-08f7c78e-e0e7-4b28-a64c-d7362c96fa95"
  Normal   Pulled                  5m48s (x5 over 7m28s)   kubelet                  Container image "emqx/emqx:4.4.1" already present on machine
  Normal   Created                 5m48s (x5 over 7m28s)   kubelet                  Created container emqx
  Normal   Started                 5m48s (x5 over 7m28s)   kubelet                  Started container emqx
  Warning  Unhealthy               5m43s (x3 over 7m23s)   kubelet                  Readiness probe failed: Get "http://192.168.24.100:8081/status": dial tcp 192.168.24.100:8081: connect: connection refused
  Warning  BackOff                 2m22s (x22 over 7m17s)  kubelet                  Back-off restarting failed container

Is your feature request related to a problem? Please describe.
v1/beat2 支持加载插件吗?

• 例如

apiVersion: apps.emqx.io/v1beta3
kind: EmqxPlugin
metadata:
  name: emqx-lwm2m
  namespace: default
spec:
  selector:
      "foo": "bar"
  pluginName: emqx_lwm2m
  config:
      "lwm2m.lifetime_min": "1s"
      "lwm2m.lifetime_max": "86400s"
      "lwm2m.bind.udp.1": "0.0.0.0:5683"
      "lwm2m.bind.udp.2": "0.0.0.0:5684"
      "lwm2m.bind.udp.3": "0.0.0.0:5685"
      "lwm2m.xml_dir": "/opt/emqx/etc/lwm2m_xml"

if so, then which params of the CRD should be set ?

It seems that spec.coreTemplate.spec.extraContainers field was lost in the last update (2.1.2).

To Reproduce

1. Have an emqx resource with an extra container declared in spec.coreTemplate.spec.extraContainers.
2. upgrade to latest emqx-operator v2.1.2
3. extra container will no longer be created by operator

Expected behavior
The extra container should be present even after the upgrade.

Anything else we need to know?:

Environment details::

• Kubernetes version: v1.24.10-gke.2300
• Cloud-provider/provisioner: GKE
• emqx-operator version: v2.1.2
• Install method: helm

Describe the bug
When an instance of EMQX is removed, the relevant PVC is still lingering. That prevents an EMQX instance to be recreated as it runs into mounting problems.

To Reproduce

> kubectrl delete emqx emqx
> kubectrl get emqx
No resources found in infra namespace.
> kubectl get PVC
emqx-core-data-emqx-core-0      Bound    pvc-03cd9d0f-010c-4fef-bdc2-edbfa4ff4f29   20Mi       RWO            azurefile-csi-nfs

Expected behavior
PVC should be removed together with the EMQX instance

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.24.6
• Cloud-provider/provisioner: Azure / AKS
• emqx-operator version: 2.1.1
• Install method: helm

Describe the bug
The original emqx-operator-webhook-service has the following config:

spec:
  selector:                                                                                                                                                │
    control-plane: controller-manager 

which is too vague. On my k8s cluster, with additional operators installed this service selector targets also other operator pods causing problems when the request goes to other operators.

To Reproduce
Steps to reproduce the behavior:

• have another pod with label: control-plane: controller-manager, for example, deploy emqx operator in a namespace in which a grafana operator exists(https://github.com/grafana-operator/grafana-operator).

Expected behavior
The service selector for the webhook service should be more restrictive, for example:

spec:
  selector:                                                                                                                                                │
    control-plane: controller-manager
    app.kubernetes.io/name: emqx-operator

Environment details::

• Kubernetes version: v1.22.15-gke.100
• Cloud-provider/provisioner: GKE
• emqx-operator version: 2.0.1
• Install method: helm

Describe the bug
A clear and concise description of what the bug is.

kubectl logs -f emqx-2
Defaulted container "emqx" out of: emqx, reloader
!!!!!!
WARNING: Default (insecure) Erlang cookie is in use.
Please set node.cookie in /opt/emqx/etc/emqx.conf or override from environment variable EMQX_NODE_COOKIE
NOTE: Use the same config value for all nodes in the cluster.
!!!!!!
Starting emqx on node [email protected]
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
EMQ X Broker 4.4.14 is running now!
Stop http:management listener on 0.0.0.0:8081 successfully.
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
rpc error: code = NotFound desc = an error occurred when try to find container "40ca01dce6a94173b829ba3655f0bdc8f896ccc5f3992ca4c80efc0d1aa

To Reproduce
Steps to reproduce the behavior:
emqx.yaml

apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
  name: emqx
  labels:
    "apps.emqx.io/instance": "emqx"
spec:
  replicas: 3
  persistent:
    spec:
      storageClassName: local-path
      resources:
        requests:
          storage: 100Mi
      accessModes:
        - ReadWriteOnce
  template:
    metadata:
      labels:
        "apps.emqx.io/instance": "emqx"
    spec:
      emqxContainer:
        image:
          repository: emqx/emqx
          version: 4.4.14
        emqxConfig:
          name: emqx
          cluster.discovery: dns
          cluster.dns.type: srv
          cluster.dns.app: emqx
          cluster.dns.name: emqx-headless.default.svc.cluster.local
          listener.tcp.external: "1883"
          node.process_limit: "2097152"
          node.max_ports: "1048576"
        emqxACL:
          - '{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.'
          - '{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}.'
          - '{allow, all, subscribe, ["$SYS/brokers/+/clients/#"]}.'
          - '{allow, all}.'
        ports:
            - containerPort: 8081
            - containerPort: 18083
            - containerPort: 1883
            - containerPort: 11883
            - containerPort: 8083
            - containerPort: 8084
        volumeMounts:
          - name: emqx-http-vol
            mountPath: /opt/emqx/etc/plugins/emqx_auth_http.conf
            subPath: emqx_auth_http.conf
      volumes:
        - name: emqx-http-vol
          configMap:
            name: emqx-http
            items:
              - key: emqx_auth_http.conf
                path: emqx_auth_http.conf
  serviceTemplate:
    metadata:
      name: emqx
      namespace: default
      labels:
        "apps.emqx.io/instance": "emqx"
    spec:
      type: LoadBalancer
      selector:
        "apps.emqx.io/instance": "emqx"
      ports:
        - name: "http-management-8081"
          port: 8081
          protocol: "TCP"
          targetPort: 8081
        - name: "http-dashboard-18083"
          port: 18083
          protocol: "TCP"
          targetPort: 18083
        - name: "mqtt-tcp-1883"
          protocol: "TCP"
          port: 1883
          targetPort: 1883

Expected behavior
A clear and concise description of what you expected to happen.

$ kubectl logs -f emqx-2
Defaulted container "emqx" out of: emqx, reloader
!!!!!!
WARNING: Default (insecure) Erlang cookie is in use.
Please set node.cookie in /opt/emqx/etc/emqx.conf or override from environment variable EMQX_NODE_COOKIE
NOTE: Use the same config value for all nodes in the cluster.
!!!!!!
Starting emqx on node [email protected]
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
EMQ X Broker 4.4.14 is running now!
Stop http:management listener on 0.0.0.0:8081 successfully.
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
rpc error: code = NotFound desc = an error occurred when try to find container "40ca01dce6a94173b829ba3655f0bdc8f896ccc5f3992ca4c80efc0d1aafdb22": not found

$ kubectl get  EmqxBroker emqx -o json | jq '.status' 
{
  "conditions": [
    {
      "lastTransitionTime": "2023-02-16T06:55:40Z",
      "lastUpdateTime": "2023-02-16T07:02:58Z",
      "message": "All resources are ready",
      "reason": "ClusterReady",
      "status": "True",
      "type": "Running"
    }
  ],
  "currentStatefulSetVersion": "emqx-974f94d58",
  "emqxNodes": [
    {
      "node": "[email protected]",
      "node_status": "Running",
      "otp_release": "24.3.4.2/12.3.2.2",
      "version": "4.4.14"
    },
    {
      "node": "[email protected]",
      "node_status": "Running",
      "otp_release": "24.3.4.2/12.3.2.2",
      "version": "4.4.14"
    },
    {
      "node": "[email protected]",
      "node_status": "Running",
      "otp_release": "24.3.4.2/12.3.2.2",
      "version": "4.4.14"
    }
  ],
  "readyReplicas": 3,
  "replicas": 3
}

Anything else we need to know?:

Environment details::

• Kubernetes version:
  kubectl version
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9+k3s1", GitCommit:"4dcf0a33bc05159688ad4358aa6d5d4321a77277", GitTreeState:"clean", BuildDate:"2022-12-21T00:52:22Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9+k3s1", GitCommit:"4dcf0a33bc05159688ad4358aa6d5d4321a77277", GitTreeState:"clean", BuildDate:"2022-12-21T00:52:22Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud-provider/provisioner: none

• emqx-operator version: apps.emqx.io/v1beta4

• Install method: e.g. helm/static manifests
  helm repo add emqx https://repos.emqx.io/charts
  helm repo update
  helm install emqx-operator emqx/emqx-operator --namespace emqx-operator-system --create-namespace

Describe the bug
I have a three node emqx-ee cluster that created by emqx-opeartor which 1.2.7 version;
When just one node running failed for some reason, for example, file direction permission, My whole emqx-ee cluster will be inaccessible， for the 1883\8883 ports will be disappear in the default service.

To Reproduce

1. create a multiple node emqx-ee cluster;
2. get into one node, change the data direction permission to 400;
3. watch the default service's change.

Expected behavior
just part of node running failed, the emqx-ee cluster should be also accessible.
the service also should be like as the blow example:

Anything else we need to know?:
the latest emqx-operator version also has this same problem.

Environment details::

• Kubernetes version: 1.22.3
• emqx-operator version: 1.2.7-ecp.1
• Install method: e.g. helm/static manifests helm

Is your feature request related to a problem? Please describe.
We are operating emqx broker together with a companion container for cloudsql-proxy.

Describe the solution you'd like
Because of this, it would be very useful to be able to specify additional containers in the EmqxBroker crd.

Describe alternatives you've considered
Can't think of an alternative atm.

Hello there!
Thanks for the great operator software.

I'm trying to deploy this operator and it's not possible to start reload sidecars on raspberry pi nodes as version is hardcoded to emqx-operator-reloader:0.0.1 one - which compiled without arm support.

Is there any way to define this package version somewhere in the operator config?

Describe the bug

Operator doesn't have correctly configured service and/or containerPort, consequently metrics endpoint doesn't work.

To Reproduce

kubectl apply -f https://github.com/emqx/emqx-operator/releases/download/1.1.5/emqx-operator-controller.yaml

emqx-operator-controller-manager-metrics-service has one port (https/8443), this port points to targetPort https on emqx-operator-controller-manager-... pod. The container in the pod doesn't have any such containerPort, because of that Prometheus metrics aren't usable.

Expected behavior

The pod should have containerPort: 8080 exposed (according to pod logs, this is where /metrics actually is) and service should be adjusted accordingly (point to this port instead of 8443).

Environment details::

• Kubernetes version: 1.23
• Cloud-provider/provisioner: k3s
• emqx-operator version: 1.1.4
• Install method: static manifest

Describe the bug
when use helm uninstall the emqx-operator release, the emqxplugins.apps.emqx.io crd is not remove.

To Reproduce
Steps to reproduce the behavior:
see the picture above。
Expected behavior
all the crds removed, when use helm uninstall the emqx-operator release.

Anything else we need to know?:

Environment details::

• Kubernetes version:
  ~ % kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

% helm version version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"dirty", GoVersion:"go1.16.6"}

• Cloud-provider/provisioner:minikube
• emqx-operator version: master branch
• Install method: e.g. helm

Describe the bug
A clear and concise description of what the bug is.
when i use this yaml create emqx-ee，could not create successfully

#emqx-ee.yaml
apiVersion: apps.emqx.io/v1beta2
kind: EmqxEnterprise
metadata:
  name: emqx-ee
spec:
  serviceAccountName: "emqx"
  image: emqx/emqx-ee:4.4.0
  replicas: 3
  labels:
    cluster: emqx
  storage:
    storageClassName: hostpath
    resources:
      requests:
        storage: 20Mi
    accessModes:
    - ReadWriteOnce
  env:
    - name: "EMQX_LOG__LEVEL"
      value: "debug"
  emqxTemplate:
    # acl: []
    plugins: 
      - name: emqx_management
        enable: true
      - name: emqx_recon
        enable: true
      - name: emqx_retainer
        enable: true
      - name: emqx_dashboard
        enable: true
      - name: emqx_telemetry
        enable: true
      - name: emqx_rule_engine
        enable: true
      - name: emqx_bridge_mqtt
        enable: false
    # modules: 
    #   
    #   - name: internal_acl
    #     enable: true
    #     configs:
    #       acl_rule_file: etc/acl.conf
    #   - name: presence
    #     enable: true
    #     configs:
    #       qos: 0
    #   - name: recon
    #     enable: true
    #     configs: {}
    #   - name: retainer
    #     enable: true
    #     configs:
    #       expiry_interval: 0
    #       max_payload_size: 1MB
    #       max_retained_messages: 0
    #       storage_type: ram


    listener:
      type: NodePort
      ports:
        mqtt: 1883
        mqtts: 8883
        ws: 8083
        wss: 8084
        dashboard: 18083
        api: 8081

This is the error screenshot

To Reproduce

kubeclt apply -f emqx-ee.yaml

Anything else we need to know?:

Environment details::

• Kubernetes version:v1.22.4 (docker-desktop)
• emqx-operator version: v1.1.1
• Install method: helm

Describe the bug
Dash board reports - some error occoured

Logs show:

2022-09-12 11:41:09.622 BST2022-09-12T10:41:09.621898+00:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /listeners, reason: {badmatch,{error,{'EXIT',{{badkey,<<"listeners">>},[{erlang,map_get,[<<"listeners">>,#{}],[{error_info,#{module => erl_erts_errors}}]},{emqx_config,get_schema_mod,1,[{file,"emqx_config.erl"},{line,505}]},{emqx_listeners,do_list_raw,0,[{file,"emqx_listeners.erl"},{line,92}]},{emqx_listeners,list_raw,0,[{file,"emqx_listeners.erl"},{line,71}]},{emqx_mgmt_api_listeners,do_list_listeners,0,[{file,"emqx_mgmt_api_listeners.erl"},{line,519}]}]}}}}, stacktrace: [{emqx_mgmt_api_listeners,listener_status_by_id,2,[{file,"emqx_mgmt_api_listeners.erl"},{line,496}]},{emqx_mgmt_api_listeners,listener_status_by_id,1,[{file,"emqx_mgmt_api_listeners.erl"},{line,472}]},{emqx_mgmt_api_listeners,list_listeners,2,[{file,"emqx_mgmt_api_listeners.erl"},{line,342}]},{minirest_handler,apply_callback,3,[{file,"minirest_handler.erl"},{line,111}]},{minirest_handler,handle,2,[{file,"minirest_handler.erl"},{line,44}]},{minirest_handler,init,2,[{file,"minirest_handler.erl"},{line,27}]},{cowboy_handler,execute,2,[{file,"cowboy_handler.erl"},{line,41}]},{cowboy_stream_h,execute,3,[{file,"cowboy_stream_h.erl"},{line,318}]},{cowboy_stream_h,request_process,3,[{file,"cowboy_stream_h.erl"},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

To Reproduce

Followed installation steps in Google K8s - GKE

Attepted to portforward to the dashboard and login with admin / public

Expected behavior
To be able to login.

Anything else we need to know?:

Environment details::

• Kubernetes version:

1.22.11-gke.400

• Cloud-provider/provisioner:

Google Cloud

• emqx-operator version:

V5

• Install method: e.g. helm/static manifests

https://github.com/emqx/emqx-operator/blob/main/docs/en_US/getting-started/getting-started.md

EMQX_HOST: $(POD_NAME).$(STS_HEADLESS_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local

this is the operator generated

however my domain of k8s is k8s.com not cluster.local

emqx-operator ：1.2.0
emqx： 4.4.1

Hey guys,

I am trying to run your operator on a local Kubernetes cluster, but get an error in the pod's log:
kubectl logs emqx-0
2022-01-21T22:58:52.565992+00:00 [error] Error creating /opt/emqx/data/configs: permission denied

Do you have any idea how I could fix this?
BR, Christian

Describe the bug
I have configured EMQX as following

    apiVersion: apps.emqx.io/v2alpha1
    kind: EMQX
    metadata:
      name: emqx
    spec:
      image: emqx/emqx:5.0.6
      coreTemplate:
        metadata:
          name: emqx-core
          labels:
            apps.emqx.io/instance: emqx
            apps.emqx.io/db-role: core
          annotations:
        spec:
          replicas: 1
          volumeClaimTemplates:
            resources:
              requests:
                storage: 1Gi
            accessModes:
              - ReadWriteOnce
          ports:
            - containerPort: 1883
          envFrom:
            - prefix: EMQX_
              configMapRef:
                name: emqx-config
            - prefix: EMQX_AUTHENTICATION__0__
              secretRef:
                name: emqx-pgsql-connection
            - prefix: EMQX_AUTHORIZATION__SOURCES__0__
              secretRef:
                name: emqx-pgsql-connection
          resources:
            limits:
              # cpu: 100m
              memory: 256Mi
            # requests:
            #   cpu: 100m
            #   memory: 256Mi
      dashboardServiceTemplate:
        metadata:
          name: emqx-dashboard
        spec:
          selector:
            apps.emqx.io/db-role: core
          ports:
            - name: "dashboard-listeners-http-bind"
              protocol: TCP
              port: 18083
              targetPort: 18083
      listenersServiceTemplate:
        metadata:
          name: emqx-listeners
        spec:
          ports:
            - name: "tcp-default"
              protocol: TCP
              port: 1883
              targetPort: 1883

When the emqx-core-0 starts it crashes with following error:

mkdir: cannot create directory ‘/opt/emqx/data/configs’: Permission denied                                                                │
│ Stream closed EOF for my-namespace/emqx-core-0 (emqx)      

To Reproduce
Steps to reproduce the behavior:
I just uncommented volumeClaimTemplates section under coreTemplate [also removed hardcoded storageClassName so that the default is selected] - removed replicantTemplate section, as not required - removed commented code

Expected behavior
This error should not come when using "sample" config values.

Not sure, how to fix this - looks like the pod has no write permission on volume mounted - either it controller has bug or config is wrong.

Environment details::

• Kubernetes version: 1.23.10
• Cloud-provider/provisioner: Linode
• emqx-operator version: 2.0.0
• Install method: Helm + Helmfile

Describe the bug
emqx-operator-controller-manager pod keep crashing with the below logs after deploy the emqx opensource broker to cluster. here we use kustomize.

Logs:

I1103 14:44:33.965126       1 request.go:601] Waited for 1.003207335s due to client-side throttling, not priority and fairness, request: GET:https://172.0.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
65
2022-11-03T14:44:35Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
64
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
63
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
62
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
61
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
60
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
59
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
58
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
57
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
56
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
55
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
54
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
53
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
52
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
51
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
50
2022-11-03T14:44:35Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/validate-apps-emqx-io-v2alpha1-emqx"}
49
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-apps-emqx-io-v2alpha1-emqx"}
48
2022-11-03T14:44:35Z	INFO	setup	starting manager
47
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook.webhooks	Starting webhook server
46
2022-11-03T14:44:35Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
45
2022-11-03T14:44:35Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
44
2022-11-03T14:44:35Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
43
I1103 14:44:35.824603       1 leaderelection.go:248] attempting to acquire leader lease emqx/19fd6fcc.emqx.io...
42
2022-11-03T14:44:35Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
41
2022-11-03T14:44:35Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
40
I1103 14:45:07.039257       1 leaderelection.go:258] successfully acquired lease emqx/19fd6fcc.emqx.io
39
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "source": "kind source: *v1beta3.EmqxPlugin"}
38
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "source": "kind source: *v1beta3.EmqxBroker"}
37
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker"}
36
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "source": "kind source: *v2alpha1.EMQX"}
35
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX"}
34
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin"}
33
2022-11-03T14:45:07Z	INFO	Starting EventSource	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "source": "kind source: *v1beta3.EmqxEnterprise"}
32
2022-11-03T14:45:07Z	INFO	Starting Controller	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise"}
31
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "worker count": 1}
30
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "worker count": 1}
29
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "worker count": 1}
28
2022-11-03T14:45:07Z	INFO	Starting workers	{"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "worker count": 1}
27
2022-11-03T14:45:07Z	INFO	Observed a panic in reconciler: runtime error: index out of range [0] with length 0	{"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "eMQX": {"name":"mycluster1-emqx","namespace":"emqx"}, "namespace": "emqx", "name": "mycluster1-emqx", "reconcileID": "7b73e597-435a-45b0-b9e5-57fe41c34757"}
26
panic: runtime error: index out of range [0] with length 0 [recovered]
25
	panic: runtime error: index out of range [0] with length 0
24
23
goroutine 492 [running]:
22
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
21
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
20
panic({0x1847520, 0xc001c7a030})
19
	/usr/local/go/src/runtime/panic.go:838 +0x207
18
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*coreUpdateStatus).nextStatus(0xc0003240f0, 0xc00015ca00, 0xc00011a800?)
17
	/workspace/controllers/apps/v2alpha1/status_machine.go:163 +0x2c5
16
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*emqxStatusMachine).NextStatus(...)
15
	/workspace/controllers/apps/v2alpha1/status_machine.go:111
14
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).updateStatus(0xc000601c00, 0xc00011a800)
13
	/workspace/controllers/apps/v2alpha1/emqx_controller.go:160 +0x2f0
12
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).Reconcile(0xc000601c00, {0x1ba82c8, 0xc00086d020}, {{{0xc000404630?, 0x10?}, {0xc000404620?, 0x40d7e7?}}})
11
	/workspace/controllers/apps/v2alpha1/emqx_controller.go:75 +0x134
10
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1ba8220?, {0x1ba82c8?, 0xc00086d020?}, {{{0xc000404630?, 0x1878040?}, {0xc000404620?, 0x4041f4?}}})
9
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xc8
8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001f1680, {0x1ba8220, 0xc000932580}, {0x179bae0?, 0xc000529380?})
7
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x33c
6
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001f1680, {0x1ba8220, 0xc000932580})
5
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x1d9
4
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
3
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
2
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x325

Environment details::

• Kubernetes version: 1.24.6
• Cloud-provider/provisioner: Azure
• emqx-operator version: 2.0.1
• Install method: static manifests

The first time i run helm install everything goes smoothly, i.e. the CRD are installed and all the rest deployed.

when i run helm install on a different namespace, setting the value skipCRDs to true, i get an error of ownership in the cluster role ... how to fix?

Discussed in #171

Hi,
How can I add extra ENV in the EmqxBroker yaml config?
So I can enable the mysql auth plugin and setup the right settings.
Should I do that with ENV or is there another way to add the configs related to the plugin?

@Rory-Z can you maybe help me out with a sample config where the emqx mysql auth plugin is added? I tried to get the env somehow in the manifest but I doesn't get it to work.