Comments (10)
Rory-Z
commented on August 15, 2024
Sorry about, could you please give me some log for reloader emqx_auth_mysql ? this is very helpful in solving this problem
from emqx-operator.
stayhungry001
commented on August 15, 2024
2022-12-03T14:57:47.837206+00:00 [notice] Application: emqx_auth_mysql. Exited: stopped. Type: temporary.
2022-12-03T14:57:47.837400+00:00 [info] [Plugins] Stop plugin emqx_auth_mysql successfully
2022-12-03T14:57:47.838600+00:00 [error] [Plugins] Write File "/mounted/plugins/data/loaded_plugins" Error: erofs
2022-12-03T14:57:47.847704+00:00 [debug] Adding Defaults
2022-12-03T14:57:47.847835+00:00 [debug] Right Hand Side Substitutions
2022-12-03T14:57:47.847910+00:00 [debug] Applying Datatypes
2022-12-03T14:57:47.848200+00:00 [debug] Validation
2022-12-03T14:57:47.848365+00:00 [debug] Applied 1:1 Mappings
2022-12-03T14:57:47.848514+00:00 [debug] Running translation for emqx_auth_mysql.server
2022-12-03T14:57:47.848935+00:00 [debug] Running translation for emqx_auth_mysql.password_hash
from emqx-operator.
stayhungry001
commented on August 15, 2024
This log is from EMQX cluster. Another question: where to get the log of emqx operator?
from emqx-operator.
stayhungry001
commented on August 15, 2024
What other logs do you want to know? The plugins were not restarted since I remove emqx oprator, and no more message dropped due to Not authorized anymore.
from emqx-operator.
Rory-Z
commented on August 15, 2024
The plugins were not restarted since I remove emqx oprator, and no more message dropped due to Not authorized anymore.
So stranger, could you please update EMQX Operator to 1.2.8 and try again ?
If this issue is still, please let me know the EMQX container log, the EMQX reloader container log, the EMQX Operator container log ( you can get EMQX Operator pod in emqx-operator-system namespace ), and any other information you think would be useful, thanks
from emqx-operator.
stayhungry001
commented on August 15, 2024
Log from EMQX cluster：
2022-12-03T14:57:47.442744+00:00 [debug] [email protected]:55270 [MQTT] SEND PUBACK(Q0, R0, D0, PacketId=63997, ReasonCode=0)
2022-12-03T14:57:47.837206+00:00 [notice] Application: emqx_auth_mysql. Exited: stopped. Type: temporary.
2022-12-03T14:57:47.837400+00:00 [info] [Plugins] Stop plugin emqx_auth_mysql successfully
2022-12-03T14:57:47.838600+00:00 [error] [Plugins] Write File "/mounted/plugins/data/loaded_plugins" Error: erofs
2022-12-03T14:57:47.847704+00:00 [debug] Adding Defaults
2022-12-03T14:57:47.847835+00:00 [debug] Right Hand Side Substitutions
2022-12-03T14:57:47.847910+00:00 [debug] Applying Datatypes
2022-12-03T14:57:47.848200+00:00 [debug] Validation
2022-12-03T14:57:47.848365+00:00 [debug] Applied 1:1 Mappings
2022-12-03T14:57:47.848514+00:00 [debug] Running translation for emqx_auth_mysql.server
2022-12-03T14:57:47.848935+00:00 [debug] Running translation for emqx_auth_mysql.password_hash
2022-12-03T14:57:47.849036+00:00 [debug] Applied Translations
2022-12-03T14:57:48.002993+00:00 [debug] [email protected]:36522 [MQTT] RECV <<50,192,2,0,35,50,4,...>>
2022-12-03T14:57:48.003252+00:00 [debug] [email protected]:36522 [MQTT] RECV PUBLISH(Q1, R0, D0, Topic=202112281201315616176/publish, PacketId=30344, Payload=<<35,35,27,2,48,xxxxxxx>>)
2022-12-03T14:57:48.003925+00:00 [warning] [email protected]:36522 [Channel] Cannot publish message to 202112281201315/publishdue to Not authorized.
2022-12-03T14:57:48.004047+00:00 [debug] [email protected]:36522 [MQTT] SEND PUBACK(Q0, R0, D0, PacketId=30344, ReasonCode=135)
2022-12-03T14:57:48.032847+00:00 [info] [Plugins] Started plugins: [emqx_auth_mysql]
2022-12-03T14:57:48.032936+00:00 [info] [Plugins] Load plugin emqx_auth_mysql successfully
from emqx-operator.
stayhungry001
commented on August 15, 2024
And I am sorry , because this is the production environment. It is a risk to reproduce it.
from emqx-operator.
stayhungry001
commented on August 15, 2024
1.6701544253086896e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "stationemqx-emqx-2-retainer", "namespace": "prod-stationemqx-2"}
1.6701544253088424e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "exchange-emqx-dashboard", "namespace": "prod-exchangeemqx"}
1.670154425308951e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "exchange-emqx-retainer", "namespace": "prod-exchangeemqx"}
1.6701544253090365e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqxbroker-rule-engine", "namespace": "pre-exchange-emqx"}
1.6701544253091166e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqxbroker-management", "namespace": "pre-exchange-emqx"}
1.670154425317561e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"pre-exchange-emqx","name":"emqxbroker-plugins-config","uid":"f727a182-72b3-43dd-b787-bd83432963a2","apiVersion":"v1","resourceVersion":"76717074"}, "reason": "Updated", "message": "Update resource successfully"}
1.6701544258570118e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544264061954e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544269666483e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544269753249e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqx-prometheus", "namespace": "prod-stationemqx-2"}
1.6701544269754632e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "exchange-emqxauthmysql", "namespace": "pre-exchange-emqx"}
1.6701544269840934e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"pre-exchange-emqx","name":"emqxbroker-plugins-config","uid":"f727a182-72b3-43dd-b787-bd83432963a2","apiVersion":"v1","resourceVersion":"76717081"}, "reason": "Updated", "message": "Update resource successfully"}
1.6701544275951638e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_auth_mysql to container emqx successfully"}
1.6701544282024267e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_auth_mysql to container emqx successfully"}
1.6701544288377151e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_auth_mysql to container emqx successfully"}
1.670154428846684e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqxbroker-retainer", "namespace": "pre-exchange-emqx"}
1.6701544288468287e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "stationemqx-emqx-2-management", "namespace": "prod-stationemqx-2"}
1.670154428855722e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"prod-stationemqx-2","name":"stationemqx-emqx-2-plugins-config","uid":"71036272-d8fd-40c4-91d8-9165d2445d89","apiVersion":"v1","resourceVersion":"76717096"}, "reason": "Updated", "message": "Update resource successfully"}
1.670154429457018e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-stationemqx-2","name":"stationemqx-emqx-2","uid":"3c156e18-af31-4df6-9d5a-929cd3878a4a","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715936"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.670154430069236e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-stationemqx-2","name":"stationemqx-emqx-2","uid":"3c156e18-af31-4df6-9d5a-929cd3878a4a","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715936"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544307623181e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-stationemqx-2","name":"stationemqx-emqx-2","uid":"3c156e18-af31-4df6-9d5a-929cd3878a4a","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715936"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544307697613e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqxbroker-dashboard", "namespace": "pre-exchange-emqx"}
1.6701544307801445e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"pre-exchange-emqx","name":"emqxbroker-plugins-config","uid":"f727a182-72b3-43dd-b787-bd83432963a2","apiVersion":"v1","resourceVersion":"76717111"}, "reason": "Updated", "message": "Update resource successfully"}
1.6701544313120747e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_dashboard to container emqx successfully"}
1.6701544318441486e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_dashboard to container emqx successfully"}
1.6701544323835957e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"pre-exchange-emqx","name":"emqxbroker","uid":"0c47dbe0-7cde-4d47-ba5a-4b7fffe37847","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715938"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_dashboard to container emqx successfully"}
1.6701544323907194e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "exchange-emqx-rule-engine", "namespace": "prod-exchangeemqx"}
1.670154432390865e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "exchange-emqx-management", "namespace": "prod-exchangeemqx"}
1.6701544323993552e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"prod-exchangeemqx","name":"exchange-emqx-plugins-config","uid":"5c035d72-1cc0-4134-9185-afbab1ba631e","apiVersion":"v1","resourceVersion":"76717123"}, "reason": "Updated", "message": "Update resource successfully"}
1.6701544330627728e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-exchangeemqx","name":"exchange-emqx","uid":"ce8af907-a8d4-46e5-91d3-0f8b155d654c","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715942"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544337655463e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-exchangeemqx","name":"exchange-emqx","uid":"ce8af907-a8d4-46e5-91d3-0f8b155d654c","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715942"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544344218457e+09 DEBUG events Normal {"object": {"kind":"EmqxBroker","namespace":"prod-exchangeemqx","name":"exchange-emqx","uid":"ce8af907-a8d4-46e5-91d3-0f8b155d654c","apiVersion":"apps.emqx.io/v1beta3","resourceVersion":"76715942"}, "reason": "Exec", "message": "exec emqx_ctl plugins reload emqx_management to container emqx successfully"}
1.6701544344314559e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "emqx-prometheus", "namespace": "prod-exchangeemqx"}
1.6701544344316096e+09 DEBUG controller.emqxplugin Reconcile EmqxPlugin {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxPlugin", "name": "station-emqxauthmysql", "namespace": "prod-stationemqx-2"}
1.6701544344402394e+09 DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"prod-stationemqx-2","name":"stationemqx-emqx-2-plugins-config","uid":"71036272-d8fd-40c4-91d8-9165d2445d89","apiVersion":"v1","resourceVersion":"76717142"}, "reason": "Updated", "message": "Update resource successfully"}
from emqx-operator.
Rory-Z
commented on August 15, 2024
Hi, @stayhungry001 I commend update EMQX Operator to 1.2.8, the 1.2.0 is to old, in 1.2.0, EMQX Operator use emqx_ctl to manager EMQX Plugins, it is very inefficient, and in some boundary conditions will applied wrong plug-in configuration
And in 1.2.8, we solved thie error log: 2022-12-03T14:57:47.838600+00:00 [error] [Plugins] Write File "/mounted/plugins/data/loaded_plugins" Error: erofs
But update EMQX Operator will make restart EMQX Pod, so be carefully, I think you should test it.
from emqx-operator.
Rory-Z
commented on August 15, 2024
This issue has been automatically marked as stale because it has not had recent activity.
It will be closed.
Thank you for your contributions.
from emqx-operator.
Related Issues (20)
- New core nodes never become ready when replicant replicas set to 0
- Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference HOT 2
- EMQX operator is parsing config into garbage value HOT 2
- AWS EKS Fargate Deployment error mkdir: cannot create directory ‘/opt/emqx/data/configs’: Permission denied HOT 2
- Blue-green update doesn't remove nodes from cluster HOT 15
- No ability to specify a PDB? HOT 4
- 最早创建的emqx pod所在节点挂掉会导致emqx集群服务中断 HOT 1
- EMQX Config patches takes precendence on statefulset changes HOT 1
- Blue-green deployment and scale-up not working if ACL authorization file comes from an extra volume mount HOT 8
- EMQX Cluster stuck on `CoreNodesProgressing` HOT 9
- Add support for EMQX `.spec.config` from existing Secret or Configmap HOT 4
- [Docs] get pods with json format missing fields
- Support for topologySpreadConstraints HOT 2
- EMQX Operator compatibility problem in K8S HOT 1
- Deploying with changes to listener causes deployment to fail HOT 7
- Permanent reconcile of emqx-listeners loadbalancer HOT 31
- Custom acl.conf created via Dashboard is not propageted to new nodes
- Listener(load balancer) is losing - has not ready endpoints (addresses) after emqx pod restart HOT 19
- Permanent reconcile on emqx-core pods HOT 7
- Make revisionHistoryLimit available for the operator deployment in the chart HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emqx-operator.