Giter Club home page Giter Club logo

Comments (12)

ruifung avatar ruifung commented on May 24, 2024 2

Commands worked for first table, but failed when i tried it with Tab = emqx_ds_session.

Since based on the discussion above, it appears this is some kind of inconsistent database state, I've exported the data, blow away the cluster PVCs after stopping it, and restarted it then imported the data export.

Followed by checking if the schemas are still listed from the diagnostic command above. This appears to have resolved the issue.

I thank you for your time in responding to this.

from emqx.

ruifung avatar ruifung commented on May 24, 2024

Any suggestions on how to proceed would be appreciated, since I have absolutely no idea how it got into this state.

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

could you share the commands you use?

do you run the force leave commands on one of the alive nodes?

running_nodes =>
                      ['emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
                       'emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
                       'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local'],

It should print something, shouldn't be silent.

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

If you are not running production,

Assume the node emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local is the stale node that you want to remove from the cluster.

Have you tried something like following?

emqx_ctl cluster force-leave 'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local'

from emqx.

ruifung avatar ruifung commented on May 24, 2024

That's exactly it, when I run the force-leave command, it does nothing with no output.

emqx@emqx-core-f7b96944f-0:/opt/emqx$ emqx_ctl cluster force-leave 'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local'
emqx@emqx-core-f7b96944f-0:/opt/emqx$

And yes, I run the force-leave commands on one of the alive nodes as the stopped nodes are from instances that no longer exist because of updates. (The emqx operator creates new instances on changes.)

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

could you check the container logs, it may print to the container logs or emqx logs?

from emqx.

ruifung avatar ruifung commented on May 24, 2024

Huh, you're right, it did log an error to the container log.

2024-04-11T14:45:22.489186+00:00 [error] msg: ctl_command_crashed, function: cluster, module: emqx_mgmt_cli, reason: {error,{no_exists,emqx_enhanced_authn_scram_mnesia}}, stacktrace: [{mria_lib,ensure_ok,1,[{file,"mria_lib.erl"},{line,200}]},{emqx_mgmt_cli,cluster,1,[{file,"emqx_mgmt_cli.erl"},{line,120}]},{emqx_ctl,run_command,2,[{file,"emqx_ctl.erl"},{line,128}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,589}]}]

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

could you run

emqx eval 'io:format("~p",[ets:tab2list(schema)])'

and send us the outputs?

from emqx.

ruifung avatar ruifung commented on May 24, 2024 
emqx@emqx-core-f7b96944f-0:/opt/emqx$ emqx eval 'io:format("~p",[ets:tab2list(schema)])'
[{schema,emqx_shared_subscription,
     [{name,emqx_shared_subscription},
      {type,bag},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_shared_subscription},
      {attributes,[group,topic,subpid]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065685467428384,-576460752303423261,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,209748}}}}]},
 {schema,emqx_retainer_index,
     [{name,emqx_retainer_index},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,retained_index},
      {attributes,[key,expiry_time]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{dets,[{auto_save,1000}]},
           {ets,
               [compressed,
                {read_concurrency,true},
                {write_concurrency,true}]}]},
      {cookie,
          {{1693065687469494982,-576460752303422813,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{19,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,225842}}}}]},
 {schema,emqx_delayed,
     [{name,emqx_delayed},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,true},
      {record_name,delayed_message},
      {attributes,[key,delayed,msg]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065678111567111,-576460752303423007,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{2,61},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,237902}}}}]},
 {schema,emqx_enhanced_authn_scram_mnesia,
     [{name,emqx_enhanced_authn_scram_mnesia},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,user_info},
      {attributes,[user_id,stored_key,server_key,salt,is_superuser]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065684176525817,-576460752303423357,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{15,10},
           {'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
               {1701,660370,348172}}}}]},
 {schema,emqx_trie,
     [{name,emqx_trie},
      {type,ordered_set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_trie},
      {attributes,[key,count]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065685361027907,-576460752303423230,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,200749}}}}]},
 {schema,emqx_app,
     [{name,emqx_app},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_app},
      {attributes,
          [name,api_key,api_secret_hash,enable,desc,expired_at,created_at]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065678325425274,-576460752303422911,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,252630}}}}]},
 {schema,emqx_exclusive_subscription,
     [{name,emqx_exclusive_subscription},
      {type,set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,exclusive_subscription},
      {attributes,[topic,clientid]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065685725162238,-576460752303423070,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,234938}}}}]},
 {schema,emqx_route_filters,
     [{name,emqx_route_filters},
      {type,ordered_set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,routeidx},
      {attributes,[entry,unused]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,auto}]}]},
      {cookie,
          {{1695117719694849871,-576460752303417790,1},
           'emqx@emqx-core-699f577dd8-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,217434}}}}]},
 {schema,emqx_admin_jwt,
     [{name,emqx_admin_jwt},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_admin_jwt},
      {attributes,[token,username,exptime,extra]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065678755233574,-576460752303422815,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,253496}}}}]},
 {schema,emqx_banned_rules,
     [{name,emqx_banned_rules},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,banned},
      {attributes,[who,by,reason,at,until]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1712595007635830501,-576460752303422896,1},
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{7,3},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,241171}}}}]},
 {schema,cluster_rpc_mfa,
     [{name,cluster_rpc_mfa},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,cluster_rpc_mfa},
      {attributes,[tnx_id,mfa,created_at,initiator]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065681678657421,-576460752303423485,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,256395}}}}]},
 {schema,emqx_routing_node,
     [{name,emqx_routing_node},
      {type,set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,routing_node},
      {attributes,[name,const]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065685544069539,-576460752303423165,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,213404}}}}]},
 {schema,emqx_ds_session,
     [{name,emqx_ds_session},
      {type,set},
      {ram_copies,[]},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {rocksdb_copies,
          ['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,session},
      {attributes,[id,iterators]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065686470964222,-576460752303422335,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{15,10},
           {'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
               {1701,660370,165129}}}}]},
 {schema,emqx_admin,
     [{name,emqx_admin},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_admin},
      {attributes,[username,pwdhash,description,role,extra]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065679931265846,-576460752303423484,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,254543}}}}]},
 {schema,bpapi,
     [{name,bpapi},
      {type,set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,bpapi},
      {attributes,[key,version]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065687063855530,-576460752303422910,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,258793}}}}]},
 {schema,emqx_ds_iterator_ref,
     [{name,emqx_ds_iterator_ref},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {rocksdb_copies,
          ['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,iterator_ref},
      {attributes,[ref_id,it_id,start_time]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1695117720120122338,-576460752303417630,1},
           'emqx@emqx-core-699f577dd8-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{15,10},
           {'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
               {1701,660370,131478}}}}]},
 {schema,emqx_activated_alarm,
     [{name,emqx_activated_alarm},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,true},
      {record_name,activated_alarm},
      {attributes,[name,details,message,activate_at]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065685986815759,-576460752303422431,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{2,61},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,255649}}}}]},
 {schema,emqx_acl,
     [{name,emqx_acl},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_acl},
      {attributes,[who,rules]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065683472063593,-576460752303423294,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,255685}}}}]},
 {schema,emqx_psk,
     [{name,emqx_psk},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,psk_entry},
      {attributes,[psk_id,shared_secret,extra]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065678067829202,-576460752303423103,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,231205}}}}]},
 {schema,emqx_channel_registry,
     [{name,emqx_channel_registry},
      {type,bag},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,channel},
      {attributes,[chid,pid]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065687167321371,-576460752303422239,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,237971}}}}]},
 {schema,emqx_banned,
     [{name,emqx_banned},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,banned},
      {attributes,[who,by,reason,at,until]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065685843327984,-576460752303423069,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,244950}}}}]},
 {schema,mria_schema,
     [{name,mria_schema},
      {type,ordered_set},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,mria_schema},
      {attributes,[mnesia_table,shard,storage,config]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065677861775661,-576460752303423327,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,262931}}}}]},
 {schema,cluster_rpc_commit,
     [{name,cluster_rpc_commit},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,cluster_rpc_commit},
      {attributes,[node,tnx_id]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065682855116971,-576460752303423358,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,257318}}}}]},
 {schema,'$mria_rlog_sync',
     [{name,'$mria_rlog_sync'},
      {type,set},
      {ram_copies,[]},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {null_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,'$mria_rlog_sync'},
      {attributes,[reply_to,shard,extra]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065678029072848,-576460752303423231,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,259774}}}}]},
 {schema,schema,
     [{name,schema},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,schema},
      {attributes,[table,cstruct]},
      {user_properties,
          [{mnesia_backend_types,
               [{rocksdb_copies,mnesia_rocksdb},
                {null_copies,mria_mnesia_null_storage}]}]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065677601354987,-576460752303423390,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{171,4},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,200725}}}}]},
 {schema,emqx_route,
     [{name,emqx_route},
      {type,bag},
      {ram_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_copies,[]},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,route},
      {attributes,[topic,dest]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
      {cookie,
          {{1693065685628943507,-576460752303423166,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,219865}}}}]},
 {schema,emqx_dashboard_monitor,
     [{name,emqx_dashboard_monitor},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,true},
      {record_name,emqx_monit},
      {attributes,[time,data]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065679589317207,-576460752303422719,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{2,61},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,237940}}}}]},
 {schema,emqx_retainer_index_meta,
     [{name,emqx_retainer_index_meta},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,retained_index_meta},
      {attributes,[key,read_indices,write_indices,reindexing,extra]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{dets,[{auto_save,1000}]},
           {ets,
               [compressed,
                {read_concurrency,true},
                {write_concurrency,true}]}]},
      {cookie,
          {{1693065687310684089,-576460752303422909,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{19,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,224323}}}}]},
 {schema,emqx_authn_scram_mnesia,
     [{name,emqx_authn_scram_mnesia},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,user_info},
      {attributes,[user_id,stored_key,server_key,salt,is_superuser]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1700095721479745429,-576460752303422206,1},
           'emqx@emqx-core-794b5bd5b5-1.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{14,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,250613}}}}]},
 {schema,emqx_deactivated_alarm,
     [{name,emqx_deactivated_alarm},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,true},
      {record_name,deactivated_alarm},
      {attributes,[activate_at,name,details,message,deactivate_at]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065686101778828,-576460752303423388,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{2,61},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,237925}}}}]},
 {schema,emqx_telemetry,
     [{name,emqx_telemetry},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,telemetry},
      {attributes,[id,uuid]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065688182842201,-576460752303422143,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,208394}}}}]},
 {schema,emqx_trace,
     [{name,emqx_trace},
      {type,set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,emqx_trace},
      {attributes,
          [name,type,filter,enable,payload_encode,extra,start_at,end_at]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[]},
      {cookie,
          {{1693065687234401845,-576460752303423005,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,205979}}}}]},
 {schema,emqx_retainer_message,
     [{name,emqx_retainer_message},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,retained_message},
      {attributes,[topic,msg,expiry_time]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,
          [{dets,[{auto_save,1000}]},
           {ets,
               [compressed,
                {read_concurrency,true},
                {write_concurrency,true}]}]},
      {cookie,
          {{1693065687345372571,-576460752303422814,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{19,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,222014}}}}]},
 {schema,emqx_authn_mnesia,
     [{name,emqx_authn_mnesia},
      {type,ordered_set},
      {ram_copies,[]},
      {disc_copies,
          ['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
           'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
      {disc_only_copies,[]},
      {load_order,0},
      {access_mode,read_write},
      {majority,false},
      {index,[]},
      {snmp,[]},
      {local_content,false},
      {record_name,user_info},
      {attributes,[user_id,password_hash,salt,is_superuser]},
      {user_properties,[]},
      {frag_properties,[]},
      {storage_properties,[{ets,[{read_concurrency,true}]}]},
      {cookie,
          {{1693065684858268518,-576460752303422591,1},
           'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
      {version,
          {{18,18},
           {'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
               {1712,595312,251641}}}}]}]ok

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

Thanks for the info, so the problem is there is one deprecated table (emqx_enhanced_authn_scram_mnesia) since 5.3.1 was used by old version of EMQX on old nodes. However this table are no longer in use by new nodes.

Now when you call nodes to leave, the cluster complains because the old deleted nodes has the copy which shouldn't be deleted as they are the only nodes holding the original copy of the data.

Did you upgrade the EMQX cluster with emqx-operator? could you recall the command you use for us to reproduce?

ping @Rory-Z to get some attention as it was deplyed and managed by Emqx-operator.

on the other hand, while we are investigating are you willing to run some hacking command to get these nodes out of your cluster?
I won't recommend to do that if you have live nodes. prefer to reproduce the issue in a staging env and try out the hacking commands.

from emqx.

ruifung avatar ruifung commented on May 24, 2024

I do not believe I will be able to reliably reproduce this on demand now, given that I updated to 5.3.1 over 5 months ago. And for the most part I just update the EMQX CRD spec in git and it gets applied to my k8s cluster automatically where the operator proceeds to update the emqx cluster.

I have no issue with running command hacks on it as it's a homelab environment and not anything production-critical. It would be appreciated as it would likely be easier then having to set up all the authentication credentials again for MQTT and updating various IoT devices using them.

from emqx.

qzhuyan avatar qzhuyan commented on May 24, 2024

without reproducing the issue it is hard to find a proper commands for you to remove the dead nodes.

if possible, I think it is easier just to recreate your home lab cluster.

But if you want to try we have some hacks: NOTE: it may corrupt you entire data cause cluster refuse to come up.

  1. In the emqx container, Open emqx remote console with
emqx remote_console
  1. copy & paste following commands
DeadNodes = ['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
                       'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
                       'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local'].
rr(mnesia_schema).
Tab = emqx_enhanced_authn_scram_mnesia.
lists:foreach(fun(Node) -> mnesia:add_table_copy(Tab, Node, ram_copies) end, mria:running_nodes()),
[{K,#cstruct{disc_copies = DC, ram_copies = RC} = V}] = ets:lookup(mnesia_gvar, {Tab, cstruct}).
ets:insert(mnesia_gvar, {K,V#cstruct{disc_copies = DC--DeadNodes, ram_copies = RC--DeadNodes}}).
{atomic, ok} = mnesia:delete_table(Tab).

  1. then exit the remote console with `Ctl-G' then 'q' enter...

  2. then retry with force-leave commands...

  3. you will hit other tables, then just repeat 1-4 with other tables. Maybe it will be easier with another layer of scripts.

from emqx.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.