Comments (5)
github-actions
commented on July 21, 2024
👋 Thanks for opening this issue!
Get help or engage by:
/help: to print help messages./assignme: to assign this issue to you.
from openraft.
drmingdrmer
commented on July 21, 2024
A panic is a logic bug.
Could you provide the debug level logs? This way I can see what's happening when it panicked.
from openraft.
web3creator
commented on July 21, 2024
2023-11-04T03:03:21.296813Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main: openraft::core::raft_core: raft node is initializing
2023-11-04T03:03:21.296848Z INFO tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup: openraft::engine::engine_impl: startup: state: RaftState { vote: UTime { data: Vote { leader_id: LeaderId { term: 1, node_id: 7395302447718541494 }, committed: true }, utime: Some(Instant { tv_sec: 464375, tv_nsec: 700082041 }) }, committed: Some(LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }), purged_next: 10001, log_ids: LogIdList { key_log_ids: [LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }] }, membership_state: MembershipState { committed: EffectiveMembership { log_id: None, membership: Membership { configs: [], nodes: {} }, voter_ids: {} }, effective: EffectiveMembership { log_id: None, membership: Membership { configs: [], nodes: {} }, voter_ids: {} } }, snapshot_meta: SnapshotMeta { last_log_id: Some(LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }), last_membership: StoredMembership { log_id: None, membership: Membership { configs: [], nodes: {} } }, snapshot_id: "10000-2-10000-1" }, server_state: Learner, accepted: Accepted { leader_id: LeaderId { term: 0, node_id: 0 }, log_id: None }, io_state: IOState { building_snapshot: false, vote: Vote { leader_id: LeaderId { term: 1, node_id: 7395302447718541494 }, committed: true }, flushed: LogIOId { leader_id: LeaderId { term: 0, node_id: 0 }, log_id: None }, applied: Some(LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }), purged: Some(LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }) }, snapshot_streaming: None, purge_upto: Some(LogId { leader_id: LeaderId { term: 10000, node_id: 2 }, index: 10000 }) }
2023-11-04T03:03:21.297285Z INFO tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup: openraft::engine::engine_impl: startup: is_leader: true
2023-11-04T03:03:21.297298Z INFO tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup: openraft::engine::engine_impl: startup: is_voter: false
2023-11-04T03:03:21.299352Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup:calc_server_state: openraft::raft_state: states is_member=false is_leader=true is_leading=true
2023-11-04T03:03:21.299383Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup: openraft::engine::handler::server_state_handler: update_server_state_if_changed id=7395302447718541494 prev_server_state=Learner server_state=Learner
2023-11-04T03:03:21.300196Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup: openraft::engine::handler::replication_handler: openraft::engine::handler::replication_handler::ReplicationHandler<_>::update_local_progress upto=10000-2-10000
2023-11-04T03:03:21.300225Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:startup:initiate_replication: openraft::engine::handler::replication_handler: openraft::engine::handler::replication_handler::ReplicationHandler<_>::initiate_replication progress=VecProgress { quorum_set: Joint { data: [], _p: PhantomData<(u64, alloc::vec::Vec<u64>)> }, granted: None, voter_count: 0, vector: [], stat: Stat { update_count: 0, move_count: 0, is_quorum_count: 0 } }
2023-11-04T03:03:21.300296Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands: openraft::core::raft_core: queued commands: start...
2023-11-04T03:03:21.300307Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands: openraft::core::raft_core: queued commands: RebuildReplicationStreams { targets: [] }
2023-11-04T03:03:21.300317Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands: openraft::core::raft_core: queued commands: end...
2023-11-04T03:03:21.300325Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands: openraft::core::raft_core: run command: RebuildReplicationStreams { targets: [] }
2023-11-04T03:03:21.300346Z DEBUG tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands: openraft::core::raft_core: condition: None
2023-11-04T03:03:21.300382Z INFO tokio-runtime-worker ThreadId(06) new{cluster=morse_brain}:RaftCore{id=7395302447718541494 cluster=morse_brain}:main:run_engine_commands:remove_all_replication: openraft::core::raft_core: remove all replication
thread 'tokio-runtime-worker' panicked at /Users/simonyi/.cargo/git/checkouts/openraft-0677bbb78cc57964/4b78dd3/openraft/src/core/raft_core.rs:797:13:
internal error: entered unreachable code: it has to be a leader!!!
from openraft.
drmingdrmer
commented on July 21, 2024
What steps did you take to arrive at this particular status?
Did you manually fakes these status for Openraft?
From the provided log, it appears that the membership is empty:
Membership { configs: [], nodes: {} }, voter_ids: {} }
and the vote is non-empty:
Vote { leader_id: LeaderId { term: 1, node_id: 7395302447718541494 }
The lack of initialization for Openraft is evident, since an empty membership is not permissible when initializing Openraft.
Reason and Solution
Openraft's handling of this unexpected scenario seems inconsistent. In this context:
calc_server_state()returnsLearner.- But
is_leader()returns true.
Given that calc_server_state() returns Learner, Openraft does not initialize the Leader state, which accounts for the encountered panic.
To resolve this issue, I will modify calc_server_state() to return Lender, which should help achieve internal consistency.
Added 2023-11-06:
To resolve this issue, I will update calc_server_state() and is_leader() so that they return consistent result according to the following table:
For node-2:
-
Node-2 with vote
(term=1, node_id=2, committed=true):- is a leader if it is present in config, either a voter or non-voter.
- is a learner if it is absent in config.
-
Node-2 with vote
(term=1, node_id=2, committed=false):- is a candidate if it is present in config, either a voter or non-voter.
- is a learner if it is absent in config.
-
Node-3 with vote
(term=1, node_id=99, committed=false|true):- is a follower if it is a voter in config,
- is a learner if it is a non-voter or absent in config.
| vote \ membership | Voter | Non-voter | Absent |
|---|---|---|---|
| (term=1, node_id=2, committed=true) | leader | leader | learner |
| (term=1, node_id=2, committed=false) | candidate | candidate | learner |
| (term=1, node_id=99, committed=true) | follower | learner | learner |
| (term=1, node_id=99, committed=false) | follower | learner | learner |
from openraft.
drmingdrmer
commented on July 21, 2024
The following PR is merged. In your case, the node should start as a Learner now. Please confirm that the issue is addressed.
from openraft.
Related Issues (20)
- Feature: Monoio Runtime HOT 3
- Add sync primitives to `AsyncRuntime` trait HOT 1
- Hide `RaftStorage` and similar types from generic arguments to `Raft`/`RaftInner` HOT 7
- RaftMsg::ExternalRequest: should not rely on RaftLogStorage and RaftNetworkFactory HOT 3
- Add random number generation to `Runtime` HOT 5
- Ditch `async_trait` in favor of the official `async_fn_in_trait` feature HOT 6
- Refactor: Move `client_resp_channels` from `LeaderData` to new `tx_...`-field in `RaftCore` HOT 2
- Linearizable read with `ReadIndex` HOT 1
- about single node HOT 6
- Cluster Bootstrap and Node Initialization Sequence HOT 4
- Tracking issue for examples update HOT 1
- Update example `raft-kv-memstore` to use `storage-v2` HOT 2
- Update example `raft-kv-rocksstore` to use `storage-v2` HOT 1
- Move examples into the workspace HOT 1
- Install Snapshot v1 api HOT 3
- Observe state changes in a Raft node HOT 6
- Split metrics into data metrics and server metrics HOT 5
- About automatic remove HOT 3
- Backing up the WAL HOT 6
- Raft Core Panicking HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openraft.