Comments (4)
👋 Thanks for opening this issue!
Get help or engage by:
/help
: to print help messages./assignme
: to assign this issue to you.
from openraft.
What script were you using to produce this issue?
I think you're using one of the shell script in the example.
You can just push the modify code base to a branch so I can reproduce this issue using your branch.
And please attach the full log in DEBUG level so I can see what was happening.
And I don't understand why you believe that step two/three should fail.
Adding non-voter without join them to a cluster should always work.
from openraft.
And I don't understand why you believe that step two/three should fail.
sorry, my bad, they should be step 4/5.
Adding non-voters with different NodeId always works, but after upgrading them to voter, killing the leader, the whole cluster stopped.
I think you can reproduce it with the mem kv example. To express myself better, I redescribe it in Chinese:
如何重现
有三个节点,node0/1/2,对应的 RaftNetwork 地址分别是 addr 0/1/2,启动这三个节点时,传给 openraft::Raft::new 的 NodeId 分别是 0/1/2
- 启动 node0,作为单节点 cluster。
- 启动 node1,作为 learner。
- 启动 node2,作为 learner。
- 在 node0 上调用 add_learner,但是用 NodeId = 2 来添加 node1 (addr1)
- 在 node0 上调用 add_learner,用 NodeId = 1 来添加 node2 (addr2)
- 在 node0 上调用 change_membership,把两个新加进来的 node 升级为 voter
- kill node0, the leader
期待结果
下面两种结果都可以接受:
- 步骤 4/5 直接报错。因为 change_membership 日志中记录 id 和 raft 启动时的 id 不符,这导致了在 learner 升级成 voter 且 leader 挂掉之后整个集群不再工作。或者
- 选举正常工作,哪怕 node1/2 的 id 被改掉了。
相比之下 1 会简单一些,2 可能需要在 learner 被加入 cluster 时去做个判断,并更新自己的 id,目测这个需要梳理很多逻辑来保证更新 id 的正确性。
实际结果
node 1/2 stop working, "openraft::core::tick: Tick fails to send, receiving end quit: channel closed"
from openraft.
OK, I see the root cause of this problem is this, node-2 expected itself to be a follower but it is acutally a leader:
thread 'main' panicked at openraft/openraft/src/engine/engine_impl.rs:793:9:
assertion failed: self.internal_server_state.is_following()
Because you mistakenly configured a wrong network addresses for node-3: the endpoint address of node-3 is the address of node-2.
When node-1 is killed and node-2 becomes the leader, node-2 tries to replicate a message to node-3 by replicating it to itself.
And Openraft only allows a follower to receive replication messages.
Because node-2 itself is a leader, the assertion fails.
from openraft.
Related Issues (20)
- semi-async `RaftLogReader::try_get_log_entries` HOT 3
- Non-blocking `RaftStateMachine::apply()` HOT 5
- `AsyncRuntime::unbounded_mpsc` HOT 2
- `AsyncRuntime::watch` HOT 3
- Update quickstart to include Responder for RaftTypeConfig HOT 2
- Mention `server_metrics()` and `data_metrics()` in `FAQ How to get notified when the server state changes` HOT 1
- Metrics: add last acked heartbeat timestamp for each follower/learner HOT 2
- Timekeeping in standard units HOT 3
- Generalize `AnyError` HOT 1
- Direction towards allocation-free operations HOT 22
- Change a voter to learner should not affect existing learners. HOT 1
- Uniform `AsyncRuntime` channel interfaces HOT 2
- Add test suite to ensure the correctness of `AsyncRuntime` implementation. HOT 6
- Migrate Rust toolchain action from `actions-rs/toolchain` to an action that is still maintained HOT 1
- New release to remove version pin of proc-macro2 HOT 3
- Logs spam from `openraft::engine::handler::vote_handler` HOT 3
- Improper use of `Raft::metrics()` can block the RaftCore async task and underlying OS thread HOT 1
- panic in `src/membership/membership.rs:309:51` HOT 5
- Missing log replay on single-node restart (bug?) HOT 4
- Add an API that blocks until logs replay is done if `save_committed()` is implemented HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openraft.