After the new backoff feature was implemented with <a class="issue-link js-issue-link"

👋 Thanks for opening this issue! Get help or engage by: <ul dir

Possible issue with backoff when multiple nodes are unreachable about openraft HOT 4 CLOSED

kevlu93 commented on September 22, 2024

Possible issue with backoff when multiple nodes are unreachable

from openraft.

Comments (4)

github-actions commented on September 22, 2024

👋 Thanks for opening this issue!

Get help or engage by:

/help : to print help messages.
/assignme : to assign this issue to you.

from openraft.

drmingdrmer commented on September 22, 2024

You can not kill two nodes in a cluster of only three nodes. A majority requires at least two nodes.
The whole cluster will inevitably become unresponsive, whether or not using an "Unreachable" error and backoff.

The leader has received two Unreachable errors for all of the target node it replicates to. Thus by the design, the leader decide not to send anything out. This is as expected.

There should be several warning messages in the log, emitted by the two logging statements below.

openraft/openraft/src/replication/mod.rs

Lines 255 to 262 in 828ad87

 if let Some(b) = &mut self.backoff { 

 let duration = b.next().unwrap_or_else(|| { 

 tracing::warn!("backoff exhausted, using default"); 

 Duration::from_millis(500) 

 }); 

 self.backoff_drain_events(Instant::now() + duration).await?; 

 }

openraft/openraft/src/replication/mod.rs

Lines 475 to 481 in 828ad87

 pub async fn backoff_drain_events(&mut self, until: Instant) -> Result<(), ReplicationClosed> { 

 let d = until - Instant::now(); 

 tracing::warn!( 

 interval = debug(d), 

 "{} backoff mode: drain events without processing them", 

 func_name!() 

 );

from openraft.

kevlu93 commented on September 22, 2024

Thanks for the explanation. Yes I see those two warning messages, but are there any messages emitted indicating that the leader recognizes that all nodes are unreachable and it should no longer send anything out? I looked through the logs and wasn't seeing anything obvious. Also, would the same behavior occur if the leader receives multiple Network or Timeout errors from all of the target nodes?

from openraft.

drmingdrmer commented on September 22, 2024

No such log message indicating that all nodes are unreachable.

And the critical state is not that all nodes are unreachable, but that a quorum of nodes are unreachable.

There is no such a log message because it may misleading: the leader can not be sure if it is a problem with the target node or a problem with the leader node.

To get a correct view of the cluster state, you must contact each node directly.

from openraft.

Recommend Projects

Possible issue with backoff when multiple nodes are unreachable about openraft HOT 4 CLOSED

Comments (4)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	if let Some(b) = &mut self.backoff {
	let duration = b.next().unwrap_or_else(\|\| {
	tracing::warn!("backoff exhausted, using default");
	Duration::from_millis(500)
	});

	self.backoff_drain_events(Instant::now() + duration).await?;
	}

	pub async fn backoff_drain_events(&mut self, until: Instant) -> Result<(), ReplicationClosed> {
	let d = until - Instant::now();
	tracing::warn!(
	interval = debug(d),
	"{} backoff mode: drain events without processing them",
	func_name!()
	);