flexible reply for sending entries about openraft HOT 9 CLOSED

So this is the partial success reply? send_append_entries() has to be able to return what log entries are replicated and what not, instead of a single bool.

There are two parts to this:

• timeout handling for large batches
• partial success reporting

For the former, it's sufficient to pass TTL into the method instead of timeouting on the outside. The Network implementation can then apply the timeout to individual messages/sub-batches.

The latter is the possibility to advance commit index before the entire batch is processed. Here, the Network implementation might return early, indicating Success(LogIndex) instead of plain Success, so the caller can adjust the commit index and do the needful before diving back to Network with remaining entries to process.

Alternatively, it could be possible to completely decouple it (streaming API), so the Network would send back the status via a callback batch-wise, similar to the updates done to the logging already. I.e., there would be no big difference between local logging and log replication. Potentially, even the same callback type could be used.

from openraft.

I think this would be a really neat feature that improves network reliability and would work well hand in hand with unreachable state or even initial joins.

One thing we've been doing in tremor is using dynamic backpressure from the downstream system to throttle messages (either drop which isn't reasonable here, or slow down sending which might work well).

Perhaps a similar logic could be added here? I think with a timeout hint it could even be done inside the Network implementation and not force openraft to make a decision, a downstream system could send back a "please slow don" message that the network protocol implements, and then the sender goes and increases timeouts or adds delays as required.

from openraft.

I think with a timeout hint it could even be done inside the Network implementation and not force openraft to make a decision

Exactly, this was the intention :-).

from openraft.

Many messaging systems use always increasing message IDs and sparse akcs to allow for batching (in this fashion) without needing to have full batches to be confirmed. I think we can "steal" the idea here. The basic idea is not every message has to be acknowledged but any acknowledgement given means this and everything before was received and processed. I think that would satisfied @schreter's requirement and be quite scalable as we still have 1 ack / batch in most cases even during partial acks

from openraft.

Certainly! At the moment, openraft follows a request/response design. Implementing a streaming pattern as you proposed would bring about increased flexibility and efficiency.

I have been focusing on refactoring the storage API to a stream-like interface, after which the network API will be revised accordingly.

from openraft.

Ja no hurry, I'm just thinking out loud :)

from openraft.

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

from openraft.

@schreter :

I think @Licenser 's requirement is to stop sending out anything if a node is known to be unreachable(send_append_entries() returns Unreachable), and the case you mentioned is another problem(send_append_entries() returns NetworkError).

Pass in a ttl to send_append_entries() would be great!

And with current framework, openraft can not deal with a partial successful result, and it will always resend the full bulk of entries again.

For now, maybe it is feasible to let the network implementation return a timeout hint back to openraft therefore the next time openraft will use a bigger timeout.

I think you were talking about a stream like send-entries API, with which the a big bulk of entries is split into small chunks.
Let me add it when the storage v2 refactoring is done.

from openraft.

I think with a timeout hint it could even be done inside the Network implementation and not force openraft to make a decision

Exactly, this was the intention :-).

So this is the partial success reply? send_append_entries() has to be able to return what log entries are replicated and what not, instead of a single bool.

from openraft.