Backing up the WAL about openraft HOT 6 CLOSED

Just my 2c: I don't think there is a generic way to back up logs. We'll also back up to S3, but that's just one of the options. We must support 5 different hyperscalers, all with different APIs. From openraft POV, the log is continuous and how you cut it for backup is your beer. In our case, once a log segment is filled, it can be backed up. Once openraft tells us to purge, it's removed locally (unless the backup didn't yet finish).

from openraft.

Hi @drmingdrmer, would you be open to incorporating that logic into openraft? This way, users wouldn't have to replicate the same work.

from openraft.

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

from openraft.

In databend, we just export all of the log and the snapshot to a file and backup it. The following is the server side export implementation:
https://github.com/datafuselabs/databend/blob/313b40ed3ff1cfe001bc6e2f806773d9001861d0/src/meta/service/src/store/store_inner.rs#L394

Fix the above url

from openraft.

@drmingdrmer Do you do this in a continuous basis for databend or is this happens after every couple of hours?

from openraft.

@drmingdrmer Do you do this in a continuous basis for databend or is this happens after every couple of hours?

It is a cronjob and is executed by schedule. It is not a incremental backup :(

from openraft.