Comments (9)
Lines 84 to 87 in 1e5465d
from raft.
- n1 becomes leader in term 3; it has an existing log entry at index 3969 from term 2, and it replicates a barrier entry (term 3) at index 3970
- n3 receives an append request from n1 for just the barrier entry, and refuses it because it doesn't have an entry at index 3969 (based on logLastIndex)
- n1 tries again, sending a request for two entries (indices 3969 and 3970)
- this time, there is at least a trace on n3 of an entry at index 3969 (presumably not 3970), also with term 2, because n3 finds a slot that tracks such an entry in its refcount hashmap for the in-memory log
It seems like the refcount hashmap on n3 is getting out of sync with the value reported by logLastIndex...
from raft.
Ahh, and importantly:
- n3 is the leader in term 2, and its disk write for the entry at 3969 fails, causing it to step down
So I think the cleanup after the disk write failure must be buggy, missing a refcount decrement maybe
from raft.
It's possible that this was exposed by #460
from raft.
Ahh, and importantly:
- n3 is the leader in term 2, and its disk write for the entry at 3969 fails, causing it to step down
So I think the cleanup after the disk write failure must be buggy, missing a refcount decrement maybe
My 2c: when struct raft_io
reports a disk write failure (i.e. raft_io->append()
's callback fires with a non-zero status
), struct raft
should somehow schedule a raft_io->append
retry later in the future, rather than rolling back its own in-memory log and rely on the leader to resend everything and retrigger the whole behavior. Rolling back is always hard and it's easy to miss details, while retrying should be relatively straighforward (or at least it limits any rollback logic to the internals of struct raft_io
).
In case the write failure is ENOSPC
the follower struct raft
could also take note of that and stop accepting new AppendEntries messages until the situation resolves.
from raft.
Hmm, couldn't the same goal be achieved by refusing to append the entries to the in-memory log until they've been persisted?
In any case, I'd like to fully understand what's going on to cause this specific broken invariant; there might be more to the story than what I've been able to deduce so far...
from raft.
This seems to be somewhat reproducible in the scheduled runs, e.g. https://github.com/canonical/jepsen.dqlite/actions/runs/5977971227/job/16219112428
from raft.
https://github.com/canonical/jepsen.dqlite/actions/runs/6091570581/job/16528374348
from raft.
(fixed by #483)
from raft.
Related Issues (20)
- missing getrandom on centos 7 HOT 1
- [question] Usage of raft_add, raft_remove, and raft_assign HOT 2
- [question] When is the first leader elected? HOT 2
- recvAppendEntries: Assertion `r->state == RAFT_FOLLOWER || r->state == RAFT_CANDIDATE` failed HOT 9
- [question] Forwarding request to leader HOT 8
- Potential use-after-free in handling of raft_transfer HOT 5
- Fix the 32-bit CI HOT 1
- can't build on m1 mac (`<linux/xxxxx.h>` is missing) HOT 3
- v1.x RFC: pull based approach HOT 5
- raft_start(): io: closed segment xxx is past last snapshot xxx HOT 4
- [question] Understanding how to add new servers HOT 15
- ./configure fails when no external dependency is found HOT 4
- Assertion: src/uv_truncate.c:168: UvTruncate: Assertion `index < uv->append_next_index' failed. HOT 3
- src/replication.c:457: getRequest: Assertion `req->type == type' failed. HOT 3
- Segment writes blocked when taking a snapshot. HOT 3
- Jepsen: Another truncate-related assertion failure
- [suggestion] add lock/unlock function pointers to raft_io (and use them) HOT 1
- `uvOsFallocateEmulation` taking too long time HOT 1
- CI: `xfs` test failures HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raft.