Comments (3)
Fencing: take a look at the setting and code around maximum_lag_on_failover
. Each time the loop runs, we log the last XLOG position for the primary Postgres. With maximum_lag_on_failover
, you can tailor the maximum bytes between the recorded value and a follower deemed healthy enough to failover.
Quorum: take a look at this method https://github.com/compose/governor/blob/master/helpers/etcd.py#L81. With it, we use the built in prevExist=false
, it is a check and set action built into etcd. We rely on it to etcd's RAFT and prevExit to ensure there are no ties.
from governor.
Thanks for the fast reply!
Fencing: The maximum_lag_on_failover
parameter makes sense in terms of preventing the failed node from becoming a candidate for failover, but what prevents a rogue node from coming back at some point and haproxy starting to send traffic to it? The stonith story is a good illustration of why the fencing mechanism needs to ensure that any rogue node is reset to a known clean state (e.g with a reboot via IPMI)
Quorum: okay, so it sounds like etcd's RAFT ensures that a race condition between multiple nodes cannot end in a tie or multiple concurrent masters?
from governor.
Great questions.
Haproxy status checks rely on etcd as a single source of truth. If a rogue node can take over the leader key, it would start receiving reads and writes. The maximum_lag_on_failover
is one convention used to prevent that -- particularly when all nodes are shut down and nodes are started one at a time. Take a look at the blocker method is_healthiest_node
(https://github.com/compose/governor/blob/master/helpers/postgresql.py#L125). The is_healthiest_node
method is called before any node tries to take over as a leader.
If an etcd namespace is shared between Postgres Governor clusters, Governor doesn't prevent the cluster from comparing themselves to nodes of other clusters. It'd be awesome if you could code something up to prevent rogue clusters from trying to overthrow the current cluster's etcd namespace.
Take a look at the code and throw some scenarios at the code base. If you have a scenario of a rogue takeover that can be recreated, we'd love to solve for that situation.
from governor.
Related Issues (20)
- database system identifier differs between the primary and standby? HOT 2
- etcd returns 500 internal server error on ubuntu which causes postgres to crash. HOT 1
- 404 error causing the postgres to go down
- Use python-etcd client library HOT 1
- Fatal: requested timeline 8 is not a child of this server's history HOT 3
- Local Docker cluster with Governor on board HOT 1
- Make governor a module
- PostgreSQL + haproxy with multiple IP HOT 3
- not catching ssl timeout exception HOT 1
- rewind ex-leader before joining again HOT 2
- [Errno 32] Broken pipe HOT 1
- non atomic has_lock() and update_lock()
- cannot easily "go build" golang-custom-raft; maybe we should have a new project? HOT 1
- golang-custom-raft: add abilty to execute post-election script HOT 14
- New GB build tool based off of golang-custom-raft branch with a new name - hapg HOT 2
- golang-custom-raft: If a PG process is unhealthy - it can kill governor
- golang-custom-raft: maximum_lag_on_failover doesn't work as it should
- help:the connect info in the recover.conf are "None"
- replication slots failing when names contain dashes
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from governor.