Comments (13)
Faktory doesn't support multi-server clustering. You can set up Redis replicas for data redundancy but only one Faktory can be operational at a time.
from faktory.
Oh that's not great. It's a bit scary to have one point of failure like that. I believe we contacted you about this earlier and you said it would be possible to have two servers behind a load balancer, but I'll double check that.
from faktory.
You can but they need to run completely independent. For instance if you have two with the same configuration, they will both fire cron jobs. There's no concept of primary/standby or cluster leadership.
from faktory.
Hi again, we updated our configuration to have a "primary" and "secondary" servers, with only the primary scheduling cron jobs. This seems to work for scheduling jobs, but for some reason we're seeing jobs complete and the being enqueued again, about half an hour after the job completed. Any ideas about what's would be the cause of that? the servers are sharing one redis (elasticache) instance.
Any help would be most appreciated!
from faktory.
I think 30 minutes is the default job reservation time. If you fetch job A from server 1 but don't ack it within 30 minutes, Faktory will re-enqueue job A for re-execution under the assumption that the worker failed somehow.
from faktory.
Interesting. I wonder why it's not sending ack then. Also doesn't seem like it should be caused by a multi-server setup.
from faktory.
Is it possible your worker is FETCHing from Server 1 but ACKing to Server 2?
from faktory.
It is possible, but wouldn't it update the same redis record?
from faktory.
Faktory keeps a list of outstanding jobs in memory, keyed by JID. If it's not in its records, it'll return an error. You can argue that Faktory should go to Redis for this operation but as Faktory is not designed to run in parallel, that's the way it works today.
from faktory.
Hmmm I'm not sure how to proceed here then... would it be possible to change the implementation there? do you agree that
there should be a capability of having redundancy on the Faktory server?
from faktory.
Faktory does not support clustering. You can use Redis replicas to get a real-time backup. I'm not planning any changes here, redundancy is worthwhile but would add a tremendous amount of complexity to the system. Are you trying to solve a real problem or an imaginary one? Is Faktory reliability really an issue for you?
from faktory.
It's not an issue we faced, but I wouldn't call it imaginary. We have redundancy with almost every part of our infrastructure, being proactive about mitigating issues rather than waiting until something fails.
Faktory reliability is an issue since if the server goes down enqueueing jobs will fail, which will result in errors affecting our users directly.
Our other option is to have a fallback of saving jobs to another database if the enqueueing fails, but I would consider that a hacky workaround. Open to any suggestions you might have.
from faktory.
The one option you can roll yourself is two shards. You can run two independent Faktorys in two different AZs/DCs and have your clients use some algorithm to choose their nearest Faktory. If that Faktory is unavailable, they can push to the backup Faktory.
But at the end of the day, Faktory is a primary datastore. Just as you don't really expect your app to be usable when postgres goes down, you should expect similar semantics here. I haven't had a report of Faktory crashing in over a year now? So I hope you can test and verify Faktory's reliability.
from faktory.
Related Issues (20)
- Feature suggestion: Unwrap ActiveJob wrapped Job names in the WebUI HOT 6
- Handling server upgrades? HOT 4
- Custom data stripped on batch callbacks for trace metadata propagation HOT 1
- Feature Request: Unique jobs even after completion HOT 2
- Ent Faktory Batches Guarantees HOT 1
- Does Faktory have any notion of job cost? HOT 3
- Fail jobs immediately when worker crashes HOT 6
- Add direct TLS support HOT 2
- Publish enterprise release binary for Linux to aid in testing HOT 1
- Faktory Enterprise shutting down messages in the log file - no explanation as to why Faktory is shutting down.
- REST API for UI HOT 2
- [Feature Request] Include Redis RTT in statsd metrics that are emitted HOT 3
- [Feat Request] Improve the reliability of Faktory server when overloaded with connections HOT 1
- Support Env Var expansion in toml config files HOT 3
- Question: What is the unit of "time" for Ent Metrics HOT 6
- Crashlooping on start HOT 16
- deb packages for arm64 HOT 3
- bzip2 deprecated
- Faktory crashed with fatal error: concurrent map writes HOT 3
- Is it possible to run the faktory server in a HA setup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faktory.