Comments (6)
you can lower the reservation time for that job type using the “reserve_for” attribute. The default is 1800 seconds but lowering it to 60 seconds or 300 seconds will get you quicker retries. Also, keep in mind that if your job is crashing the worker process, you don’t want to retry too quickly or your app will spend a significant amount of time simply booting the worker process.
from faktory.
@mperham Thanks for reply. I do a job that sending http request to check order status every 10 seconds in 10 mins. That means this job will sits in busy
for 10 mins. But When I restart faktory for app upgrade in production, the job won't be retried until reach reserve_for
. So I lost my order status in the remain time because of the every 10 seconds order query function won't be called.
I saw the doc says:
Upon seeing "terminate", the worker process should wait up to N seconds for any remaining jobs to finish. After 25 seconds (see below), the worker should send FAIL to Faktory for those lingering jobs (so they'll restart) and exit.
Then I test for this but what I found is that the worker seems don't send FAIL to Faktory for those lingering jobs (so they'll restart)
, because code is hangs in https://github.com/contribsys/faktory_worker_go/blob/main/manager.go#L122:
func (mgr *Manager) Terminate(reallydie bool) {
mgr.mut.Lock()
defer mgr.mut.Unlock()
if mgr.state == "terminate" {
return
}
mgr.Logger.Info("Shutting down...")
mgr.state = "terminate"
close(mgr.done)
mgr.fireEvent(Shutdown)
-> mgr.shutdownWaiter.Wait()
mgr.Pool.Close()
mgr.Logger.Info("Goodbye")
if reallydie {
os.Exit(0) // nolint:gocritic
}
}
Until the busy jobs finish, then mgr.Pool.Close()
.
I try to figure out a workaround, I write code as the following:
faktoryMgr.On(worker.Shutdown, func(manager *worker.Manager) error {
manager.Pool.With(func(conn *faktory.Client) error {
var job *faktory.Job
var e error
for {
job, e = conn.Fetch("labors")
if e != nil {
return e
}
if job == nil {
return nil
}
conn.Fail(job.Jid, errors.New("force labors jobs fail because of app restarting"), nil)
}
})
return nil
})
However, the Fail
method causes the task to fail directly, rather than pushing it into retries
.
from faktory.
It sounds like your deploy is killing the worker process instead of giving it time to finish pending jobs and shut down cleanly. The Faktory worker library does attempt to FAIL unfinished jobs so it can immediately restart them after restart but it can’t do this if you don’t give it time to shutdown. It’s common to make this mistake with Kubernetes.
from faktory.
I give it time to shutdown, but I found that mgr.shutdownWaiter.Wait()
will wait up to 10 mins in my case. Because I use channel
to force job running for 10 minutes. What I wonder is that if there is a method to force job to retry immediately?
from faktory.
Ah right, it looks like FWR does but FWG doesn't have a hard shutdown timer. It will wait minutes for jobs to finish. That's not really desired; I should implement a 30 sec hard shutdown.
from faktory.
Fixed.
from faktory.
Related Issues (20)
- Feature suggestion: Unwrap ActiveJob wrapped Job names in the WebUI HOT 6
- Handling server upgrades? HOT 4
- Custom data stripped on batch callbacks for trace metadata propagation HOT 1
- Feature Request: Unique jobs even after completion HOT 2
- Ent Faktory Batches Guarantees HOT 1
- Does Faktory have any notion of job cost? HOT 3
- Add direct TLS support HOT 2
- Publish enterprise release binary for Linux to aid in testing HOT 1
- Faktory Enterprise shutting down messages in the log file - no explanation as to why Faktory is shutting down.
- REST API for UI HOT 2
- [Feature Request] Include Redis RTT in statsd metrics that are emitted HOT 3
- [Feat Request] Improve the reliability of Faktory server when overloaded with connections HOT 1
- Support Env Var expansion in toml config files HOT 3
- Question: What is the unit of "time" for Ent Metrics HOT 6
- Crashlooping on start HOT 16
- deb packages for arm64 HOT 3
- bzip2 deprecated
- Faktory crashed with fatal error: concurrent map writes HOT 3
- Is it possible to run the faktory server in a HA setup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faktory.