Comments (5)
mperham
commented on August 20, 2024
Logging about staging is on main but has not been released.
It sounds like that error is with faktory_worker_ruby, not Faktory itself.
from faktory.
mperham
commented on August 20, 2024
Can you please include the full backtrace and any initializer code for faktory_worker_ruby?
from faktory.
Deekor
commented on August 20, 2024
We have no initializer code.
Backtrace:
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:77 :in `block (2 levels) in pop`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:69 :in `loop`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:69 :in `block in pop`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:68 :in `synchronize`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool/timed_stack.rb:68 :in `pop`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool.rb:125 :in `checkout`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool.rb:107 :in `block in with`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool.rb:106 :in `handle_interrupt`
[GEM_ROOT]/gems/connection_pool-2.4.1/lib/connection_pool.rb:106 :in `with`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/job.rb:117 :in `block in client_push`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/middleware/chain.rb:129 :in `block in invoke`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/middleware/batch.rb:14 :in `call`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/middleware/chain.rb:131 :in `block in invoke`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/middleware/chain.rb:134 :in `invoke`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/job.rb:116 :in `client_push`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/job.rb:80 :in `perform_async`
[GEM_ROOT]/gems/faktory_worker_ruby-1.3.0/lib/faktory/job.rb:132 :in `perform_async`
[PROJECT_ROOT]/app/services/sendgrid/webhooks/handler.rb:24 :in `block in process_webhook`
[PROJECT_ROOT]/app/services/sendgrid/webhooks/handler.rb:12 :in `each`
[PROJECT_ROOT]/app/services/sendgrid/webhooks/handler.rb:12 :in `process_webhook`
[PROJECT_ROOT]/app/controllers/webhooks_controller.rb:8 :in `sendgrid`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/basic_implicit_render.rb:6 :in `send_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/abstract_controller/base.rb:215 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/rendering.rb:165 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/abstract_controller/callbacks.rb:234 :in `block in process_action`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:99 :in `run_callbacks`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/abstract_controller/callbacks.rb:233 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/rescue.rb:23 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/instrumentation.rb:67 :in `block in process_action`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206 :in `block in instrument`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24 :in `instrument`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206 :in `instrument`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/instrumentation.rb:66 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal/params_wrapper.rb:259 :in `process_action`
[GEM_ROOT]/gems/activerecord-7.0.8/lib/active_record/railties/controller_runtime.rb:27 :in `process_action`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/abstract_controller/base.rb:151 :in `process`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal.rb:188 :in `dispatch`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_controller/metal.rb:251 :in `dispatch`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:49 :in `dispatch`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:32 :in `serve`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:50 :in `block in serve`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32 :in `each`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32 :in `serve`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:852 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/rack/browser_monitoring.rb:38 :in `traced_call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266 :in `context`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/cookies.rb:704 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/etag.rb:27 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/conditional_get.rb:40 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/head.rb:12 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:27 :in `block in call`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:99 :in `run_callbacks`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:26 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/debug_exceptions.rb:28 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/show_exceptions.rb:29 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/railties-7.0.8/lib/rails/rack/logger.rb:40 :in `call_app`
[GEM_ROOT]/gems/railties-7.0.8/lib/rails/rack/logger.rb:25 :in `block in call`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99 :in `block in tagged`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:37 :in `tagged`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99 :in `tagged`
[GEM_ROOT]/gems/railties-7.0.8/lib/rails/rack/logger.rb:25 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/remote_ip.rb:93 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/request_id.rb:26 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/runtime.rb:22 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/activesupport-7.0.8/lib/active_support/cache/strategy/local_cache_middleware.rb:29 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/executor.rb:14 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-2.2.8/lib/rack/sendfile.rb:110 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/actionpack-7.0.8/lib/action_dispatch/middleware/host_authorization.rb:131 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/rack-cors-2.0.1/lib/rack/cors.rb:102 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/railties-7.0.8/lib/rails/engine.rb:530 :in `call`
[GEM_ROOT]/gems/newrelic_rpm-9.6.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:99 :in `call`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/configuration.rb:252 :in `call`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/request.rb:77 :in `block in handle_request`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/thread_pool.rb:340 :in `with_force_shutdown`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/request.rb:76 :in `handle_request`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/server.rb:443 :in `process_client`
[GEM_ROOT]/gems/puma-5.6.7/lib/puma/thread_pool.rb:147 :in `block in spawn_thread`
from faktory.
mperham
commented on August 20, 2024
You're running in puma so the Faktory client defaults to a pool of 5 connections. I'm assuming you were doing some load testing and had many puma threads pushing a lot of jobs to Faktory at the same time. It has nothing to do with staging.
I've increased the default pool size from 5 to 20. Pools are lazy so it will only create 20 if you actually need that many. You can run FWR's main branch to pick up that change and verify it fixes your issue.
from faktory.
Deekor
commented on August 20, 2024
It was a normal traffic event for us, but we do process a ton of traffic and a lot of that traffic pushes jobs to faktory. The weird things is the staging ENV throwing the exact same error at the exact same time.. it shouldnt have had anywhere close to the same traffic as prod.
from faktory.
Related Issues (20)
- Feature suggestion: Unwrap ActiveJob wrapped Job names in the WebUI HOT 6
- Handling server upgrades? HOT 4
- Custom data stripped on batch callbacks for trace metadata propagation HOT 1
- Feature Request: Unique jobs even after completion HOT 2
- Ent Faktory Batches Guarantees HOT 1
- Does Faktory have any notion of job cost? HOT 3
- Fail jobs immediately when worker crashes HOT 6
- Add direct TLS support HOT 2
- Publish enterprise release binary for Linux to aid in testing HOT 1
- Faktory Enterprise shutting down messages in the log file - no explanation as to why Faktory is shutting down.
- REST API for UI HOT 2
- [Feature Request] Include Redis RTT in statsd metrics that are emitted HOT 3
- [Feat Request] Improve the reliability of Faktory server when overloaded with connections HOT 1
- Support Env Var expansion in toml config files HOT 3
- Question: What is the unit of "time" for Ent Metrics HOT 6
- Crashlooping on start HOT 16
- deb packages for arm64 HOT 3
- bzip2 deprecated
- Faktory crashed with fatal error: concurrent map writes HOT 3
- Is it possible to run the faktory server in a HA setup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faktory.