general-cbic / poolex Goto Github PK
View Code? Open in Web Editor NEWA library for managing pools of workers
License: MIT License
A library for managing pools of workers
License: MIT License
Poolex's tests must cover cases of handling sudden death of workers. The current implementation has crash logs and error messages while tests are running.
There is no big problem, but it will be nice to make logs clean.
Example of the test pipeline:
It's inconvenient to describe the spec manually :(
Hey, I'm using Poolex to control workers that send commands to kubernetes pods. When the worker exits the pod is stopped so as to avoid orphan pods piling up.
To reduce costs I'd like to have a lower workers_count
and a larger max_overflow
to scale up to meet workloads. Even when a docker image is cached on the kubernetes node, depending on the image it can take a few seconds for the pod become ready.
When into overflow, I'd like to amortise the cost of starting a pod by keeping the worker+pod alive and idle for a configurable grace period. The overflowed idle worker either becomes busy with a new command, or the period expires and the worker is killed, taking the pod with it.
I have an implementation that I'd be happy to firm up into a worthy contribution if you think this is a good idea:
develop...spacebat:poolex:grace-for-idle-workers
Let's push it via Github actions
I didn't create a test, but I think when a worker dies Poolex doesn't verify if there are new callers waiting for a worker:
https://github.com/general-CbIC/poolex/blob/main/lib/poolex.ex#L449-L482
We do here for example:
https://github.com/general-CbIC/poolex/blob/main/lib/poolex.ex#L401
I also think that Poolex wouldn't need to make a difference between a "temporary worker" and a "regular worker", and it could make the code easier (less complex). We only need to verify the number the workers and the configuration params:
If a worker dies, we remove it from the list of workers, and then:
Similarly, when a worker is released:
I hope it helps
Hi!
When the function you use in Poolex.run
raise a timeout, it returns :all_workers_are_busy
. I did a test alexcastano@c84a84b
If I have time, I will finish the PR with the solution. However, I think the timeout option can be misleading. I expected to be the timeout for the whole process, but it is only used to get the worker. The documentation is correct, but you have to check the documentation: https://hexdocs.pm/poolex/Poolex.html#t:run_option/0. Maybe we can find a better name to avoid confusion, like :checkout_timeout
?
Ecto used to have :pool_timeout
, but it was deprecated. In Ecto :timeout
includes the queue time and the query time: https://hexdocs.pm/ecto/changelog.html#backwards-incompatible-changes-1 . This was my expectation for poolex too :)
Thanks
Feature inspired by poolboy.
Poolex
should be able to create a configurable number of workers in excess of a constant value (if the load has increased and there are no free workers left in the pool).
Reasons are described here.
Currently implementations are configured via config :poolex, ...
. We need to get away from this in favor of initialization parameters.
Poolex.State.t()
not found during documentation generation.
❯ mix docs
Generating docs...
warning: documentation references type "Poolex.State.t()" but it is hidden or private
lib/poolex.ex:178: Poolex.get_state/1
View "html" docs at "doc/index.html"
warning: documentation references type "Poolex.State.t()" but it is hidden or private
lib/poolex.ex:178: Poolex.get_state/1
View "epub" docs at "doc/poolex.epub"
Hello!
It seems like the library doesn't handle the shutdown process. The pool dies but the workers are still alive, which is very inconvenient. I took a quick look at poolboy and it uses a supervisor where the workers are placed to handle the shutdown process correctly. Do you have a plan to do something similar?
Thank you!
run/3
will return {:error, reason :: atom()}
on errorsrun!/3
will raise on errorsOne thing has been lacking from poolboy
and it'd be really cool to see it make its way to Poolex
.
It's the ability to manage workers as "groups" per pool.
For example:
Using Poolboy, we were achieving 1. by calling :poolboy.stop(:pool_id)
, and for example this function is missing for Poolex
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.