Comments (6)
Yup, that’s a race condition. Thanks for finding that bugger. I can wrap get_resource with the lock which should solve the problem. I’ll have this fixed today.
from cuttlepool.
Resolved in PR #37.
from cuttlepool.
from cuttlepool.
Yes, I think you're right. The lock is really meant to be used to prevent race conditions when accessing the queue, not individual resources. _get
and _make_resource
should return the wrapped resource as opposed to the tracker. If you are making changes, a PR would be greatly appreciated.
from cuttlepool.
(..sorry for answering late - I'd some days "off"...)
The problem with the function "get_resource" as you have it in the version 0.9 is the call to "ping()":
def get_resource(self, resource_wrapper=None):
...
with self._lock:
...
# Ensure resource is active.
if not self.ping(rtracker.resource):
...
So, the pool it locked until the function ping() returns which may take a long time if - for example - one node of the Resource-Cluster is down. Think of a case where 10 free resources are in the pool, 11 client threads did call "get_resource()" and "ping()" has 30s timeout until it notices that the resource has moved to an other host. The last client call would have to wait for 300s (!) until it gets
a fesh resource....
I suggest the following:
-
Change the function "_get()" to return both, the _ResourceTracker object and the wrapped resource:
def _get(self, timeout, resource_wrapper_class): with self_lock: ... return rtracker, rtracker.wrap_resource(self, resource_wrapper_class)
-
Change the function "_make_resource()" to return both, the_ResourceTracker object and the wrapped resource:
def _make_resource(self, resource_wrapper_class): with self_lock: ... return rtracker, rtracker.wrap_resource(self, resource_wrapper_class)
-
Change the function "get_resource()" so that "_ResourceTracker" and the wrapped resource are stored when calling "_get()" or "_make_resource()". Example:
try: # try the get a resource from the pool. Do not wait; pool is empty at very first call rtracker, wrapedResource = self._get(0, resource_wrapper) except PoolEmptyError: pass
-
Remove the statement
with self._lock:
at line 300 in the function "get_resource()" - this is not needed anymore.
I append a changed version of the file - but sorry - it was not based on the "0.9" version of your's,,,,
from cuttlepool.
Overall, this looks good. I think if the wrapped resource is being returned by _make_resource
and _get
, then the rtracker
doesn't need to be returned as well. Can you make a pull request with these changes?
from cuttlepool.
Related Issues (20)
- Migrate tests to pytest
- Update changelog for best practices HOT 1
- Clean up bare exceptions
- Use ping instead of try/except in _close_connection.
- Improper use of RLock
- Use mock connection object for tests
- Use custom exceptions instead of built-in ones
- Raise error if timeout argument is not an int
- Offset use of threading locks to queue.Queue's native support
- empty_pool() should wait for all connections to be returned to the pool HOT 1
- Thread locks result in permanent locks HOT 1
- Change directions for running tests
- CuttlePool __del__ method should call empty_pool() HOT 1
- Make default values global
- Allow subclasses of PoolConnection to be used by CuttlePool.
- Add __repr__ methods
- Move documentation to readthedocs.io
- Update README to use different example than sqlite3
- the attribute of the sqlite3 which is "check_same_thread" not supporting? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cuttlepool.