spenceforce / cuttlepool Goto Github PK

Reduces number of import statements needed when cuttlepool is imported if access to different cursors is desired.

It currently calls _close_connections() which no longer exists

With mock sql connection objects, the previous testing instructions are invalid.

_close_connections closes the connections, but does not empty _reference_pool which can cause issues if get_connection() is called.

Currently RLock is instantiated every time it is needed. The proper usage is for a connection pool object to have one RLock object that handles all locking instead.

Ensuring that ping() and normalize_connection() are implemented will help ensure connection objects aren't modified/closed when needed.

Test public API and use environment variables to determine which type of sql to use.

Python's queue.Queue already implements thread locking, so any locking done by CuttlePool should be aimed at internal processes unrelated to the underlying Queue.

A mild change for potentially improved usability in development (for cuttlepool or extensions).

Pytest is easier to work with.

Design cuttle pool to work with any sql driver.

Keep references to all connections in CuttlePool object whether they are in the queue or not?

Would make it easier to prevent improper things being passed in and would simplify dealing with the size increments/decrements.

A connection's cursor class can be modified when out and about which can cause undesired behavior when it is later retrieved from the pool as it will be expected to create regular cursors but will not. When it's returned to the CuttlePool object it's cursorclass attribute should be set to the base cursor class.

To allow connection arguments to be modified, a mechanism has to be in place to close incoming connections instead of recycling them.

The class CuttlePool has a race-condition in its method get_resource.

Setup:

• the pool contains one available resource
• no resources are in use
• two threads are using the pool

Sequence of events:

1. thread-1:: calls "get_resource"
2. thread-1:: the method "_get()" returns the object "_ResourceTracker-1" and move this
   object to the part of the "_reference_queue" that contains the resources that
   are in use.
3. thread-1:: calls "self.ping" using the resource from "_ResourceTracker-1" as argument.
   Note: at this point in time a call to the method "available()" of the object
   "_ResourceTracker-1" returns "True" because the "weakref" to the wraped resource
   has not yet been established. This happens later in "get_resource" with a call to
   the method "wrap_resource".
4. thread-2:: calls "get_resource"
5. thread-2:: The pool's "empty()" return "True" and so "_harvest_lost_resources" is called to look
   for resources that haven been properly returned to pool.
6. thread-2:: "_harvest_lost_resources" loops the part of the "_reference_queue" that contains the
   "_ResourceTracker" objects of resources that are in use. It finds "_ResourceTracker-1"
   and calls the "available" method which returns "True".
   The method "_harvest_lost_resources" then returns the object "_ResourceTracker-1" to the
   part of the "_reference_queue" that contains the available resources.
7. thread-2:: The method "_get()" returns "_ResourceTracker-1"

As a result, boths thread are using the same resource.

The thread locks used in the CuttlePool class cause deadlocks when combined with the queue.Queue's built in locks. Related to #27.

A CuttlePool instance should be able to accept a subclass of PoolConnection for use on all calls to get_connection(). get_connection() should also accept a subclass of PoolConnection to supersede the default connection wrapper.

There are a few bare exceptions like:

try:
    ...
except:  # bare exception
    ...

These should catch more specific exceptions.

Is it possible to set normalize_connection() based on the original connection properties?

I want to use the cuttlepool in multi thread envrionment, but it seems that it doesn't support the option of the sqlite3 ,"check_same_thread : False" .
Am I wrong or Do I have to look for other libraries?
self.pool = SQLitePool(factory=sqlite3.connect,capacity=4,database='/mnt/config/test.db',isolation_level=None,check_same_thread=False)

It is not working at all

sqlite3 connections do not work across threads. Use different example like some kind of socket server or something along those lines.

Follow guidelines here: http://keepachangelog.com/en/1.0.0/

Here's the paragraph:

CuttlePool is imported and subclassed. The normalize_connection() method takes a Connection object as a parameter and changes it's properties. This is important because a Connection object can be modified while it's outside of the pool and any modifications made during that time

The final sentence is unfinished.

Using ping will check if the connection is open and it moves the burden of proper exception handling to the user. It's impossible to determine the user's needs given any SQL driver so it's best left to them to decide what's best. Related to #21