aio-libs / aiocache Goto Github PK

Lot of bugs found, need to improve testing with the keys_attribute thing

Wasnt bugs at all but feature. Should rethink about its behavior:

• We want to call the function if no keys are passed? Should it work without keys? Maybe not...

Its time to start adding some checks to the operations (without taking into account latencies with external storages) since caching should be as transparent as posible for the users.

keys_attribute, key and so are confusing, lets rename them to something more understandable

Not enough with ConnectionError, the original function should be called if the cache is not working properly.

Also a proposal would be to pass a list of exceptions to forward/catch?

http://stackoverflow.com/questions/11875770/how-to-overcome-datetime-datetime-not-json-serializable-in-python

https://github.com/jettify/aiogibson

Some operations need to have knowledge of the keys stored in the storage. LRUPolicy is tracking the keys in memory but this is wrong because the state is lost when the process exits.

Should modify the behavior so the operations that change db status keep a list of the existing keys also in the same storage

Custom serializer class expects serialize/deserialize methods. Seems reasonable to move them to expect dumps/loads since pickle, json, marshmallow and others use that.

Provide a memcached implementation following the BaseCache specification

All backends must support cleaning the cache.

This is not as easy at seems, should every instance keep a list of keys that has added so it removes only those? Should it work by namespace? What if another key starts by that namespace unintentionally? we would remove that and it's wrong... Is it time to start using different databases for each namespace (check how to do that with memcached)?

https://github.com/aio-libs/aiopg
https://github.com/MagicStack/asyncpg

Check if there are other async libs

Examples are a good way to cover use cases, lets use them as acceptance. To do so we need:

• Add asserts to check correct calls
• Add cleanups after execution
• Make pytest run them

Once the refactor is done, backends will be just simple wrappers for connecting with the underlying clients like aioredis, aiomcached, dict, etc.

Try to think of a mixins approach in order be able to call the client commands as if you were using it straightforward.

Now that mget, mset support has been added. A decorator for setting/retrieving multiple keys can be implemented.

The decorator should be able to retrieve the available keys and just query for the missing ones.

Policies should be transformed to Hook mechanism, this way the user can implement whatever behavior he wants. The naming will change so it will introduce a breaking change. Also, in order for the user to implement interesting actions, some data the hooks should receive is:

• key
• value
• time of execution (only post)
• CMD executed

TODO:

• Rename Policy to a name that indicates more generic behavior like Plugin, Hook or so. Remember to update docs
• Document well that the user can use this to implement whatever policy/behavior they would like BUT they should be careful with consistency/race conditions when executing this from multiple clients (if they are interacting with the storage).
• Remove the LRUPolicy and leave only the Default one. LRUPolicy can be an example of custom Plugin (say there that there can be inconsistency problems if different instances are using the same).
• All operations must have a pre and a post hook that are always executed (user will decide wether to use and execute htem or not).

function must be called anyway and not propagate exception if connection refused.

def cached(cache=None,...):
    cache = get_cache(cache=cache, ...)
    # ...
    def cached_decorator(fn):
        # ...
    return cached_decorator

Basically, get_cache returns new instance of backend for every call.
This mean that every decorated function will have its own instance of backend
and that every such function will create at least one connection to some
remote resource (redis, memcached, etc) (only if its not MemoryBackend)
and this might become a problem.

This will give the option to combine with other input args

Using RedisCache, SimpleMemoryCache and company as user classes, we are facing some design problems like reusing code, making serializer aware of the backend being used is difficult, etc.

A new design will be done were a main Cache class will be used. This class will contain the three main components:

• Backend
• Serializer
• Policy

The class will expose the interface to interact with the cache and wrap the backend calls with the serializer and policy calls.

There is an encoding attr defined for serializer because in some cases it is needed for aioredis. To keep the interface clean, the encoding part should be done in the serializer layer. This will help also in the case of memcached implementation

When saving 1, it retrieves "1"

TODO:

• Add tests for each serializer to check output is the same for different types and backends
• All backends must behave the same

Add instructions for contributors. Before that, tests should be able to be run with docker env (needing to install redis and memcached for running them sucks).

A file for CHANGES should also be added for keeping track together with CONTRIBUTORS file.

For debug purposes it would be nice to be able to disable the cache. Something like "AIOCACHE_ENABLED=0" so caching would be disabled for all the calls.

https://github.com/aio-libs/aiomysql

Check if there are other async libs

Some serializers may not be compatible with a given backend. This error would be better if its raised during instantiation time rather than when calling the actual operations. The error should also return which serializers are supported.

if a namespace is passed, it should create a new instance of the default config but with the new namespace. Things to do:

• Use either classes or instances for serializers and policies.
• Check whether SimpleMemoryBackend can accept extra args/kwargs.
• set_defaults should use strings, this way other classes from different modules can be used without causing cyclic imports and other problems.
• Reuse backends (connections) if there is already an existing one. -> The create_pool should be outside, maybe a post_init for backends? ...
• Improve settings module to pass args to the backend and the policy

Autodoc not working in RTD...

Right now it only works with functions returning dicts and knows the keys if a param called keys is passed. Some modifications to improve its flexibility:

• Param to the decorator to choose which param is receiving the keys.
• Check how to make it work when returning pairs.
• also cached decorator can be modified to support which attribute wants to be used as a key.

Clients may need some functionality that is not supported by the exposed interface. Add a new .raw function to access the client directly.

Right now caching can only be done explicitly. A decorator should be provided.

Right now many loops are being used to process the keys in the mget and mset calls. This should be improved

This would help to avoid problems like race conditions, dogpile effect, etc.

Some ideas:

• https://bitbucket.org/zzzeek/dogpile.cache/
• https://pypi.python.org/pypi/HermesCache

References:

• http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
• http://antirez.com/news/101
• https://aphyr.com/posts/283-jepsen-redis

Couple of things to keep in mind:

• Keep this as decoupled as possible, I don't know yet all use cases but for consistency, the approaches described below are not the most recommended. Since the main use case now is avoid stampede and avoid race conditions, we are "fine".
• Should add disclaimers that the lock doesn't aim for consistency.

Redis

For now I'm thinking about going for using the simple approach described in https://redis.io/topics/distlock#correct-implementation-with-a-single-instance (lease time should be able to be passed by the user).

Memcached

https://bluxte.net/musings/2009/10/28/simple-distributed-lock-memcached/
https://github.com/memcached/memcached/wiki/ProgrammingTricks#avoiding-stampeding-herd

Memory

Just go with asyncio.Lock

• async cache calls
• No need to transform your async calls to sync because decorator is sync (check if we can define sync decorators on reuse it for sync functions). https://gist.github.com/dlebech/c16a34f735c0c4e9b604
• multiple backends, multiple policies, multiple serializers
• easy interface
• example of redis/memcached which type of data they have stored...

The option should be compatible with the decorator too. Lets put a simple case:

Imagine we want to store a maximum number of keys for each different function call. By specifying a max per backend, we may end up doing lots of cache misses because we are deleting keys from other functions.

There isn't much problem in general because each decorator instantiates its own instance of the backend. The only edge case is when using the config_default_cache where the developer may put a small size for the keys and then use it for multiple function calls. Maybe a warning could be displayed if the developer is configuring the cache with the max_keys option.

Challenges:

• How to remove oldest key for each backend. To keep it simple, right now it won't be persistent. Each backend will keep a list of keys to track which has to be removed. This raises a "problem" with distributed systems like Redis: The number of keys won't be for the storage but for each backend/machine... Is this the behavior expected?
• Deal correctly with max_keys per backend/decorator: Should propagate the arg to the init of the backend instance

The package is too silent =/

Most of the methods for each backend are doing the same. Try to just use one test file and passing the tests for each backend available.

Also Redis and Memcached should run in containers when launching the tests.