spscommerce / redlock-py Goto Github PK

View Code? Open in Web Editor NEW

570.0 33.0 118.0 29 KB

Redis distributed locks in Python

License: MIT License

Python 100.00%

redlock-py's Introduction

redlock-py - Redis distributed locks in Python

This python lib implements the Redis-based distributed lock manager algorithm described in this blog post.

To create a lock manager:

dlm = Redlock([{"host": "localhost", "port": 6379, "db": 0}, ])

To acquire a lock:

my_lock = dlm.lock("my_resource_name",1000)

Where the resource name is an unique identifier of what you are trying to lock and 1000 is the number of milliseconds for the validity time.

The returned value is False if the lock was not acquired (you may try again), otherwise an namedtuple representing the lock is returned, having three fields:

validity, an integer representing the number of milliseconds the lock will be valid.
resource, the name of the locked resource as specified by the user.
key, a random value which is used to safe reclaim the lock.

To release a lock:

dlm.unlock(my_lock)

It is possible to setup the number of retries (by default 3) and the retry delay (by default 200 milliseconds) used to acquire the lock.

Both dlm.lock and dlm.unlock raise a exception MultipleRedlockException if there are errors when communicating with one or more redis masters. The caller of dlm should use a try-catch-finally block to handle this exception. A MultipleRedlockException object encapsulates multiple redis-py.exceptions.RedisError objects.

Disclaimer: This code implements an algorithm which is currently a proposal, it was not formally analyzed. Make sure to understand how it works before using it in your production environments.

Further Reading: http://redis.io/topics/distlock http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html http://antirez.com/news/101 https://medium.com/@talentdeficit/redlock-unsafe-at-any-time-40ceac109dbb#.uj9ffh96x

redlock-py's People

Contributors

Stargazers

Watchers

Forkers

sebastianmika mpba hellp rcoup chenokay jeremywangjun wowuwowuwo nick-merrill adgaudio ffix agronholm hongphi peterfarrell 2tim rakesh-roshan mat128 idjaw octopart flstar jonahbull stephanerobert xiaonuogantan advance512 pingviini ladder1984 githai ajvpot sunday2333 localbushel page1990 zondahuman zhaxiancheng jango2015 bigxing abdullah353 seayuehao cockcrow payshangjj whitefirer sichuyoudang jdstmporter dmitryzv thientu yangluoshen whyshan giosg figecheng-vrviu arjanvandervelde geojacobm6 leafsummer huyidao625 caojianfa fork-org sujithps wangjie07 alexcharlton fastfishio tiangles lufenggui lancerchiang onetwo1 zodgeneralkripton shinepengwei chriscslaughter martinezml2 michaelmanganiello-eb warren-wong zzzz123321 sharupoff sksitou yangxjupup diepmv mingfeipan yuyigogo max1314 tachyonicproject 853695319 liuzemeeting redisclub thanlonsmith tsingjyujing ct1104 jayvdb cicelprg lynda-li gtfd333 samsu wangpanjun chyidl sptkcode prohui evillegenius bukekangli vincentor shaikhanas1993 toywei halbow kgavrilchik asyncins jinwik

redlock-py's Issues

Question: how do you keep the lock

Once A has acquired the lock, set with a timeout of 10 seconds for example.
How can A keep the lock, within the 10 seconds, to prevent B to take it?
I see no method to renew the lock.

I expect A to keep the lock as long as necessary, then B takes it only if A fails (and ttl will remove the lock) or A releases the lock.

Thanks

Should raise `ValueError` and be more specific about exceptions in `def lock_instance`

This is just a suggestion. The following code in redlock/__init__.py:

class Redlock(object):
    ......

    def lock_instance(self, server, resource, val, ttl):
        try:
            return server.set(resource, val, nx=True, px=ttl)
        except:
            return False

should handle the case where ttl is not an int. Raising a ValueError('ttl is not an int') seems to be a good choice. Additionally, it might be beneficial to change the api so that it raises certain redis exceptions in the except clause so that the caller knows that something is wrong (instance is down, etc.), i.e., letting the caller know exactly why a lock cannot be obtained

Add a license

Hi
I'd like to fork and make some changes to redlock-py, but I can't if an open-source license is not assigned to it.
Thanks in advance for your support

Fails with non-default setting of redis

Hello,

I ran redis on port number 9000, and tried to use redlock-py then it failed. It reproduces with the following code:

import redlock

rl = redlock.Redlock({"host": "localhost", "port": 9000})
lock = rl.lock("test", 100)
rl.unlock(lock)

and the following is the error message I get:

ERROR:root:Error unlocking resource test in server StrictRedis<ConnectionPool<Connection<host=None,port=6379,db=0>>>
Traceback (most recent call last):
  File "/home/kato/.pyenv/versions/3.5.1/lib/python3.5/site-packages/redis/connection.py", line 439, in connect
    sock = self._connect()
  File "/home/kato/.pyenv/versions/3.5.1/lib/python3.5/site-packages/redis/connection.py", line 494, in _connect
    raise err
  File "/home/kato/.pyenv/versions/3.5.1/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

It looks it is looking at the default port when it tries to lock.

Lock method does not work

Hi,
I'm not sure what am I doing wrong but I haven't being able to use this package. It is exactly what I need but for some reason it throws an error when I try to lock a resource.

redis_version:2.4.10

python version 3.5.2

This is my code

from redlock import Redlock
dlm = Redlock([{"host": "REDIS_SERVER_IP", "port": 6379, "db": 0}, ])

Acquire lock for 1 minute

my_lock = dlm.lock("this_is_my_key",1000)

ERROR:root:Error unlocking resource this_is_my_key in server StrictRedis<ConnectionPool<Connection<host=X.X.X.X,port=6379,db=0>>>
Traceback (most recent call last):
File "/home/redis/apps/Proj/env/lib/python3.5/site-packages/redlock/init.py", line 80, in unlock_instance
server.eval(self.unlock_script, 1, resource, val)
File "/home/redis/apps/Proj/env/lib/python3.5/site-packages/redis/client.py", line 1932, in eval
return self.execute_command('EVAL', script, numkeys, *keys_and_args)
File "/home/redis/apps/Proj/env/lib/python3.5/site-packages/redis/client.py", line 573, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/redis/apps/Proj/env/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/home/redis/apps/Proj/env/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'EVAL'

Thanks in advanced!

Possible to check on lock status?

Hi all,

First thank you all for the great work with redis-lock, truly useful.

One question, how can I check if a resource is locked or not? I understand that I should go check in Redis database if this "recourse" key exists or not. But it seems I have to implement this myself? Or is there already existing way to do this?

Thank you very much.

// Bingwen

Redis Cluster

Hi,

Is this approach still compatible with the Redis Cluster specification? If so, does the usage is still the same (just pick IP address of one of the available nodes) ? Thanks.

Add watching dog to extend TTL

I'd like to implement a function like we can extend a lock every N secs, the purpose to do that is designed for some transaction that can't control the timeout, but we also want the lock be released while the server is down.

Do you have any advice?

retry_count = 0 or retry_delay = 0 not works, may be a bug?

when i dont want it to retry then set retry_count =0, actually it will use the default value (retry 3 times) , may be it is a bug there (L68-L69). (return default if "is None" may fix it)
https://github.com/SPSCommerce/redlock-py/blob/master/redlock/__init__.py#L68-L69

Add context manager support

It would be nice to use it as context manager. Like the following:

with Redlock([{"host": "localhost", "port": 6379, "db": 0}, ]) as dlm:
    do_sync_stuff()

NameError: name 'basestring' is not defined

Getting the following issue when using redlock with python 3.4

Traceback (most recent call last):
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/redlock/init.py", line 27, in init
if isinstance(connection_info, basestring):
NameError: name 'basestring' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/kp/.virtualenvs/celery_test/bin/celery", line 11, in
sys.exit(main())
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/main.py", line 30, in main
main()
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/base.py", line 465, in setup_app_from_commandline
self.app = self.find_app(app)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/base.py", line 485, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/app/utils.py", line 229, in find_app
sym = symbol_by_name(app, imp=imp)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/bin/base.py", line 488, in symbol_by_name
return symbol_by_name(name, imp=imp)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/kombu/utils/init.py", line 92, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
return imp(module, package=package)
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/importlib/init.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 2254, in _gcd_import
File "", line 2237, in _find_and_load
File "", line 2226, in _find_and_load_unlocked
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/Users/kp/repo/playground/celery_test/task.py", line 12, in
dlm = Redlock([{'host': app.conf.REDIS_HOST, 'port': app.conf.REDIS_PORT, 'db': app.conf.REDIS_DB }, ])
File "/Users/kp/.virtualenvs/celery_test/lib/python3.4/site-packages/redlock/init.py", line 33, in init
raise Warning(str(e))
Warning: name 'basestring' is not defined