Giter Club home page Giter Club logo

ratelimiter's Introduction

RateLimiter

PyPI Version Build Status Python Version License

Simple Python module providing rate limiting.

Overview

This package provides the ratelimiter module, which ensures that an operation will not be executed more than a given number of times on a given period. This can prove useful when working with third parties APIs which require for example a maximum of 10 requests per second.

Usage

Decorator

from ratelimiter import RateLimiter

@RateLimiter(max_calls=10, period=1)
def do_something():
    pass

Context Manager

from ratelimiter import RateLimiter

rate_limiter = RateLimiter(max_calls=10, period=1)

for i in range(100):
    with rate_limiter:
        do_something()

Callback

The callback is called in its own thread, so your callback may use sleep without delaying the rate limiter.

import time

from ratelimiter import RateLimiter

def limited(until):
    duration = int(round(until - time.time()))
    print('Rate limited, sleeping for {:d} seconds'.format(duration))

rate_limiter = RateLimiter(max_calls=2, period=3, callback=limited)

for i in range(3):
    with rate_limiter:
        print('Iteration', i)

Output:

Iteration 0
Iteration 1
Rate limited, sleeping for 3 seconds
Iteration 2

asyncio

The RateLimiter object can be used in an async with statement on Python 3.5+. Note that the callback must be a coroutine in this context. The coroutine callback is not called in a separate thread.

import asyncio
import time

from ratelimiter import RateLimiter

async def limited(until):
    duration = int(round(until - time.time()))
    print('Rate limited, sleeping for {:d} seconds'.format(duration))

async def coro():
    rate_limiter = RateLimiter(max_calls=2, period=3, callback=limited)
    for i in range(3):
        async with rate_limiter:
            print('Iteration', i)

loop = asyncio.get_event_loop()
loop.run_until_complete(coro())

License

Original work Copyright 2013 Arnaud Porterie
Modified work Copyright 2016 Frazer McLean

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

ratelimiter's People

Contributors

icecrime avatar pmav99 avatar razerm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

ratelimiter's Issues

bug in implementation

I believe this implementation is broken if you enter/aenter the class twice before aexit'ing (share instance with multiple classes). The lock should be from enter/aenter through exit/aexit. This would also mean that you can't use the sync lock in exit with aexit

needs update for python3.11: module 'asyncio' has no attribute 'coroutine'

Hi,
as reported in the Debian bug tracking system ratelimiter does not work with python3.11:

Python 3.11.1 (main, Dec  7 2022, 08:49:13) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ratelimiter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andreas/debian-maintain/salsa/python-team/packages/python-ratelimiter/ratelimiter.py", line 36, in <module>
    class RateLimiter(object):
  File "/home/andreas/debian-maintain/salsa/python-team/packages/python-ratelimiter/ratelimiter.py", line 127, in RateLimiter    
    __aexit__ = asyncio.coroutine(__exit__)
                ^^^^^^^^^^^^^^^^^
AttributeError: module 'asyncio' has no attribute 'coroutine'. Did you mean: 'coroutines'?

Kind regards, Andreas.

Remove exec for async support

Not that exec() is a huge problem in this context, but perhaps a more elegant way to support Python < 3.5 would be to move __aenter__ into a different module and import the module on runtime. E.g.

import sys

if sys.version_info < (3, 5):
    raise ImportError("Only support python versions >= 3.5")

import time
import asyncio

async def __aenter__(self):
    if self._alock is None:
        self._init_async_lock()

    with await self._alock:
        # We want to ensure that no more than max_calls were run in the allowed
        # period. For this, we store the last timestamps of each call and run
        # the rate verification upon each __enter__ call.
        if len(self.calls) >= self.max_calls:
            until = time.time() + self.period - self._timespan
            if self.callback:
                asyncio.ensure_future(self.callback(until))
            sleeptime = until - time.time()
            if sleeptime > 0:
                await asyncio.sleep(sleeptime)
        return self

and

class RateLimiter(object):
    """Provides rate limiting for an operation with a configurable number of
    requests for a time period.
    """
    # snip

    if PY35:
        import _async
        __aenter__ = _async.__aenter__
        __aexit__ = asyncio.coroutine(__exit__)

Maintainers wanted

If you have the time to merge PRs and make releases, please email me at the address on my profile. Please include any links to existing projects you maintain.

If you are a user of ratelimiter looking for a maintained alternative, I recommend rush

What is _alock?

I think those lines are not useful

        self._alock = None

        # Lock to protect creation of self._alock
        self._init_lock = threading.Lock()

bug in callback triggering possibly related to threading

I have a function that uses the requests library that I am decorating with a callback. The callback seems to be triggered more frequently than one would expect. The behavior works as expected when the decorated function returns a string.

import requests
from ratelimiter import RateLimiter


def callback_message(until):
    print('Rate limit reached, sleeping')


@RateLimiter(max_calls=30, period=15, callback=callback_message)
def thing1():
    return requests.get("https://www.google.com")


@RateLimiter(max_calls=30, period=15, callback=callback_message)
def thing2():
    return "thing2"


print("Testing with requests")
for i in range(100):
    if i % 10 == 0:
        print(i)
    thing1()

print("Testing with simple function")
for i in range(100):
    if i % 10 == 0:
        print(i)
    thing2()

This is the output:

Testing with requests
0
10
20
30
Rate limit reached, sleeping
Rate limit reached, sleeping
40
Rate limit reached, sleeping
50
60
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
70
Rate limit reached, sleeping
Rate limit reached, sleeping
80
Rate limit reached, sleeping
Rate limit reached, sleeping
90
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping
Rate limit reached, sleeping

Testing with simple functions
0
10
20
30
Rate limit reached, sleeping
40
50
60
Rate limit reached, sleeping
70
80
90
Rate limit reached, sleeping

Fix warning that coroutine decorator is deprecated since Python 3.8

Our project currently uses ratelimiter-1.2.0.post0, the latest version on PyPI.

Python 3.8 has been released and we're adding a py28 CI test for our project, which resulted in the following warning:

  /home/travis/virtualenv/python3.8.0/lib/python3.8/site-packages/ratelimiter.py:127: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    __aexit__ = asyncio.coroutine(__exit__)

While the master branch has unreleased commits that weren't in the version we're testing, I believe the warning would still occur on the latest master commit at the following line:

__aexit__ = asyncio.coroutine(RateLimiter.__exit__)

Is it possible to update the code to no longer create the warning and release a new version to PyPI with the update?

Files missing on PyPI

While creating a Debian package for ratelimiter (as described on https://bugs.debian.org/880661), I found that the non-empty files in tests/ are missing, and that only one of the two README files (which appear to have diverged) are provided.

Please consider adding the tests via MANIFEST.in (eg. as described on stackoverflow), and having one up-to-date README that is shipped the same way.

Broken algorithm

The algorithm will always wait too much.

import time
from ratelimiter import RateLimiter

rate_limiter = RateLimiter(max_calls=1, period=1)

start = time.time()
for i in range(3):
    with rate_limiter:
        print("%.02f" % (time.time()  - start))
        time.sleep(1)

outputs

0.00
2.00
4.01

I would expect it to output 0 1 2

Non-UTF8 characters

I have had issues installing this package with pip due to the presence of non-UTF8 characters in README.rst. This is for version 1.2.0.post0. Specifically, the quote marks around 'LICENSE' and 'AS IS' are the italicised variety and apparently non-standard.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.