Retries count about tenacity HOT 11 OPEN

@mhindery I don't think you need a closure. You can just reference the function directly:

>>> import tenacity
>>> @tenacity.retry
... def x():
...     print(x.retry.statistics)
...
>>> x()
{'start_time': 743610.875068273, 'attempt_number': 1, 'idle_for': 0}

from tenacity.

Is it possible to access the number of retries which have occurred?

If you want the overall retry count for all tasks running an f function, you should use f.retry.statistics["attempt_number"] - active_task_count, where this non-local active_task_count should be increased only at the first attempt of each task (which might be quite hard to do).

If you want the retry count for a single task, I'm afraid it's not possible with the retry.statistics, unless you're not calling the same function more than once, since its context is lexical, and the retry count in this case is part of a dynamic context. If you have several tasks running the same decorated function, the statistics is mixed.

This example shows this behavior and includes a solution for the second problem using contextvars:

import asyncio, contextvars, datetime, random, tenacity
attempt = contextvars.ContextVar("attempt")

@tenacity.retry
async def f(n):
    attempt.set(attempt.get(0) + 1)
    values = [n, datetime.datetime.utcnow(), attempt.get(),
              f.retry.statistics["attempt_number"]]
    print(" ".join(map(str, values)))
    await asyncio.sleep(random.random())
    raise tenacity.TryAgain

async def call():
    for n in range(5):
        asyncio.ensure_future(f(n))

asyncio.run(call())

The result:

0 2019-08-08 20:39:30.991814 1 1
1 2019-08-08 20:39:30.992007 1 1
2 2019-08-08 20:39:30.992135 1 1
3 2019-08-08 20:39:30.992247 1 1
4 2019-08-08 20:39:30.992353 1 1
4 2019-08-08 20:39:30.993940 2 6
1 2019-08-08 20:39:30.994081 2 6
0 2019-08-08 20:39:30.994182 2 6
3 2019-08-08 20:39:30.994273 2 6
2 2019-08-08 20:39:30.994358 2 6
1 2019-08-08 20:39:31.074111 3 7
2 2019-08-08 20:39:31.128905 3 8
3 2019-08-08 20:39:31.206795 3 9
2 2019-08-08 20:39:31.284957 4 10
0 2019-08-08 20:39:31.289798 3 11
4 2019-08-08 20:39:31.516829 3 12
1 2019-08-08 20:39:31.652761 4 13
0 2019-08-08 20:39:31.713638 4 14
1 2019-08-08 20:39:31.772541 5 15
3 2019-08-08 20:39:31.841446 4 16
1 2019-08-08 20:39:32.051496 6 17
4 2019-08-08 20:39:32.062492 4 18
2 2019-08-08 20:39:32.174466 5 19
3 2019-08-08 20:39:32.236457 5 20
1 2019-08-08 20:39:32.303372 7 21
0 2019-08-08 20:39:32.323257 5 22
4 2019-08-08 20:39:32.607363 5 23
4 2019-08-08 20:39:32.632185 6 24
3 2019-08-08 20:39:32.810179 6 25
1 2019-08-08 20:39:32.930142 8 26
2 2019-08-08 20:39:33.020101 6 27
3 2019-08-08 20:39:33.045923 7 28
1 2019-08-08 20:39:33.065823 9 29
4 2019-08-08 20:39:33.120670 7 30
2 2019-08-08 20:39:33.143518 7 31
0 2019-08-08 20:39:33.270527 6 32
0 2019-08-08 20:39:33.440524 7 33
4 2019-08-08 20:39:33.497371 8 34
0 2019-08-08 20:39:33.543119 8 35
0 2019-08-08 20:39:33.692038 9 36
3 2019-08-08 20:39:33.778935 8 37
0 2019-08-08 20:39:33.793688 10 38
0 2019-08-08 20:39:33.925558 11 39
2 2019-08-08 20:39:33.956431 8 40
1 2019-08-08 20:39:34.039304 10 41
4 2019-08-08 20:39:34.475522 9 42
1 2019-08-08 20:39:34.536371 11 43
0 2019-08-08 20:39:34.696280 12 44
2 2019-08-08 20:39:34.750217 9 45
3 2019-08-08 20:39:34.754077 9 46
1 2019-08-08 20:39:34.921033 12 47
4 2019-08-08 20:39:35.233154 10 48
0 2019-08-08 20:39:35.268114 13 49
2 2019-08-08 20:39:35.517181 10 50
3 2019-08-08 20:39:35.529995 10 51
1 2019-08-08 20:39:35.534860 13 52
0 2019-08-08 20:39:35.671832 14 53
4 2019-08-08 20:39:35.922999 11 54
1 2019-08-08 20:39:36.212128 14 55
4 2019-08-08 20:39:36.289996 12 56
3 2019-08-08 20:39:36.305002 11 57
2 2019-08-08 20:39:36.358790 11 58
0 2019-08-08 20:39:36.472697 15 59
1 2019-08-08 20:39:36.489496 15 60
1 2019-08-08 20:39:36.558043 16 61
3 2019-08-08 20:39:36.678750 12 62
4 2019-08-08 20:39:36.728588 13 63
2 2019-08-08 20:39:37.223850 12 64
2 2019-08-08 20:39:37.240736 13 65
0 2019-08-08 20:39:37.251575 16 66
4 2019-08-08 20:39:37.400507 14 67
2 2019-08-08 20:39:37.428611 14 69
1 2019-08-08 20:39:37.428857 17 69
4 2019-08-08 20:39:37.526645 15 70
3 2019-08-08 20:39:37.557577 13 71
1 2019-08-08 20:39:37.683582 18 72
1 2019-08-08 20:39:37.815516 19 73
0 2019-08-08 20:39:37.939429 17 74
2 2019-08-08 20:39:37.962317 15 75
2 2019-08-08 20:39:38.092201 16 76
4 2019-08-08 20:39:38.141038 16 77

If you sort these lines, you'll see the attempt context variable is the attempt number for that dynamic context. The number of retries is just attempt.get() - 1.

For the first approach (the retry count summing retries from all tasks), you'll need to increase the global active_task_count when attempt.get() == 1.

from tenacity.

Each tenacity.Retrying has a statistics attribute that you can use to access the number of retries.

I'll let this bug open to make sure its documened somewhere.

from tenacity.

Something related which might be helpful for people looking at this; We wanted to have access to the statistics during the execution, and used a closure for that. Skeleton code for it (which for example uses a statistic to put into a header of a request) looks like this:

def doRequest(url):
    # Closure structure to access retry statistics
    @retry(reraise=True)
    def doRequest_closure(url):
        try:
            headers.update({"AttemptNumber": str(doRequest_closure.retry.statistics["attempt_number"])})
        except:
            pass
        return r = requests.post(url, headers=headers)
    return doRequest_closure(url)

from tenacity.

Feels like wrapping it with another decorator might be a pretty way of doing it.

e.g.

@tenacity.retry
@tenacity.pass_statistics
def foo(statistics, value):
    print(f'Attempt #{statistics.attempt_number}: {value}'

from tenacity.

All of the above one's are correct in their own cases :) I think a decorator could be useful, as this functionality of accessing statistics during execution looks like something not that uncommon to me.

@jd this indeed works if you have a function, but not in the case we used it in (a static method on a class). :

class Utils(object):
    # works
    @staticmethod
    def doRequest():
        # Closure structure to access retry statistics
        @tenacity.retry(reraise=True)
        def doRequest_closure():
            print(doRequest_closure.retry.statistics)
        return doRequest_closure()

    # doesn't work
    @staticmethod
    @tenacity.retry(reraise=True, stop=tenacity.stop_after_attempt(1))
    def doRequest2():
        print(doRequest2.retry.statistics)

The doesn't work case raises this error:

<ipython-input-8-a41ae46bbd83> in doRequest2()
     13     @tenacity.retry(reraise=True, stop=tenacity.stop_after_attempt(1))
     14     def doRequest2():
---> 15         print(doRequest2.retry.statistics)

NameError: global name 'doRequest2' is not defined

from tenacity.

You should define it as a classmethod or a method, not a staticmethod. It calls itself inside a class, so it's not static.

from tenacity.

I don't mind putting in a small amount of time to support a @tenacity.pass_statistics for what it's worth.

from tenacity.

@naphta that's not needed, see the discussion.

from tenacity.

@mhindery I don't think you need a closure. You can just reference the function directly:

>>> import tenacity
>>> @tenacity.retry
... def x():
...     print(x.retry.statistics)
...
>>> x()
{'start_time': 743610.875068273, 'attempt_number': 1, 'idle_for': 0}

@jd but in this case the function x needs to access to attributes defined outside of their own scope. But what happens in the case that you cant modify the original function and add just the retry decorator? I think in that cases could be nice to have or another decorator, or maybe there is a way to use a custom callback for tihs?

from tenacity.

With respect to global vs local retry stats, retry.statistics is local like you'd expect. Perhaps not for async functions, but I suspect most are not using async.

from tenacity import retry, stop_after_attempt

@retry(stop=stop_after_attempt(3))
def foo():
    attempt = foo.retry.statistics['attempt_number']
    print(f"Attempt {attempt}...")
    if attempt < 3:
        raise Exception("Failed")

foo()
foo()

prints

Attempt 1...
Attempt 2...
Attempt 3...
Attempt 1...
Attempt 2...
Attempt 3...

from tenacity.