"Rate limited, will retry in X" question about riot-lol-api HOT 7 CLOSED

Heya,

You'll get blacklisted if you get too many 429.

What this library does is slowly reduce the concurrency as you get closer to the limit, so that no matter what happens, you only get ONE 429 per bucket.
This seems to be acceptable (I serve 10M+ requests a day with no issues, and get rate limited once pretty much every minute per region. YMMV but it's a good start).

What will get you blacklisted for sure however is to not respect the Rate Limit header and keep sending requests to the API.

tl;dr: one 429 is fine, more will be an issue.

from riot-lol-api.

Yup, you're definitely having issues with the BR league.

The default rate limit for this region is super low (80 requests at the top of my head?) and the service is pretty slow, so sometimes requests arrive out of order and can mess the concurrency a little.

My advice for this situation would be to use the throttler.

Throttle the BR "league" endpoints, something like (please confirm, can't test from here):

riotLolApi.setThrottler("BR1", "league", 10);

This will reserve 10 "units" of safe-space, which should also ensure concurrency has dropped to 1 by the time you start getting rate limited.

Happy hacking!

from riot-lol-api.

Thank you for your response.

Sure enough I got a 403 back and I just feel I cannot trust the script to run on its own.

Perhaps I am not using the library right.

In a for loop I make my request()'s which the library automatically puts in a queue. And I'm getting multiple "Rate limited, will retry" messages per bucket.

I let the script run until it was completely rate limited and then I captured the log and have attached it here. Please let me know if this is normal.

In the source it says it's configured for dev keys (I have a dev key for now) until the first request tells it otherwise but a default concurrency of 20 (which the script seems to keep going back to) doesn't seem suitable for dev keys.

I hardcoded the concurrency to 1 in the source just to see and it still was giving me two 429s per bucket.

I'm trying to see if configuring throttling can get it to a point that prevents 429s for dev keys, but this just seems weird for default behavior if I'm using it correctly.

log.txt

from riot-lol-api.

I've used the method signature that allows me to set it for all regions and then did that for all the types of calls that I make. I hope that help solves the blacklisting problem.

It's reassuring to know that even a production script like yours can hit a 429 once a minute and be fine.

Thanks for the help.

from riot-lol-api.

Cool, hope everything will be alright for you!

from riot-lol-api.

Hey @nbs :-)
Did you managed to solve that issue? we have lots of 429 too. On our setup we have regional load balancer and in each region (2 in total - US & EU) we have node with this (wonderful, I have to say) library. each node server have 8 instance. we currently have around 10 million +/- requests per day (+ 2-7 hours of downtime from the blacklist) to Riot with throttler setup and we still got 429 (for example today we had 7,300 +/- 429s from eune, league endpoint)

Any help on that issue will be very appreciated, Thanks in advanced!

from riot-lol-api.

@cohenu I've opened #26 to talk about your use case :)

from riot-lol-api.