Comments (11)
I can progress this if no one is currently working on it. And handle to specific cases.
- Standard
Retry-After
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.37 - and custom Github https://developer.github.com/v3/rate_limit/
- or in alternative https://developer.twitter.com/en/docs/ads/general/guides/rate-limiting
from rust-cookbook.
I'd suggest implementing just one version with a comment that you can do all the other ways easily by adapting that code.
from rust-cookbook.
What kind of example did you have in mind here? The way rate limiting is handled will necessarily vary depending on which API you are communicating with. For example, GitHub always replies with a X-RateLimit-Remaining
header and you'll get a 403 if you go over your limit.
So a hypothetical request to some GitHub API might go like this:
- Send request
- deserialize response
- if deserializing fails, check header for
X-RateLimit-Remaining
and returnRateLimitError
if equal to "0" - else check status code and do other error handling to return an appropriate error
Or were you thinking of patterns people can use to make sure they don't go over their rate limit (e.g. the Leaky Bucket algorithm)?
from rust-cookbook.
@konstin you suggested this one initially. Any thoughts?
from rust-cookbook.
The correct solution would be a crate that is hooked into hyper and/or rewquest. This crate would delay all requests until you're allowed to requests again while logging to the user. Unfortunately, no stable crate for that currently exists.
There a two custom solutions: Github's api is friendly enough telling you the number of remaining requests, so you'd just check that header and sleep if the number gets to low.
As most apis aren't that helpfull though, the best solution is essentially trial and error, which I've quickly sketched out below. Consider the code as pseudocode, it's only meant to explain the control flow.
fn rate_limit_agnostic_request(url: Url) -> Result<String, Box<Error>> {
loop {
let mut reponse = reqwest::get(&url)?;
if reponse.status().is_success() {
let mut string = String::new();
reponse.read_to_string(&mut string)?;
return Ok(string);
} else if reponse.status() == TooManyRequests {
info!("Rate limit hit, sleeping for 5 seconds");
sleep(Seconds::new(5)); // TODO: Find the right functions for that
} else { // Some other error
return Err(from::From("Some other error"));
}
}
}
from rust-cookbook.
Side note: I wouldn't label this easy. Doing this right is surprisingly difficult, esp. when it comes to parallelism. This is made even more difficult by the fact that many services do not even adhere the http standard, which is to return a 429 and a Retry-After
. Github returns a 403, twitter its famous 420 and I'm sure there a enough apis just returning a json with an error message.
from rust-cookbook.
So fun fact, I was actually working on solving this problem myself and found the ratelimit
crate, which makes it actually fairly stratightforward to solve this problem in a proactive (as opposed to reactive) way. I actually submitted brayniac/ratelimit#19 to make the API for this crate easier to work with to solve these types of use cases.
I think it'd be good to have two different rate-limiting examples: one that takes the proactive approach (rate limiting on the client side, only submitting requests at a certain rate) and one that takes the reactive approach (creating a handler that will retry rate-limited requests). They both have different use cases; code that sends requests based upon events would prefer a reactive method, whereas code that sends batches of requests would prefer a proactive method (potentially paired with a reactive method).
It might make sense to add ways of handling the reactive method to reqwest (for all of the sorts of non-standard variations listed), although I think that there should be some examples for the ratelimit
crate (at least once the new API lands) to cover this.
from rust-cookbook.
Hi @clarcharr,
Thanks for looking into this!
We would certainly love to have both types of rate limiting examples as long as these are relatively bytesized (the draw fractal seams to be near our pain threshold). Originally we have hoped to present some trivial example without additional crates.
At the moment the cookbook tries to follow crates covered by Libz Blitz initiative. These crates are guaranteed to be of (or soon reach) high quality which is crucial for learning/reference resource such as cookbook.
So I would be hesitant to add another relatively new crate just yet. But feel free to submit ratelimit
for inclusion on Libz Blitz thread (inclusion of relatively new crate wouldn't certainly be a precedent) π
from rust-cookbook.
In that case Retry-After
(supported in Hyper hyperium/hyper#998) or the Custom headers of one the other two APIs?
Another question is about how this will be tested, since httpbin.org doesn't have a way to test this, and the other ones require an user account (not an issue during the implementation, but).
from rust-cookbook.
Personally I would go with github api. It's rate limited (the limit is much stricter if unauthorized)
from rust-cookbook.
At this moment we are looking for volunteer to finish up the PR by @jbolila
from rust-cookbook.
Related Issues (20)
- Indexing a map with a non-Copy structural key
- Replace lazy_static with once_cell
- Text refers to "Expand" button, actually it's "Show hidden lines"
- The header for the bitfield page is incorrect
- querying-the-web-api examples don't work due to missing user-agent-headers
- `GUI` createβs introduce and contrast
- Use clap-derive instead of raw clap for argument parsing HOT 1
- 3.1 Decompress a tarball while removing a prefix from the paths with E0107 HOT 1
- Some documents are unavailable
- Mime crate badges show CSV's version HOT 1
- Missing (hard to find?) edit link to github
- Add recipe to building custom format-like macros
- Is this repo abandoned? HOT 5
- Bug: Sample Code Doesn't Work
- replace crossbeam's scoped threads with std's scoped threads
- Dead links in Mathematics section
- rusqlite first example does not compile HOT 2
- clap-basic.md is stale, wrong
- Clap basic got a wrong demo
- update Log Messages/Configure Logging sections HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rust-cookbook.