The ml-downloader from malaire

ml-downloader's Introduction

ml-downloader

Simple blocking downloader, featuring:

retries with custom delays
custom interval between successful downloads for rate limiting
hash check (optional)
based on reqwest

Examples

Simple usage

Create Downloader with default configuration and then download one file.

use ml_downloader::Downloader;

let mut downloader = Downloader::new()?;
let bytes = downloader.get("https://example.com/").send()?;
# Ok::<(), ml_downloader::Error>(())

Custom configuration

Create Downloader with

"foobar/1.0" as USER_AGENT
1.0 - 1.1 seconds interval between successful downloads
two retries after failed download
- 2.0 - 2.2 seconds delay after initial failure
- 5.0 - 5.5 seconds delay after 2nd failure

use ml_downloader::Downloader;

let mut downloader = Downloader::builder()
    .reqwest(|cb| cb.user_agent("foobar/1.0"))
    .interval(1.0, 1.1)
    .retry_delays(&[(2.0, 2.2), (5.0, 5.5)])
    .build()?;

# Ok::<(), ml_downloader::Error>(())

ml-downloader's People

Contributors

Stargazers

Watchers

ml-downloader's Issues

The default client has 30 seconds timeout

See seanmonstar/reqwest#1343

Downloading large files will fail without an public API to tweak the timeout!

Crash on unwrap()

Hi!
Thank you for your interesting crate!

I tried it with the updated code from hhtpmock example for a simple get json request :

example

    let m = server.mock(|when, then| {
        when.method(GET)
            .path("/users");
        then.status(200)
            .header("content-type", "application/json")
            .json_body(json!({ "name": "Hans" }));
    });

The request:

    let mut response = Request::get(&format!("http://{}/users", server.address()))
        .body(())
        .unwrap()
        .send()
        .unwrap();

    let user: Value =
        serde_json::from_str(&response.text().unwrap()).expect("cannot deserialize JSON");

Works well.

But

let mut downloader = Downloader::builder()
                .reqwest(|cb| cb.user_agent("foobar/1.0"))
                .interval(1.0, 1.1)
                .retry_delays(&[(2.0, 2.2), (5.0, 5.5)])
                .build().expect("Cant build Downloader");
            let bytes = downloader.get(&url).send().expect("Cant download");

crashed:

thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'called Option::unwrap()on aNonevaluecalledOption::unwrap()on aNone value', ', .cargo/registry/src/index.crates.io-6f17d22bba15001f/ml-downloader-0.1.0/src/lib.rs/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ml-downloader-0.1.0/src/lib.rs::398398::4747

    fn send_once(&mut self) -> Result<Bytes, Error> {
        let response = self.inner.try_clone().unwrap().send()?;     << This unwrap()

I don`t know why but now the response is None...
Anyway, it looks like a better solution to remove this unwrap and respond with some Error code to prevent crashing .

Cheers!

Recommend Projects

malaire / ml-downloader Goto Github PK

ml-downloader's Introduction

ml-downloader

Examples

Simple usage

Custom configuration

ml-downloader's People

Contributors

Stargazers

Watchers

Forkers

ml-downloader's Issues

The default client has 30 seconds timeout

Crash on unwrap()

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent