To try to keep snatch "simple as possible", could we maybe use https://github.com/mitsuhiko/indicatif instead of a custom "progress bar implementation" ?

Currently, the algorithm consists in merging chunks when those are downloaded.
We can improve this part in merging neighbor chunks, once those have been downloaded, which will reduce the time to merge chunks.

We must check that the remote host communicates using at least HTTP v1.1 to be able to download specific chunks from a given file.

Now that the first release is available we should :

• create a tag to the good commit ;
• provide a binary at least for OSX ;
• check behaviour on other platform.

Every chunk of the file must be downloaded using a thread.

I think we should add a --force or --true flag to the CLI in order to allow integration of snatch into automated task (like installation scripts).

Snatch is a tool that download efficiently a remote content, splitting this one into chunks and download each of those in a thread.

But, there can exists some problems if the server canno't send in a request the length of the remote content, or if the server does not use HTTP 1.1 (+).
If the program meet those "problems", it stops.
But, is a default behaviour for those remote contents can be to download the remote content, in a single thread?

Compared to 0.9, I guess hyper dropped openssl dependency and made it a pure rust library.

I wonder when snatch will move on to hyper 0.11 and give an option to decide whether to use openssl. OpenSSL can potentially introduce security issue.

This will give users an option to use pure Rust binary.

Is there a plan to support BitTorrent download?

We need the total length of the remote content to calculate each chunk size.

Using the current CLI version, we canno't visualize the status of each download.
A better alternative is to improve the CLI using multi progress bar.

To manage development routines, it is important to create cargo profiles like test, development, or release.

Hi, I like this tool! very cool!

It would be nice if snatch could provide a alternate progress bar like the -a option in tool axel. Because I will use snatch -t40 or -t50 to download big stuff, and the screen cannot fit all that many separated progress bars :)

Thanks!

I love snatch but I have few notes regarding CLI.

• Remove -u, why not just snatch http://example.com/bigfile.mp4 ?
• Optional -f, why not deduce filename from URL?
• Support for multiple URLs, something like snatch http://example.com/a.mp4 http://example.com/b.mp4

Requesting a resource that's protected by Basic Authorization creates an empty file and then crashes out.

snatch https://httpbin.org/basic-auth/user/passwd -d
# [DEBUG_MODE] version: 0.1.2
# [DEBUG_MODE] file: passwd
# [DEBUG_MODE] threads: 4
# Waiting a response from the remote server... OK !
# Remote content length: 0 MB
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:323
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Following discussion on #20

To download as fast as possible the remote content we need to know the maximum number of available threads on the local machine.

It is sad...

The project needs some unit tests - each new feature has to be tested.

This is the output when I try to compile:

    Updating git repository `https://github.com/derniercri/snatch.git`
   Compiling ansi_term v0.9.0
   Compiling lazy_static v0.2.2
   Compiling bitflags v0.7.0
   Compiling winapi-build v0.1.1
   Compiling libc v0.2.18
   Compiling num_cpus v1.2.0
   Compiling httparse v1.2.1
   Compiling winapi v0.2.8
   Compiling gcc v0.3.40
   Compiling openssl v0.7.14
   Compiling pkg-config v0.3.8
   Compiling openssl-sys v0.7.17
   Compiling argparse v0.2.1
   Compiling rustc-serialize v0.3.22
   Compiling semver v0.1.20
   Compiling typeable v0.1.2
   Compiling openssl-sys-extras v0.7.14
   Compiling rustc_version v0.1.7
   Compiling unicase v1.4.0
   Compiling unicode-normalization v0.1.2
   Compiling language-tags v0.2.2
   Compiling traitobject v0.0.1
   Compiling kernel32-sys v0.2.2
   Compiling time v0.1.35
   Compiling pbr v1.0.0-alpha.2
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/pb.rs:436:13: 436:16 error: binary assignment operation `+=` cannot be applied to type `std::string::String` [E0368]
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/pb.rs:436             out += repeat!(" ", width - s.len());
                                                                                            ^~~
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/pb.rs:436:13: 436:16 help: run `rustc --explain E0368` to see a detailed explanation
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/multi.rs:206:17: 206:20 error: binary assignment operation `+=` cannot be applied to type `std::string::String` [E0368]
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/multi.rs:206                 out += &move_cursor_up(self.nlines);
                                                                                                   ^~~
.cargo/registry/src/github.com-1ecc6299db9ec823/pbr-1.0.0-alpha.2/src/multi.rs:206:17: 206:20 help: run `rustc --explain E0368` to see a detailed explanation
error: aborting due to 2 previous errors
error: failed to compile `snatch v0.1.1 (https://github.com/derniercri/snatch.git#9e30b500)`, intermediate artifacts can be found at `/tmp/cargo-install.LV86csGNNryo`

Caused by:
  Could not compile `pbr`.

To learn more, run the command again with --verbose.

Is there a way to pin pbr to a stable version or something like that?

It would be nice to know total download speed, as well as size of already downloaded content.

Each chunk represents a part of the downloaded file. To merge those chunks, it may to put them in correct order.

I tried to do a thread count of 4, and then a thread count of 64, and to the naked eye, for a small file, it seemed indistinguishable. What is the optimal thread count?

program get blocked forever when any chunk-downloading thread panics.
The problem is that MultiBar::listen method blocks until all bars call finish() or finish_print .

On file creation the file name may exceed the length the OS file system allows. In this case the program currently hard panics could this be changed?

I tried to install Snatch on Ubuntu 16.04, following error occurs:

Build failed, waiting for other jobs to finish...
error: failed to compile `snatch v0.1.1 (https://github.com/derniercri/snatch.git#f447f7ca)`, intermediate artifacts can be found at `/tmp/cargo-install.MvwyPs1F5EQJ`

Caused by:
  failed to run custom build command for `openssl-sys-extras v0.7.14`
process didn't exit successfully: `/tmp/cargo-install.MvwyPs1F5EQJ/release/build/openssl-sys-extras-5c7e4d8925825f00/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
PROFILE = Some("release")
TARGET = Some("x86_64-unknown-linux-gnu")
debug=false opt-level=3
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
HOST_CC = None
CC = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-fPIC" "-o" "/tmp/cargo-install.MvwyPs1F5EQJ/release/build/openssl-sys-extras-5c7e4d8925825f00/out/src/openssl_shim.o" "-c" "src/openssl_shim.c"
cargo:warning=src/openssl_shim.c:1:26: fatal error: openssl/hmac.h: No such file or directory
cargo:warning=compilation terminated.
ExitStatus(ExitStatus(256))


command did not execute successfully, got: exit code: 1



--- stderr
thread 'main' panicked at 'explicit panic', .cargo/registry/src/github.com-1ecc6299db9ec823/gcc-0.3.40/src/lib.rs:1006
note: Run with `RUST_BACKTRACE=1` for a backtrace.

To build and test the latest version of Snatch, it will be useful to add a continuous integration service, like Travis CI.

The first version of the CLI must be able to download a file given an URL.

Exemple :
snatch http://domain.com/myFile

https://crates.io/search?q=snatch

If you want to grab the name snatch on crates.io.

Also it would be easier for others to install it by:

cargo install snatch

Add in Readme, for macOS El Capitan + Sierra:

export OPENSSL_INCLUDE_DIR=`brew --prefix openssl`/include
export OPENSSL_LIB_DIR=`brew --prefix openssl`/lib
export DEP_OPENSSL_INCLUDE=`brew --prefix openssl`/include

cargo install --git https://github.com/derniercri/snatch.git
Updating git repository https://github.com/derniercri/snatch.git
Updating registry https://github.com/rust-lang/crates.io-index
Downloading num_cpus v1.2.0
Downloading pbr v1.0.0-alpha.2
Downloading argparse v0.2.1
Downloading hyper v0.9.14
Downloading ansi_term v0.9.0
Downloading libc v0.2.18
Downloading kernel32-sys v0.2.2
Downloading time v0.1.35
Downloading winapi v0.2.8
Downloading winapi-build v0.1.1
Downloading rustc-serialize v0.3.22
Downloading unicase v1.4.0
Downloading url v1.2.4
Downloading httparse v1.2.1
Downloading openssl-verify v0.1.0
Downloading traitobject v0.0.1
Downloading language-tags v0.2.2
Downloading solicit v0.4.4
Downloading mime v0.2.2
Downloading cookie v0.2.5
Downloading typeable v0.1.2
Downloading log v0.3.6
Downloading openssl v0.7.14
Downloading rustc_version v0.1.7
Downloading semver v0.1.20
Downloading idna v0.1.0
Downloading matches v0.1.4
Downloading unicode-normalization v0.1.2
Downloading unicode-bidi v0.2.3
Downloading bitflags v0.7.0
Downloading openssl-sys v0.7.17
Downloading lazy_static v0.2.2
Downloading openssl-sys-extras v0.7.14
Downloading pkg-config v0.3.8
Downloading gcc v0.3.40
Downloading hpack v0.2.0
Compiling unicode-normalization v0.1.2
Compiling language-tags v0.2.2
Compiling pkg-config v0.3.8
Compiling log v0.3.6
Compiling matches v0.1.4
Compiling semver v0.1.20
Compiling rustc-serialize v0.3.22
Compiling unicode-bidi v0.2.3
Compiling argparse v0.2.1
Compiling ansi_term v0.9.0
Compiling lazy_static v0.2.2
Compiling idna v0.1.0
Compiling httparse v1.2.1
Compiling rustc_version v0.1.7
Compiling unicase v1.4.0
Compiling traitobject v0.0.1
Compiling bitflags v0.7.0
Compiling mime v0.2.2
Compiling libc v0.2.18
Compiling openssl-sys v0.7.17
Compiling gcc v0.3.40
Compiling typeable v0.1.2
Compiling winapi v0.2.8
Compiling winapi-build v0.1.1
Compiling kernel32-sys v0.2.2
Compiling time v0.1.35
Compiling hpack v0.2.0
Compiling num_cpus v1.2.0
Compiling url v1.2.4
Compiling pbr v1.0.0-alpha.2
Compiling solicit v0.4.4
Compiling openssl-sys-extras v0.7.14
Build failed, waiting for other jobs to finish...
error: failed to compile snatch v0.1.1 (https://github.com/derniercri/snatch.git#9e30b500), intermediate artifacts can be found at /var/folders/7c/fp0pspqj27l2frkfv567nmfh0000gn/T/cargo-install.PRrg6M7dBiun

Caused by:
failed to run custom build command for openssl-sys-extras v0.7.14
process didn't exit successfully: /var/folders/7c/fp0pspqj27l2frkfv567nmfh0000gn/T/cargo-install.PRrg6M7dBiun/release/build/openssl-sys-extras-5c7e4d8925825f00/build-script-build (exit code: 101)
--- stdout
TARGET = Some("x86_64-apple-darwin")
OPT_LEVEL = Some("3")
PROFILE = Some("release")
TARGET = Some("x86_64-apple-darwin")
debug=false opt-level=3
HOST = Some("x86_64-apple-darwin")
TARGET = Some("x86_64-apple-darwin")
TARGET = Some("x86_64-apple-darwin")
HOST = Some("x86_64-apple-darwin")
CC_x86_64-apple-darwin = None
CC_x86_64_apple_darwin = None
HOST_CC = None
CC = None
HOST = Some("x86_64-apple-darwin")
TARGET = Some("x86_64-apple-darwin")
HOST = Some("x86_64-apple-darwin")
CFLAGS_x86_64-apple-darwin = None
CFLAGS_x86_64_apple_darwin = None
HOST_CFLAGS = None
CFLAGS = None
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-fPIC" "-o" "/var/folders/7c/fp0pspqj27l2frkfv567nmfh0000gn/T/cargo-install.PRrg6M7dBiun/release/build/openssl-sys-extras-5c7e4d8925825f00/out/src/openssl_shim.o" "-c" "src/openssl_shim.c"
cargo:warning=src/openssl_shim.c:1:10: fatal error: 'openssl/hmac.h' file not found
cargo:warning=#include <openssl/hmac.h>
cargo:warning= ^
cargo:warning=1 error generated.
ExitStatus(ExitStatus(256))

command did not execute successfully, got: exit code: 1

--- stderr
thread 'main' panicked at 'explicit panic', .cargo/registry/src/github.com-1ecc6299db9ec823/gcc-0.3.40/src/lib.rs:1006
note: Run with RUST_BACKTRACE=1 for a backtrace.

This is with the brew version of openssl already installed.

Snatch built from devel HEAD (fac87df) cannot download these files even with -t1.

• https://github.com/derniercri/snatch/archive/0.1.1.zip
• https://github.com/derniercri/snatch/archive/0.1.1.tar.gz
• https://github.com/derniercri/snatch/archive/devel.zip

Output:

# Waiting a response from the remote server... OK !
# Remote content length: 384.57 KB
Downloading 1 chunks:
Chunk 0 ▎ |##############################################################################################################################################| 100.00 % 352.65 KB/s
thread '<unnamed>' panicked at 'Cannot download the chunk 0, due to error Invalid Status provided', src/download.rs:167
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Done! Your download is available in 0.1.1.zip

when downloading files less than 1 mb it is displayed as 0 mb due to division truncation. snatch should either display it kilobytes or display size as floating points

A comment from HackerNews highlight the idea that each chunk has to be downloaded from a mirror.
Actually, this is a great idea to avoid the traffic jam of the targeted server.

A good observation from slamb in the Reddit post announcement is:

It looks like your chunk GET requests are not conditioned on an etag, or even on a Last-Modified header.
So if the server doesn't serve the same file every time (such as if the file were replaced on the server
halfway through), you'll blindly put together parts of different files without any warning.
This checking shouldn't be hard to add; I think it'd be better to do it and warn the user in this case.

I agree with this comment, and I think we should warn the user as a first case, and implement a better solution for a next release.

Instead of forcing the user to specify a filename, we can (if no filename has been given) infer it from the given URL. In cas the url doesn't really give a filename we should use a default one such as index.html.

I think this would improve user experience.

I got a file from someone to download a file from his dropbox account.

It's not easy to extract the exact link so I go the link that I was handed (with Firefox), click on "Download" combobox -> "Direct download" option, then I'm prompted to sign up, but I click on the small link that says "no thanks, take me to the download", and then Firefox starts downloading. Then I go to Firefox's download manager, locate the download -> right click -> Copy download link.

Then I go to the terminal and I get this error from snatch:

$ snatch --url https://dl.dropboxusercontent.com/content_link/onXmWanH62azyJzz85nZe7mkv8kLr8PMPX71uXtmCW15vtnwBEV6iTra4DGuqhAI/file?dl=1
# Waiting a response from the remote server... OK !
[ERROR] Cannot get the content length of the remote content, from the server.

I've also tried to Pause or Cancel the download before I launch this command, but the result is the same (and yes, firefox allows me to pause and resume the download, and Firefox shows me the amount of bytes left, so no issue with the Content-Length header from the server I guess).

Hey there, I wanted to give snatch a try and so I followed the README to install it.

It it suggested to do as follow : cargo install --git https://github.com/derniercri/snatch.git but once installed I noticed that it was the master version which got installed since:

▶ snatch -V       
Usage:
    snatch [OPTIONS]
snatch: Unknown short option "V"

So I deleted it and reinstalled it with cargo install --git https://github.com/derniercri/snatch.git --branch devel and it solved it for me. :-)

▶ snatch -V
Snatch 0.1.2