ovyerus / bandsnatch Goto Github PK

View Code? Open in Web Editor NEW

55.0 2.0 2.0 156 KB

A CLI batch downloader for your Bandcamp collection.

License: MIT License

Rust 91.56% Dockerfile 2.06% Shell 0.63% Nix 5.75%

bandcamp bandcamp-downloader bulk-download music rust hacktoberfest

bandsnatch's Introduction

bandsnatch

A CLI batch downloader for your Bandcamp collection.

Bandsnatch is a Rust tool for downloading all of your Bandcamp purchases all at once in your desired format, and being able to be run multiple times when you buy new releases.

This project is heavily inspired by Ezwen's bandcamp-collection-downloader, which I used myself before this, specifically existing to help me learn Rust, but also to add some improvements over it that I've wanted.

State of the Project

This tool is still currently a work in progress, so bugs and other weirdness may occur. If anything weird happens or something breaks, please open an issue about it with information and reproduction steps if possible. Specifically testing use of this with large collections would be very helpful to see if there's any areas that I need to improve in.

If you're a developer poking around in the code, please note that this is my first proper project written using Rust, so code quality may be subpar, especially in terms of memory usage. If you have any ideas to improve the project in general I'd love to hear them.

Usage

The most basic usage is along the lines of bandsnatch run -f <format> <username>, as it will try to automatically fetch cookies from a local cookies.json. But if this fails you can provide the -c option with a path to a cookies file to use.

For more advanced usage, you can run bandsnatch run -h to get output similar to the following.

Run Bandsnatch to download your collection

Usage: bandsnatch run [OPTIONS] --format <AUDIO_FORMAT> <USER>

Arguments:
  <USER>  Name of the user to download releases from (must be logged in through cookies) [env: BS_USER=]

Options:
  -f, --format <AUDIO_FORMAT>   The audio format to download the files in [env: BS_FORMAT=] [possible values: flac, wav, aac-hi, mp3-320, aiff-lossless, vorbis, mp3-v0, alac]
  -c, --cookies <COOKIES_FILE>  [env: BS_COOKIES=]
      --debug                   Enables some extra debug output in certain scenarios [env: BS_DEBUG=]
  -d, --dry-run                 Return a list of all tracks to be downloaded, without actually downloading them
  -F, --force                   Ignores any found cache file and instead does a from-scratch download run [env: BS_FORCE=]
  -j, --jobs <JOBS>             The amount of parallel jobs (threads) to use [env: BS_JOBS=] [default: 4]
  -n, --limit <LIMIT>           Maximum number of releases to download. Useful for testing [env: BS_LIMIT=]
  -o, --output-folder <FOLDER>  The folder to extract downloaded releases to [env: BS_OUTPUT_FOLDER=] [default: ./]
  -h, --help                    Print help information

Besides these options, you can also use environment variables with the option name in SCREAMING_SNAKE_CASE, prefixed with BS_, so that if set up correctly you can just run bandsnatch run and have it automatically download your collection to the folder you want.

Example

bandsnatch run -c ./cookies.json -f flac -o ./Music ovyerus

This would download my entire music collection into a local "Music" folder, and also create a bandcamp-collection-downloader.cache in the same directory, which then gets read on future runs in order to skip items it has already retrieved.

Authentication

Because Bandsnatch does not manage logging into Bandcamp itself, you need to provide it the authentication cookies. For Firefox users, you can extract a cookies.json with the Cookie Quick Manager extension, and on Chrome, you can use the Get cookies.txt LOCALLY extension, to extract the cookies in the Netscape format, which Bandsnatch also supports.

If you don't provide the --cookies option, Bandsnatch will attempt to automatically find a file named cookies.json or cookies.txt in the local directory and load it.

Installing

Binary builds of Bandsnatch are available on our releases page for Windows, Mac (both ARM & Intel), and Linux (various architectures).

Nix flake

If you use Nix, Bandsnatch is available as a flake. You can try it out without installing via nix run or nix shell:

nix run github:ovyerus/bandsnatch -- --help
nix shell github:ovyerus/bandsnatch

You can install it permanently with nix profile install, or by adding it to your NixOS/Home Manager configuration.

Homebrew

brew install ovyerus/tap/bandsnatch

Scoop

scoop bucket add ovyerus https://github.com/Ovyerus/bucket
scoop install bandsnatch

AUR

Bandsnatch is also available on the AUR. Either use your favourite AUR helper, or you can install it manually via the following:

git clone https://aur.archlinux.org/bandsnatch.git
cd bandsnatch
makepkg -si

NetBSD (unofficial)

Bandsnatch is also available from the official NetBSD repositories, but is not maintaned by myself.

pkgin install bandsnatch

Crate

cargo install bandsnatch

From source

Pull this repository and run cargo build --release, and look for the bandsnatch binary in ./target/release/.

License

This program is licensed under the MIT license (see LICENSE or https://opensource.org/licenses/MIT).

bandsnatch's People

Contributors

Stargazers

Watchers

Forkers

akiyukiokayasu russelg

bandsnatch's Issues

Auto detect user to pull releases for

The identities.fan properties on the pagedata blob can probably be used to reliably automatically pull the user's name/page url without needing to be provided via --user. This can be retrieved from the homepage.

Strange message is output during downloading

Thank you for making great software. This seems to be working for me.

However, a strange message appeared, so I thought I would report it here.
Cookie.json was exported from Cookie Quick Manager.
Here is the command I used. I ran it on macOS 13.2.

bandsnatch -c cookies.json -f flac MY_NAME

thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Error("missing field `download_type`", line: 1, column: 3752)', src/api/mod.rs:188:91
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: bandsnatch::main::{{closure}}::{{closure}}::{{closure}}
   4: tokio::runtime::task::harness::Harness<T,S>::poll
   5: std::thread::local::LocalKey<T>::with
   6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: tokio::macros::scoped_tls::ScopedKey<T>::set
   9: tokio::runtime::scheduler::multi_thread::worker::run
  10: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  11: tokio::runtime::task::harness::Harness<T,S>::poll
  12: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I get this message, but the download seems to be working correctly.
Therefore, there may be no harm done, but I reported it just in case.

Again, your software is very useful. Thank you for making it.

Debugging subcommands

Bandsnatch should have a bunch of different subcommands which grab different information from Bandcamp and displays it as JSON for debugging/testing purposes.

Better logging

Currently logging kinda sucks big time. Here's a list of some improvements

Show information about files/account being used right at the start like bandcamp-collection-downloader does.
- Format
- Username
- Cookies path
- Output path
- etc.
Remove wonky debug logs that screw with the progress.
Use the log stuff we have already for these.
Have another progress bar/line for progress through the user's collection

Replace invalid folder names.

Currently Bandsnatch has an error when it comes across a release with characters in its name which results in the release being skipped.
We should either have a map of replacement characters to use, which are visually similar, or just remove them straight up.

Previous fix was good, but not perfect

As mentioned in a previous issue, I was stuck at 104 releases of ~950, now it's down to 13, so great improvement

-> https://deadsara.bandcamp.com/album/self-titled
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("data did not match any variant of untagged enum StringOrNumber", line: 1, column: 2246)', ~/.cargo/registry/src/github.com-1ecc6
299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://stuntrock.bandcamp.com/album/were-not-assholes-were-just-smarter-than-you-split-release-with-tuyo
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 3492)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://stuntrock.bandcamp.com/album/fuck-canada-fuck-america-split-release-with-venetian-snares
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 3464)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://stuntrock.bandcamp.com/album/this-is-stunt-rock-volume-one-in-memory-of-bob-seger
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 3443)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/technoptimistic
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1879)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/red-eyes-ep
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1862)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.
2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/the-cult-of-remute
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1892)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/electronic-lifestyle
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1899)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://stuntrock.bandcamp.com/album/this-is-stunt-rock-volume-two-in-memory-of-tray-loren
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 3442)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://65daysofstatic.bandcamp.com/album/utopian-frequencies-disquiet-vinyl-bundle
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 4133)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/theme-tunes-for-10-games-never-made
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1958)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://redrumrecordz.bandcamp.com/album/threnody-bewildering-thoughts-1995
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 4210)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

-> https://remute.bandcamp.com/album/the-sound-of-money-radio-edit
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: Error("invalid type: null, expected a string", line: 1, column: 1936)', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/bandsnatch-0.2.0/src/api/mod.rs:194:74

Because the JSON isn't anonymized, I'm not posting it here, the locations seem to be off too, so it's hard to pin point the exact offending key value pair. I apparently have a bandcamp key left for a few of the remute's, so there is a way to test it.

Disallowed . or space at end of directory/filename

I was moving my collection to an NTFS partition, and then I ran into the limitation that Windows doesn't allow files/folders to end in a space or a period (.), quick fix would be to add a legal character like - or _, that way you'd know it was slightly altered to work Windows.

Outdated `html5ever` version required by `soup` produces future-incompatibilities warning

Just as an heads-up. Building with Rust-1.72.0 spits-out the following warning:

   Compiling bandsnatch v0.3.0 (/usr/pkgsrc/wip/bandsnatch/work/bandsnatch-0.3.0)
warning: use of deprecated method `chrono::TimeZone::datetime_from_str`: use `DateTime::parse_from_str` instead
  --> src/api/structs/digital_item.rs:50:34
   |
50 |             Some(d) => match Utc.datetime_from_str(d, FORMAT) {
   |                                  ^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(deprecated)]` on by default

warning: function `command` is never used
  --> src/cmds/release.rs:18:14
   |
18 | pub async fn command(_args: Args) -> Result<(), Box<dyn std::error::Error>> {
   |              ^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: `bandsnatch` (bin "bandsnatch") generated 2 warnings
    Finished release [optimized] target(s) in 9m 01s
warning: the following packages contain code that will be rejected by a future version of Rust: html5ever v0.22.5
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`

Error when running

I have downloaded to Linux Mint 21.2, I have made the file executable, created a cookies.json file containing the bandcamp.com cookie and I am using the following command:

./bandsnatch run -f flac <username>

After running that command I get:

thread 'main' panicked at 'Failed to deserialise collection page data blob.: Error("invalid type: null, expected u16", line: 1, column: 68813)', src/api/mod.rs:76:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Please can you help?