Giter Club home page Giter Club logo

humblebundle-downloader's Introduction

Humble Bundle Downloader

PyPI PyPI

Download all of your content from your Humble Bundle Library!

The first time this runs it may take a while because it will download everything. After that it will only download the content that has been updated or is missing.

Features

  • support for Humble Trove (--trove flag)
  • downloads new and updated content from your Humble Bundle Library on each run (only check for updates if using --update)
  • cli command for easy use (downloading will also work on a headless system)
  • works for SSO and 2FA accounts
  • optional progress bar for each item downloaded (--progress flag)
  • optional filter by file types using an include or exclude list (--include/--exclude flag)
  • optional filter by platform types like video, ebook, etc... (--platform flag)

Install

pip install humblebundle-downloader

Instructions

1. Getting cookies

First thing to do is get your account cookies. This can be done by getting a browser extension that lets you see or export your cookies.

  • Method 1 (recommended)

    • Get the value of the cookie called _simpleauth_sess and pass that value using -s "COOKIE_VALUE"
  • Method 2

    • Export the cookies in the Netscape format using an extension.
      If your exported cookie file is not working, it may be a formatting issue, this can be fixed by running the command curl -b cookies.orig.txt --cookie-jar cookies.txt http://bogus

2. Downloading your library

Use the following command to download your Humble Bundle Library:
hbd --cookie-file cookies.txt --library-path "Downloaded Library" --progress

This directory structure will be used:
Downloaded Library/Purchase Name/Item Name/downloaded_file.ext

Notes

  • Inside your library folder a file named .cache.json is saved and keeps track of the files that have been downloaded. This way running the download command again pointing to the same directory will only download new or updated files.
  • Use --help with all hbd commands to see available options
  • Find supported platforms for the --platform flag by visiting your Humble Bundle Library and look under the Platform dropdown
  • Download select bundles by using the -k or --keys flag. Find these keys by going to your Purchases section, click on a products and there should be a downloads?key=XXXX in the url.

humblebundle-downloader's People

Contributors

rytilahti avatar xtream1101 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

humblebundle-downloader's Issues

Crash on FileNotFoundError

Whenever I try to run the script it crashes after a while with a FileNotFoundError (see traceback below). This is using v0.0.8. Any ideas?

Traceback (most recent call last):
File "c:\users\micro\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\users\micro\anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\micro\Anaconda3\Scripts\hbd.exe_main
.py", line 7, in
File "c:\users\micro\anaconda3\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
platform_include=cli_args.platform,
File "c:\users\micro\anaconda3\lib\site-packages\humblebundle_downloader\download_library.py", line 116, in download_library
with open(filename, 'wb') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: '.\Humble Double Fine Bundle\Psychonauts\Psychonauts.com/psychonauts-linux-05062013-bin'

"Malformed" cookies.txt causes confusing traceback

I'd like to suggest two small changes as a result of my experience getting started with hbd:

  1. Log a warning about the cookie file not being in Netscape format when employing the "original cookie method" fallback, OR do a format check on the file data to verify it isn't a not-correct-enough Netscape format file. Something very basic and crude like if '\tTRUE\t' in f.read().upper(): raise SomeError() would do the trick, I'd think.
  2. Mention in README.md that the cookie loader is pedantic and the below curl command can fix such problems if you have them.

Backstory follows:

As in issues #29 and #30 I used the Cookie Quick Manager add-on for Firefox (configured to export using the Netscape template) to obtain my cookies.txt. However, it seems http.cookiejar.MozillaCookieJar is too picky about the file format to load my file as-is, and the resulting LoadError exception caused hbd to silently fall back on the "original cookie method," ultimately resulting in the requests module raising an uncaught exception about leading/trailing whitespace in the cookie header. I guess from the newline at the end of the file.

I was ultimately able to resolve my problem by round-tripping the cookies through curl; diffing the two cookie files makes me think the problem has to do with lines that rely on tail matching....

curl -b cookies.orig.txt --cookie-jar cookies.txt http://bogus

Failed to get list of purchases

As using the recommended Firefox extension didn't work (see issue #29), using the Cookie Quick Manager extension worked enough to move the program to the next stage.
Now, the only output I am getting is: Failed to get list of purchases with no extra output to determine what is going wrong.
Is there some sort of way I could go about debugging this?

Option to only get new files

Currently it will always check every link to see if there is an update. but add a flag --update that will check every link, otherwise just check for files not in the cache

Organize folders by title instead of by bundle

It would be great to have the option to not create folders for each bundle and instead create them based on the title of the thing.

Instead of Downloaded Library/Purchase Name/Item Name/downloaded_file.ext you'd just get Downloaded Library/Item Name/downloaded_file.ext.

Right now you can end up downloading the same files twice if you have the same content in multiple bundles.

Torrent Download Support?

Just curious, does this downloader support downloading the corresponding torrents for supported games?

image

In theory, archiving the torrent files is one way of reducing the storage complexity while still ensuring future access to the files (under the assumption that there will still be a generous seeder still around by the day you decide to download the torrent). If not, it's totally cool, but the help flags didn't seem to mention it, and it seems to be rather the distinct from a "platform" selection

cookies missing

Hi,
I fed the script with my key (-s) and the script crashed.
am I missing something?

Traceback (most recent call last):
  File "~/.local/bin/hbd", line 8, in <module>
    sys.exit(cli())
  File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 84, in cli
    DownloadLibrary(
  File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 63, in start
    self.purchase_keys = self.purchase_keys if self.purchase_keys else self._get_purchase_keys()  # noqa: E501
  File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 397, in _get_purchase_keys
    raise Exception("Unable to download user-data, cookies missing?")
Exception: Unable to download user-data, cookies missing?

Support Using Original File Names

Given the bundle Humble Survive This Bundle, for the game Shelter 2, three zips can be downloaded:

image

https://dl.humble.com/OSX_Shelter2_Build20150708.zip
https://dl.humble.com/Linux_Shelter2_Build20150708.zip
https://dl.humble.com/PC_Shelter2_Build20150708.zip

However, hbd appears to use the game name Shelter 2.zip instead for the resulting download file.

hbd should attempt to maintain the original file name, this would also resolve #5.

Platform flag no longer works

I used this successfully on a previous version, but have recently updated to version 0.2.1. I went to one of my current libraries (ebooks) with an existing json file and tried to do a new download to update just the ebooks. Using the "-p ebook" flag had no effect and the downloader started downloading everything from all platforms.

In order to download my newly purchased bundles, I had to use the --keys flag and download each with its own key.

Syntax Errors

I'm trying to use this tool, but I'm running into a syntax error:

Traceback (most recent call last):
  File "/usr/local/bin/hbd", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/humblebundle_downloader/cli.py", line 97, in cli
    from .download_library import DownloadLibrary
  File "/usr/local/lib/python2.7/site-packages/humblebundle_downloader/download_library.py", line 338
    ), end='\r')
          ^
SyntaxError: invalid syntax```

`hbd --help` does not show available options

Running hbd -h or hbd --help doesn't show all available options (with which to get more detailed help on how to use them or what they're for). It only shows:

usage: hbd [-h] {download} ...

positional arguments:
  {download}
    download  Download content in your humble bundle library

optional arguments:
  -h, --help  show this help message and exit

Expected: a list of available options, as with most CLI's.
Version: 0.3.4, installed from pip install humblebundle-downloader

Without any more info from hbd --help, one must inspect source to see what options there are...

Option Rebuild .cache.json from already downloaded files

Hi @xtream1101, love HBD saved me lots of work.

Would it be possible to add a feature that will generate a new .cache.json based on the files already downloaded?

I moved my downloaded files to a different storage drive but not the .cache.json file, Then I deleted the original folder without backing up the .cache.json even though I had it in my mind to do so. I couldn't recover the file because windows doesn't send large amounts of files/data to the recycle bin. My Library is 1.05 TB so far and I have more bundles I have acquired since "the incident."

--exclude option is inverted

When I run the following command, the --exclude mobi epub option seems to be inverted, it is downloading only .mobi and .epub files, and is excluding .pdf files:

$ hbd download --cookie-file ~/humblebundle/hbd-cookies.txt --library-path ~/humblebundle/ --exclude mobi epub --progress
Checking bundle: Humble Book Bundle - Design Like a Designer by Adams Media
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Typography for the People/typographyforthepeople.mobi
	100% [==================================================]
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Typography for the People/typographyforthepeople.epub
	100% [==================================================]
Skipping the file typographyforthepeople.pdf
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Cultivating Creativity - Daily Rituals for Visual Inspiration/cultivatingcreativity.mobi
	100% [==================================================]
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Cultivating Creativity - Daily Rituals for Visual Inspiration/cultivatingcreativity.epub
	100% [==================================================]
Skipping the file cultivatingcreativity.pdf

Can't specify platform when adding the "--trove" switch

Specifying the platform for download works just fine when downloading from your library:
hbd download --cookie-file hbd-cookies.txt --library-path "HumbleBundleGames" --platform Windows --progress

However, when adding the "--trove" switch, it will download all platforms:
hbd download --cookie-file hbd-cookies.txt --library-path "HumbleBundleGames" --platform Windows --trove --progress

Is this intended behavior, an issue?

Do not overwrite older files

From /u/TSPhoenix

Keep all versions of a file, rather then overwrite like the tool currently does.

Possible solution is to append the date of the old url to the old file name (also must update the cache data with this new file

KeyError: 'last-modified'

First off, thank you for this app. It's worked really well to download the majority of my library.

After many (18-20+) hours of downloading, the downloader crashed with this error:

Checking bundle: Humble Image Comics Bundle 2 - Image Firsts
Checking bundle: Humble Comics Bundle - Image Comics featuring Creators OWN Worlds
Traceback (most recent call last):
  File "/usr/local/bin/hbd", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/cli.py", line 82, in cli
    DownloadLibrary(
  File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 66, in start
    self._process_order_id(order_id)
  File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 207, in _process_order_id
    self._process_product(order_id, bundle_title, product)
  File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 272, in _process_product
    'url_last_modified': product_r.headers['Last-Modified'],
  File "/usr/local/lib/python3.9/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'last-modified'

I tried to resume but it broke at the exact same spot. Any thoughts? Can I provide any specifics to help debug? Is there any way to skip this one so I can continue downloading the rest?

Unable to download user-data, cookies missing?

Using the recommended Firefox Export Cookies extension to download the cookies results with the following error:

Traceback (most recent call last):
  File "/home/james/.local/bin/hbd", line 8, in <module>
    sys.exit(cli())
  File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 82, in cli
    DownloadLibrary(
  File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 57, in start
    self.purchase_keys = self.purchase_keys if self.purchase_keys else self._get_purchase_keys()  # noqa: E501
  File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 380, in _get_purchase_keys
    raise Exception("Unable to download user-data, cookies missing?")
Exception: Unable to download user-data, cookies missing?

I managed to get around this particular issue by using the Cookie Quick Manager plugin and changing the output to Netscape in the settings.

Add icon per folder

Here's the script I used (all but your parts are public domain). It does more than just save images though. It creates a sanatized version of orders.json (which is a dict of order_id to order):

import json
from urllib.parse import urlparse
from urllib.request import urlretrieve
import os.path


j = json.load(open("orders.json"))
library_path = ""


def _clean_name(dirty_str):
    allowed_chars = (' ', '_', '.', '-', '[', ']')
    clean = []
    for c in dirty_str.replace('+', '_').replace(':', ' -'):
        if c.isalpha() or c.isdigit() or c in allowed_chars:
            clean.append(c)

    return "".join(clean).strip().rstrip('.')


ids_to_delete = []

for id, values in j.items():
    #values.pop("amount_spent", None)
    values.pop('gamekey', None)
    values.pop('uid', None)
    values.pop('all_coupon_data', None)
    values.pop('tpkd_dict', None)
    values.pop('path_ids', None)
    
    bundle_title = _clean_name(values['product']['human_name'])
    
    subs_to_delete = []
    
    subproducts = values['subproducts']
    for sub in subproducts:
        product_title = _clean_name(sub['human_name'])
        product_folder = os.path.join(library_path, bundle_title, product_title)
        
        if os.path.exists(product_folder):
            sub['path'] = product_folder
            
            icon_url = sub['icon']
            
            if icon_url is not None:
                ext = urlparse(icon_url).path.rsplit('.')[-1]
                icon_path = os.path.join(product_folder, 'folder.' + ext)
                if not os.path.exists(target_path):
                    urlretrieve(url = icon_url, filename=icon_path)
        
        sub.pop('icon', None)
        
        dl_counts = 0
        for dl in sub['downloads']:
            to_remove = []
            dl_struct = dl['download_struct']
            for dls in dl_struct:
                if 'url' not in dls:
                    if 'filename' not in dls:
                        to_remove.append(dls)
                    continue
                    
                p = urlparse(dls['url']['web'])
                fname = p.path.rsplit("/", maxsplit=1)[-1]
                dls['filename'] = fname
                
                del dls['url']
            for t in to_remove:
                dl_struct.remove(t)
            
            if dl_struct:
                dl_counts += 1
        
        if not dl_counts:
            subs_to_delete.append(sub)
    
    for sub in subs_to_delete:
        subproducts.remove(sub)
    
    if not subproducts:
        ids_to_delete.append(id)

for id in ids_to_delete:
    del j[id]

json.dump(j, open("all-orders-sanatized.json", "w+"), indent=4)

Clean up alignment of progress percentage

The --progress percentage currently is left aligned, making the progress bar jump to the right as the download completes, e.g.:

        0% [                                                  ]
        10% [=====                                             ]
        100% [==================================================]

I managed to reproduce this with python 3.7, macOS 11.4 in a zsh terminal.

Size of library/dry run

It would be useful if the script could calculate the total size of all files downloaded. This makes it easier to prepare the necessary storage needed.

Humble Trove URL returning HTTP 400

In an effort to get the latest Trove games from my subscription, the Trove URL has been updated to throw 400 errors without the full parameters.

trove_base_url = 'https://www.humblebundle.com/api/v1/trove/chunk?index={idx}' # noqa: E501

The URL now needs the parameters "property" and "direction".

Example

https://www.humblebundle.com/api/v1/trove/chunk?property=popularity&direction=desc&index=0

Incomplete downloads are not handled

This download failed, but the partially downloaded file is sitting in its folder as if nothing went wrong.

Possible solutions:

  • Files that are current being written should have a .part extension (e.g. BS2RM_3_1_02.apk). The .part extension should be removed when the download has successfully completed.
  • Catch exceptions that cause a download to fail. Delete the partially downloaded file.
  • Catch exceptions that cause a download to fail. Add a .part extension to the partially downloaded file.

Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_Linux_1395095896.zip: 100% [==================================================]
Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_windows_1395095896.zip: 100% [==================================================]
Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_mac_1396042209.dmg: 100% [==================================================]
Traceback (most recent call last):Smoking Mirror/BS2RM_3_1_02.apk: 30% [===============                                   ]
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 425, in _error_catcher
    yield
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 507, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\http\client.py", line 454, in read
    n = self.readinto(b)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\http\client.py", line 498, in readinto
    n = self.fp.readinto(b)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 564, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 529, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jesse\AppData\Local\Programs\Python\Python38\Scripts\hbd.exe\__main__.py", line 7, in <module>
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
    download_library(
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 128, in download_library
    for data in item_r.iter_content(chunk_size=4096):  # noqa E501
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Edge case (Invicible Vol 1 epub)

Invincible Vol. 1 has two different EPUB downloads in the library. One is about 43 MB while the other is about 100 MB. I haven't been able to find which bundle the larger file is associated with and I have to assume that this is one of those cases where the file was silently replaced by either Humble Bundle or Image. Being a hoarder, I'd appreciate it if the downloader could download all versions, but if it walking through bundles, I'm not sure how it would catch this.

KeyError: 'url' while downloading

Downloading: 2.apk: 100% [==================================================]
Downloading: 3.zip: 100% [==================================================]
Checking bundle: 9
Checking bundle: G
Downloading: s.exe: 100% [==================================================]
Downloading: X.dmg: 100% [==================================================]
Downloading: X.zip: 100% [==================================================]
Downloading: X.sh: 100% [==================================================]
Traceback (most recent call last):
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jesse\AppData\Local\Programs\Python\Python38\Scripts\hbd.exe\__main__.py", line 7, in <module>
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 65, in cli
    download_library(
  File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 64, in download_library
    url = file_type['url']['web']
KeyError: 'url'

Android apks might not contain the game

This is just a FYI ticket, feel free to close if you don't see this in scope for this project.

Usecase: Using this tool to create a backup of ones whole Humble Library

Unfortunately, some Android apks downloaded from Humble only contain a downloader (see F-Droid's explanation of this feature) and not the whole game. I neither know how to detect this (without either launching the apk or by dissecting the apk with something like apktool) nor how to fetch the obb files automatically...

Games which do this:

  • Grim Fandango Remastered (was at least in "Humble Mobile Bundle - Great Adventures" and "Humble Mobile Bundle 17")

(might add to this issue if I find more)

Haven't been able to use this since HB redid the trove setup

I setup hbd earlier this year and had it running for a while, but shortly after the change, the trove was no longer downloadable (might be permanent given the client). Worse though, is that I haven't been able to use this at all. No matter if I'm providing the cookie via argument or file, whether I'm asking to download by key or platform it keeps crashing and suggesting that I'm not including a cookie. I've made the suggested change to line 174 (new link for JSON data), but this hasn't resolved my issue.

Exception: Unable to download user-data, cookies missing?

Using Python 3.10 (virtualenv), installed hbd through pip.

  • Obtained the cookie through Chrome (since it easily exposes them), copied it into the clipboard as is.
  • Copied its value (quoted)
  • Ran hbd -s $cookie -l library --progress

Got error:
Exception: Unable to download user-data, cookies missing?

Am I doing something wrong or this is just not working currently?

Please note I did a quick search (while logged into my Humble Bundle account, into Library) and I could not find the tag hbd is trying to find in the code.

   logger.debug("Library request: " + str(library_r))
        library_page = parsel.Selector(text=library_r.text)
        user_data = library_page.css('#user-home-json-data').xpath('string()').extract_first()  #i noqa: E501
        if user_data is None:
            raise Exception("Unable to download user-data, cookies missing?")

Can't launch hbd command

After installing using pip install humblebundle-downloader, I can't run hbd.

I'm running the commands in a PowerShell session on Windows 10, and python 3.9.

hbd --help returns the error The term 'hbd' is not recognized as the name of a cmdlet, function, script file, or operable program..
python hbd returns the error C:\Users\wildcardjoker\AppData\Local\Microsoft\WindowsApps\python.exe: can't open file 'C:\Users\wildcardjoker\source\humblebundle-downloader\build\lib\humblebundle_downloader\hbd': [Errno 2] No such file or directory

It's probably a very basic thing that I'm missing, however I'm not very familiar with python and am unsure how to proceed.

Is there a way to circumvent throttling?

First of all thank you for creating this amazing tool!

I have a pretty big library and for hours it downloaded fine, but now it downloads at an extremely slow speed.

Screenshot_20210422_124848

A small pdf takes about half an hour.

However, when I download over the humble bundle interface, the speed is a lot faster (about 5 mbps).

What I tried

I started using the cookie file, while also copying the headers of my browser session to the requests session (self.session.headers.update(HEADERS) - logging enabled, so I see the correct headers are sent), but this seemed to have no effect.

Not installing under pipx

I realize this is more "feature request" than "bug report" since pipx isn't mentioned as a supported install method.

pipx handles making each pip install it's own venv to prevent dependency hell between conflicting apps. It's a utility around pip that just makes life easier. Project page is here: https://pypa.github.io/pipx/

Normally pipx should just work for any pip program, but I'm getting the following error:

pipx install humblebundle-downloader
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-yd3man9q/humblebundle-downloader_2c6b9a727e3f4ba08c3f97fb0eef3bab/setup.py", line 8, in <module>
          setup(
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/__init__.py", 
line 155, in setup
          return distutils.core.setup(**attrs)
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 109, in setup
          _setup_distribution = dist = klass(attrs)
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 458, in __init__
          _Distribution.__init__(
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 293, in __init__
          self.finalize_options()
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 850, in finalize_options
          for ep in sorted(loaded, key=by_order):
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 849, in <lambda>
          loaded = map(lambda e: e.load(), filtered)
        File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 196, in load
          return functools.reduce(getattr, attrs, module)
      AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'        
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Error installing humblebundle-downloader.

This is on Ubuntu 20.04 LTS,, x86-64. Install DID work under pip proper.

Remove or don't create empty folders

Here is how I am using HBD.

  1. I download the files on a windows laptop I use most of the time
  2. then I copy the downloaded files to a remote storage drive on my Mac
  3. I delete the files from the laptop but leave the cache file
  4. when I rerun the HBD it will create empty folders for ones I have already downloaded and moved to the other computer/drive

The network share isn't always connected for various reasons so that is why I dl to the laptop drive. Anyway t would be nice if there were only folders for new and updated things rather then all of them, currently that is 173 folders and only a small percent have anything.

I know my use case is different from most bit it would be a nice feature.

Script crash when Humble Bundle returns error 404

Hi, I discovered this scrip a few weeks ago and i'm loving it, the good feeling that comes from having all your files in your own hard drive instead of depending on someone to keep em online.

Praise aside I found a "bug", I noticed that the script crashed while downloading, not a big deal, ran it again and didn't thing twice of it, but today i noticed that the thing is crashing on an specific bundle, so i looked up the key for the bundle it was having trouble with and found it was "HUMBLE OVERWHELMINGLY POSITIVE BUNDLE 2", grabbed the key and started testing.

It crashed as soon as it finished downloading it and I went onto the humble page to try to download it manually and voila, the file is missing. (also tried the torrent file and is also corrupted)

I will now provide the error code:

I:\Humble Bundle>hbd download -c cookies.txt -l "Tests" --progress -k [Bundle key here] --update Checking bundle: Humble Overwhelmingly Positive Bundle 2 Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Windows_v110.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Linux_v110.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Mac_v110_ModLauncher_Fix.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\ModLauncher.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\Momodora - Reverie Under the Moonlight\momodora_reverieunderthemoonlight_windows.zip 100% [==================================================] Traceback (most recent call last): File "C:\Users\alfer\AppData\Local\Programs\Python\Python38\Scripts\hbd-script.py", line 11, in <module> load_entry_point('humblebundle-downloader==0.3.2', 'console_scripts', 'hbd')() File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 82, in cli DownloadLibrary( File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 66, in start self._process_order_id(order_id) File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 207, in _process_order_id self._process_product(order_id, bundle_title, product) File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 272, in _process_product 'url_last_modified': product_r.headers['Last-Modified'], File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\requests\structures.py", line 54, in __getitem__ return self._store[key.lower()][1] KeyError: 'last-modified'

If I can provide any other useful information I would love to help.

Missing Downloads - maybe because of the /

No url found: Humble PC Android Bundle 12/Costume Quest
No url found: Humble Bundle - PC and Android 8/Jack Lumber
No url found: Humble Bundle - PC and Android 8/AaaaaAAaaaAAAaaAAAAaAAAAA for the Awesome
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble eBook Bundle/Signal to Noise
No url found: Humble Indie Bundle 2/Osmos
No url found: Humble Indie Bundle 6/Dustforce DX
No url found: Humble Indie Bundle 7/Indie Game - The Movie
No url found: Humble Indie Bundle 9/FTL - Faster than Light
No url found: Humble Mobile Bundle 7/Horn
No url found: Humble Mobile Bundle 8/TowerMadness 2
No url found: Humble Weekly Bundle - Presented by Rock Paper Shotgun/World Of Goo Soundtrack

This are the bundles that are not downloaded. Maybe its because of the / in the name. The rest works perfectly, thanks for the awesome script!

Downloading trove results in error message

When attempting to download using the command hbd -s "my cookie value" --library-path "Downloaded Library" --progress --trove, I get the error:

`Only checking the Humble Trove...
Traceback (most recent call last):
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\Scripts\hbd-script.py", line 33, in
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\cli.py", line 95, in cli
).start()
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\download_library.py", line 67, in start
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\download_library.py", line 185, in _get_trove_products
page_content = trove_r.json()
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value]

<title>405 Method Not Allowed</title>

405 Method Not Allowed

The method GET is not allowed for this resource.

: 0`

Running the command without --trove works as expected.

How to download bundle if I wasn't logged in when I purchased?

When I purchased a bundle, i only provided an email address.
So I received a link by email to download the bundle without the intermediate login step.
When I run

hbd gen-cookies --cookie-file hbd-cookies.txt

Then paste the email link in the browser location bar, nothing happens, the chrome browser instance does not get closed, despite seemingly having all the required cookies set.

Has anyone managed to successfully download an 'anonymous' bundle (one purchased without a login).

Issue downloading library

Not sure what exactly is happening and verified cookie file is functioning as needed.

Traceback (most recent call last):
File "/home/.local/bin/hbd", line 8, in
sys.exit(cli())
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 82, in cli
DownloadLibrary(
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 66, in start
self._process_order_id(order_id)
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 207, in _process_order_id
self._process_product(order_id, bundle_title, product)
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 272, in _process_product
'url_last_modified': product_r.headers['Last-Modified'],
File "/usr/lib/python3/dist-packages/requests/structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'last-modified'

Dowload fails since upgrading to the newest version

C:\Users\David>hbd download --cookie-file hbd-cookies.txt --library-path K:\Humble_Bundle --progress
Traceback (most recent call last):
  File "C:\Python27\Scripts\hbd-script.py", line 11, in <module>
    load_entry_point('humblebundle-downloader==0.2.2', 'console_scripts', 'hbd')()
  File "c:\python27\lib\site-packages\humblebundle_downloader\cli.py", line 97, in cli
    from .download_library import DownloadLibrary
  File "c:\python27\lib\site-packages\humblebundle_downloader\download_library.py", line 338
    ), end='\r')
          ^
SyntaxError: invalid syntax

Error during download

While downloading with the problem it crashed and produced following error:

`Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 1322, in getresponse
response.begin()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "c:\users\david\appdata\local\programs\python\python38-32\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\util\retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dl.humble.com', port=443): Max retries exceeded with url: /batman_inferno_flac.zip?gamekey=UfzcTBCyZwY45Ksf&ttl=1580402318&t=6f3c35d65e6caaeca64b3ffd85070fa0 (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\david\appdata\local\programs\python\python38-32\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\David\AppData\Local\Programs\Python\Python38-32\Scripts\hbd.exe_main
.py", line 7, in
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
DownloadLibrary(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 48, in start
self._process_order_id(order_id)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 59, in _process_order_id
self._process_product(order_id, bundle_title, product)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 98, in _process_product
product_r = requests.get(url, stream=True)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dl.humble.com', port=443): Max retries exceeded with url: /batman_inferno_flac.zip?gamekey=UfzcTBCyZwY45Ksf&ttl=1580402318&t=6f3c35d65e6caaeca64b3ffd85070fa0 (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)')))`

"--trove" don't work for some games

Hi !
First, thank you for this great script ! I just used it to download the Humble Trove and I noticed that the script doesn't try to download certain games. I tried to restart the script several times which allowed me to download some missing games, but for the list below, nothing helps. The script doesn't want to download them. I don't see any error messages, it just doesn't seem to detect them.

Tested on Manjaro Linux, 11/04/2020 :

- Trine Enchanted Edition
- Deadlight
- The Penumbra Collection
- Amnesia: A Machine for Pigs
- The Bridge
- ETHEREAL
- SanctuaryRPG: Black Edition
- A Short Hike
- FOTONICA
- MirrorMoon EP
- Once Upon A Crime In The West
- macdows 95
- Broken Sword: Director's Cut
- Anna's Quest
- Broken Sword 2 - the Smoking Mirror
- Chaos on Deponia
- DON'T GIVE UP
- Blackguards
- Roman Sands
- Dear Esther

I also noticed an another problem: when the script has finished downloading the Humble Trove, it then tries to download the rest of the purchases made on Humble Bundle.

:)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.