xtream1101 / humblebundle-downloader Goto Github PK
View Code? Open in Web Editor NEWDownload you Humble Bundle Library
License: MIT License
Download you Humble Bundle Library
License: MIT License
Asked for by /u/Stenthal
This is the subscription service which has access to DRM free content
Running hbd -h
or hbd --help
doesn't show all available options (with which to get more detailed help on how to use them or what they're for). It only shows:
usage: hbd [-h] {download} ...
positional arguments:
{download}
download Download content in your humble bundle library
optional arguments:
-h, --help show this help message and exit
Expected: a list of available options, as with most CLI's.
Version: 0.3.4, installed from pip install humblebundle-downloader
Without any more info from hbd --help
, one must inspect source to see what options there are...
After installing using pip install humblebundle-downloader
, I can't run hbd.
I'm running the commands in a PowerShell session on Windows 10, and python 3.9.
hbd --help
returns the error The term 'hbd' is not recognized as the name of a cmdlet, function, script file, or operable program.
.
python hbd
returns the error C:\Users\wildcardjoker\AppData\Local\Microsoft\WindowsApps\python.exe: can't open file 'C:\Users\wildcardjoker\source\humblebundle-downloader\build\lib\humblebundle_downloader\hbd': [Errno 2] No such file or directory
It's probably a very basic thing that I'm missing, however I'm not very familiar with python and am unsure how to proceed.
When I purchased a bundle, i only provided an email address.
So I received a link by email to download the bundle without the intermediate login step.
When I run
hbd gen-cookies --cookie-file hbd-cookies.txt
Then paste the email link in the browser location bar, nothing happens, the chrome browser instance does not get closed, despite seemingly having all the required cookies set.
Has anyone managed to successfully download an 'anonymous' bundle (one purchased without a login).
Just curious, does this downloader support downloading the corresponding torrents for supported games?
In theory, archiving the torrent files is one way of reducing the storage complexity while still ensuring future access to the files (under the assumption that there will still be a generous seeder still around by the day you decide to download the torrent). If not, it's totally cool, but the help flags didn't seem to mention it, and it seems to be rather the distinct from a "platform" selection
In an effort to get the latest Trove games from my subscription, the Trove URL has been updated to throw 400 errors without the full parameters.
The URL now needs the parameters "property" and "direction".
Example
https://www.humblebundle.com/api/v1/trove/chunk?property=popularity&direction=desc&index=0
Similar as the SSO issue. need new way to detect when the user is fully logged in
It would be great to have the option to not create folders for each bundle and instead create them based on the title of the thing.
Instead of Downloaded Library/Purchase Name/Item Name/downloaded_file.ext
you'd just get Downloaded Library/Item Name/downloaded_file.ext
.
Right now you can end up downloading the same files twice if you have the same content in multiple bundles.
The --progress
percentage currently is left aligned, making the progress bar jump to the right as the download completes, e.g.:
0% [ ]
10% [===== ]
100% [==================================================]
I managed to reproduce this with python 3.7, macOS 11.4 in a zsh terminal.
When I run the following command, the --exclude mobi epub
option seems to be inverted, it is downloading only .mobi
and .epub
files, and is excluding .pdf
files:
$ hbd download --cookie-file ~/humblebundle/hbd-cookies.txt --library-path ~/humblebundle/ --exclude mobi epub --progress
Checking bundle: Humble Book Bundle - Design Like a Designer by Adams Media
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Typography for the People/typographyforthepeople.mobi
100% [==================================================]
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Typography for the People/typographyforthepeople.epub
100% [==================================================]
Skipping the file typographyforthepeople.pdf
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Cultivating Creativity - Daily Rituals for Visual Inspiration/cultivatingcreativity.mobi
100% [==================================================]
Downloading: /home/pieter/humblebundle/Humble Book Bundle - Design Like a Designer by Adams Media/Cultivating Creativity - Daily Rituals for Visual Inspiration/cultivatingcreativity.epub
100% [==================================================]
Skipping the file cultivatingcreativity.pdf
Using Python 3.10 (virtualenv), installed hbd through pip.
hbd -s $cookie -l library --progress
Got error:
Exception: Unable to download user-data, cookies missing?
Am I doing something wrong or this is just not working currently?
Please note I did a quick search (while logged into my Humble Bundle account, into Library) and I could not find the tag hbd is trying to find in the code.
logger.debug("Library request: " + str(library_r))
library_page = parsel.Selector(text=library_r.text)
user_data = library_page.css('#user-home-json-data').xpath('string()').extract_first() #i noqa: E501
if user_data is None:
raise Exception("Unable to download user-data, cookies missing?")
C:\Users\David>hbd download --cookie-file hbd-cookies.txt --library-path K:\Humble_Bundle --progress
Traceback (most recent call last):
File "C:\Python27\Scripts\hbd-script.py", line 11, in <module>
load_entry_point('humblebundle-downloader==0.2.2', 'console_scripts', 'hbd')()
File "c:\python27\lib\site-packages\humblebundle_downloader\cli.py", line 97, in cli
from .download_library import DownloadLibrary
File "c:\python27\lib\site-packages\humblebundle_downloader\download_library.py", line 338
), end='\r')
^
SyntaxError: invalid syntax
First of all thank you for creating this amazing tool!
I have a pretty big library and for hours it downloaded fine, but now it downloads at an extremely slow speed.
A small pdf takes about half an hour.
However, when I download over the humble bundle interface, the speed is a lot faster (about 5 mbps).
I started using the cookie file, while also copying the headers of my browser session to the requests session (self.session.headers.update(HEADERS)
- logging enabled, so I see the correct headers are sent), but this seemed to have no effect.
No url found: Humble PC Android Bundle 12/Costume Quest
No url found: Humble Bundle - PC and Android 8/Jack Lumber
No url found: Humble Bundle - PC and Android 8/AaaaaAAaaaAAAaaAAAAaAAAAA for the Awesome
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Botanicula Debut/Kooky
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble Mo-boo-ile Bundle/The Spookening
No url found: Humble eBook Bundle/Signal to Noise
No url found: Humble Indie Bundle 2/Osmos
No url found: Humble Indie Bundle 6/Dustforce DX
No url found: Humble Indie Bundle 7/Indie Game - The Movie
No url found: Humble Indie Bundle 9/FTL - Faster than Light
No url found: Humble Mobile Bundle 7/Horn
No url found: Humble Mobile Bundle 8/TowerMadness 2
No url found: Humble Weekly Bundle - Presented by Rock Paper Shotgun/World Of Goo Soundtrack
This are the bundles that are not downloaded. Maybe its because of the / in the name. The rest works perfectly, thanks for the awesome script!
This download failed, but the partially downloaded file is sitting in its folder as if nothing went wrong.
Possible solutions:
Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_Linux_1395095896.zip: 100% [==================================================]
Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_windows_1395095896.zip: 100% [==================================================]
Downloading: Broken Sword 2 - The Smoking Mirror/BrokenSword2_mac_1396042209.dmg: 100% [==================================================]
Traceback (most recent call last):Smoking Mirror/BS2RM_3_1_02.apk: 30% [=============== ]
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 425, in _error_catcher
yield
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 507, in read
data = self._fp.read(amt) if not fp_closed else b""
File "c:\users\jesse\appdata\local\programs\python\python38\lib\http\client.py", line 454, in read
n = self.readinto(b)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\http\client.py", line 498, in readinto
n = self.fp.readinto(b)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 564, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 529, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Jesse\AppData\Local\Programs\Python\Python38\Scripts\hbd.exe\__main__.py", line 7, in <module>
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
download_library(
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 128, in download_library
for data in item_r.iter_content(chunk_size=4096): # noqa E501
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
Here's the script I used (all but your parts are public domain). It does more than just save images though. It creates a sanatized version of orders.json (which is a dict of order_id
to order
):
import json
from urllib.parse import urlparse
from urllib.request import urlretrieve
import os.path
j = json.load(open("orders.json"))
library_path = ""
def _clean_name(dirty_str):
allowed_chars = (' ', '_', '.', '-', '[', ']')
clean = []
for c in dirty_str.replace('+', '_').replace(':', ' -'):
if c.isalpha() or c.isdigit() or c in allowed_chars:
clean.append(c)
return "".join(clean).strip().rstrip('.')
ids_to_delete = []
for id, values in j.items():
#values.pop("amount_spent", None)
values.pop('gamekey', None)
values.pop('uid', None)
values.pop('all_coupon_data', None)
values.pop('tpkd_dict', None)
values.pop('path_ids', None)
bundle_title = _clean_name(values['product']['human_name'])
subs_to_delete = []
subproducts = values['subproducts']
for sub in subproducts:
product_title = _clean_name(sub['human_name'])
product_folder = os.path.join(library_path, bundle_title, product_title)
if os.path.exists(product_folder):
sub['path'] = product_folder
icon_url = sub['icon']
if icon_url is not None:
ext = urlparse(icon_url).path.rsplit('.')[-1]
icon_path = os.path.join(product_folder, 'folder.' + ext)
if not os.path.exists(target_path):
urlretrieve(url = icon_url, filename=icon_path)
sub.pop('icon', None)
dl_counts = 0
for dl in sub['downloads']:
to_remove = []
dl_struct = dl['download_struct']
for dls in dl_struct:
if 'url' not in dls:
if 'filename' not in dls:
to_remove.append(dls)
continue
p = urlparse(dls['url']['web'])
fname = p.path.rsplit("/", maxsplit=1)[-1]
dls['filename'] = fname
del dls['url']
for t in to_remove:
dl_struct.remove(t)
if dl_struct:
dl_counts += 1
if not dl_counts:
subs_to_delete.append(sub)
for sub in subs_to_delete:
subproducts.remove(sub)
if not subproducts:
ids_to_delete.append(id)
for id in ids_to_delete:
del j[id]
json.dump(j, open("all-orders-sanatized.json", "w+"), indent=4)
From /u/TSPhoenix
Keep all versions of a file, rather then overwrite like the tool currently does.
Possible solution is to append the date of the old url to the old file name (also must update the cache data with this new file
Using the recommended Firefox Export Cookies extension to download the cookies results with the following error:
Traceback (most recent call last):
File "/home/james/.local/bin/hbd", line 8, in <module>
sys.exit(cli())
File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 82, in cli
DownloadLibrary(
File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 57, in start
self.purchase_keys = self.purchase_keys if self.purchase_keys else self._get_purchase_keys() # noqa: E501
File "/home/james/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 380, in _get_purchase_keys
raise Exception("Unable to download user-data, cookies missing?")
Exception: Unable to download user-data, cookies missing?
I managed to get around this particular issue by using the Cookie Quick Manager plugin and changing the output to Netscape in the settings.
This is just a FYI ticket, feel free to close if you don't see this in scope for this project.
Usecase: Using this tool to create a backup of ones whole Humble Library
Unfortunately, some Android apks downloaded from Humble only contain a downloader (see F-Droid's explanation of this feature) and not the whole game. I neither know how to detect this (without either launching the apk or by dissecting the apk with something like apktool) nor how to fetch the obb files automatically...
Games which do this:
(might add to this issue if I find more)
Hi !
First, thank you for this great script ! I just used it to download the Humble Trove and I noticed that the script doesn't try to download certain games. I tried to restart the script several times which allowed me to download some missing games, but for the list below, nothing helps. The script doesn't want to download them. I don't see any error messages, it just doesn't seem to detect them.
Tested on Manjaro Linux, 11/04/2020 :
- Trine Enchanted Edition
- Deadlight
- The Penumbra Collection
- Amnesia: A Machine for Pigs
- The Bridge
- ETHEREAL
- SanctuaryRPG: Black Edition
- A Short Hike
- FOTONICA
- MirrorMoon EP
- Once Upon A Crime In The West
- macdows 95
- Broken Sword: Director's Cut
- Anna's Quest
- Broken Sword 2 - the Smoking Mirror
- Chaos on Deponia
- DON'T GIVE UP
- Blackguards
- Roman Sands
- Dear Esther
I also noticed an another problem: when the script has finished downloading the Humble Trove, it then tries to download the rest of the purchases made on Humble Bundle.
:)
Asked for by /u/JimMcKeeth
Filter what files types are included or excluded when downloading
The linked Chrome plugin does supply a cookie file that can be used with hbd. Comparing the output of the two linked plugins, I got found that while the plugin exports in the correct Netscape structure it is improperly escaping the values. see daftano/cookies.txt#1
Chrome 83 on Windows 10
Currently it will always check every link to see if there is an update. but add a flag --update
that will check every link, otherwise just check for files not in the cache
Specifying the platform for download works just fine when downloading from your library:
hbd download --cookie-file hbd-cookies.txt --library-path "HumbleBundleGames" --platform Windows --progress
However, when adding the "--trove" switch, it will download all platforms:
hbd download --cookie-file hbd-cookies.txt --library-path "HumbleBundleGames" --platform Windows --trove --progress
Is this intended behavior, an issue?
When attempting to download using the command hbd -s "my cookie value" --library-path "Downloaded Library" --progress --trove
, I get the error:
`Only checking the Humble Trove...
Traceback (most recent call last):
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\Scripts\hbd-script.py", line 33, in
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\cli.py", line 95, in cli
).start()
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\download_library.py", line 67, in start
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\humblebundle_downloader\download_library.py", line 185, in _get_trove_products
page_content = trove_r.json()
File "C:\Users\Ethan\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value]
Running the command without --trove works as expected.
I realize this is more "feature request" than "bug report" since pipx isn't mentioned as a supported install method.
pipx handles making each pip install it's own venv to prevent dependency hell between conflicting apps. It's a utility around pip that just makes life easier. Project page is here: https://pypa.github.io/pipx/
Normally pipx should just work for any pip program, but I'm getting the following error:
pipx install humblebundle-downloader
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-yd3man9q/humblebundle-downloader_2c6b9a727e3f4ba08c3f97fb0eef3bab/setup.py", line 8, in <module>
setup(
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/__init__.py",
line 155, in setup
return distutils.core.setup(**attrs)
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 109, in setup
_setup_distribution = dist = klass(attrs)
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 458, in __init__
_Distribution.__init__(
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 293, in __init__
self.finalize_options()
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 850, in finalize_options
for ep in sorted(loaded, key=by_order):
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/dist.py", line 849, in <lambda>
loaded = map(lambda e: e.load(), filtered)
File "/home/user/.local/pipx/shared/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 196, in load
return functools.reduce(getattr, attrs, module)
AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Error installing humblebundle-downloader.
This is on Ubuntu 20.04 LTS,, x86-64. Install DID work under pip
proper.
I'm trying to use this tool, but I'm running into a syntax error:
Traceback (most recent call last):
File "/usr/local/bin/hbd", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python2.7/site-packages/humblebundle_downloader/cli.py", line 97, in cli
from .download_library import DownloadLibrary
File "/usr/local/lib/python2.7/site-packages/humblebundle_downloader/download_library.py", line 338
), end='\r')
^
SyntaxError: invalid syntax```
Hi @xtream1101, love HBD saved me lots of work.
Would it be possible to add a feature that will generate a new .cache.json based on the files already downloaded?
I moved my downloaded files to a different storage drive but not the .cache.json file, Then I deleted the original folder without backing up the .cache.json even though I had it in my mind to do so. I couldn't recover the file because windows doesn't send large amounts of files/data to the recycle bin. My Library is 1.05 TB so far and I have more bundles I have acquired since "the incident."
Found by /u/Qazerowl
Hi, I discovered this scrip a few weeks ago and i'm loving it, the good feeling that comes from having all your files in your own hard drive instead of depending on someone to keep em online.
Praise aside I found a "bug", I noticed that the script crashed while downloading, not a big deal, ran it again and didn't thing twice of it, but today i noticed that the thing is crashing on an specific bundle, so i looked up the key for the bundle it was having trouble with and found it was "HUMBLE OVERWHELMINGLY POSITIVE BUNDLE 2", grabbed the key and started testing.
It crashed as soon as it finished downloading it and I went onto the humble page to try to download it manually and voila, the file is missing. (also tried the torrent file and is also corrupted)
I will now provide the error code:
I:\Humble Bundle>hbd download -c cookies.txt -l "Tests" --progress -k [Bundle key here] --update Checking bundle: Humble Overwhelmingly Positive Bundle 2 Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Windows_v110.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Linux_v110.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\SOMA_Mac_v110_ModLauncher_Fix.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\SOMA\ModLauncher.zip 100% [==================================================] Downloading: Tests\Humble Overwhelmingly Positive Bundle 2\Momodora - Reverie Under the Moonlight\momodora_reverieunderthemoonlight_windows.zip 100% [==================================================] Traceback (most recent call last): File "C:\Users\alfer\AppData\Local\Programs\Python\Python38\Scripts\hbd-script.py", line 11, in <module> load_entry_point('humblebundle-downloader==0.3.2', 'console_scripts', 'hbd')() File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 82, in cli DownloadLibrary( File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 66, in start self._process_order_id(order_id) File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 207, in _process_order_id self._process_product(order_id, bundle_title, product) File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 272, in _process_product 'url_last_modified': product_r.headers['Last-Modified'], File "c:\users\alfer\appdata\local\programs\python\python38\lib\site-packages\requests\structures.py", line 54, in __getitem__ return self._store[key.lower()][1] KeyError: 'last-modified'
If I can provide any other useful information I would love to help.
When using
hbd --progress
I noticed that it continues downloading 1epub, 1.cbz and 1.pdf and continues to overwrite those files as seen here: https://i.imgur.com/saHpqIx.png and this is the download-location: https://i.imgur.com/oDYfBWn.png
It would be useful if the script could calculate the total size of all files downloaded. This makes it easier to prepare the necessary storage needed.
Not sure what exactly is happening and verified cookie file is functioning as needed.
Traceback (most recent call last):
File "/home/.local/bin/hbd", line 8, in
sys.exit(cli())
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 82, in cli
DownloadLibrary(
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 66, in start
self._process_order_id(order_id)
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 207, in _process_order_id
self._process_product(order_id, bundle_title, product)
File "/home/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 272, in _process_product
'url_last_modified': product_r.headers['Last-Modified'],
File "/usr/lib/python3/dist-packages/requests/structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'last-modified'
Here is how I am using HBD.
The network share isn't always connected for various reasons so that is why I dl to the laptop drive. Anyway t would be nice if there were only folders for new and updated things rather then all of them, currently that is 173 folders and only a small percent have anything.
I know my use case is different from most bit it would be a nice feature.
I setup hbd earlier this year and had it running for a while, but shortly after the change, the trove was no longer downloadable (might be permanent given the client). Worse though, is that I haven't been able to use this at all. No matter if I'm providing the cookie via argument or file, whether I'm asking to download by key or platform it keeps crashing and suggesting that I'm not including a cookie. I've made the suggested change to line 174 (new link for JSON data), but this hasn't resolved my issue.
First off, thank you for this app. It's worked really well to download the majority of my library.
After many (18-20+) hours of downloading, the downloader crashed with this error:
Checking bundle: Humble Image Comics Bundle 2 - Image Firsts
Checking bundle: Humble Comics Bundle - Image Comics featuring Creators OWN Worlds
Traceback (most recent call last):
File "/usr/local/bin/hbd", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/cli.py", line 82, in cli
DownloadLibrary(
File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 66, in start
self._process_order_id(order_id)
File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 207, in _process_order_id
self._process_product(order_id, bundle_title, product)
File "/usr/local/lib/python3.9/site-packages/humblebundle_downloader/download_library.py", line 272, in _process_product
'url_last_modified': product_r.headers['Last-Modified'],
File "/usr/local/lib/python3.9/site-packages/requests/structures.py", line 54, in __getitem__
return self._store[key.lower()][1]
KeyError: 'last-modified'
I tried to resume but it broke at the exact same spot. Any thoughts? Can I provide any specifics to help debug? Is there any way to skip this one so I can continue downloading the rest?
Invincible Vol. 1 has two different EPUB downloads in the library. One is about 43 MB while the other is about 100 MB. I haven't been able to find which bundle the larger file is associated with and I have to assume that this is one of those cases where the file was silently replaced by either Humble Bundle or Image. Being a hoarder, I'd appreciate it if the downloader could download all versions, but if it walking through bundles, I'm not sure how it would catch this.
Hi,
I fed the script with my key (-s) and the script crashed.
am I missing something?
Traceback (most recent call last):
File "~/.local/bin/hbd", line 8, in <module>
sys.exit(cli())
File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/cli.py", line 84, in cli
DownloadLibrary(
File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 63, in start
self.purchase_keys = self.purchase_keys if self.purchase_keys else self._get_purchase_keys() # noqa: E501
File "~/.local/lib/python3.8/site-packages/humblebundle_downloader/download_library.py", line 397, in _get_purchase_keys
raise Exception("Unable to download user-data, cookies missing?")
Exception: Unable to download user-data, cookies missing?
Found by /u/Qazerowl
Current solution is to use normal email login.
Fix is to check the url after login differently
While downloading with the problem it crashed and produced following error:
`Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 1322, in getresponse
response.begin()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 303, in begin
version, status, reason = self._read_status()
File "c:\users\david\appdata\local\programs\python\python38-32\lib\http\client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "c:\users\david\appdata\local\programs\python\python38-32\lib\socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\urllib3\util\retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dl.humble.com', port=443): Max retries exceeded with url: /batman_inferno_flac.zip?gamekey=UfzcTBCyZwY45Ksf&ttl=1580402318&t=6f3c35d65e6caaeca64b3ffd85070fa0 (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\david\appdata\local\programs\python\python38-32\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\david\appdata\local\programs\python\python38-32\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\David\AppData\Local\Programs\Python\Python38-32\Scripts\hbd.exe_main.py", line 7, in
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
DownloadLibrary(
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 48, in start
self._process_order_id(order_id)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 59, in _process_order_id
self._process_product(order_id, bundle_title, product)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\humblebundle_downloader\download_library.py", line 98, in _process_product
product_r = requests.get(url, stream=True)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "c:\users\david\appdata\local\programs\python\python38-32\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dl.humble.com', port=443): Max retries exceeded with url: /batman_inferno_flac.zip?gamekey=UfzcTBCyZwY45Ksf&ttl=1580402318&t=6f3c35d65e6caaeca64b3ffd85070fa0 (Caused by SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2607)')))`
Downloading: 2.apk: 100% [==================================================]
Downloading: 3.zip: 100% [==================================================]
Checking bundle: 9
Checking bundle: G
Downloading: s.exe: 100% [==================================================]
Downloading: X.dmg: 100% [==================================================]
Downloading: X.zip: 100% [==================================================]
Downloading: X.sh: 100% [==================================================]
Traceback (most recent call last):
File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\jesse\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Jesse\AppData\Local\Programs\Python\Python38\Scripts\hbd.exe\__main__.py", line 7, in <module>
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\cli.py", line 65, in cli
download_library(
File "c:\users\jesse\appdata\local\programs\python\python38\lib\site-packages\humblebundle_downloader\download_library.py", line 64, in download_library
url = file_type['url']['web']
KeyError: 'url'
I'd like to suggest two small changes as a result of my experience getting started with hbd
:
if '\tTRUE\t' in f.read().upper(): raise SomeError()
would do the trick, I'd think.README.md
that the cookie loader is pedantic and the below curl
command can fix such problems if you have them.Backstory follows:
As in issues #29 and #30 I used the Cookie Quick Manager add-on for Firefox (configured to export using the Netscape template) to obtain my cookies.txt. However, it seems http.cookiejar.MozillaCookieJar is too picky about the file format to load my file as-is, and the resulting LoadError exception caused hbd
to silently fall back on the "original cookie method," ultimately resulting in the requests module raising an uncaught exception about leading/trailing whitespace in the cookie header. I guess from the newline at the end of the file.
I was ultimately able to resolve my problem by round-tripping the cookies through curl
; diffing the two cookie files makes me think the problem has to do with lines that rely on tail matching....
curl -b cookies.orig.txt --cookie-jar cookies.txt http://bogus
As using the recommended Firefox extension didn't work (see issue #29), using the Cookie Quick Manager extension worked enough to move the program to the next stage.
Now, the only output I am getting is: Failed to get list of purchases
with no extra output to determine what is going wrong.
Is there some sort of way I could go about debugging this?
Whenever I try to run the script it crashes after a while with a FileNotFoundError (see traceback below). This is using v0.0.8. Any ideas?
Traceback (most recent call last):
File "c:\users\micro\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\users\micro\anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\micro\Anaconda3\Scripts\hbd.exe_main.py", line 7, in
File "c:\users\micro\anaconda3\lib\site-packages\humblebundle_downloader\cli.py", line 89, in cli
platform_include=cli_args.platform,
File "c:\users\micro\anaconda3\lib\site-packages\humblebundle_downloader\download_library.py", line 116, in download_library
with open(filename, 'wb') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: '.\Humble Double Fine Bundle\Psychonauts\Psychonauts.com/psychonauts-linux-05062013-bin'
Currently it will first download all trove products, then it will look at the rest of your purchases
From /u/Rot_Grub
I used this successfully on a previous version, but have recently updated to version 0.2.1. I went to one of my current libraries (ebooks) with an existing json file and tried to do a new download to update just the ebooks. Using the "-p ebook" flag had no effect and the downloader started downloading everything from all platforms.
In order to download my newly purchased bundles, I had to use the --keys flag and download each with its own key.
Given the bundle Humble Survive This Bundle
, for the game Shelter 2
, three zips can be downloaded:
https://dl.humble.com/OSX_Shelter2_Build20150708.zip
https://dl.humble.com/Linux_Shelter2_Build20150708.zip
https://dl.humble.com/PC_Shelter2_Build20150708.zip
However, hbd
appears to use the game name Shelter 2.zip
instead for the resulting download file.
hbd
should attempt to maintain the original file name, this would also resolve #5.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.