mayeulc / hb-downloader Goto Github PK
View Code? Open in Web Editor NEWThis project forked from bluetyson/hb-downloader
An automated utility to download your Humble Bundle purchases.
License: MIT License
This project forked from bluetyson/hb-downloader
An automated utility to download your Humble Bundle purchases.
License: MIT License
Latest git
[2018/04/18 03:45:10] [ Config] write_md5=True
[2018/04/18 03:45:10] [ Config] read_md5=True
[2018/04/18 03:45:10] [ Config] force_md5=False
[2018/04/18 03:45:10] [ Config] ignore_md5=False
[2018/04/18 03:45:10] [ Config] debug=True
[2018/04/18 03:45:10] [ Config] download_location=/home/parker/backup
[2018/04/18 03:45:10] [ Config] cookie_filename=cookies.txt
[2018/04/18 03:45:10] [ Config] chunksize=8192000
[2018/04/18 03:45:10] [ Config] resume_downloads=True
[2018/04/18 03:45:10] [ Config] Platform audio=False
[2018/04/18 03:45:10] [ Config] Platform ebook=False
[2018/04/18 03:45:10] [ Config] Platform windows=False
[2018/04/18 03:45:10] [ Config] Platform mac=False
[2018/04/18 03:45:10] [ Config] Platform linux=True
[2018/04/18 03:45:10] [ Config] Platform android=True
[2018/04/18 03:45:10] [ Config] Platform asmjs=False
Enter your Authy token: <authycode>
Traceback (most recent call last):
File "./hb-downloader.py", line 41, in <module>
hapi.login(ConfigData.username, ConfigData.password, ConfigData.authy_token)
File "/home/parker/build/hb-downloader/humble_api/humble_api.py", line 137, in login
error_msg, request=request, response=response, authy_required=authy_required)
humble_api.exceptions.humble_captcha_exception.HumbleCaptchaException: Are you sure you're not a robot? Please try again.
Hello MayeulC
i faced the situation that i wanted to run hb-downloader without cd-ing in to the hb directory.
By default i got this error:
oceanic@batch:[~]$: /opt/hb-downloader/hb-downloader.py
Humble Bundle Downloader v0.5.0
This program is not affiliated nor endorsed by Humble Bundle, Inc.
For any suggestion or bug report, please create an issue at:
https://github.com/MayeulC/hb-downloader/issues
Traceback (most recent call last):
File "/opt/hb-downloader/hb-downloader.py", line 22, in <module>
Configuration.load_configuration("hb-downloader-settings.yaml")
File "/opt/hb-downloader/configuration.py", line 46, in load_configuration
with open(config_file, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'hb-downloader-settings.yaml'
I edited ./hb-downloader/hb-downloader.py
Line 22:
old: Configuration.load_configuration("hb-downloader-settings.yaml")
new: Configuration.load_configuration(workdir + "/hb-downloader-settings.yaml")
and in the header section i added:
import os
workdir = (os.path.dirname(os.path.abspath(__file__)))
not sure if the is the way to do it correctly or not first try understanding python
hope this is useful to someone
thank you
Can this script download from the tribe? If not, it would be a very useful feature.
Whether we perform API calls asynchronously or in parallel, that would be of tremendous help speeding up the script, especially the initial purchase information, that's needed by hb-downloader list
among others. As the workload only makes API calls, making them asynchronously could easily speed things up 50 times, depending on the number of purchases. That's a conservative estimate, and it could speed things up a lot more on slower connections.
The other elephant in the room is hashing and downloading, that can be done in parallel (download new files while checking already downloaded ones). This could be really helpful on slower machines. Next steps could be multithreading hashing, and performing multiple download simultaneously.
So, here is the roadmap:
I am not sure what would be the best way to perform this. However, it would be nice if:
A couple references:
I discovered that in addition to the familiar "games, ebooks, audio" platforms, there is now a "video" platform. By default, hb-downloader
does not list or download anything from the video platform. A simple change like below to hb-downloader-settings.yaml
does successfully download the videos when a plain command of hb-downloader.py download
is ran, though, so it seems support should be easy to add?
download-platforms:
audio: True
ebook: True
windows: True
mac: True
linux: True
android: True
asmjs: False
video: True
At some point, the downloads start to fail, and the script downloads every file with a 0B size.
I actually have had a pretty good idea of the cause for a while now: this is probably due to the links being generated by humblebundle.com expiring.
There are two ways this could be fixed: generate links just before downloading instead of ahead-of-time, or regenerate them when there is an issue.
The first solution probably adds some latency, but that's probably not a big problem. The second one is more hacky, and probably more complicated to implement.
There is another, probably better way: generate small batches of links, and investigate the timeout.
-- Assuming links are indeed the issue here.
Originally asked by @tobias-p in #4.
if possible, could you alter the downloader, so that everything is downloaded by bundle?
For example, I have single games bought in hb where it is ok that the downloader creates a folder called like the game, but another game within a bundle is also downloaded in just a folder called like the game itself.
For example:
bought bioshock in hb store: then download to "/destination/bioshock/..." is ok
got bioshock within a bundle: then download should go to "/destination/bundlename/bioshock/..."this would make it easier to check out, which bundles are alreday downloaded.
The major issue I see with this if using a naive implementation, is that it would take two times as much disk space, and bandwidth to download duplicate games.
I would rather create a new folder, "by-purchase", or something like this, containing one subfolder for each purchase, and symlinks to the actual files.
Proposed organization, that just adds a "by-purchase" folder:
.
├── by-purchase
│ ├── Humble indie bundle X
│ │ ├── product01 -> ../../product01
│ │ └── product02 -> ../../product02
│ └── Humble store product 01
│ └── product01 -> ../../product01
├── product01
│ ├── audio
│ ├── linux
│ ├── mac
│ └── windows
└── product02
├── ebook
├── linux
└── windows
See alternative trees at the end of the post.
Now, one question: symlinks, or hardlinks?
I have an issue with symlinks, though. On windows, as far as I know, only ntfs supports them (maybe vfat as well), and before the creators update, you had to be root/administrator to create one. Maybe we can try to do so, and fallback to .lnk
if that doesn't work.
by-purchase
by-bundle
, by-store-purchase
, by-monthly
by-product
that has the current contents.I would make this optional (though probably enabled by default), behind a configuration switch. When disabled, she structure should remain the current one.
This might not be easily backwards-compatible if the above feature (storing products in a by-products
folder) is implemented. If so, I would probably always leave the downloads in by-product
.
For automatic installation and requirement management a setup.py would be a nice enhancement for people not that familiar with python or that can't (or just won't) RTFM. ¯_(ツ)_/¯
A few useful features would be to:
I have a few more ideas down the line with what we could do with that... :)
Would be cool if you could work directly in a calibre library instead of downloading to a folder.
I mean, you select a folder that is a calibre library.
Skip downloading books that are already downloaded in that library.
Get metadata from Humblebundle?
Add multiple formats to the same entry.
I'd appreciate an option to group downloaded files first by platform as in the example tree below.
.
|--------Audio
| |
| |--Product01
|
|--------Windows
|
|--Product02
It would be nice to download games from the humble trove as well. Unfortunately, that requires an active subscription to develop, as far as I know.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.