minomino / mindl Goto Github PK

View Code? Open in Web Editor NEW

71.0 71.0 14.0 3.85 MB

A downloader for various sites and services.

License: GNU Affero General Public License v3.0

Go 97.08% Makefile 2.26% Python 0.66%

mindl's People

Contributors

Stargazers

Watchers

Forkers

rikimtasu hiroppy2001 darkmusic bob-fu nutjob funagi yuiiuy wenshihken usagi123 pietuszko tkw25 cwavs uprprc777 raf555

mindl's Issues

window immediately closes

As soon as I run the executable through command, the window immediately closes upon running. Now I may just be an idiot, but I don't think this was how mindl intended to work.

Reading URLs from a file

Should be fairly trivial. If a file and command line URLs are passed, simply merge the lists.

How do you use this tool for local file instead of URL?

I have already downloaded an eBook on my PC via Bookwalker's Windows Application.
And the files are stored locally but they're encrypted as you can see in this screenshot: prntscr.com/uqn7l5
So, how do I use the tool to jump straight to the decryption stage instead of downloading files from URL and risking an account ban?

Mac: Permission Denied error

I'm running Mac OS 10.11.6 (El Capitan) and I tried to run mindl so I can use it on ebookjapan. It gave me a "Permission Denied" error. I probably did something wrong so could someone please enlighten me on how exactly to start it up and run it in Terminal?

Support other BinB Reader sites

Since I've kept BinB Reader stuff separate from the BookLive plugin, it should be fairly easy to add support for other sites. The problem is I need an account with a bought book to make sure things properly work. If anyone ever reads this and wishes some other site that uses this reader was supported and wouldn't mind lending me their account, get in touch.

In the mean time, I'll implement what I can.

Cmoa (シーモア)
AnimateBookstore (アニメイトブックストア)

Zipping downloaded files

Haven't added the feature in the rewrite. The standard library has a package for it, so it should be fairly easy. As opposed to the legacy mindl, it should be implemented by the download manager rather than by the plugins.

Set up cross-compilation

Right now I've set up Appveyor to compile for Windows on tag pushes, but Go can be set up for cross-compilation. I'm thinking a "dist" rule in the makefile to compile for Windows, Linux and OSX, both for 32/64 bit, then set up a Python script to zip and tar the stuff.

booklive unscrambling doesn't work

I've tried downloading the demo pages from a couple of booklive novels and I get a bunch of scrambled images, some of which are repeated (basically, with the exception of page 1, each couple of pages appears twice, i.e. image 2 is the same as image 4, image 3 the same as 5, 6 is 8, 7 is 9...)

example:

page 3

page 5

Unable to use with bookwalker

Sorry to open this if the program is no longer being actively developed, but I cannot get it to work with bookwalker.

this is on Windows 10 64bit

I cd into the directory with the program ofc, then run

mindl -d -z -o [email protected] -o password=password https://global.bookwalker.jp/ded443a3d3-9fdb-4902-b7bd-69c34967cb5b/

I get the following output

(20:37:00 ERRO) Found no handler for: https://global.bookwalker.jp/ded443a3d3-9fdb-4902-b7bd-69c34967cb5b/
(20:37:00 FATA) No plugins to select from.

Am I missing anything in the arguments or command? or is this project not currently working anymore?

Thanks

Hangs at starting download

Here is the console error:

(19:31 INFO) URL is being handled by plugin: eBookJapan v0.1
(19:31 INFO) Starting download...
(19:33:45 CRITICAL) [eBookJapan] Failed to find the controller. If it's a reader
 link, make sure it hasn't expired.
(19:33 CRITICAL) An uncaught exception was raised while downloading.
viewLayer qualitylayer None
Traceback (most recent call last):
  File "C:\Python\Python35-32\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python\Python35-32\lib\site-packages\mindl\__main__.py", line 117, in
 <module>
    main(parser.parse_args())
  File "C:\Python\Python35-32\lib\site-packages\mindl\__main__.py", line 106, in
 main
    dm.start_download()
  File "C:\Python\Python35-32\lib\site-packages\mindl\download_manager.py", line
 74, in start_download
    raise e
  File "C:\Python\Python35-32\lib\site-packages\mindl\download_manager.py", line
 40, in start_download
    for dl in self._plugin.downloader():
  File "C:\Python\Python35-32\lib\site-packages\mindl\plugins\ebookjapan.py", li
ne 119, in downloader
    raise e
  File "C:\Python\Python35-32\lib\site-packages\mindl\plugins\ebookjapan.py", li
ne 116, in downloader
    self.wait.until(EC.visibility_of_element_located((By.XPATH, "//div[@id='cont
roller']")))
  File "C:\Python\Python35-32\lib\site-packages\selenium\webdriver\support\wait.
py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

And here is the ghostdriver.log:

[INFO  - 2016-09-03T23:31:36.089Z] GhostDriver - Main - running on port 49860
[INFO  - 2016-09-03T23:31:37.337Z] Session [8eecbb40-722e-11e6-9041-c7422ee2d8e1] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0","webSecurityEnabled":false}
[INFO  - 2016-09-03T23:31:37.337Z] Session [8eecbb40-722e-11e6-9041-c7422ee2d8e1] - page.customHeaders:  - {}
[INFO  - 2016-09-03T23:31:37.337Z] Session [8eecbb40-722e-11e6-9041-c7422ee2d8e1] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"windows-7-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"},"phantomjs.page.settings.userAgent":"Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0"}
[INFO  - 2016-09-03T23:31:37.337Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: 8eecbb40-722e-11e6-9041-c7422ee2d8e1
[ERROR - 2016-09-03T23:31:51.671Z] Session [8eecbb40-722e-11e6-9041-c7422ee2d8e1] - page.onError - msg: SyntaxError: JSON Parse error: Unrecognized token '?'

  phantomjs://platform/console++.js:263 in error
[ERROR - 2016-09-03T23:31:51.671Z] Session [8eecbb40-722e-11e6-9041-c7422ee2d8e1] - page.onError - stack:
  parse (undefined:3)
  methodCaller_emscripten$$val_$std$$wstring$ (:3)
  __emval_call_method (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:1)
  W (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:6)
  B (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:6)
  E (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:6)
  ba (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:7)
  Di (https://br.ebookjapan.jp/br/reader/common/js/br_ems-7h.js:13)
  dynCall_iii_11 (:2)
  decrypt (:6)
  __decrypt (:1)
  decryption (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_engine-7h-min.js:3)
  (anonymous function) (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_engine-7h-min.js:3)
  l (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_util-7h-min.js:2)
  fireWith (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_util-7h-min.js:2)
  d (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_util-7h-min.js:3)
  d (https://br.ebookjapan.jp/br/reader/viewer/js/viewer_util-7h-min.js:3)

  phantomjs://platform/console++.js:263 in error

Support for Jump Plus

Hey, this is a really nice plugin. I used it to extract around 10 volumes through BookLive, but I now need it for a different site. So I'm wondering if its possible to use this for the Jump Plus service.

Links:

If I extract it normally, it looks like this. Fully puzzle-like image.

The good thing is, Jump Plus is a free magazine so it doesn't require user details. Perhaps you could test it easily as well because of that.

Thanks!

EBJ updated.

Just a heads up, if you're still working on this. Even the canvas dump scripts don't seem to work, it seems.

Image quality

When comparing the results of mindl with a canvas dump script, mindl seems to have more artifacts. I chose a random series to compare. It's only really noticeable on color images.

It's not too significant, but here's the zip of mindl's png 100 rip (001) with the canvas dump (000) if you want to look into it. It's noticeable on flat colors.

https://jii.moe/VJRExXkVz.zip
https://jii.moe/4JJI_zyEG.zip

invalid syntax with Python 3.4.4

I am using with Python 3.4.4, mindl 0.3, pillow 3.4.2, and PhantomJS 1.9.8.
I've got these console errors and I didn't have any ghostdriver.log.

For eBookJ

python -m mindl -o [email protected] -o password=mypassword "https://br.xxxxxxx.jp/br/reader/viewer/view.html?sessionid=******&keydata=******&shopID=exxxxxxxJapan"
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\lib\site-packages\mindl\__main__.py", line 22, in <module>
    main()
  File "C:\Python34\lib\site-packages\mindl\cli.py", line 117, in main
    pm = PluginManager()
  File "C:\Python34\lib\site-packages\mindl\plugin_manager.py", line 40, in __in
it__
    module = importlib.import_module(modname)
  File "C:\Python34\lib\importlib\__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "C:\Python34\lib\site-packages\mindl\plugins\animatebookstore.py", line 2
6, in <module>
    import mindl.plugins.binb as binbapi
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\__init__.py", line 1, i
n <module>
    from .binb_api import BinBApi, BinBApiError, SERVERTYPE_SBC, SERVERTYPE_STAT
IC, USER_AGENT
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\binb_api.py", line 294
    params = dict(cid=self.cid, k=self.k, **self._kwargs, **kwargs)
                                                        ^
SyntaxError: invalid syntax

For BL!

python -m mindl "https://xxxxxxxxxx.jp
/bviewer/?cid=*******&rurl=*******"
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\lib\site-packages\mindl\__main__.py", line 22, in <module>
    main()
  File "C:\Python34\lib\site-packages\mindl\cli.py", line 117, in main
    pm = PluginManager()
  File "C:\Python34\lib\site-packages\mindl\plugin_manager.py", line 40, in __in
it__
    module = importlib.import_module(modname)
  File "C:\Python34\lib\importlib\__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "C:\Python34\lib\site-packages\mindl\plugins\animatebookstore.py", line 2
6, in <module>
    import mindl.plugins.binb as binbapi
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\__init__.py", line 1, i
n <module>
    from .binb_api import BinBApi, BinBApiError, SERVERTYPE_SBC, SERVERTYPE_STAT
IC, USER_AGENT
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\binb_api.py", line 294
    params = dict(cid=self.cid, k=self.k, **self._kwargs, **kwargs)
                                                        ^
SyntaxError: invalid syntax

For BL!

python -m mindl "https://bookxxxx.jp
/product/index/title_id/*******/vol_no/001"
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python34\lib\site-packages\mindl\__main__.py", line 22, in <module>
    main()
  File "C:\Python34\lib\site-packages\mindl\cli.py", line 117, in main
    pm = PluginManager()
  File "C:\Python34\lib\site-packages\mindl\plugin_manager.py", line 40, in __in
it__
    module = importlib.import_module(modname)
  File "C:\Python34\lib\importlib\__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "C:\Python34\lib\site-packages\mindl\plugins\animatebookstore.py", line 2
6, in <module>
    import mindl.plugins.binb as binbapi
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\__init__.py", line 1, i
n <module>
    from .binb_api import BinBApi, BinBApiError, SERVERTYPE_SBC, SERVERTYPE_STAT
IC, USER_AGENT
  File "C:\Python34\lib\site-packages\mindl\plugins\binb\binb_api.py", line 294
    params = dict(cid=self.cid, k=self.k, **self._kwargs, **kwargs)
                                                        ^
SyntaxError: invalid syntax

Speed tracker is messed up at times.

The speed calculation isn't very good and needs a revamp. More of a minprogress issue than anything, but also needs fine-tuning here.

EDIT: For whatever reason, it's really messed up on Linux.

The directory name is invalid.

Hey, so I'm trying to rip a novel off BookLive, because of its name, it's having an issue.

The title is "僕のヒーローアカデミア雄英白書 I 1-A：授業参観" as you can see there's a special symbol ： and because of that I'm getting errors such as this:

(14:19:47 ERRO) Worker #6 panicked: mkdir downloads\僕のヒーローアカデミア雄英白書 I 1-A:授業参観第01巻: The directory name is invalid.

If it's possible, please solve this issue, I don't have the required knowledge for Go, I'd look into it otherwise. Thanks.

File naming using templates

Allowing for file and directory naming using templates should be possible. text/template provides a powerful templating system which could be applied to mindl.