Problem with download from Storytel about audiobook-dl HOT 41 OPEN

The site api has been updated. The Storytel support has to be reimplemented.

from audiobook-dl.

I belive I tracked down the problem. It's indeed TLS fingerprinting related. The urllib3 default SSL context will set the SSL_OP_TLSEXT_PADDING option which will result in a padding extension in the ClientHello in many OpenSSL configurations. When unsetting the option (no padding extension) it seems to work fine.

I'll update the code tomorrow.

from audiobook-dl.

I've updated the storytell_cf branch. Please test if it's working for you. If you're also using other sources, please test them too (disabled the padding extension for all HTTPS requests).

from audiobook-dl.

I can confirm that.

from audiobook-dl.

try the old version

from audiobook-dl.

I’ve try to update after have the error the first time

from audiobook-dl.

I can confirm error

from audiobook-dl.

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

from audiobook-dl.

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

I did the last rewrite but currently don't have any active storytel subscription. If someone can share his login credentials I can have a look on what needs to be adjusted.

from audiobook-dl.

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

I did the last rewrite but currently don't have any active storytel subscription. If someone can share his login credentials I can have a look on what needs to be adjusted.

Hi, i cant provide credential, how can i send it to you?

from audiobook-dl.

You should be able to see my email in my profile now.

from audiobook-dl.

Mail send

from audiobook-dl.

Got them, thank you.

API didn't change but it seems like they added cloudflare as proxy and it's detecting and blocking (CAPTCHA page) requests made by python/requests. Detection seems to be more sophisticated (suspecting TLS fingerprinting, header order checking or something simmilar). IP Reputation seems to be relevant too (blocked from my server, working fine when proxing it via my Desktop). Not sure how well the current cloudflare solver proxies are working, I'll try to fix it without having to use one.

from audiobook-dl.

Got them, thank you.

API didn't change but it seems like they added cloudflare as proxy and it's detecting and blocking (CAPTCHA page) requests made by python/requests. Detection seems to be more sophisticated (suspecting TLS fingerprinting, header order checking or something simmilar). IP Reputation seems to be relevant too (blocked from my server, working fine when proxing it via my Desktop). Not sure how well the current cloudflare solver proxies are working, I'll try to fix it without having to use one.

Thanks a lot, we wait for your update and if needed make some tests

from audiobook-dl.

You can try the following change to make sure you've the same problem as me (blocked by cloudflare).
https://github.com/jo1gi/audiobook-dl/pull/108/files
It probably won't fix the blocking, it's just giving a better error message.

from audiobook-dl.

You can try the following change to make sure you've the same problem as me (blocked by cloudflare). https://github.com/jo1gi/audiobook-dl/pull/108/files It probably won't fix the blocking, it's just giving a better error message.

Hi I tried your code and yes i got nicer error message :-) I will se if I can help

from audiobook-dl.

No problem actually for me

from audiobook-dl.

I will test it later to night

from audiobook-dl.

I got this error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='www.storytel.com', port=443): Max retries exceeded with url: /api/login.action?m=1&token=guestsv&userid=-1&version=24.22&terminal=android&locale=sv&deviceId=995f2562-0e44-4410-b1b9-8d08261f33c4&kidsMode=false (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

from audiobook-dl.

not use https.

from audiobook-dl.

This version is working perfectly for me, thanks!

from audiobook-dl.

@nianhbg the change should have no impact on the certificate validation. If you just use the master version you should get the same error.

from audiobook-dl.

Ok thanks I got it working :-D

from audiobook-dl.

I've updated the storytell_cf branch. Please test if it's working for you. If you're also using other sources, please test them too (disabled the padding extension for all HTTPS requests).

im sorry if im stupid, but how do i install that branch?

from audiobook-dl.

Clone the repo and use git checkout storytell_cf to switch to the branch.

from audiobook-dl.

i have gotten that far, but python3 setup.py install on root does nothing

from audiobook-dl.

i have gotten that far, but python3 setup.py install on root does nothing

Remove the previous installation with pip uninstall audiobook-dl

Then

pip install "git+https://github.com/jo1gi/audiobook-dl.git@storytell_cf"

from audiobook-dl.

Few bugs I needed to fix (Windows):
1.

pip uninstall audiobook-dl
pip install git+https://github.com/jo1gi/audiobook-dl.git@storytell_cf

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/output/output.py#L113 not working on windows, changed to 256.

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/sources/source/__init__.py#L180

ssl_context.check_hostname = False

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/sources/source/__init__.py#L188

session.verify = False

from audiobook-dl.

@thewh1teagle thank you for the feedback.

What python and windows version are you running?

ssl_context.check_hostname = False and session.verify = False should not be necessary and disables HTTPS security.
Any chance that you have some fancy security software installed which intercepts web traffic?

from audiobook-dl.

I get the same SSL errors as #106 (comment) and the ones I guess #106 (comment) tries to avoid with turning off verifications.

requests changed how certificates are loaded in 2.32.3. Not clear if it is a bug or not.
psf/requests#6730
If using requests 2.32.2 everything works.

With 2.32.3 it worked for me (Mac OS Sonoma, pyenv, 3.11.9) after adding either a
ssl_context.load_default_certs()
or

import certifi
ssl_context.load_verify_locations(certifi.where())

or something similar in the create_session function

from audiobook-dl.

Thank you @currently-off-my-rocker
You're absolutly right.
I looked into this, and requests introduced a regression with the 2.32.0 release which breaks the usage of custom SSLContext's (Which we need for the cloudflare). It's fixed in 2.32.3, but breaks certifiate loading. I added the ssl_context.load_default_certs() workaround in #113.

Also fixed Windows support via #114
Please test the current master branch or the https://github.com/jo1gi/audiobook-dl/releases/tag/v0.7.4 pre release.

from audiobook-dl.

i installed it using pip install "git+https://github.com/jo1gi/audiobook-dl.git".
audiobook-dl https://www.storytel.com/nl/nl/books/book
Traceback (most recent call last):
File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python39\Scripts\audiobook-dl.exe_main.py", line 4, in
File "C:\Python39\lib\site-packages\audiobookdl_init.py", line 1, in
from .utils.audiobook import AudiobookFile, Chapter, AudiobookMetadata, Cover, Audiobook, Result, Series, BookId
File "C:\Python39\lib\site-packages\audiobookdl\utils\audiobook.py", line 33, in
class AudiobookFile:
File "C:\Python39\lib\site-packages\audiobookdl\utils\audiobook.py", line 41, in AudiobookFile
headers: MutableMapping[str, str | bytes] = Factory(dict)
TypeError: unsupported operand type(s) for |: 'type' and 'type'

from audiobook-dl.

@zaourzag please try again, made some adjustments for older python versions

from audiobook-dl.

@zaourzag please try again, made some adjustments for older python versions

audiobook-dl https://www.storytel.com/nl/nl/books/book
Finding compatible source
Traceback (most recent call last):
File "C:\Python39\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python39\Scripts\audiobook-dl.exe_main.py", line 7, in
File "C:\Python39\lib\site-packages\audiobookdl_main.py", line 33, in main
process_url(url, options, config)
File "C:\Python39\lib\site-packages\audiobookdl_main.py", line 51, in process_url
source = source_class(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\storytel.py", line 107, in init
super().init(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init_.py", line 42, in init
self.session: requests.Session = self.create_session(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init.py", line 196, in create_session
ssl_context: SSLContext = self.create_ssl_context(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init_.py", line 182, in create_ssl_context
ssl_context: SSLContext = urllib3.util.create_urllib3_context() # type: ignore[attr-defined]
AttributeError: module 'urllib3.util' has no attribute 'create_urllib3_context'.

from audiobook-dl.

perhaps i should just use a newer python version

from audiobook-dl.

Please test the current master branch or the https://github.com/jo1gi/audiobook-dl/releases/tag/v0.7.4 pre release.

tag v0.7.4 works on my setup.

from audiobook-dl.

perhaps i should just use a newer python version

Yes, that would help, can you check which exact python and urlib3 version you're running currently?

from audiobook-dl.

im running python v3.9.13 and urllib3 1.26.19.
edit it was the urllib3 version. i did pip install --upgrade urllib3 and it fixed it

from audiobook-dl.

is it maybe an idea to add an optional flareresolver "proxy"? it essentially bypasses cloudflare blocks.

from audiobook-dl.

is it maybe an idea to add an optional flareresolver "proxy"? it essentially bypasses cloudflare blocks.

If it ever becomes mandatory, that's probably the logical solution. For now it shouldn't be necessary.

from audiobook-dl.

so an update, I have been using my laptop. it now throws a blocked by cloudflare error. so i moved to my desktop. it can download just fine. both devices are on the same network and on python 3.9. so i dont know the trickery cloudflare employs here

from audiobook-dl.