Comments (2)
I have installed an empty enviroment with conda and requests:
python 3.12.2
requests 2.31.0
urllib3 1.26.18
And we have no errors, when I install urllib3 2.1.0, the program crashes, Below is a simple test program that crashes.
python 3.12.2
requests 2.31.0
urllib3 2.1.0
import requests
from io import BytesIO
class Test():
def __init__(self):
self.proxy = None
self.headers = {}
self.headers['Connection'] = 'close'
self.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36'
self.headers['Accept'] = '*/*'
self.headers['Accept-Encoding'] = 'gzip, deflate'
self.session = requests.Session()
def get(self, url):
self.session.headers = self.headers
r = self.session.get(url, proxies=self.proxy, allow_redirects=True, stream=True)
raw_content = r.raw.read()
content_length = len(raw_content)
r.raw._fp = BytesIO(raw_content)
content=r.text
with open('test.html', 'w', encoding='utf-8') as fo:
fo.write(content)
lurl = 'https://www.nolon.es/es/es/comprar?MainUseId=&Invest=False&ClassId=&HighlightTypeId=&PropertyTypeId=&DistrictId=30&CountyId=&AreaRange.MinValue=&AreaRange.MaxValue=&PriceRange.MinValue=0&PriceRange.MaxValue=&CampaignId=&PropertyId=&ResultsPerPage=27&SortOrderBy=DescendingDate&SortOrderBy=DescendingDate'
Test().get(lurl)`
This is the error I get:
traceback (most recent call last):
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 712, in _error_catcher
yield
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 833, in _raw_read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(42084 bytes read, -21042 more expected)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 934, in stream
data = self.read(amt=amt, decode_content=decode_content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 877, in read
data = self._raw_read(amt)
^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 811, in _raw_read
with self._error_catcher():
File "/opt/miniconda3/envs/test/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/urllib3/response.py", line 729, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(42084 bytes read, -21042 more expected)', IncompleteRead(42084 bytes read, -21042 more expected))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/datos1/test/test.py", line 28, in
Test().get(lurl)
File "/home/datos1/test/test.py", line 23, in get
content=r.text
^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 923, in text
if not self.content:
^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/envs/test/lib/python3.12/site-packages/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(42084 bytes read, -21042 more expected)', IncompleteRead(42084 bytes read, -21042 more expected))
from requests.
You must never call r.text
or r.content
after consuming the underlying response data. That's you're actual bug. I suspect requests and urllib3 were silently returning nothing but you're code has always been the issue
from requests.
Related Issues (20)
- permission denied regression reading extracted certs with multiple users HOT 1
- 一个while True requests运行的爬虫程序,会报错:HTTPSConnectionPool(host='***', port=443): Max retries exceeded with url: /aa (Caused by SSLError(OSError(24, 'Too many open files'))) HOT 1
- Same code works with Python 3.10 but not with 3.11 HOT 2
- With setuptools 72.0.0 requests can no longer be installed from git HOT 5
- Documentation of what to override when using custom adapter HOT 1
- What should be Overwritten to ignore all verification in a custom adapter? HOT 2
- Request for py3.7 support HOT 1
- Support for py3.7 HOT 1
- Doc: missing example HOT 2
- QUIC over Socks5/UDP capable proxy is not working. HOT 3
- Multiple path separators causes bad requests
- Any ipv6 request spuriously stays stuck for a long time, sometimes never progressing HOT 1
- Time between request sent and first byte received (time to first byte) HOT 1
- Import time regression
- Add Warning for Incorrect Keyword Arguments in POST Requests (e.g., 'header' instead of 'headers') HOT 1
- Cannot close the proxy
- Requests changes the URL on GET HOT 5
- Environment variables take precedence over the explicitly passed proxies argument in session.get() HOT 1
- Python requests: No module named 'werkzeug.wrappers.json' HOT 1
- AttributeError: 'Request' object has no attribute 'authorize' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requests.