Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failures with urllib3 2.0.4 #784

Open
3 tasks done
mweinelt opened this issue Sep 20, 2023 · 0 comments
Open
3 tasks done

Test failures with urllib3 2.0.4 #784

mweinelt opened this issue Sep 20, 2023 · 0 comments
Labels
bug help wanted We can't figure this out, if you can, then please help!

Comments

@mweinelt
Copy link

Problem description

We are building smart-open 0.4.0 for nixpkgs, thereby running the test suite. After updating urllib3 from 1.26.16 to 2.0.4 we started seeing failures in the smart-open testsuite.

=========================== short test summary info ============================
FAILED smart_open/tests/test_http.py::HttpTest::test_https_seek_forward - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRe...
FAILED smart_open/tests/test_http.py::HttpTest::test_seek_from_current - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRe...
FAILED smart_open/tests/test_http.py::HttpTest::test_seek_from_end - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRe...
FAILED smart_open/tests/test_http.py::HttpTest::test_seek_from_start - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRe...
======= 4 failed, 368 passed, 4 skipped, 27 warnings in 70.45s (0:01:10) =======
Test protocol 
smart-open> _______________________ HttpTest.test_https_seek_forward _______________________
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open> >               yield
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:710: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = , amt = 131072
smart-open> 
smart-open>     def _raw_read(
smart-open>         self,
smart-open>         amt: int | None = None,
smart-open>     ) -> bytes:
smart-open>         """
smart-open>         Reads `amt` of bytes from the socket.
smart-open>         """
smart-open>         if self._fp is None:
smart-open>             return None  # type: ignore[return-value]
smart-open>     
smart-open>         fp_closed = getattr(self._fp, "closed", False)
smart-open>     
smart-open>         with self._error_catcher():
smart-open>             data = self._fp_read(amt) if not fp_closed else b""
smart-open>             if amt is not None and amt != 0 and not data:
smart-open>                 # Platform-specific: Buggy versions of Python.
smart-open>                 # Close the connection when no data is returned
smart-open>                 #
smart-open>                 # This is redundant to what httplib/http.client _should_
smart-open>                 # already do.  However, versions of python released before
smart-open>                 # December 15, 2012 (http://bugs.python.org/issue16298) do
smart-open>                 # not properly close the connection in all cases. There is
smart-open>                 # no harm in redundantly calling close.
smart-open>                 self._fp.close()
smart-open>                 if (
smart-open>                     self.enforce_content_length
smart-open>                     and self.length_remaining is not None
smart-open>                     and self.length_remaining != 0
smart-open>                 ):
smart-open>                     # This is an edge case that httplib failed to cover due
smart-open>                     # to concerns of backward compatibility. We're
smart-open>                     # addressing it here to make sure IncompleteRead is
smart-open>                     # raised during streaming, so all calls with incorrect
smart-open>                     # Content-Length are caught.
smart-open> >                   raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
smart-open> E                   urllib3.exceptions.IncompleteRead: IncompleteRead(58 bytes read, 10 more expected)
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:835: IncompleteRead
smart-open> 
smart-open> The above exception was the direct cause of the following exception:
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open> >               yield from self.raw.stream(chunk_size, decode_content=True)
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:816: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:940: in stream
smart-open>     data = self.read(amt=amt, decode_content=decode_content)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:911: in read
smart-open>     data = self._raw_read(amt)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:813: in _raw_read
smart-open>     with self._error_catcher():
smart-open> /nix/store/15djm2sxy19yyvicydzwrzfchq7nkmh1-python3-3.11.5/lib/python3.11/contextlib.py:155: in __exit__
smart-open>     self.gen.throw(typ, value, traceback)
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open>                 yield
smart-open>     
smart-open>             except SocketTimeout as e:
smart-open>                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
smart-open>                 # there is yet no clean way to get at it from this context.
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except BaseSSLError as e:
smart-open>                 # FIXME: Is there a better way to differentiate between SSLErrors?
smart-open>                 if "read operation timed out" not in str(e):
smart-open>                     # SSL errors related to framing/MAC get wrapped and reraised here
smart-open>                     raise SSLError(e) from e
smart-open>     
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except (HTTPException, OSError) as e:
smart-open>                 # This includes IncompleteRead.
smart-open> >               raise ProtocolError(f"Connection broken: {e!r}", e) from e
smart-open> E               urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:727: ProtocolError
smart-open> 
smart-open> During handling of the above exception, another exception occurred:
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @responses.activate
smart-open>     def test_https_seek_forward(self):
smart-open>         """Did the seek forward over HTTPS work?"""
smart-open>         responses.add_callback(responses.GET, HTTPS_URL, callback=request_callback)
smart-open>     
smart-open>         with smart_open.open(HTTPS_URL, "rb") as fin:
smart-open>             fin.seek(10)
smart-open> >           read_bytes = fin.read(size=10)
smart-open> 
smart-open> smart_open/tests/test_http.py:142: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> smart_open/http.py:183: in read
smart-open>     bytes_read = self._read_buffer.fill(self._read_iter)
smart-open> smart_open/bytebuffer.py:155: in fill
smart-open>     for more_bytes in source:
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open>                 yield from self.raw.stream(chunk_size, decode_content=True)
smart-open>             except ProtocolError as e:
smart-open> >               raise ChunkedEncodingError(e)
smart-open> E               requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:818: ChunkedEncodingError
smart-open> _______________________ HttpTest.test_seek_from_current ________________________
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open> >               yield
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:710: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = , amt = 131072
smart-open> 
smart-open>     def _raw_read(
smart-open>         self,
smart-open>         amt: int | None = None,
smart-open>     ) -> bytes:
smart-open>         """
smart-open>         Reads `amt` of bytes from the socket.
smart-open>         """
smart-open>         if self._fp is None:
smart-open>             return None  # type: ignore[return-value]
smart-open>     
smart-open>         fp_closed = getattr(self._fp, "closed", False)
smart-open>     
smart-open>         with self._error_catcher():
smart-open>             data = self._fp_read(amt) if not fp_closed else b""
smart-open>             if amt is not None and amt != 0 and not data:
smart-open>                 # Platform-specific: Buggy versions of Python.
smart-open>                 # Close the connection when no data is returned
smart-open>                 #
smart-open>                 # This is redundant to what httplib/http.client _should_
smart-open>                 # already do.  However, versions of python released before
smart-open>                 # December 15, 2012 (http://bugs.python.org/issue16298) do
smart-open>                 # not properly close the connection in all cases. There is
smart-open>                 # no harm in redundantly calling close.
smart-open>                 self._fp.close()
smart-open>                 if (
smart-open>                     self.enforce_content_length
smart-open>                     and self.length_remaining is not None
smart-open>                     and self.length_remaining != 0
smart-open>                 ):
smart-open>                     # This is an edge case that httplib failed to cover due
smart-open>                     # to concerns of backward compatibility. We're
smart-open>                     # addressing it here to make sure IncompleteRead is
smart-open>                     # raised during streaming, so all calls with incorrect
smart-open>                     # Content-Length are caught.
smart-open> >                   raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
smart-open> E                   urllib3.exceptions.IncompleteRead: IncompleteRead(58 bytes read, 10 more expected)
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:835: IncompleteRead
smart-open> 
smart-open> The above exception was the direct cause of the following exception:
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open> >               yield from self.raw.stream(chunk_size, decode_content=True)
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:816: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:940: in stream
smart-open>     data = self.read(amt=amt, decode_content=decode_content)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:911: in read
smart-open>     data = self._raw_read(amt)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:813: in _raw_read
smart-open>     with self._error_catcher():
smart-open> /nix/store/15djm2sxy19yyvicydzwrzfchq7nkmh1-python3-3.11.5/lib/python3.11/contextlib.py:155: in __exit__
smart-open>     self.gen.throw(typ, value, traceback)
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open>                 yield
smart-open>     
smart-open>             except SocketTimeout as e:
smart-open>                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
smart-open>                 # there is yet no clean way to get at it from this context.
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except BaseSSLError as e:
smart-open>                 # FIXME: Is there a better way to differentiate between SSLErrors?
smart-open>                 if "read operation timed out" not in str(e):
smart-open>                     # SSL errors related to framing/MAC get wrapped and reraised here
smart-open>                     raise SSLError(e) from e
smart-open>     
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except (HTTPException, OSError) as e:
smart-open>                 # This includes IncompleteRead.
smart-open> >               raise ProtocolError(f"Connection broken: {e!r}", e) from e
smart-open> E               urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:727: ProtocolError
smart-open> 
smart-open> During handling of the above exception, another exception occurred:
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @responses.activate
smart-open>     def test_seek_from_current(self):
smart-open>         responses.add_callback(responses.GET, URL, callback=request_callback)
smart-open>         reader = smart_open.http.SeekableBufferedInputBase(URL)
smart-open>     
smart-open>         reader.seek(10)
smart-open> >       read_bytes = reader.read(size=10)
smart-open> 
smart-open> smart_open/tests/test_http.py:79: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> smart_open/http.py:183: in read
smart-open>     bytes_read = self._read_buffer.fill(self._read_iter)
smart-open> smart_open/bytebuffer.py:155: in fill
smart-open>     for more_bytes in source:
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open>                 yield from self.raw.stream(chunk_size, decode_content=True)
smart-open>             except ProtocolError as e:
smart-open> >               raise ChunkedEncodingError(e)
smart-open> E               requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:818: ChunkedEncodingError
smart-open> _________________________ HttpTest.test_seek_from_end __________________________
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open> >               yield
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:710: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = , amt = 131072
smart-open> 
smart-open>     def _raw_read(
smart-open>         self,
smart-open>         amt: int | None = None,
smart-open>     ) -> bytes:
smart-open>         """
smart-open>         Reads `amt` of bytes from the socket.
smart-open>         """
smart-open>         if self._fp is None:
smart-open>             return None  # type: ignore[return-value]
smart-open>     
smart-open>         fp_closed = getattr(self._fp, "closed", False)
smart-open>     
smart-open>         with self._error_catcher():
smart-open>             data = self._fp_read(amt) if not fp_closed else b""
smart-open>             if amt is not None and amt != 0 and not data:
smart-open>                 # Platform-specific: Buggy versions of Python.
smart-open>                 # Close the connection when no data is returned
smart-open>                 #
smart-open>                 # This is redundant to what httplib/http.client _should_
smart-open>                 # already do.  However, versions of python released before
smart-open>                 # December 15, 2012 (http://bugs.python.org/issue16298) do
smart-open>                 # not properly close the connection in all cases. There is
smart-open>                 # no harm in redundantly calling close.
smart-open>                 self._fp.close()
smart-open>                 if (
smart-open>                     self.enforce_content_length
smart-open>                     and self.length_remaining is not None
smart-open>                     and self.length_remaining != 0
smart-open>                 ):
smart-open>                     # This is an edge case that httplib failed to cover due
smart-open>                     # to concerns of backward compatibility. We're
smart-open>                     # addressing it here to make sure IncompleteRead is
smart-open>                     # raised during streaming, so all calls with incorrect
smart-open>                     # Content-Length are caught.
smart-open> >                   raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
smart-open> E                   urllib3.exceptions.IncompleteRead: IncompleteRead(10 bytes read, 58 more expected)
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:835: IncompleteRead
smart-open> 
smart-open> The above exception was the direct cause of the following exception:
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open> >               yield from self.raw.stream(chunk_size, decode_content=True)
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:816: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:940: in stream
smart-open>     data = self.read(amt=amt, decode_content=decode_content)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:911: in read
smart-open>     data = self._raw_read(amt)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:813: in _raw_read
smart-open>     with self._error_catcher():
smart-open> /nix/store/15djm2sxy19yyvicydzwrzfchq7nkmh1-python3-3.11.5/lib/python3.11/contextlib.py:155: in __exit__
smart-open>     self.gen.throw(typ, value, traceback)
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open>                 yield
smart-open>     
smart-open>             except SocketTimeout as e:
smart-open>                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
smart-open>                 # there is yet no clean way to get at it from this context.
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except BaseSSLError as e:
smart-open>                 # FIXME: Is there a better way to differentiate between SSLErrors?
smart-open>                 if "read operation timed out" not in str(e):
smart-open>                     # SSL errors related to framing/MAC get wrapped and reraised here
smart-open>                     raise SSLError(e) from e
smart-open>     
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except (HTTPException, OSError) as e:
smart-open>                 # This includes IncompleteRead.
smart-open> >               raise ProtocolError(f"Connection broken: {e!r}", e) from e
smart-open> E               urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(10 bytes read, 58 more expected)', IncompleteRead(10 bytes read, 58 more expected))
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:727: ProtocolError
smart-open> 
smart-open> During handling of the above exception, another exception occurred:
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @responses.activate
smart-open>     def test_seek_from_end(self):
smart-open>         responses.add_callback(responses.GET, URL, callback=request_callback)
smart-open>         reader = smart_open.http.SeekableBufferedInputBase(URL)
smart-open>     
smart-open>         reader.seek(-10, whence=smart_open.constants.WHENCE_END)
smart-open>         self.assertEqual(reader.tell(), len(BYTES) - 10)
smart-open> >       read_bytes = reader.read(size=10)
smart-open> 
smart-open> smart_open/tests/test_http.py:96: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> smart_open/http.py:183: in read
smart-open>     bytes_read = self._read_buffer.fill(self._read_iter)
smart-open> smart_open/bytebuffer.py:155: in fill
smart-open>     for more_bytes in source:
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open>                 yield from self.raw.stream(chunk_size, decode_content=True)
smart-open>             except ProtocolError as e:
smart-open> >               raise ChunkedEncodingError(e)
smart-open> E               requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(10 bytes read, 58 more expected)', IncompleteRead(10 bytes read, 58 more expected))
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:818: ChunkedEncodingError
smart-open> ________________________ HttpTest.test_seek_from_start _________________________
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open> >               yield
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:710: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = , amt = 131072
smart-open> 
smart-open>     def _raw_read(
smart-open>         self,
smart-open>         amt: int | None = None,
smart-open>     ) -> bytes:
smart-open>         """
smart-open>         Reads `amt` of bytes from the socket.
smart-open>         """
smart-open>         if self._fp is None:
smart-open>             return None  # type: ignore[return-value]
smart-open>     
smart-open>         fp_closed = getattr(self._fp, "closed", False)
smart-open>     
smart-open>         with self._error_catcher():
smart-open>             data = self._fp_read(amt) if not fp_closed else b""
smart-open>             if amt is not None and amt != 0 and not data:
smart-open>                 # Platform-specific: Buggy versions of Python.
smart-open>                 # Close the connection when no data is returned
smart-open>                 #
smart-open>                 # This is redundant to what httplib/http.client _should_
smart-open>                 # already do.  However, versions of python released before
smart-open>                 # December 15, 2012 (http://bugs.python.org/issue16298) do
smart-open>                 # not properly close the connection in all cases. There is
smart-open>                 # no harm in redundantly calling close.
smart-open>                 self._fp.close()
smart-open>                 if (
smart-open>                     self.enforce_content_length
smart-open>                     and self.length_remaining is not None
smart-open>                     and self.length_remaining != 0
smart-open>                 ):
smart-open>                     # This is an edge case that httplib failed to cover due
smart-open>                     # to concerns of backward compatibility. We're
smart-open>                     # addressing it here to make sure IncompleteRead is
smart-open>                     # raised during streaming, so all calls with incorrect
smart-open>                     # Content-Length are caught.
smart-open> >                   raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
smart-open> E                   urllib3.exceptions.IncompleteRead: IncompleteRead(58 bytes read, 10 more expected)
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:835: IncompleteRead
smart-open> 
smart-open> The above exception was the direct cause of the following exception:
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open> >               yield from self.raw.stream(chunk_size, decode_content=True)
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:816: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:940: in stream
smart-open>     data = self.read(amt=amt, decode_content=decode_content)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:911: in read
smart-open>     data = self._raw_read(amt)
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:813: in _raw_read
smart-open>     with self._error_catcher():
smart-open> /nix/store/15djm2sxy19yyvicydzwrzfchq7nkmh1-python3-3.11.5/lib/python3.11/contextlib.py:155: in __exit__
smart-open>     self.gen.throw(typ, value, traceback)
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @contextmanager
smart-open>     def _error_catcher(self) -> typing.Generator[None, None, None]:
smart-open>         """
smart-open>         Catch low-level python exceptions, instead re-raising urllib3
smart-open>         variants, so that low-level exceptions are not leaked in the
smart-open>         high-level api.
smart-open>     
smart-open>         On exit, release the connection back to the pool.
smart-open>         """
smart-open>         clean_exit = False
smart-open>     
smart-open>         try:
smart-open>             try:
smart-open>                 yield
smart-open>     
smart-open>             except SocketTimeout as e:
smart-open>                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
smart-open>                 # there is yet no clean way to get at it from this context.
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except BaseSSLError as e:
smart-open>                 # FIXME: Is there a better way to differentiate between SSLErrors?
smart-open>                 if "read operation timed out" not in str(e):
smart-open>                     # SSL errors related to framing/MAC get wrapped and reraised here
smart-open>                     raise SSLError(e) from e
smart-open>     
smart-open>                 raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
smart-open>     
smart-open>             except (HTTPException, OSError) as e:
smart-open>                 # This includes IncompleteRead.
smart-open> >               raise ProtocolError(f"Connection broken: {e!r}", e) from e
smart-open> E               urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/dxqvdy0rggdg8zqja3m3wf34dx5wqq3q-python3.11-urllib3-2.0.4/lib/python3.11/site-packages/urllib3/response.py:727: ProtocolError
smart-open> 
smart-open> During handling of the above exception, another exception occurred:
smart-open> 
smart-open> self = 
smart-open> 
smart-open>     @responses.activate
smart-open>     def test_seek_from_start(self):
smart-open>         responses.add_callback(responses.GET, URL, callback=request_callback)
smart-open>         reader = smart_open.http.SeekableBufferedInputBase(URL)
smart-open>     
smart-open>         reader.seek(10)
smart-open>         self.assertEqual(reader.tell(), 10)
smart-open> >       read_bytes = reader.read(size=10)
smart-open> 
smart-open> smart_open/tests/test_http.py:61: 
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> smart_open/http.py:183: in read
smart-open>     bytes_read = self._read_buffer.fill(self._read_iter)
smart-open> smart_open/bytebuffer.py:155: in fill
smart-open>     for more_bytes in source:
smart-open> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
smart-open> 
smart-open>     def generate():
smart-open>         # Special case for urllib3.
smart-open>         if hasattr(self.raw, "stream"):
smart-open>             try:
smart-open>                 yield from self.raw.stream(chunk_size, decode_content=True)
smart-open>             except ProtocolError as e:
smart-open> >               raise ChunkedEncodingError(e)
smart-open> E               requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(58 bytes read, 10 more expected)', IncompleteRead(58 bytes read, 10 more expected))
smart-open> 
smart-open> /nix/store/0ywyi98av0590s5idss3c13zas1cd1a0-python3.11-requests-2.31.0/lib/python3.11/site-packages/requests/models.py:818: ChunkedEncodingError

Steps/code to reproduce the problem

Install requests, with urllib3>2 and run the test suite.

Versions

  • Linux-6.1.46-x86_64-with-glibc2.37
  • Python 3.11.5 (main, Aug 24 2023, 12:23:19) [GCC 12.3.0]
  • smart_open 6.4.0

Checklist

Before you create the issue, please make sure you have:

  • Described the problem clearly
  • Provided a minimal reproducible example, including any required data
  • Provided the version numbers of the relevant software
@mpenkov mpenkov added bug help wanted We can't figure this out, if you can, then please help! labels Sep 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug help wanted We can't figure this out, if you can, then please help!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mweinelt @mpenkov