Cleanup of forward_local fails when requests fails

[daniel-falk]

Describe the bug
The cleanup for the forward_local context manager fails if using requests within the context manager and an exception is raised from requests. Even if I catch the exception within the context manager, and the context manager returns normally, a ThreadException is raised in the context managers cleanup. This then leaves the forwarded port in use such that it is not possible to use it again.

To Reproduce
Use this code to create a tunnel to a server with a self signed certificate. The requests request will fail to validate the certificate and raise an exception. This exception is caught, the execution continues and then a new exception is raised in the context manager cleanup. The port is never released.

import fabric
import requests
from time import sleep

sshargs = {
    "host": "192.168.0.195",
    "user": "root",
    "connect_kwargs": {
        "password": "root"
    }
}
local_port = 32345

with fabric.Connection(**sshargs).forward_local(local_port, remote_port=443):
   sleep(1) # Wait for the tunnel to start working
   try:
       auth = requests.auth.HTTPDigestAuth(sshargs["user"], sshargs["connect_kwargs"]["password"])
       response = requests.get(
         f"https://127.0.0.1:{local_port}", auth=auth, verify=True
       )
       print("RESPONSE: ", response)
       print(response.text)
   except requests.exceptions.SSLError as e:
       print("GOT ERROR: ", e)
   print("Done")

The output of the command is:

GOT ERROR:  HTTPSConnectionPool(host='127.0.0.1', port=32345): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
Done
Traceback (most recent call last):

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File ~/.config/spyder-py3/temp.py:19
    with fabric.Connection(**sshargs).forward_local(local_port, remote_port=443):

  File ~/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py:142 in __exit__
    next(self.gen)

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/connection.py:1003 in forward_local
    raise wrapper.value

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/invoke/util.py:209 in run
    self._run()  # type: ignore

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py:102 in _run
    raise ThreadException(exceptions)

ThreadException: 
Saw 1 exceptions within threads (ConnectionResetError):


Thread args: {}

Traceback (most recent call last):

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/invoke/util.py", line 209, in run
    self._run()  # type: ignore

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py", line 130, in _run
    empty_sock = self.read_and_write(

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py", line 151, in read_and_write
    data = reader.recv(chunk_size)

ConnectionResetError: [Errno 104] Connection reset by peer

Note how "Done" is printed before the exception, so the exception is in the context manager code.

The next time I try to use the port in the same way, I get OSError: [Errno 98] Address already in use.

This does not seem to happen every time, but at least half of the times.

Expected behaviour

• There should not be an exception raised from the cleanup
• Regardless of any errors, the port should be released

Environment
I don't think the issue is specific to a narrow environment as we have seen this on multiple Linux environments, but here's what I used to replicate it now:

• Linux danielfa-ThinkPad-T590 6.2.0-36-generic #37~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 9 15:34:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• Python 3.10.13

Python libraries:

• fabric==3.2.2
• invoke==2.2.0
• requests==2.31.0
• requests-oauthlib==1.3.1
• requests-toolbelt==1.0.0

[daniel-falk]

To clarify, I think it is related to the fact that requests uses a thread pool to execute the requests, which is not obvious at a first glance.