[BUG] WebSocket send/receive intermittently throwing SSL error on SelectorEventLoopPolicy

[dolfies]

Describe the bug
Seemingly randomly, when sending or receiving from a WebSocket, I'll get an error with the reason BoringSSL SSL_read: Call would block, errno 10035.

Both of the following tracebacks are observed:

Traceback (most recent call last):
  File "D:\Workspace\discord.py-self\discord\gateway.py", line 699, in _sendstr
    await self.socket.asend(data.encode('utf-8'))
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\requests\websockets.py", line 104, in asend
    return await self.loop.run_in_executor(None, self.send, payload, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\requests\websockets.py", line 62, in send
    return self.curl.ws_send(payload, flags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\curl.py", line 366, in ws_send
    self._check_error(ret, "WS_SEND")
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\curl.py", line 125, in _check_error
    raise error
curl_cffi.curl.CurlError: Failed to WS_SEND, ErrCode: 1, Reason: 'BoringSSL SSL_read: Call would block, errno 10035'. This may be a libcurl error, See https://curl.se/libcurl/c/libcurl-errors.html first for more details.

and (more commonly):

Traceback (most recent call last):
  File "D:\Workspace\discord.py-self\discord\gateway.py", line 663, in poll_event
    msg, flags = await asyncio.wait_for(self.socket.arecv(), timeout=self._max_heartbeat_timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\asyncio\tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\requests\websockets.py", line 101, in arecv
    return await self.loop.run_in_executor(None, self.recv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\requests\websockets.py", line 47, in recv
    chunk, frame = self.curl.ws_recv()
                   ^^^^^^^^^^^^^^^^^^^
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\curl.py", line 355, in ws_recv
    self._check_error(ret, "WS_RECV")
  File "D:\Workspace\discord.py-self\.venv\Lib\site-packages\curl_cffi\curl.py", line 125, in _check_error
    raise error
curl_cffi.curl.CurlError: Failed to WS_RECV, ErrCode: 56, Reason: 'BoringSSL SSL_read: Call would block, errno 10035'. This may be a libcurl error, See https://curl.se/libcurl/c/libcurl-errors.html first for more details.

To Reproduce

# This is hard to consistently reproduce, but I'm just using the arecv() and asend() coroutines (possibly at the same time, unsure if related):

from curl_cffi.requests import AsyncSession, WebSocket

session = AsyncSession()
ws = await session.ws_connect('wss://example.com')
await ws.asend(b'hello')
while True:
    await ws.arecv()

Expected behavior
According to Microsoft docs, error 10035 is nonfatal and should simply be retried (I think?). I would do this at the application level, but this error is wrapped by curl into its own error so I can't just check the error number.

Versions

• OS: Windows 11 23H2
• curl_cffi version v0.6.0b7

Additional context
I'm using the async session with the selector event loop policy. I cannot reproduce this with the proactor event loop policy, despite the warnings.

[yifeikong]

It seems that we should check the return code when using .ws_send and .ws_recv and retry them when 10035 is raised.

Regardless of that, the asyncio implementation is not elegant at all, I wish there were APIs integrating curl_multi_ and curl_ws_ functions, but unfortunately, the WS API in curl is still experimental and not complete.

See my question here: curl/curl#11867

[dolfies]

It seems that we should check the return code when using .ws_send and .ws_recv and retry them when 10035 is raised.

Would this be in the Python implementation? I was going to PR a fix using regex but I was hoping there was a more low-level place to do it.

Regardless of that, the asyncio implementation is not elegant at all, I wish there were APIs integrating curl_multi_ and curl_ws_ functions, but unfortunately, the WS API in curl is still experimental and not complete.

Yeah it's unfortunate. This is definitely a lot better than nothing though, since Cloudflare does provide a JA3 fingerprint for WebSocket traffic for enterprise customers :(

[yifeikong]

It's here:

curl_cffi/curl_cffi/curl.py

Lines 362 to 367 in aefc247

	def ws_send(self, payload: bytes, flags: CurlWsFlag = CurlWsFlag.BINARY) -> int:
	n_sent = ffi.new("int *")
	buffer = ffi.from_buffer(payload)
	ret = lib.curl_ws_send(self._curl, buffer, len(buffer), n_sent, 0, flags)
	self._check_error(ret, "WS_SEND")
	return n_sent[0]

Does it work for Cloudflare?

[yifeikong]

Some also reported that not only websockets returned the same error:

#106 (comment)

[dolfies]

Some also reported that not only websockets returned the same error:

#106 (comment)

That's error 10053, not 10035 :)

According to MS docs, that's ECONNABORTED: Software caused connection abort. An established connection was aborted by the software in your host computer, possibly due to a data transmission time-out or protocol error. So I don't think they're directly related.

[dolfies]

It's here:

curl_cffi/curl_cffi/curl.py

Lines 362 to 367 in aefc247

	def ws_send(self, payload: bytes, flags: CurlWsFlag = CurlWsFlag.BINARY) -> int:
	n_sent = ffi.new("int *")
	buffer = ffi.from_buffer(payload)
	ret = lib.curl_ws_send(self._curl, buffer, len(buffer), n_sent, 0, flags)
	self._check_error(ret, "WS_SEND")
	return n_sent[0]

Yeah that's where I was looking, but I didn't think it'd be very clean since the error is wrapped and all we have is the string BoringSSL SSL_read: Call would block, errno 10035 in the error buffer, not the error number directly.

Does it work for Cloudflare?

Other than these minor issues, it seems to be working perfectly. Though it probably isn't as performant as aiohttp due to the libcurl API issues you mentioned.

[dolfies]

Unfortunately, I'm now able to reproduce this with the proactor event loop as well :(
Seems a lot less common though

[yifeikong]

That's error 10053, not 10035 :)

You have sharp eyes 😄