Requesting a site by its IP address instead of hostname raises OpenSSL.SSL.Error: [('SSL routines', '', 'tlsv1 alert internal error')]

[PATAPOsha]

Description

Requesting a site by its IP address instead of hostname raises OpenSSL.SSL.Error: [('SSL routines', '', 'tlsv1 alert internal error')]

2023-07-28 09:58:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://20.83.139.214/> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'tlsv1 alert internal error')]>]
2023-07-28 09:58:19 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://20.83.139.214/> (failed 2 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'tlsv1 alert internal error')]>]
2023-07-28 09:58:19 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://20.83.139.214/> (failed 3 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'tlsv1 alert internal error')]>]

Example spider code:

from scrapy import Spider, Request


class TestSpider(Spider):
    name = "test"

    USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"

    def start_requests(self):
        headers = {
            'Host': 'www.temu.com',
            "Cache-Control": "max-age=0",
            "User-Agent": self.USER_AGENT,
            "Accept-Encoding": "gzip, deflate, br",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
            "Upgrade-Insecure-Requests": "1",
        }
        url = "https://20.83.139.214/"
        yield Request(
            url,
            dont_filter=True,
            headers=headers,
            callback=self.parse,
        )

    def parse(self, response, **kwargs):
        return

Same request made with Python requests library, providing verify=False param works fine.

import requests

headers = {
    'Host': 'www.temu.com',
    'sec-ch-ua': '"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'dnt': '1',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'sec-fetch-site': 'none',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-user': '?1',
    'sec-fetch-dest': 'document',
    'accept-language': 'en-US,en;q=0.9',
}

url = "https://20.83.139.214/"
response = requests.get(
    url,
    # proxies=proxies,
    headers=headers,
    verify=False
)
print(response.status_code)
print(response.text)

In Scrapy docs it is said that Scrapy's default ScrapyClientContextFactory has already disabled certificate verification. So, I assume that the problem is somewhere in hostname verification. I didn't find how to disable hostname verification it Twisted.
One interesting notice: if you put your local MITM proxy with SSL decryption enabled like Charles - the request from Scrapy will succeed.

Steps to Reproduce

1. Run the example code from the description

Expected behavior: successful request

Versions

Scrapy       : 2.9.0
lxml         : 4.9.3.0
libxml2      : 2.9.4
cssselect    : 1.2.0
parsel       : 1.7.0
w3lib        : 2.1.1
Twisted      : 22.10.0
Python       : 3.9.13 (main, Oct 27 2022, 14:57:01) - [Clang 13.1.6 (clang-1316.0.21.2.5)]
pyOpenSSL    : 23.2.0 (OpenSSL 3.1.1 30 May 2023)
cryptography : 41.0.2
Platform     : macOS-12.4-arm64-arm-64bit

[Gallaecio]

Have you considered using HTTP instead of HTTPS, or do you need HTTPS?

[wRAR]

FWIW adding an /etc/hosts entry and using the domain name works, though I haven't made sure it connects to the correct IP.

[PATAPOsha]

Have you considered using HTTP instead of HTTPS, or do you need HTTPS?

The site redirects all http requests to https. That's not what I need.

I ended up with following custom ClientContextFactory. Always overriding hostname from "IP-style to "domain-style":

class TemuClientContextFactory(ScrapyClientContextFactory):
    def creatorForNetloc(self, hostname: bytes, port: int) -> "ClientTLSOptions":  # pylint: disable=W0613
        hostname = b"www.temu.com"
        return ScrapyClientTLSOptions(
            hostname.decode("ascii"),
            self.getContext(),
            verbose_logging=self.tls_verbose_logging,
        )

[wRAR]

So it's something inside ClientTLSOptions and I wonder if we can override that.