Investigate speeding up MockServer()

[wRAR]

On my machine, with coverage enabled and h2 installed (both of these things add significant additional time), a single call of tests.mockserver.MockServer(), at least in FeedExportTest.run_and_export(), takes up to 6.5s, which is likely the main reason why our tests, and especially feed export tests (which start the mockserver many times per test function via assertExported*() and in the worst case assertExported()) are so slow. The slowest test, tests/test_feedexport.py::FeedExportTest::test_export_feed_export_fields, actually reaches the 120s limit here because it calls MockServer() 2*2*6=24 times.

[wRAR]

Looks like the main problem I experience is because on this machine trying to resolve an invalid domain (which we do on every import of tests) takes several seconds before getting the resolving error. It looks specific to systemd-resolved in my case (edit: this was a misconfiguration, not intended behavior), but can be possible in other configurations too. So as one improvement we should only call

scrapy/tests/__init__.py

Line 31 in 532cc8a

socket.getaddrinfo("non-existing-host", 80)

when running tests or at least not inside the mockserver process.

[wRAR]

The next step is doing something about the coverage-introduced slowness, it takes about 2 seconds to initialize mockserver under coverage run while it's almost instant otherwise, not sure if anything simple can be done here (also not sure how is coverage being enabled for subprocesses as we don't seem to do anything for that).

[wRAR]

Without coverage and h2 the start up time of mockserver here is about 0.4s, under coverage run but without h2 it's about 1.2s, under coverage run with h2 it's about 2.3s, and cProfile clearly shows that importing hpack.huffman_table takes about 1.1s (but it's instant without coverage). It doesn't seem possible to tell Twisted to not try importing HTTP2 modules and passing --source=scrapy doesn't make coverage ignore that file. Another slow coverage-specific thing is importing html.entities (about 0.3s).