Make brotlipy a hard Scrapy dependency

[Gallaecio]

When we added support to Scrapy (#2535), we did so without making brotlipy a hard dependency, seemingly because there seem to be some potential for installation issues.

However, given that most web browsers support Brotli (br) compression out of the box, I think we should consider revisiting that. It would also slightly simplify our code.

Related to #4697

[wRAR]

Related: #4267

[wRAR]

How about doing this in 2.7?

[wRAR]

Related: https://www.lapierrebikes.com/page-data/de-de/haendler/page-data.json sends content-encoding: br even with no Accept-Encoding from the client, so Scrapy silently fails to decode it with brotli not installed. I'm not saying it's a good reason to install brotli by default but it's still a use case where it helps.

[Gallaecio]

It seems we can do https://github.com/python-hyper/brotlicffi#using-brotlicffi-in-projects for PyPy support, and for Windows I think https://docs.scrapy.org/en/latest/intro/install.html#windows already covers everything needed, although it would be great to have a Windows user confirm so.

[Gallaecio]

@kmike Any thoughts on this? If we are going ahead, we could mark this as a good first issue.

[kmike]

It seems brotli is popular and maintained (e.g. there are wheels uploaded recently), even if last release was in 2020. That's probably fine to add it as a dependency. On the other hand, it's still a possible source of install errors in various environments, as it's a binary package. So, no strong opinion on that.

[wRAR]

google/brotli#944 suggests there will be a new release soon. But so far we decided not to make it a dep as it's a binary module.