Improve how combining DEFAULT_REQUEST_HEADERS with the Referer middleware is handled

[Gallaecio]

Currently, setting the Referer header through the DEFAULT_REQUEST_HEADERS setting has no effect, because the Referer spider middleware effectively prevents the default request header downloader middleware from redefining the header (since setdefault is used by both, and the referer spider middleware runs first).

I kind of expected the opposite to happen, for DEFAULT_REQUEST_HEADERS to take priority.

I think we should either:

• Let DEFAULT_REQUEST_HEADERS take priority.
• Log a warning if Referer is defined in DEFAULT_REQUEST_HEADERS without also disabling the referer spider middleware.

[kmike]

What's the use case of setting Referrer through DEFAULT_REQUEST_HEADERS?

[Gallaecio]

I think on some websites setting a Referer header with a static value like a Google URL can improve your success rate, but it is a good question, usually Referer makes more sense on a per-request basis.