issue #6323: add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter

[bloodforcream]

fixes #6323

SpiderLoggerAdapter prioritizes self.extra over logged extra so it works as it worked before.
spider in extra will always be self

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.88%. Comparing base (1d11ea3) to head (bd43233).
Report is 37 commits behind head on master.

❗ Current head bd43233 differs from pull request most recent head 9852d9d. Consider uploading reports for the commit 9852d9d to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6324      +/-   ##
==========================================
+ Coverage   83.92%   88.88%   +4.95%     
==========================================
  Files         161      162       +1     
  Lines       11972    11981       +9     
  Branches     1865     1930      +65     
==========================================
+ Hits        10048    10649     +601     
+ Misses       1588      980     -608     
- Partials      336      352      +16     

Files	Coverage Δ
scrapy/spiders/__init__.py	93.84% <100.00%> (+0.09%)	⬆️
scrapy/utils/spider_logger_adapter.py	100.00% <100.00%> (ø)

... and 26 files with indirect coverage changes

[wRAR]

This should go into scrapy.utils.log

[bloodforcream]

I agree. That's where I tried to put it first but I get a circular import error.

..\scrapy\__init__.py:15: in <module>
    from scrapy.spiders import Spider
..\scrapy\spiders\__init__.py:16: in <module>
    from scrapy.utils.log import SpiderLoggerAdapter
..\scrapy\utils\log.py:13: in <module>
    from scrapy.settings import Settings
..\scrapy\settings\__init__.py:23: in <module>
    from scrapy.settings import default_settings
..\scrapy\settings\default_settings.py:322: in <module>
    USER_AGENT = f'Scrapy/{import_module("scrapy").__version__} (+https://scrapy.org)'
E   AttributeError: partially initialized module 'scrapy' has no attribute '__version__' (most likely due to a circular import)

Maybe i'm wrong and it's solvable

[wRAR]

I still don't like having a separate module for this, I see these options:

1. Move it to scrapy.spiders (probably no technical downsides as it's not useful anywhere else?).
2. Remove the SpiderLoggerAdapter import from the scrapy.spiders top-level, moving it under if TYPE_CHECKING and inside def logger().
3. (likely a worse idea) Change some other imports in the loop.

I would go with 2 in general but probably with 1 in this case.

@Gallaecio second opinion?

[wRAR]

Is it possible to add a test that checks that the Spider logging methods work as expected?

[bloodforcream]

Is it possible to add a test that checks that the Spider logging methods work as expected?

Do you mean a test that's making sure general logging messages(not "extra") of Spider work as expected?
I'll try to code them today/tomorrow

[bloodforcream]

Suggested change

	return SpiderLoggerAdapter(logger, {"spider": self})
	return SpiderLoggerAdapter(logger, {"spider": self.name})

Maybe also change self to self.name? So we get actual spider name in logs instead of <Spider 'spider_name' at 0x1b2fc089580>.

Probably not the best idea since it's gonna change current behavior

[wRAR]

Probably not the best idea since it's gonna change current behavior

Yeah

[bloodforcream]

Added more tests