Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set spider name in the stats logs. #6091

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

roykyle8
Copy link

@roykyle8 roykyle8 commented Oct 9, 2023

When the multiple crawlers are run through CrawlerProcess, logstats are not clear as which spider these stats are for. Adding a spider name in the log.

@wRAR
Copy link
Member

wRAR commented Oct 9, 2023

I think fixing #3163 is better than adding the spider name to the message itself (and it's not specific to logstats).

@wRAR
Copy link
Member

wRAR commented Oct 9, 2023

Hmm or is this a different problem? Do you only have problems with the logstats message when you run several spiders?

@roykyle8
Copy link
Author

roykyle8 commented Oct 30, 2023

@wRAR Ideally we should have a spider name in every log otherwise its very difficult to find which spider the log relates to however even if its implemented, I'm not sure how it will be implemented into stats logger as stat separate classe linked to a spider instance and is not a part of spider.

@kmike
Copy link
Member

kmike commented Nov 8, 2023

It seems it'd be nice to include the spider name to all log messages, optionally. Probably, it's a matter of allowing to include spider_name variable in the LOG_FORMAT option (https://docs.scrapy.org/en/latest/topics/settings.html#log-format), when the spider name is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants