How to reduce the file size of the Scrapy log?

[xunqirui]

I runed my spider by using scrapydweb.
After a few days latter, I found my spider's logfile was so big, and it took me a lot of time to open it.
Can you give me some advice?
Thank you : )

[my8100]

1. Visit the corresponding Stats page in scrapydweb, then switch to the Log categorization tab,
   and find out the amount of different level of logs.
2. You may also need to set up LOG_LEVEL of your spider to control the logging level, see docs.

BTW, there's no need to open the original log of the spider as scrapydweb provides the Stats page.

[Digenis]

Also see logrotation through the built-in logging module:
scrapy/scrapy#3235

logrotate(8) is also an option scrapy/scrapy#3628
but make sure you test it thoroughly with scrapy.
It gave me issues with corruption of the first bytes of every new scrapyd log file
(which is written by twisted.python.log, not the built-in logging module).

[xunqirui]

@my8100
I changed my LOG_LEVEL, and the log has been decreased a lot.
Then I will try to use LogParser, I find it is very useful.
Thank you!!

[xunqirui]

@Digenis
when I found the log file was so big, I also wanted to use logrotation, but I didn't find a better way.
your suggestions also help me a lot.
thank you!

[my8100]

@xunqirui
But logrotation would break LogParser.

[xunqirui]

@my8100
The LogParser is very useful.
If the logrotation would break LogParser, I will not use the logrotation.
I have already changed the LOG_LEVEL, and the log has been decreased a lot, it has met my needs.
thank you for reminding me!!!!

[my8100]

@xunqirui
Don‘t forget to add a space after the punctuation mark.
Add a new line if needed.
: )

[xunqirui]

@my8100
Haha
I haven't chatted with others in English for a long time.
I will re-edit it.
: )