Improve unhandled exception handling for Crawler*.crawl()

[wRAR]

As we recently found in #6038, the following well-known snippet from docs and tests:

process = CrawlerProcess()
process.crawl(MySpider)
process.start()

doesn't handle exceptions raised in crawl() correctly, as that method returns a Deferred and we should add an errback to it instead of relying on https://docs.twisted.org/en/latest/core/howto/defer.html#unhandled-errors, either in our code or in examples.

[namansharma18899]

Hey @wRAR I'm on point with the description.

I think I can work on it so I'll really appreciate if you can assign this issue to me.

Also I had just one doubt, this issue only focuses on adding errbacks in the docs right ??

[wRAR]

"either in our code or in examples" so no, ideally it should be done in the code.

[wRAR]

Related (but is about scrapy crawl): #2171

[sa2415]

Could I work on this?

[wRAR]

Sure.

[antarahebbar]

Hi! @yueric985 and I are students are we are working on a project for a class that requires us to make an open-source contribution to a library. We will be working on this issue as an example.