Retry on ContentFetchError #437

dgoiko · 2020-02-03T22:03:20Z

Allows fetching a WebURL again when it fails. Features:

Stores the number of repetitions with WebURL in order to limit
Maximun number of repetitions configurable with ConfigCrawler.
onContentFetchError(Page) deprecated, created onContentFetchError(Page, Throwable) to pass information about the fetching error.
onContentFetchErrorNotFinal can abort reschedule in subclases based on exception or Page
provides protected method so subclases can re-schedule urls.
Handles HttpHostConnectException and UnknownHostException individually.

closes #99

HttpHostConnectException and UnknownHostException, as they're relevant to the crawling task. Crawler can be configured to retry if HttpHostConnectException . It is disabled by default, even if maxRetries > 0.

dgoiko added 2 commits February 3, 2020 22:48

configurable retry on ContentFetchError

d37c3ed

onContentFetchErrorNotFinal can abort re-schedule

dcc169b

dgoiko changed the title ~~Retry on error~~ Retry on ContentFetchError Feb 3, 2020

dgoiko added 2 commits February 4, 2020 00:06

HttpHostConnectException and UnknownHostException individually handled

5851ba9

HttpHostConnectException and UnknownHostException, as they're relevant to the crawling task. Crawler can be configured to retry if HttpHostConnectException . It is disabled by default, even if maxRetries > 0.

maxRetries converted in short to save space in sleepycat

46c3c01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry on ContentFetchError #437

Retry on ContentFetchError #437

dgoiko commented Feb 3, 2020 •

edited

Retry on ContentFetchError #437

Are you sure you want to change the base?

Retry on ContentFetchError #437

Conversation

dgoiko commented Feb 3, 2020 • edited

dgoiko commented Feb 3, 2020 •

edited