Fix Save/Load State

[ikreymer]

• Fixes state serialization, which was missing the done list. Instead, adds a 'finished' list computed from the seen list, minus failed and queued URLs.
• Also adds serialization support for 'extraSeeds', seeds added dynamically from a redirect (via Handle seed redirects #475). Extra seeds are added to Redis and also included in the serialization.

Fixes #491

[tw4l]

Very nice, tested locally and working well. Thanks for the tests and cleanup here too.

[tw4l]

I guess only other question is, could we not go back to using done rather than introducing a new finished, since we're already handling it as an int or array or strings for backwards compatibility's sake? No strong opinion here, but it seems we could just undo the breaking change rather than renaming.

[ikreymer]

I guess only other question is, could we not go back to using done rather than introducing a new finished, since we're already handling it as an int or array or strings for backwards compatibility's sake? No strong opinion here, but it seems we could just undo the breaking change rather than renaming.

Yeah, I thought about that, technically could populate done from the pages list (it is supposed to be a full json dict rather than a single URL), but that requires either 1) reading and parsing pages.jsonl, and in case there's an issue (though there shouldn't be, have seen it if text is added) or 2) re-adding the done key in redis which would only be used for this.
But, that previous format still required that done be a JSON string but really we just need a single URL so this is more efficient, but perhaps later can switch to reading directly from pages.jsonl as well.

So overall, this is a better format, and chose a new key name to avoid confusion.