- Log html page for failing selectors
- Save cookies to redis
- Use cookie container for dynamic pages
- Add support for saving url in schema
- Split up crawler builder and runner
- Think about creating DataDiscoveryCrawler and DataExtractionCrawler
- Use channels instead of blocking collection
- Implement throttling
- Save logs to Seq
- Write tests