Command-line-based web crawler configured by JSON configurations, defining what data fields to scrape from the visited websites and how to export them as JSON.
This crawler was designed for motion portals in Pirate-Party wikis. This examples given extract motions of a party assembly and exports them to JSON to be used on pirat.ly and Spickerrr.
$ npm install -g scraperrr
Several example configuration are provided in the presets-folder.
$ scraperrr -c presets/gvtk131_config.json
This will export a JSON data file with the motions from the wiki.
$ scraperrr -c presets/gvtk131_config.json -o out/gv-anträge.json
The resulting json export is saved in the defined output file.
$ scraperrr -v -c presets/gvtk131_config.json
Verbose output for debugging. --verbose works as well.
$ scraperrr -p 500 -c presets/gvtk131_config.json
Politeness, defines a waiting period in miliseconds between HTTP requests. --politeness works as well.
The config format is still in development and changes occasionally. Once, it is freezed full documentation will be provided.
- Flexible configuration files for scraping websites and exporting results to a specified JSON file
- Waiting period between HTTP-requests