Skip to content

Releases: infinilabs/crawler

v0.12.0_SNAPSHOT

07 Dec 06:07
Compare
Choose a tag to compare
v0.12.0_SNAPSHOT Pre-release
Pre-release

v0.12

breaking changes
  1. pipeline config moved to dedicated section, can be reference by id
  2. elasticsearch config moved to dedicated section, can be reference by id
  3. remove indexer module, nested to elastic module
features
  1. add parse_pdf joint to parse PDF files
  2. support major elasticsearch version, auto handle API differences
improvement
  1. auto clone and update framework/vendor repo
  2. move vendor out of this repo
bugfix
  1. fix relative link was not proper resolved with https
  2. fix redirected link not handled exception

v0.11.0

25 Aug 15:04
Compare
Choose a tag to compare

v0.11

Adios Sqlite, Hola Elasticsearch!

breaking changes
  1. extract common codebase to another repo: https://github.com/infinitbyte/framework
  2. sqlite retired, elasticsearch is the first citizen
features
  1. add a new cmd static_fs to support load static files from folder
  2. auto generate elasticsearch mapping and template, no need to manual create mapping first
  3. add new backup command to support migration

v0.10.0

23 Feb 14:22
Compare
Choose a tag to compare

Only support SQLite as database .

v0.10

breaking changes
  1. refactor domain to host, api and mapping has changed
  2. refactor module, update yml settings: module->name
features
  1. dynamic create pipelines
  2. init plugin architecture
  3. support extract tags by css path
  4. add chrome fetch joint, via chrome debug protocol
  5. add auto-completion to search ui
  6. search ui support mobile
  7. support access control by github oauth
improvement
  1. remove goleveldb due to memory leak
  2. update logo
  3. remove hard coded version
  4. update task UI, support filter by status and host
  5. clean offset_canvas menu

v0.9.0

20 Oct 12:11
Compare
Choose a tag to compare
add screenshot

v0.8.0

05 Jul 08:24
Compare
Choose a tag to compare
enable index by default