dmcrawler (Douban Movie Crawler)

Author: Muchenxuan Tong demon386@gmail.com

A Clojure-based crawler for fetching short comments from Douban movie page. (e.g. http://movie.douban.com/subject/11529526/comments?sort=time)

Features:

Store context when problems happened or the program shutted down.
Parallelized crawling on a single machine.
Data are stored in the database, with the storage time attached.

Usage

Set up the DB according to the comments in src/dbcrawler/db.clj.
Set up the variable starting-movie-id in src/dbcrawler/config.clj. (Optional. You can use the default one.)
With leiningen2 installed, run the program with lein run.

License

Copyright © 2013 Muchenxuan Tong

Distributed under the Eclipse Public License.