Skip to content

Scrapes a user's public tweets and saves them in JSON format.

Notifications You must be signed in to change notification settings

LocalSense/birdeater

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Using Node.js and JQuery to Crawl Public Tweets

A talk by @benjamincoe

Birdeater

Birdeater is a command-line tool for backing up a user's public Tweets in JSON format.

Usage

First, make sure you have node package manager installed:

curl http://npmjs.org/install.sh | sh

Then run the following commands to install and run Birdeater:

npm install birdeater -g
birdeater --user=shitmydadsays

Be mindful when running it, as Twitter limits the number of requests that a single client can make per hour.

How Birdeater Works

Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML. Here's how it works:

An http connection is made to a user's public timeline using the request library:

User.prototype.loadStatuses = function(callback) {
	request({
		url: 'https://twitter.com/...',
		headers: {accept: 'application/json'}
	}, function(err, res, body) {
		var error = err || res.statusCode != 200;
		// handle error, or.
		callback( body );
	});
};

This returns an HTML representation of the tweets. JQuery is used to extract structured information from this:

jquery(body).find('.tweet').each(function() {
	var tweet = jquery(this);
	_this.tweets.push({
		text: jquery.trim( tweet.find('.js-tweet-text').text() ),
		timestamp: parseInt( tweet.find('._timestamp').attr('data-time') ),
		mentions: mentions.length > 0 ? tweet.attr('data-mentions').split(' ') : [],
		urls: urls
	});
});

I find that Node.js, coupled with JQuery, works great for building web crawlers:

  • Node.js offers a non-blocking evented paradigm. This is awesome for a web-crawler, which will tend to be I/O bound.
  • JQuery is a champ at parsing HTML. I find it much more intuitive than other libraries that I've used, such as Beautiful Soup.
  • First and foremost, it's easy to do (take a look at lib/user.js Birdeater clocks in at about 100 lines of code).

This approach has become my hammer when web scraping tasks come up.

-- Ben @benjamincoe

About

Scrapes a user's public tweets and saves them in JSON format.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published