Working-with-Twitter-Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Cornell Day of Data](https://data.research.cornell.edu/sites/default/files/CornellDayOfData2019-hr.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Working with Twitter Data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This interactive Jupyter notebook is a companion to \"Working with Twitter Data,\" a workshop led by Melanie Walsh at Cornell's 2019 Day of Data. The notebook was designed to allow participants to experiment with Twarc and Twint without prior set-up or installation.\n",
    "\n",
    "Many of the cells below include code that should be run on the command-line. These cells all begin with an exclamation point `!`. The `!` allows a Jupyter notebook to run code from a shell.\n",
    "![the command line](images/command-line.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Run a Jupyter cell**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Nice! You did it. You just ran a cell.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Installation**\n",
    "\n",
    "### Python\n",
    "\n",
    "I recommend installing Python with Anaconda: https://docs.continuum.io/anaconda/install/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Twarc](https://github.com/DocNow/twarc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install twarc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or download and open twarc as a zip file: https://github.com/DocNow/twarc/archive/master.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Twint](https://github.com/twintproject/twint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install twint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "  \n",
    "  \n",
    "  \n",
    "    \n",
    " \n",
    " \n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Set up a Twitter developer account**\n",
    "\n",
    "*Twarc won't work in this notebook unless you configure it with your own consumer key, consumer secret, access token, and access token secret.\n",
    "\n",
    "1. Create a Twitter developer account: https://developer.twitter.com/en/apps  \n",
    "\n",
    "2. Get consumer key, consumer secret, access token, and access token secret\n",
    "\n",
    "3. Open a terminal\n",
    "\n",
    "![](images/terminal.png)\n",
    "\n",
    "4. Configure twarc by entering `twarc configure` and following the prompts\n",
    "\n",
    "![](images/twarc-configure.png)\n",
    "\n",
    "Now you should be able to use twarc in this notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Tweet Collection**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect tweets based on keyword and output to a CSV file with Twint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `--search` or `-s` flag indicates scraping all tweets that include a specific keyword\n",
    "\n",
    "The `--output` or `-o` flag indicates saving the tweets to a file\n",
    "\n",
    "The `--csv` flag indicates write the file in CSV format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Search for tweets with the keyword \"demdebate\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!twint --search demdebate --output dem-debate-2019-10.csv --csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check how many tweets have been collected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An easy way to check how many tweets have been collected is to use the `wc` command with the `-l` flag, which returns the number of lines (`-l`) in a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wc -l dem-debate-2019-10.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cells below use the Python library [Pandas](https://pandas.pydata.org/), which we will not discuss in-depth today. It's included here because it's an easy way to quickly show what the the CSV file looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "#Set column width\n",
    "pandas.options.display.max_colwidth = 1000\n",
    "\n",
    "#Read CSV file\n",
    "dem_debate_tweets = pandas.read_csv('dem-debate-2019-10.csv')\n",
    "\n",
    "#Display columns in a Twint-produced CSV\n",
    "dem_debate_tweets.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show preview of CSV file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dem_debate_tweets[['date', 'time', 'username', 'tweet', 'mentions', 'retweets_count', 'hashtags', 'link']].head(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect tweets from a specific user and output to a JSON file with Twint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `--username` or `-u` flag indicates scraping a specific user's tweets\n",
    "\n",
    "The `--output` or `-o` flag indicates saving the tweets to a file\n",
    "\n",
    "The `--json` flag indicates write the file in JSON format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect tweets from @washingtonpost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!twint --username washingtonpost --output washington-post-tweets.json --json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check how many @washingtonpost tweets have been collected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect tweets from @FoxNews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!twint -u foxnews --output fox-news-tweets.json --json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check how many @FoxNews tweets have been collected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "  \n",
    "  \n",
    "  \n",
    "    \n",
    " \n",
    " \n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "  \n",
    "  \n",
    "  \n",
    "    \n",
    " \n",
    " \n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✨ Collect tweets from a Twitter Account of your Choice ✨"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!twint -u YOUR-TWITTER-ACCOUNT-HERE --output YOUR-FILENAME-HERE.json --json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "  \n",
    "  \n",
    "  \n",
    "    \n",
    " \n",
    " \n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "  \n",
    "  \n",
    "  \n",
    "    \n",
    " \n",
    " \n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect tweets based on keyword and output to a JSON file with Twarc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!twarc search demdebate > dem-debate-2019-10-twarc.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect most recent tweets from a specific user and output to a JSON file with Twarc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!twarc timeline washingtonpost > wa-po-recent.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!twarc timeline foxnews > fox-news-recent.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "fox_news_tweets = pandas.read_json('fox-news-recent.json', lines=True)\n",
    "\n",
    "#Display columns in a Twarc-produced JSON\n",
    "fox_news_tweets.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show preview of JSON file sorted by date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fox_news_tweets[['created_at', 'full_text', 'user', 'geo', 'retweet_count', 'id_str']].sort_values(by='id_str', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Tweet Analysis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twarc utilities `twarc/utils`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify Top Hashtags `twarc/utils/tags.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twarc/utils/tags.py dem-debate-2019-10-twarc.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Word Cloud `twarc/utils/wordcloud.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twarc/utils/wordcloud.py dem-debate-2019-10-twarc.json > dem-debate-2019-10-twarc.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[dem-debate-2019-10-twarc.html](dem-debate-2019-10-twarc.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twarc/utils/wordcloud.py wa-po-recent.json > wa-po-recent.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View your word cloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[wa-po-recent.html](wa-po-recent.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twarc/utils/wordcloud.py fox-news-recent.json > fox-news-recent.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View your word cloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[fox-news-recent.html](fox-news-recent.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify Top Emojis `twarc/utils/emojis.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twarc/utils/emojis.py dem-debate-2019-10-twarc.json | head -n 20"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}