W5-6 data from online apis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting data from online APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's relatively easy to get recent data from twitter. But you might have to pay for historical data.\n",
    "\n",
    "The first step is go to http://apps.twitter.com to create an app. \n",
    "You'll get a consumer_key, consumer_secrete, access_token, and accces_token_secret.\n",
    "\n",
    "These two random web pages are somewhat helpful:\n",
    "\n",
    "https://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/\n",
    "\n",
    "https://www.geeksforgeeks.org/extraction-of-tweets-using-tweepy/\n",
    "\n",
    "Then you connect to the twitter api as below. \n",
    "\n",
    "You can use my keys and tokens today. But you should create your own if you're planning to do more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These command you execute just once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tweepy\n",
    "consumer_key = \"G37xcCsQa5vkVhT9d6aFGgGWd\"\n",
    "consumer_secret = \"KNkBbqldSqjslanKD0pThNvmhbNptlPhj9YqPVjAG3Whyvfugf\"\n",
    "access_token = \"26928590-nWahsl79rYOGlzRWGGtidq1formdn6sy87Ea32E7U\"\n",
    "access_token_secret = \"toZsqPKDMp1zgwdgjWVj4YPqgLzM9PcjfsXqypeaJ8Az2\"\n",
    "# Creating the authentication object\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "# Setting your access token and secret\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "# Creating the API object while passing in auth information\n",
    "api = tweepy.API(auth) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how you do a research:\n",
    "\n",
    "(Note, if you don't use `tweet_mode='extended'` then the text of long tweets will be truncated.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "language = \"en\"\n",
    "query = \"#mtbos\"\n",
    "results = api.search(q=query, lang=language, count=100, tweet_mode='extended')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will return a list of a results.\n",
    "\n",
    "To extract the information you need, you'll want the ._json property. This will give you a dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "post = results[0]._json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "post.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "post[\"full_text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Propublica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Propublica makes some substantial data freely available from an online API.\n",
    "\n",
    "You have to request a key. But it's quick and easy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.propublica.org/datastore/apis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "my_propub_key = \"ktlRtmbsOL7c27WRGsSeUSYFJRdU04EZQZIKGSuv\"\n",
    "headers = {'X-API-Key': my_propub_key}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "def count_documents_pages(propublica_query): # This function number of pages/requests to make for JSON files based on the number of query results.\n",
    "    offset = 0\n",
    "    initial_request = requests.get(propublica_query.format(offset), headers = headers).json()\n",
    "    num_pages = math.ceil(initial_request['num_results']/20) # Each response shows 20 articles. ProPublica API supports pagination, so we need to calculate how many pages we need to request.\n",
    "    print(\"Number of documents is {}.\".format(initial_request['num_results']))\n",
    "    print(\"Number of pages is {}.\".format(num_pages))\n",
    "    return num_pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "propublica_query = \"https://api.propublica.org/congress/v1/statements/search.json?query=%22common%20core%22&offset={0}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_documents_pages(propublica_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "offset = 0\n",
    "initial_request = requests.get(propublica_query.format(offset), headers = headers).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_request.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells us how many matches our search found.\n",
    "But it only gives us the first 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_request[\"num_results\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(initial_request[\"results\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_request[\"results\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this just gives us the url for the web page we want. \n",
    "\n",
    "We can use the python `requests` module to actually get the web page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = requests.get('https://biggs.house.gov/media/press-releases/congressman-biggs-reintroduces-ending-common-core-and-expanding-school-choice')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This just gives us the html source of the web page. So there's work to do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(res.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(res.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing the offset can give us more results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "offset = 20\n",
    "second_request = requests.get(propublica_query.format(offset), headers = headers).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reddit\n",
    "\n",
    "You can download data from reddit. A page that explains how to do it is [here](https://towardsdatascience.com/how-to-use-the-reddit-api-in-python-5e05ddfd1e5c): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# note that CLIENT_ID refers to 'personal use script' and SECRET_TOKEN to 'token'\n",
    "auth = requests.auth.HTTPBasicAuth('<CLIENT_ID>', '<SECRET_TOKEN>')\n",
    "\n",
    "# here we pass our login method (password), username, and password\n",
    "data = {'grant_type': 'password',\n",
    "        'username': '<USERNAME>',\n",
    "        'password': '<PASSWORD>'}\n",
    "\n",
    "# setup our header info, which gives reddit a brief description of our app\n",
    "headers = {'User-Agent': 'MyBot/0.0.1'}\n",
    "\n",
    "# send our request for an OAuth token\n",
    "res = requests.post('https://www.reddit.com/api/v1/access_token',\n",
    "                    auth=auth, data=data, headers=headers)\n",
    "\n",
    "# convert response to JSON and pull access_token value\n",
    "TOKEN = res.json()['access_token']\n",
    "\n",
    "# add authorization to our headers dictionary\n",
    "headers = {**headers, **{'Authorization': f\"bearer {TOKEN}\"}}\n",
    "\n",
    "# while the token is valid (~2 hours) we just add headers=headers to our requests\n",
    "requests.get('https://oauth.reddit.com/api/v1/me', headers=headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = requests.get(\"https://oauth.reddit.com/r/Northwestern\",\n",
    "                   headers=headers)\n",
    "\n",
    "for post in res.json()[\"data\"][\"children\"]:\n",
    "    print(post[\"data\"][\"title\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "post[\"data\"][\"selftext\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}