1. General Metadata of a Wikipedia Article.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%store -r the_page\n",
    "\n",
    "if 'the_page' not in locals():\n",
    "    import pickle\n",
    "    print(\"Loading default data...\")\n",
    "    the_page = pickle.load(open(\"data/the_page.p\",'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Welcome!\n",
    "\n",
    "You have just opened a collection of notebooks that lets you inspect the evolution of the revision history of a Wikipedia article, up to now (From the English language edition). It also allows you to highlight **article- or word-specific conflicts as well as the productivity of any given editor.** \n",
    "\n",
    "Specifically, for the notebooks after this initial one, it interfaces with the API of a specialized service called [WikiWho](www.wikiwho.net), which provides fine-grained change information about the tokens (words) in an article. \n",
    "\n",
    "It is written in a way that you can **explore it like a Web app, without interacting with the code behind it**, or - if you choose to - click on \"edit app\" in the Juypter navigation bar and play around with the code yourself. \n",
    "\n",
    "The default introduction example is the article \"The Camp of the Saints\" (a novel), which we recommend to start with. You can enter/search an article of your choice and explore it as well. \n",
    "\n",
    "Let's first get live data of some general statistics from Wikipedias own API and a service called Xtools:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown as md\n",
    "display(md(\"---\"))\n",
    "display(md(f\"# A. Basic Info from Wikipedia\"))\n",
    "display(md(f\"***Search for an article on the English Wikipedia***\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import widgets, Output\n",
    "from IPython.display import display, clear_output\n",
    "from external.wikipedia import WikipediaDV, WikipediaAPI\n",
    "wikipedia_dv = WikipediaDV(WikipediaAPI(domain='en.wikipedia.org'))\n",
    "\n",
    "# the method that listens to the click event\n",
    "def on_button_clicked(b):\n",
    "    global the_page\n",
    "    \n",
    "    # use the out widget so the output is overwritten when two or more\n",
    "    # searches are performed\n",
    "    with out:\n",
    "        try:\n",
    "            # query wikipedia\n",
    "            search_result = wikipedia_dv.search_page(searchTerm.value)\n",
    "            the_page = wikipedia_dv.get_page(search_result)\n",
    "            %store the_page\n",
    "            clear_output()\n",
    "            display(the_page.to_frame('value'))\n",
    "            display(md(f'You selected:'))\n",
    "            display(the_page['title'])\n",
    "\n",
    "        except:\n",
    "            clear_output()\n",
    "            display(md(f'The page title *\"{searchTerm.value}\"* was not found'))\n",
    "\n",
    "# by default display the last search\n",
    "try:\n",
    "    searchTerm = widgets.Text(the_page['title'], description='Page title:')\n",
    "except:\n",
    "    searchTerm = widgets.Text(\"The Camp of the Saints\", description='Page title:')\n",
    "\n",
    "# create and display the button    \n",
    "button = widgets.Button(description=\"Search\")\n",
    "example = md(\"e.g. *The Camp of the Saints*\")\n",
    "display(searchTerm,example,button)\n",
    "\n",
    "# the output widget is used to remove the output after the search field\n",
    "out = Output()\n",
    "display(out)\n",
    "\n",
    "# set the event\n",
    "button.on_click(on_button_clicked)\n",
    "\n",
    "# trigger the event with the default value\n",
    "on_button_clicked(button)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import  widgets\n",
    "from IPython.display import display, Javascript\n",
    "\n",
    "def run_below(ev):\n",
    "    display(Javascript('IPython.notebook.execute_cells_below()'))\n",
    "\n",
    "display(md(f'If this is correct, load the data and set this as the article to explore.'))\n",
    "button = widgets.Button(description=\"Load data\", button_style='info', min_width=500)\n",
    "button.on_click(run_below)\n",
    "display(button)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown as md\n",
    "display(md(\"---\"))\n",
    "display(md(f\"# B. General Statistics \"))\n",
    "display(md(f\"Provided through the Xtools API (1)\"))\n",
    "display(md(f\"***Page: {the_page['title']}***\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown as md\n",
    "from external.xtools import XtoolsAPI, XtoolsDV\n",
    "\n",
    "xtools_api = XtoolsAPI(project = 'en.wikipedia.org')\n",
    "xtools_dv = XtoolsDV(xtools_api)\n",
    "page_info = xtools_dv.get_page_info(the_page['title'])\n",
    "page_info['assessment'] = page_info['assessment']['value']\n",
    "\n",
    "page_info = page_info.to_frame('value').rename(index={\n",
    "    'project': 'Project name',\n",
    "    'page': 'Page name',\n",
    "    'watchers': 'Watchers (2)',    'pageviews': f\"Page Views (per {page_info['pageviews_offset']} days)\",\n",
    "    'revisions': 'Revisions',\n",
    "    'editors': 'Editors',\n",
    "    'author': 'Creator of the page',\n",
    "    'created_at': 'Creation Date',\n",
    "    'created_rev_id': 'Creation revision id',\n",
    "    'modified_at': 'Last modified',\n",
    "    'last_edit_id': 'Last revision id',\n",
    "    'assessment': 'Content Assessment (3)',\n",
    "}).drop(index = ['pageviews_offset', 'author_editcount', 'secs_since_last_edit','elapsed_time'])\n",
    "\n",
    "\n",
    "display(page_info)\n",
    "display(md(\"<sup>**(1)** *A community-built service for article statistics at xtools.wmflabs.org* **(2)** *Users that added this page to their watchlist.* **(3)** *See [Wikipedia Content Assessment](https://en.wikipedia.org/wiki/Wikipedia:Content_assessment)*</sup>\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown as md\n",
    "display(md(\"---\"))\n",
    "display(md(f\"# C. Page Views\"))\n",
    "display(md(f\"Provided through the Wikimedia API\"))\n",
    "display(md(f\"***Page: {the_page['title']}***\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query request\n",
    "from external.wikimedia import WikiMediaDV, WikiMediaAPI\n",
    "wikimedia_api = WikiMediaAPI(project='en.wikipedia')\n",
    "wikimedia_dv = WikiMediaDV(wikimedia_api)\n",
    "views = wikimedia_dv.get_pageviews(the_page['title'], 'daily')\n",
    "\n",
    "# Visualization\n",
    "from visualization.views_listener import ViewsListener\n",
    "from ipywidgets import interact\n",
    "from ipywidgets.widgets import Dropdown\n",
    "\n",
    "listener = ViewsListener(views)\n",
    "interact(listener.listen, \n",
    "         begin=Dropdown(options=views.timestamp),\n",
    "         end=Dropdown(options=views.timestamp.sort_values(ascending=False)),\n",
    "         granularity=Dropdown(options=['Yearly', 'Monthly', 'Weekly', 'Daily'], value='Monthly'))\n",
    "\n",
    "# The df_plotted keeps a reference to the plotted data above\n",
    "listener.df_plotted['views'].agg({\n",
    "    'Total views': sum,\n",
    "    'Max views period': max,\n",
    "    'Min views period': min,\n",
    "    'Average views': min,}).to_frame('Value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "After we have no seen some general statistics of the article and the views it attracted, we will go on to take a look at what specific kinds of changes by which editors it was subject to over time. \n",
    "\n",
    "Click below to go to the next notebook. You can later come back to this notebook and simply enter another article name to start the process over with that new article. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.notebooks import get_next_notebook\n",
    "from IPython.display import HTML\n",
    "display(HTML(f'<a href=\"{get_next_notebook()}\" target=\"_blank\">Go to next workbook</a>'))"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "nteract": {
   "version": "0.14.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}