1_General_Metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false,
    "hide_input": false
   },
   "source": [
    "# Welcome to IWAAN*\n",
    "\\**Interactive Wikipedia Article Analysis Notebooks*\n",
    "\n",
    "This  collection of notebooks lets you analyze the evolution of the revision history of a Wikipedia article with **live data**. Among other things, it allows you to inspect **article- or editor-specific writing dynamics, productivity & conflicts.** \n",
    "The main notebooks are the current one and notebook two, with notebook three offering some additional information. \n",
    "\n",
    "Each Notebook can be **explored like a Web app, without interacting with the code behind it**. But you can also - if you choose to - click on **\"Show solution\"** after each block and work with the code directly (green buttons, comment the first line of code to use). \n",
    " \n",
    "\n",
    "**Please choose a small or medium-sized article for your first exploration to reduce computation times**. All data is fetched live, so please be patient and wait for the kernel to finish before you interact with the notebook.\n",
    "\n",
    "<img src=\"kernel_active.png\" alt=\"drawing\" width=\"300\"/>  \n",
    "\n",
    "E.g., for your very first try you should simply go with our default choice, \"The camp of the saints\". \n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "#### Troubleshooting:\n",
    "\n",
    "- Allow some time for the notebook to run fully before interacting with the interface controls. For articles with a long revision history, this could take minutes and the interaction with the controls will be slow.\n",
    "- Very large articles (\"Donald Trump\", \"World War 2\") may lead to memory overload, depending on which Binder platform you are using. You could get more memory by shutting down some previous running notebooks.\n",
    "- **All cells should run automatically** when you open the notebook and **you should not see any code**. If that is not the case, please just reload the tab in your browser. \n",
    "- After chosing a new article, please rerun the cells/modules you want to use. \n",
    "- You should not see any code when you run it for the first time. If so, please let us know by posting an issue in our Github repository: https://github.com/gesiscss/IWAAN/issues.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hide_input": true,
    "init_cell": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading default data...\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "IPython.notebook.execute_cells_below()"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "Jupyter.notebook.get_cells()"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<script>\n",
       "var update_input_visibility = function () {\n",
       "    Jupyter.notebook.get_cells().forEach(function(cell) {\n",
       "        if (cell.metadata.hide_input) {\n",
       "            cell.element.find(\"div.input\").hide();\n",
       "        }\n",
       "    })\n",
       "};\n",
       "update_input_visibility();\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<script>\n",
       "document.getElementById('notebook').scrollIntoView();\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Javascript, HTML, clear_output\n",
    "from ipywidgets import widgets, Output, interact, Layout\n",
    "from ipywidgets.widgets import Dropdown\n",
    "from datetime import datetime\n",
    "\n",
    "## SOME EXTENTIONS ##\n",
    "#%load_ext autoreload\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "if 'the_page' not in locals():\n",
    "    import pickle\n",
    "    print(\"Loading default data...\")\n",
    "    the_page = pickle.load(open(\"data/the_page.p\",'rb'))\n",
    "\n",
    "display(Javascript('IPython.notebook.execute_cells_below()'))\n",
    "\n",
    "display(Javascript('Jupyter.notebook.get_cells()'))\n",
    "\n",
    "re_hide = \"\"\"\n",
    "<script>\n",
    "var update_input_visibility = function () {\n",
    "    Jupyter.notebook.get_cells().forEach(function(cell) {\n",
    "        if (cell.metadata.hide_input) {\n",
    "            cell.element.find(\"div.input\").hide();\n",
    "        }\n",
    "    })\n",
    "};\n",
    "update_input_visibility();\n",
    "</script>\n",
    "\"\"\"\n",
    "display(HTML(re_hide))\n",
    "\n",
    "scroll_to_top = \"\"\"\n",
    "<script>\n",
    "document.getElementById('notebook').scrollIntoView();\n",
    "</script>\n",
    "\"\"\"\n",
    "display(HTML(scroll_to_top))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "%store -r the_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "### <span style=\"color:green\"> Import all modules </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'wikiwho_wrapper'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-3-e4b919e2437a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;31m# APIs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mwikiwho_wrapper\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mWikiWho\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mexternal\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwikipedia\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mWikipediaDV\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mWikipediaAPI\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mexternal\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwikimedia\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mWikiMediaDV\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mWikiMediaAPI\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'wikiwho_wrapper'"
     ]
    }
   ],
   "source": [
    "## Modules Imported ##\n",
    "\n",
    "# Display\n",
    "from IPython.display import display, Markdown as md, clear_output\n",
    "from datetime import date\n",
    "import urllib\n",
    "\n",
    "# APIs\n",
    "from wikiwho_wrapper import WikiWho\n",
    "from external.wikipedia import WikipediaDV, WikipediaAPI\n",
    "from external.wikimedia import WikiMediaDV, WikiMediaAPI\n",
    "from external.xtools import XtoolsAPI, XtoolsDV\n",
    "\n",
    "# Data Processing\n",
    "import pickle\n",
    "import pandas as pd\n",
    "\n",
    "# Visualization tools\n",
    "import qgrid\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Page views timeline\n",
    "from visualization.views_listener import ViewsListener\n",
    "\n",
    "# Change actions timeline\n",
    "from visualization.actions_listener import ActionsListener\n",
    "\n",
    "# Conflicts visualization\n",
    "from visualization.conflicts_listener import ConflictsListener, ConflictsActionListener\n",
    "from visualization.calculator_listener import ConflictCalculatorListener\n",
    "\n",
    "# Word cloud visualization\n",
    "from visualization.wordcloud_listener import WCListener, WCActionsListener\n",
    "from visualization.wordclouder import WordClouder\n",
    "\n",
    "# Wikipedia talk pages visualization\n",
    "from visualization.talks_listener import TalksListener\n",
    "from visualization.topics_listener import TopicsListener\n",
    "\n",
    "# Tokens ownership visualization\n",
    "from visualization.owned_listener import OwnedListener\n",
    "\n",
    "# Templates visualization\n",
    "from visualization.templates_listener import ProtectListener, TemplateListener\n",
    "\n",
    "# Metrics management\n",
    "from metrics.conflict import ConflictManager\n",
    "from metrics.token import TokensManager\n",
    "\n",
    "# For language selection\n",
    "from utils.lngselection import abbreviation, lng_listener"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false,
    "hide_input": false
   },
   "source": [
    "---\n",
    "\n",
    "## A. Selecting a Wikipedia article\n",
    "\n",
    "You can enter/search a Wikipedia article of your choice, or use our default example: \"The Camp of the Saints\" (a novel). \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hideOutput": false,
    "hidePrompt": false,
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "# the method that listens to the click event\n",
    "def on_button_clicked(b):\n",
    "    global wikipedia_dv\n",
    "    global the_page\n",
    "    # use the out widget so the output is overwritten when two or more\n",
    "    # searches are performed\n",
    "    with out:\n",
    "        try:            \n",
    "            # query wikipedia\n",
    "            wikipedia_dv = WikipediaDV(WikipediaAPI(lng=abbreviation(languageSelection.value)))\n",
    "            search_result = wikipedia_dv.search_page(searchTerm.value)\n",
    "            the_page = wikipedia_dv.get_page(search_result)\n",
    "            %store the_page\n",
    "            clear_output()\n",
    "            display(md(f\"The following page was found: **{the_page['title']}**\"))\n",
    "            display(md(f\"Page id: **{the_page['page_id']}**\"))\n",
    "            url = f\"{wikipedia_dv.api.base}action=query&titles={urllib.parse.quote_plus(the_page['title'])}&format=json\"\n",
    "            display(md(f\"Metadata can be found in:\"))\n",
    "            print(url)\n",
    "            #display(Javascript('Jupyter.notebook.execute_cells([8])'))\n",
    "\n",
    "        except:\n",
    "            clear_output()\n",
    "            display(md(f'The page title *\"{searchTerm.value}\"* was not found'))\n",
    "            #display(Javascript('Jupyter.notebook.execute_cells([8])'))\n",
    "\n",
    "# Language selection.\n",
    "languageSelection = Dropdown(options=['English', 'Deutsch', 'Español', 'Türkçe', 'Euskara'], value='English', description='Language:')\n",
    "\n",
    "out_main = Output()\n",
    "display(out_main)\n",
    "with out_main:\n",
    "    # by default display the last search\n",
    "    try:\n",
    "        searchTerm = widgets.Text(the_page['title'], description='Page title:')\n",
    "    except:\n",
    "        searchTerm = widgets.Text(\"The Camp of the Saints\", description='Page title:')\n",
    "\n",
    "    # Update selected language\n",
    "    initial_select = widgets.interactive(lng_listener, lng=languageSelection, search_term=searchTerm)\n",
    "    display(initial_select.children[0])\n",
    "    display(initial_select.children[1])\n",
    "\n",
    "    # create and display the button    \n",
    "    button = widgets.Button(description=\"Search\")\n",
    "    example = md(\"e.g. *The Camp of the Saints*\")\n",
    "    display(example, button)\n",
    "\n",
    "# the output widget is used to remove the output after the search field\n",
    "out = Output()\n",
    "display(out)\n",
    "\n",
    "# set the event\n",
    "button.on_click(on_button_clicked)\n",
    "\n",
    "# trigger the event with the default value\n",
    "on_button_clicked(button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "<span style=\"color: #626262\"> Try yourself! This is the code executed when  you click the 'Search' button: </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false,
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "### IMPORTANT NOTE: COMMENT THE ABOVE LINE TO EXECUTE THE CELL ###\n",
    "\n",
    "### --------------------------------------------------------------------- ###\n",
    "### TRY YOURSELF! THIS IS WHAT WILL HAPPEN WHEN YOU CLICK 'Search' BUTTON ###\n",
    "### --------------------------------------------------------------------- ###\n",
    "\n",
    "## This is the default data and used for initialization ##\n",
    "the_page = pickle.load(open(\"data/the_page.p\",'rb'))  # global\n",
    "title_default = the_page['title']\n",
    "print('The pre-filled value for the title:', title_default)\n",
    "\n",
    "## The search term you have input ##\n",
    "\n",
    "# this was extracted from the previous search box, searchTerm, which is built by\n",
    "# e.g. searchTerm = widgets.Text(the_page['title'], description='Page title:'), you could\n",
    "# know more about it through https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html\n",
    "search_language = languageSelection.value\n",
    "language_for_api = abbreviation(search_language)\n",
    "\n",
    "search_value = searchTerm.value\n",
    "print('The language you use now:', search_language)\n",
    "print('The value you input in search box:', search_value)\n",
    "\n",
    "# of course you could also update the value here, like in the \"Page title:\" box above.\n",
    "#search_value = 'Matrix Completion' \n",
    "\n",
    "## query wikipedia using WikiWho API, more details please see:                ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/wikipedia.py ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/api.py       ##\n",
    "wikipedia_dv = WikipediaDV(WikipediaAPI(lng=language_for_api)) # create an instance\n",
    "result_after_search = wikipedia_dv.search_page(search_value)\n",
    "\n",
    "print(\"The following page was found:\", result_after_search)\n",
    "\n",
    "## Get page id through get_page() method ##\n",
    "the_page = wikipedia_dv.get_page(result_after_search)  # global\n",
    "page_id = the_page['page_id']\n",
    "print('Page id:', page_id)\n",
    "\n",
    "## Metadata ##\n",
    "url = f\"{wikipedia_dv.api.base}action=query&titles={urllib.parse.quote_plus(result_after_search)}&format=json\"\n",
    "print('Metadata can be found in:', url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Click on \"Refresh All Variables\" to reload all following cells with the selected article. (may take a some time)** \n",
    "\n",
    "Alternatively, reload each module individually via their respective buttons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true,
    "init_cell": false
   },
   "outputs": [],
   "source": [
    "def restart_and_run():\n",
    "    display(HTML(\n",
    "        '''\n",
    "            <script>\n",
    "                code_show = false;\n",
    "                function restart_run_all(){\n",
    "                    IPython.notebook.kernel.restart();\n",
    "                    setTimeout(function(){\n",
    "                        IPython.notebook.execute_all_cells();\n",
    "                    })\n",
    "                }\n",
    "            </script>\n",
    "            <button onclick=\"restart_run_all()\">Refresh All Variables</button>\n",
    "        '''\n",
    "    ))\n",
    "restart_and_run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "---\n",
    "\n",
    "## B. General Statistics\n",
    "\n",
    "Thanks to the Xtools API **(1)**, we can obtain general statistics related to the articlee. This information is relevant to select an appropiate article to analyze. \n",
    "\n",
    "Our default example contains less than 1000 revisions (as of April, 2021), making it a good first case for our analysis. Articles such as *\"Chicago\"*, *\"Barack Obama\"* and *\"Climate Change\"* can have over 20K revisions, and therefore the processing will be slow and even fail due to lack of memory on the system running the notebooks (e.g.[notebooks.gesis.org](https://notebooks.gesis.org)).\n",
    "\n",
    "Wikipedia offers a [list of the pages with most revisions](https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Pages_with_the_most_revisions); **we recommend to avoid such articles for a smooth first experience with IWAAN**. Once you are confortable with our notebooks, you can tackle more complex articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false,
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "def xtools_on_click(b):\n",
    "    with out_xtools:\n",
    "        clear_output()\n",
    "        xtools_api = XtoolsAPI(lng=abbreviation(languageSelection.value))\n",
    "        xtools_dv = XtoolsDV(xtools_api)\n",
    "        page_info = xtools_dv.get_page_info(the_page['title'])\n",
    "        page_info['assessment'] = page_info['assessment']['value'] if type(page_info['assessment']) != bool else page_info['assessment']\n",
    "\n",
    "        page_info = page_info.to_frame('value').rename(index={\n",
    "            'project': 'Project name',\n",
    "            'page': 'Page name',\n",
    "            'watchers': 'Watchers (2)',    'pageviews': f\"Page Views (per {page_info['pageviews_offset']} days)\",\n",
    "            'revisions': 'Revisions',\n",
    "            'editors': 'Editors',\n",
    "            'author': 'Creator of the page',\n",
    "            'created_at': 'Creation Date',\n",
    "            'created_rev_id': 'Creation revision id',\n",
    "            'modified_at': 'Last modified',\n",
    "            'last_edit_id': 'Last revision id',\n",
    "            'assessment': 'Content Assessment (3)',\n",
    "        }).drop(index = ['pageviews_offset', 'author_editcount', 'secs_since_last_edit','elapsed_time'])\n",
    "        \n",
    "        display(md(f\"***Page: {the_page['title']} ({abbreviation(languageSelection.value).upper()})***\"))\n",
    "        url = f\"{xtools_dv.api.base}page/articleinfo/{xtools_dv.api.project}/\" + urllib.parse.quote(the_page['title'])\n",
    "        display(md(f\"Metadata can be found in\"))\n",
    "        print(url)\n",
    "        display(page_info)\n",
    "        #display(Javascript('Jupyter.notebook.execute_cells([14])'))           \n",
    "        #display(Javascript('Jupyter.notebook.execute_cells([18])'))\n",
    "        \n",
    "\n",
    "# create and display the button    \n",
    "button = widgets.Button(description=\"Get Page Info\")\n",
    "display(button)\n",
    "\n",
    "# the output widget is used to remove the output after the search field\n",
    "out_xtools = Output()\n",
    "display(out_xtools)\n",
    "\n",
    "# set the event\n",
    "button.on_click(xtools_on_click)\n",
    "\n",
    "# trigger the event with the default value\n",
    "xtools_on_click(button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "**(1)** *A community-built service for article statistics at xtools.wmflabs.org*<br>\n",
    "**(2)** *Users that added this page to their watchlist*<br>\n",
    "**(3)** *See [Wikipedia Content Assessment](https://en.wikipedia.org/wiki/Wikipedia:Content_assessment)*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "<span style=\"color: #626262\"> Try yourself! This is the code executed when  you click the 'Get Page Info' button: </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "### IMPORTANT NOTE: COMMENT THE ABOVE LINE TO EXECUTE THE CELL ###\n",
    "\n",
    "### ----------------------------------------------------------------------------- ###\n",
    "### TRY YOURSELF! THIS IS WHAT WILL HAPPEN WHEN YOU CLICK 'Get Page Info' BUTTON  ###\n",
    "### ----------------------------------------------------------------------------- ###\n",
    "\n",
    "## Define a Xtools instance, more details see:                             ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/xtools.py ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/api.py    ##\n",
    "xtools_api = XtoolsAPI(lng=abbreviation(languageSelection.value))\n",
    "xtools_dv = XtoolsDV(xtools_api)\n",
    "print('Provided through the Xtools API (1)')\n",
    "\n",
    "## The page you are insterested in ##\n",
    "print('The page that was found:', the_page['title'], f'({abbreviation(languageSelection.value).upper()})')\n",
    "\n",
    "## Get the page info through Xtools method get_page_info() ##\n",
    "page_info = xtools_dv.get_page_info(the_page['title'])\n",
    "\n",
    "## Metadata ##\n",
    "url = f\"{xtools_dv.api.base}page/articleinfo/{xtools_dv.api.project}/\" + urllib.parse.quote(the_page['title'])\n",
    "print(\"Metadata can be found in:\", url)\n",
    "\n",
    "## Use a dictionary to construct a pd.DataFrame to present the general info from Xtools ##\n",
    "dict_for_df = {\n",
    "    'Project name': page_info['project'], 'Page name': page_info['page'], 'Watchers(2)': page_info['watchers'],\n",
    "    'Page Views (per 30 days)': page_info['pageviews'], 'Revisions': page_info['revisions'], \n",
    "    'Editors': page_info['editors'], 'minor_edits': page_info['minor_edits'], 'Creator of the page': page_info['author'],\n",
    "    'Creation Date': page_info['created_at'], 'Creation revision id': page_info['created_at'],\n",
    "    'Last modified': page_info['modified_at'], 'Last revision id': page_info['last_edit_id'],\n",
    "    'Content Assessment (3)': page_info['assessment']['value'] if type(page_info['assessment']) != bool else page_info['assessment']\n",
    "}\n",
    "\n",
    "df_info = pd.DataFrame.from_dict(dict_for_df, orient='index', columns=['value'])\n",
    "display(df_info)\n",
    "\n",
    "## Some footnotes ##\n",
    "display(md('**(1)** *A community-built service for article statistics at xtools.wmflabs.org*<br>'\n",
    "           '**(2)** *Users that added this page to their watchlist*<br>'\n",
    "           '**(3)** *See [Wikipedia Content Assessment](https://en.wikipedia.org/wiki/Wikipedia:Content_assessment)*'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "---\n",
    "\n",
    "## C. Page Views\n",
    "\n",
    "We use the Wikimedia Pageviews API to display the page views of an article, which can be helpful when assesing the impact of changes in Wikipedia articles. For example, it is possible that a period with a lot of changes might due to internal politics of Wikipedia rather than an increase of interest on the topic by the general population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false,
    "hide_input": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def pageviews_button(b):\n",
    "    with out_pageviews:\n",
    "        clear_output()\n",
    "        \n",
    "        display(md(f\"***Page: {the_page['title']} ({abbreviation(languageSelection.value).upper()})***\"))\n",
    "        # Query request\n",
    "        wikimedia_api = WikiMediaAPI(lng=abbreviation(languageSelection.value))\n",
    "        wikimedia_dv = WikiMediaDV(wikimedia_api)\n",
    "        views = wikimedia_dv.get_pageviews(the_page['title'], 'daily')\n",
    "        \n",
    "        # Visualization\n",
    "        listener = ViewsListener(views)\n",
    "        inter_func = interact(listener.listen, \n",
    "                         begin=Dropdown(options=views.timestamp),\n",
    "                         end=Dropdown(options=views.timestamp.sort_values(ascending=False)),\n",
    "                         granularity=Dropdown(options=['Yearly', 'Monthly', 'Weekly', 'Daily'], value='Monthly'))\n",
    "\n",
    "        # raw data url\n",
    "        start = 19900101\n",
    "        today = datetime.today().strftime(\"%Y%m%d\")\n",
    "        end = int(today)\n",
    "        article_name = urllib.parse.quote(the_page['title'])\n",
    "        granularity = 'daily'\n",
    "\n",
    "        url = (f'{wikimedia_dv.api.base}metrics/pageviews/per-article/{wikimedia_dv.api.project}/'\n",
    "                f'all-access/all-agents/{article_name}/{granularity}/{start}/{end}')\n",
    "        display(md(f\"Metadata can be found in:\"))\n",
    "        print(url)\n",
    "                   \n",
    "        # The df_plotted keeps a reference to the plotted data above\n",
    "        pageviews_agg = listener.df_plotted['views'].agg({\n",
    "                            'Total views': sum,\n",
    "                            'Max views period': max,\n",
    "                            'Min views period': min,\n",
    "                            'Average views': min,}).to_frame('Value')\n",
    "        display(pageviews_agg)\n",
    "        \n",
    "        \n",
    "\n",
    "# create and display the button    \n",
    "button = widgets.Button(description=\"Get Pageviews\", layout=Layout(width='150px'))\n",
    "display(button)\n",
    "\n",
    "# the output widget is used to remove the output after the search field\n",
    "out_pageviews = Output()\n",
    "display(out_pageviews)\n",
    "\n",
    "# set the event\n",
    "button.on_click(pageviews_button)\n",
    "\n",
    "# trigger the event with the default value\n",
    "pageviews_button(button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "<span style=\"color: #626262\"> Try yourself! This is the code executed when  you click the  'Get Pageviews' button: </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "### IMPORTANT NOTE: COMMENT THE ABOVE LINE TO EXECUTE THE CELL ###\n",
    "\n",
    "### ----------------------------------------------------------------------------- ###\n",
    "### TRY YOURSELF! THIS IS WHAT WILL HAPPEN WHEN YOU CLICK 'Get Pageviews' BUTTON  ###\n",
    "### ----------------------------------------------------------------------------- ###\n",
    "\n",
    "## define a WikiMediaAPI instance, more details see:                          ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/wikimedia.py ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/api.py       ##\n",
    "\n",
    "\n",
    "wikimedia_api = WikiMediaAPI(lng=abbreviation(languageSelection.value))\n",
    "wikimedia_dv = WikiMediaDV(wikimedia_api)\n",
    "\n",
    "## Page of insterest ##\n",
    "print('The page that was found:', the_page['title'], f'({abbreviation(languageSelection.value).upper()})')\n",
    "\n",
    "## get pageview counts for the article, more details see:                     ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/external/wikimedia.py ##\n",
    "\n",
    "views = wikimedia_dv.get_pageviews(the_page['title'], 'daily')\n",
    "\n",
    "## Visualization, core visual code lies in ViewsListener, then the interact function    ##\n",
    "## make it interactive, mode details see:                                               ##\n",
    "## https://github.com/gesiscss/wikiwho_demo/blob/master/visualization/views_listener.py ##\n",
    "## https://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html           ##\n",
    "\n",
    "# Create a ViewListener instance with the page views counts\n",
    "listener = ViewsListener(views)\n",
    "\n",
    "# You could customize begin/end dates and granularity to generate different graphs\n",
    "# e.g. begin='20160809', end='20191020', granularity='Monthly'\n",
    "begin='20150701'\n",
    "end='20210314'\n",
    "granularity='Monthly' # 'Yearly', 'Monthly', 'Weekly' ,'Daily' \n",
    "\n",
    "# Metadata\n",
    "url = (f'{wikimedia_dv.api.base}metrics/pageviews/per-article/'\n",
    "       f\"{wikimedia_dv.api.project}/all-access/all-agents/{urllib.parse.quote(the_page['title'])}/{granularity.lower()}\"\n",
    "       f\"/{int(begin)}/{int(end)}\")\n",
    "print('Metadata can be found in:', url)\n",
    "print('(Note that: the time unit for the response data only supports granularity for \"daily\" and \"monthly\")\\n')\n",
    "\n",
    "# Time range you have selected\n",
    "print('Time range you have selected:')\n",
    "print('Start date:', begin)\n",
    "print('End date:', end)\n",
    "print('Granularity:', granularity)\n",
    "\n",
    "# Visulization\n",
    "listener.listen(begin, end, granularity)\n",
    "\n",
    "# Pageviews aggregation data. Use the attribute \"df_plotted\".\n",
    "pageviews_agg = listener.df_plotted['views'].agg({\n",
    "                    'Total views': sum,\n",
    "                    'Max views period': max,\n",
    "                    'Min views period': min,\n",
    "                    'Average views': min,}).to_frame('Value')\n",
    "\n",
    "print('Total views of this page:', pageviews_agg['Value']['Total views'])\n",
    "print('Max views during the selected period:', pageviews_agg['Value']['Max views period'])\n",
    "print('Min views during the selected period:', pageviews_agg['Value']['Min views period'])\n",
    "print('Average views during the selected period:', pageviews_agg['Value']['Average views'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# D. Templates and Protection\n",
    "\n",
    "In this step, all the [WikiWho tokens](https://www.wikiwho.net/) and associated revisions are downloaded from the [WikiWho API](https://www.wikiwho.net/). Depending on the article, this might take some time, but, once finished, it will display the period of times in which the article was on any of the following statuses:\n",
    "\n",
    "- **Semi-page edit protection**: all phases in the article’s life where edits or moves where only allowed for registered users\n",
    "- **Full-page edit protection**: all phases in the article’s life where edits or moves where only allowed for rusers with special rights, such  as  admins\n",
    "- **Good article**: as described by Wikipedia guidelines for [good articles](https://en.wikipedia.org/wiki/Wikipedia:Good_articles)\n",
    "- **Featured article**: as described by Wikipedia guidelines for [featured articles](https://en.wikipedia.org/wiki/Wikipedia:Featured_articles)\n",
    "- **Special templates**: special templates posted by editors indicating disputes, as well as bias and neutrality issues\n",
    "\n",
    "Similar to the page views, this information puts the edits, conflicts and discussions in the the following notebook in   context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def template_button(b):\n",
    "    global token_source\n",
    "    global new_template\n",
    "    global new_protect\n",
    "    global token_source\n",
    "    with template_out:\n",
    "        clear_output()\n",
    "        \n",
    "        # WikiWho API.\n",
    "        wikiwho = WikiWho(lng=abbreviation(languageSelection.value))\n",
    "        display(md(\"Downloading all_content from the WikiWhoApi...\"))\n",
    "        content = wikiwho.dv.all_content(the_page['page_id'])\n",
    "        display(md(\"Downloading revisions from the WikiWhoApi...\"))\n",
    "        revisions = wikiwho.dv.rev_ids_of_article(the_page['page_id'])\n",
    "        clear_output()\n",
    "        \n",
    "        # Wikipedia API\n",
    "        pp_log = wikipedia_dv.get_protection(the_page['title'])\n",
    "\n",
    "        # Use ConflictManager to join content and revision tables.\n",
    "        cm = ConflictManager(content,\n",
    "                             revisions, \n",
    "                             lng=abbreviation(languageSelection.value), \n",
    "                             include_stopwords=True)\n",
    "        cm.calculate()\n",
    "        clear_output()\n",
    "        token_source = cm.all_actions.copy()\n",
    "        \n",
    "        display(md(f\"***Page: {the_page['title']} ({abbreviation(languageSelection.value).upper()})***\"))\n",
    "\n",
    "        # Templateslistener\n",
    "        new_protect = ProtectListener(pp_log, lng=abbreviation(languageSelection.value))\n",
    "        \n",
    "        display(md(\"Analysing protection data...\"))\n",
    "        plot_protect = [new_protect.get_protect(level)[1] for level in [\"semi_edit\", \"semi_move\", \"fully_edit\", \"fully_move\", \"unknown\"]]\n",
    "        plot_protect = pd.concat(plot_protect)\n",
    "\n",
    "        new_template = TemplateListener(token_source, plot_protect, lng=abbreviation(languageSelection.value),\n",
    "                                       wikipediadv_api=wikipedia_dv, page=the_page)\n",
    "        new_template.listen()\n",
    "        \n",
    "        \n",
    "# create and display the button    \n",
    "button = widgets.Button(description=\"Get Templates & Protection\", layout=Layout(width='200px'))\n",
    "display(button)\n",
    "\n",
    "# the output widget is used to remove the output after the search field\n",
    "template_out = Output()\n",
    "display(template_out)\n",
    "\n",
    "# set the event\n",
    "button.on_click(template_button)\n",
    "\n",
    "# trigger the event with the default value\n",
    "template_button(button)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "\n",
    "Now that we have a base understanding of the historical context of the article, we will explore  specific kinds of changes that have affected the articles, and the editors involved in its evolution. \n",
    "\n",
    "Click below to go to the next notebook . This notebook stays open in its own tab and you can always come back to it. Or simply enter another article name to start the process over with a new article. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Go to next workbook](./2_Page_analysis.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hideOutput": false,
    "hidePrompt": false,
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "# from utils.notebooks import get_next_notebook\n",
    "# display(HTML(f'<a href=\"{get_next_notebook()}\" target=\"_blank\">Go to next workbook</a>'))\n",
    "\n",
    "\n",
    "\n",
    "scroll_to_top = \"\"\"\n",
    "<script>\n",
    "document.getElementById('notebook').scrollIntoView();\n",
    "</script>\n",
    "\"\"\"\n",
    "display(HTML(scroll_to_top))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Hide code",
  "hide_code_all_hidden": false,
  "hide_input": false,
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "nteract": {
   "version": "0.14.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}