6. Find all revisions and context involving actions on a word.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding all revisions involving a token\n",
    "\n",
    "These notebook shows how to find all the revisions that involve a given word in a document. In practice, this means, all the times that word was inserted, reinserted or removed. It also locates the left and right context of the word in that given revision. \n",
    "\n",
    "\n",
    "# 1. Select a page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "article = \"John_Logie_Baird\"\n",
    "\n",
    "# This global variable will cache the revisions tokens to avoid downloading the revisions multiple times.\n",
    "# It is particularly useful to find the information associated to the context.\n",
    "revisions_cache = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Merge all tokens with revisions (enriching the dataframe)\n",
    "\n",
    "Here, the ins and outs (columns `in` and `out` containing revisions id of insertions and deletions) of all_tokens of an article are merged with the revision metadata in order to also have the revision timestamp and the editor of the revision. \n",
    "\n",
    "Note that the merge happens twice, one for the ins and one for the outs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from wikiwho_wrapper import WikiWho\n",
    "from os.path import exists\n",
    "\n",
    "# this will print all the cases\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "# wikiwho instance\n",
    "ww = WikiWho()\n",
    "\n",
    "# get all the content\n",
    "df = ww.dv.all_content(f\"{article}\")\n",
    "\n",
    "# drop useless colummns\n",
    "df = df.drop(columns=['article_title', 'page_id']) \n",
    "\n",
    "# by default all insertions are consider reinsertions\n",
    "df['action'] = 'rein'\n",
    "\n",
    "# fix the above line, first insertions should be in\n",
    "df.loc[df['in']==-1, 'action'] = 'in'\n",
    "\n",
    "# place the o_rev_id as the real insertion\n",
    "df.loc[df['in']==-1, 'in'] = df.loc[df['in']==-1, 'o_rev_id']\n",
    "\n",
    "# get all the revision data\n",
    "revisions = ww.dv.rev_ids_of_article(f'{article}')\n",
    "\n",
    "# merge the in revisions ids with the revisions\n",
    "df = pd.merge(df,\n",
    "              revisions.rename(\n",
    "                  columns={\n",
    "                      'rev_id': 'in',\n",
    "                      'o_editor': 'in_editor',\n",
    "                      'rev_time': 'in_rev_time'\n",
    "                  })[['in', 'in_editor', 'in_rev_time']],\n",
    "              how='left', on='in')\n",
    "\n",
    "# merge the out revisions ids with the revisions\n",
    "df = pd.merge(df,\n",
    "              revisions.rename(\n",
    "                  columns={\n",
    "                      'rev_id': 'out',\n",
    "                      'o_editor': 'out_editor',\n",
    "                      'rev_time': 'out_rev_time'\n",
    "                  })[['out', 'out_editor', 'out_rev_time']],\n",
    "              how='left', on='out')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also important to keep a global variable revision history for the auxiliary methods defined below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sort the revisions\n",
    "history = revisions.sort_values('rev_time')\n",
    "\n",
    "history.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Define auxiliary methods\n",
    "\n",
    "These methods do the heavy lifting of the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import time\n",
    "\n",
    "def get_revision(article_title, rev_id, attempts=10, delay=10):\n",
    "    \"\"\" The server has a limit in the amount of requests per minute, therefore a delay \"\"\"\n",
    "    for attempt in range(0, attempts + 1):\n",
    "        try:\n",
    "            return ww.dv.specific_rev_content_by_rev_id(\n",
    "                article_title=article_title, rev_id=rev_id, out=False, _in=False)\n",
    "        except Exception as exc:\n",
    "            if attempt == attempts:\n",
    "                raise exc\n",
    "            else:\n",
    "                print(f\"Exhausted amoung of requests. Waiting {delay} seconds for the server to \"\n",
    "                      \"allow more requests...\")\n",
    "                time.sleep(delay)\n",
    "\n",
    "    \n",
    "def get_rev_history(df, article, rev_id):\n",
    "    \"\"\"Get the actions performed until certain revision\"\"\"\n",
    "    \n",
    "    _df = get_revision(article_title=article, rev_id=rev_id)\n",
    "    _df = pd.merge(\n",
    "        _df, df[['token_id', 'in', 'out']], \n",
    "        how='left', on='token_id')\n",
    "    _df = _df[(_df['in'] <= _df['rev_id'])]\n",
    "    _df.loc[_df['out'] > _df['rev_id'],'out'] = -1\n",
    "    return _df\n",
    "\n",
    "def create_download_link(df, filename, title = \"Download CSV file\", ):\n",
    "    \"\"\"Create csv and link to download the processed filed\"\"\"\n",
    "    df.to_csv(filename, quoting=csv.QUOTE_NONNUMERIC)#, sep = ';')\n",
    "    html = f'<a href=\"{filename}\" target=\"_blank\">{title}</a>'\n",
    "    return HTML(html)\n",
    "\n",
    "def vec_dt_replace(series, year=None, month=None, day=None):\n",
    "    \"\"\"This will generate a date from the above parameters. It is useful to create\n",
    "    indexes by month\"\"\"\n",
    "    return pd.to_datetime(\n",
    "        {'year': series.dt.year if year is None else year,\n",
    "         'month': series.dt.month if month is None else month,\n",
    "         'day': series.dt.day if day is None else day})\n",
    "\n",
    "def get_in_contexts(revid, tokenid):\n",
    "    \"\"\"get the left and right context of insertions and reinsertions\"\"\"\n",
    "    global revisions\n",
    "    \n",
    "    tins = len(df[df['in']==revid])\n",
    "    touts = len(df[df['out']==revid])\n",
    "    \n",
    "    try:\n",
    "        _df = revisions_cache[revid]\n",
    "    except:\n",
    "        print(f'downloading revision {revid}')\n",
    "        _df = revisions_cache[revid] = get_revision(article_title=article, rev_id=revid)\n",
    "        \n",
    "    idx = _df[_df['token_id'] == tokenid].index[0]\n",
    "    \n",
    "    return tins, touts, revid, idx, ' '.join(_df.iloc[idx - 15 : idx ]['token']), ' '.join(_df.iloc[idx + 1 : idx + 15]['token'])\n",
    "\n",
    "    \n",
    "def get_out_contexts(revid, tokenid):\n",
    "    \"\"\"get the left and right context of removes\"\"\"\n",
    "    global revisions\n",
    "    \n",
    "    tins = len(df[df['in']==revid])\n",
    "    touts = len(df[df['out']==revid])\n",
    "\n",
    "    try:\n",
    "        revid = history[history.shift(-1)['rev_id'] == revid]['rev_id'].iloc[0]\n",
    "    except Exception as e:\n",
    "        print('error')\n",
    "        print(revid)\n",
    "        print('error')\n",
    "        raise e\n",
    "    try:\n",
    "        _df = revisions_cache[revid]\n",
    "    except:\n",
    "        print(f'downloading revision {revid}')\n",
    "        _df = revisions_cache[revid] = get_revision(article_title=article, rev_id=revid)\n",
    "        \n",
    "    idx = _df[_df['token_id'] == tokenid].index[0]\n",
    "    \n",
    "    return tins, touts, revid, idx, ' '.join(_df.iloc[idx - 15 : idx ]['token']), ' '.join(_df.iloc[idx + 1 : idx + 15]['token'])\n",
    "    \n",
    "\n",
    "def search_all_tokens(df, tokens, i_o=['in','out'], with_context=False):\n",
    "    \"\"\"Search all times that actions were performed in the document\"\"\"\n",
    "    \n",
    "    # make a recursive call\n",
    "    if not isinstance(i_o, str):\n",
    "        return pd.concat((search_all_tokens(df, tokens, i_o=io, with_context=with_context) for io in i_o), axis=0)\n",
    "\n",
    "    ltokens = [token.lower() for token in tokens]\n",
    "    \n",
    "    io_sel=None\n",
    "    coi = [f'{i_o}', 'token_id', f'{i_o}_rev_time', f'{i_o}_editor', 'token', 'action']\n",
    "    if i_o == 'out':\n",
    "        _df = df.loc[(df['token'].isin(ltokens)) & (df[f'{i_o}'] != -1), coi]\n",
    "        _df['action'] = 'out'\n",
    "    elif i_o == 'in':\n",
    "        _df = df.loc[(df['token'].isin(ltokens)), coi]\n",
    "        \n",
    "    _df = _df.rename(columns = {\n",
    "        f'{i_o}': 'rev_id', \n",
    "        f'{i_o}_rev_time': 'rev_time', \n",
    "        f'{i_o}_editor': 'editor',\n",
    "    })\n",
    "\n",
    "    _df[f'rev_time']= pd.to_datetime(_df[f'rev_time'])\n",
    "    _df = _df.sort_values(['token', f'rev_time','token_id'], ascending=True)\n",
    "    _df['date'] = vec_dt_replace(_df[f'rev_time'], day=1)\n",
    "    _df['duplicated'] = _df.duplicated(subset=[f'rev_id'], keep=False)\n",
    "    \n",
    "\n",
    "    if len(_df) > 0 and with_context:\n",
    "        if i_o == 'in':\n",
    "            _df['in'], _df['out'], _df['revid_ctxt'], _df['pos'], _df['left'], _df['right'] = zip(*_df[[f'rev_id', 'token_id']].apply(\n",
    "                lambda x: get_in_contexts(revid = x[f'rev_id'], tokenid = x['token_id']), axis=1))\n",
    "        elif i_o == 'out':\n",
    "            _df['in'], _df['out'], _df['revid_ctxt'], _df['pos'], _df['left'], _df['right'] = zip(*_df[[f'rev_id', 'token_id']].apply(\n",
    "                lambda x: get_out_contexts(x[f'rev_id'], tokenid = x['token_id']), axis=1))\n",
    "    else:\n",
    "        _df['left'] = ''\n",
    "        _df['right'] = ''\n",
    "        _df['pos'] = -1\n",
    "        _df['revid_ctxt'] = -1\n",
    "\n",
    "    return _df.set_index(['date', f'rev_time'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Select the tokens that will be searched\n",
    "\n",
    "The nationality of John Loggie Baird figures among the the list of [Wikipedia Lamest Edit Wars](https://en.wikipedia.org/wiki/Wikipedia:Lamest_edit_wars). We can search for tokens in order to find all the times it has been found and in which context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = [\"Jahrhundert\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Query all revisions that inserted, reinsert or removed tokens\n",
    "\n",
    "This will take several minutes as all the revisions related to the tokens need to be downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_df = search_all_tokens(df, tokens, with_context=True)\n",
    "\n",
    "from IPython.core.display import display, HTML, clear_output\n",
    "clear_output()\n",
    "\n",
    "display(create_download_link(_df, f'data/{article} {str(tokens)}.csv'))\n",
    "display(_df.head())\n",
    "display(_df.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "from utils.notebooks import get_notebook_by_number\n",
    "\n",
    "display(HTML(f'<a href=\"{get_notebook_by_number(1)}\" target=\"_blank\">Go to first workbook</a>'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}