Analysing_OCR_corrections.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Corrections of OCRd text in Trove's newspapers\n",
    "\n",
    "The full text of newspaper articles in Trove is extracted from page images using Optical Character Recognition (OCR). The accuracy of the OCR process is influenced by a range of factors including the font and the quality of the images. Many errors slip through. Volunteers have done a remarkable job in correcting these errors, but it's a huge task. This notebook explores the scale of OCR correction in Trove.\n",
    "\n",
    "There are two ways of getting data about OCR corrections using the Trove API. To get aggregate data you can include `has:corrections` in your query to limit the results to articles that have at least one OCR correction.\n",
    "\n",
    "To get information about the number of corrections made to the articles in your results, you can add the `reclevel=full` parameter to include the number of corrections and details of the most recent correction to the article record. For example, note the `correctionCount` and `lastCorrection` values in the record below:\n",
    "\n",
    "``` json\n",
    "{\n",
    "    \"article\": {\n",
    "        \"id\": \"41697877\",\n",
    "        \"url\": \"/newspaper/41697877\",\n",
    "        \"heading\": \"WRAGGE AND WEATHER CYCLES.\",\n",
    "        \"category\": \"Article\",\n",
    "        \"title\": {\n",
    "            \"id\": \"101\",\n",
    "            \"value\": \"Western Mail (Perth, WA : 1885 - 1954)\"\n",
    "        },\n",
    "        \"date\": \"1922-11-23\",\n",
    "        \"page\": 4,\n",
    "        \"pageSequence\": 4,\n",
    "        \"troveUrl\": \"https://trove.nla.gov.au/ndp/del/article/41697877\",\n",
    "        \"illustrated\": \"N\",\n",
    "        \"wordCount\": 1054,\n",
    "        \"correctionCount\": 1,\n",
    "        \"listCount\": 0,\n",
    "        \"tagCount\": 0,\n",
    "        \"commentCount\": 0,\n",
    "        \"lastCorrection\": {\n",
    "            \"by\": \"*anon*\",\n",
    "            \"lastupdated\": \"2016-09-12T07:08:57Z\"\n",
    "        },\n",
    "        \"identifier\": \"https://nla.gov.au/nla.news-article41697877\",\n",
    "        \"trovePageUrl\": \"https://trove.nla.gov.au/ndp/del/page/3522839\",\n",
    "        \"pdf\": \"https://trove.nla.gov.au/ndp/imageservice/nla.news-page3522839/print\"\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setting things up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from operator import itemgetter  # used for sorting\n",
    "\n",
    "import altair as alt\n",
    "import pandas as pd  # makes manipulating the data easier\n",
    "import requests\n",
    "from IPython.display import FileLink, clear_output, display\n",
    "from requests.adapters import HTTPAdapter\n",
    "from requests.packages.urllib3.util.retry import Retry\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "# Make sure data directory exists\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "# Create a session that will automatically retry on server errors\n",
    "s = requests.Session()\n",
    "retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504])\n",
    "s.mount(\"http://\", HTTPAdapter(max_retries=retries))\n",
    "s.mount(\"https://\", HTTPAdapter(max_retries=retries))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Load variables from the .env file if it exists\n",
    "# Use %%capture to suppress messages\n",
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Insert your Trove API key\n",
    "API_KEY = \"YOUR API KEY\"\n",
    "\n",
    "# Use api key value from environment variables if it is available\n",
    "if os.getenv(\"TROVE_API_KEY\"):\n",
    "    API_KEY = os.getenv(\"TROVE_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Basic parameters for Trove API\n",
    "params = {\n",
    "    \"facet\": \"year\",  # Get the data aggregated by year.\n",
    "    \"zone\": \"newspaper\",\n",
    "    \"key\": API_KEY,\n",
    "    \"encoding\": \"json\",\n",
    "    \"n\": 0,  # We don't need any records, just the facets!\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_results(params):\n",
    "    \"\"\"\n",
    "    Get JSON response data from the Trove API.\n",
    "    Parameters:\n",
    "        params\n",
    "    Returns:\n",
    "        JSON formatted response data from Trove API\n",
    "    \"\"\"\n",
    "    response = s.get(\n",
    "        \"https://api.trove.nla.gov.au/v2/result\", params=params, timeout=30\n",
    "    )\n",
    "    response.raise_for_status()\n",
    "    # print(response.url) # This shows us the url that's sent to the API\n",
    "    data = response.json()\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How many newspaper articles have corrections?\n",
    "\n",
    "Let's find out what proportion of newspaper articles have at least one OCR correction.\n",
    "\n",
    "First we'll get to the total number of newspaper articles in Trove."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "233,635,348\n"
     ]
    }
   ],
   "source": [
    "# Set the q parameter to a single space to get everything\n",
    "params[\"q\"] = \" \"\n",
    "\n",
    "# Get the data from the API\n",
    "data = get_results(params)\n",
    "\n",
    "# Extract the total number of results\n",
    "total = int(data[\"response\"][\"zone\"][0][\"records\"][\"total\"])\n",
    "print(\"{:,}\".format(total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll set the `q` parameter to `has:corrections` to limit the results to newspaper articles that have at least one correction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13,585,979\n"
     ]
    }
   ],
   "source": [
    "# Set the q parameter to 'has:corrections' to limit results to articles with corrections\n",
    "params[\"q\"] = \"has:corrections\"\n",
    "\n",
    "# Get the data from the API\n",
    "data = get_results(params)\n",
    "\n",
    "# Extract the total number of results\n",
    "corrected = int(data[\"response\"][\"zone\"][0][\"records\"][\"total\"])\n",
    "print(\"{:,}\".format(corrected))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the proportion of articles with corrections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.82% of articles have at least one correction\n"
     ]
    }
   ],
   "source": [
    "print(\"{:.2%} of articles have at least one correction\".format(corrected / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be thinking that these figures don't seem to match the number of corrections by individuals displayed on the digitised newspapers home page. Remember that these figures show the **number of articles** that include corrections, while the individual scores show the **number of lines** corrected by each volunteer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of corrections by year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_facets(data):\n",
    "    \"\"\"\n",
    "    Loop through facets in Trove API response, saving terms and counts.\n",
    "    Parameters:\n",
    "        data  - JSON formatted response data from Trove API\n",
    "    Returns:\n",
    "        A list of dictionaries containing: 'term', 'total_results'\n",
    "    \"\"\"\n",
    "    facets = []\n",
    "    try:\n",
    "        # The facets are buried a fair way down in the results\n",
    "        # Note that if you ask for more than one facet, you'll have use the facet['name'] param to find the one you want\n",
    "        # In this case there's only one facet, so we can just grab the list of terms (which are in fact the results by year)\n",
    "        for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "\n",
    "            # Get the year and the number of results, and convert them to integers, before adding to our results\n",
    "            facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "\n",
    "        # Sort facets by year\n",
    "        facets.sort(key=itemgetter(\"term\"))\n",
    "    except TypeError:\n",
    "        pass\n",
    "    return facets\n",
    "\n",
    "\n",
    "def get_facet_data(params, start_decade=180, end_decade=201):\n",
    "    \"\"\"\n",
    "    Loop throught the decades from 'start_decade' to 'end_decade',\n",
    "    getting the number of search results for each year from the year facet.\n",
    "    Combine all the results into a single list.\n",
    "    Parameters:\n",
    "        params - parameters to send to the API\n",
    "        start_decade\n",
    "        end_decade\n",
    "    Returns:\n",
    "        A list of dictionaries containing 'year', 'total_results' for the complete\n",
    "        period between the start and end decades.\n",
    "    \"\"\"\n",
    "    # Create a list to hold the facets data\n",
    "    facet_data = []\n",
    "\n",
    "    # Loop through the decades\n",
    "    for decade in tqdm(range(start_decade, end_decade + 1)):\n",
    "\n",
    "        # print(params)\n",
    "        # Avoid confusion by copying the params before we change anything.\n",
    "        search_params = params.copy()\n",
    "\n",
    "        # Add decade value to params\n",
    "        search_params[\"l-decade\"] = decade\n",
    "\n",
    "        # Get the data from the API\n",
    "        data = get_results(search_params)\n",
    "\n",
    "        # Get the facets from the data and add to facets_data\n",
    "        facet_data += get_facets(data)\n",
    "\n",
    "    # Reomve the progress bar (you can also set leave=False in tqdm, but that still leaves white space in Jupyter Lab)\n",
    "    clear_output()\n",
    "    return facet_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "facet_data = get_facet_data(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert our data to a dataframe called df\n",
    "df = pd.DataFrame(facet_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1803</td>\n",
       "      <td>526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1804</td>\n",
       "      <td>619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1805</td>\n",
       "      <td>430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1806</td>\n",
       "      <td>367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1807</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   term  total_results\n",
       "0  1803            526\n",
       "1  1804            619\n",
       "2  1805            430\n",
       "3  1806            367\n",
       "4  1807            134"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So which year has the most corrections?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "term               1915\n",
       "total_results    270277\n",
       "Name: 112, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df[\"total_results\"].idxmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fact that there's more corrections in newspaper articles from 1915, might make you think that people have been more motivated to correct articles relating to WWI. But if you look at the [total number of articles per year](visualise-total-newspaper-articles-by-state-year.ipynb), you'll see that there's been more articles digitised from 1915! The raw number of corrections is probably not very useful, so let's look instead at the *proportion* of articles each year that have at least one correction.\n",
    "\n",
    "To do that we'll re-harvest the facet data, but this time with a blank, or empty search, to get the total number of articles available from each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the 'q' parameter\n",
    "# Use a an empty search (a single space) to get ALL THE ARTICLES\n",
    "params[\"q\"] = \" \"\n",
    "\n",
    "# Get facet data for all articles\n",
    "all_facet_data = get_facet_data(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the results to a dataframe\n",
    "df_total = pd.DataFrame(all_facet_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No we'll merge the number of articles by year with corrections with the total number of articles. Then we'll calculate the proportion with corrections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_df_with_total(df, df_total, how=\"left\"):\n",
    "    \"\"\"\n",
    "    Merge dataframes containing search results with the total number of articles by year.\n",
    "    This is a left join on the year column. The total number of articles will be added as a column to\n",
    "    the existing results.\n",
    "    Once merged, do some reorganisation and calculate the proportion of search results.\n",
    "    Parameters:\n",
    "        df - the search results in a dataframe\n",
    "        df_total - total number of articles per year in a dataframe\n",
    "    Returns:\n",
    "        A dataframe with the following columns - 'year', 'total_results', 'total_articles', 'proportion'\n",
    "        (plus any other columns that are in the search results dataframe).\n",
    "    \"\"\"\n",
    "    # Merge the two dataframes on year\n",
    "    # Note that we're joining the two dataframes on the year column\n",
    "    df_merged = pd.merge(df, df_total, how=how, on=\"term\")\n",
    "\n",
    "    # Rename the columns for convenience\n",
    "    df_merged.rename(\n",
    "        {\"total_results_y\": \"total_articles\"}, inplace=True, axis=\"columns\"\n",
    "    )\n",
    "    df_merged.rename({\"total_results_x\": \"total_results\"}, inplace=True, axis=\"columns\")\n",
    "\n",
    "    # Set blank values to zero to avoid problems\n",
    "    df_merged[\"total_results\"] = df_merged[\"total_results\"].fillna(0).astype(int)\n",
    "\n",
    "    # Calculate proportion by dividing the search results by the total articles\n",
    "    df_merged[\"proportion\"] = df_merged[\"total_results\"] / df_merged[\"total_articles\"]\n",
    "    return df_merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1803</td>\n",
       "      <td>526</td>\n",
       "      <td>526</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1804</td>\n",
       "      <td>619</td>\n",
       "      <td>619</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1805</td>\n",
       "      <td>430</td>\n",
       "      <td>430</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1806</td>\n",
       "      <td>367</td>\n",
       "      <td>367</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1807</td>\n",
       "      <td>134</td>\n",
       "      <td>134</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   term  total_results  total_articles  proportion\n",
       "0  1803            526             526         1.0\n",
       "1  1804            619             619         1.0\n",
       "2  1805            430             430         1.0\n",
       "3  1806            367             367         1.0\n",
       "4  1807            134             134         1.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Merge the search results with the total articles\n",
    "df_merged = merge_df_with_total(df, df_total)\n",
    "df_merged.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualise the results, showing both the number of articles with corrections each year, and the proportion of articles each year with corrections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-da3b2819245c4b34933dcbb87fdf2942\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-da3b2819245c4b34933dcbb87fdf2942\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-da3b2819245c4b34933dcbb87fdf2942\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"vconcat\": [{\"mark\": {\"type\": \"line\", \"point\": true}, \"encoding\": {\"tooltip\": [{\"type\": \"quantitative\", \"field\": \"term\", \"title\": \"Year\"}, {\"type\": \"quantitative\", \"field\": \"total_results\", \"format\": \",\", \"title\": \"Articles\"}], \"x\": {\"type\": \"quantitative\", \"axis\": {\"format\": \"c\", \"title\": \"Year\"}, \"field\": \"term\"}, \"y\": {\"type\": \"quantitative\", \"axis\": {\"format\": \",d\", \"title\": \"Number of articles with corrections\"}, \"field\": \"total_results\"}}, \"height\": 250, \"width\": 700}, {\"mark\": {\"type\": \"line\", \"color\": \"red\", \"point\": true}, \"encoding\": {\"color\": {\"value\": \"orange\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"term\", \"title\": \"Year\"}, {\"type\": \"quantitative\", \"field\": \"proportion\", \"format\": \"%\", \"title\": \"Proportion\"}], \"x\": {\"type\": \"quantitative\", \"axis\": {\"format\": \"c\", \"title\": \"Year\"}, \"field\": \"term\"}, \"y\": {\"type\": \"quantitative\", \"axis\": {\"format\": \"%\", \"title\": \"Proportion of articles with corrections\"}, \"field\": \"proportion\"}}, \"height\": 250, \"width\": 700}], \"data\": {\"name\": \"data-43ec5e7502630b0a75250de715b20249\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-43ec5e7502630b0a75250de715b20249\": [{\"term\": 1803, \"total_results\": 526, \"total_articles\": 526, \"proportion\": 1.0}, {\"term\": 1804, \"total_results\": 619, \"total_articles\": 619, \"proportion\": 1.0}, {\"term\": 1805, \"total_results\": 430, \"total_articles\": 430, \"proportion\": 1.0}, {\"term\": 1806, \"total_results\": 367, \"total_articles\": 367, \"proportion\": 1.0}, {\"term\": 1807, \"total_results\": 134, \"total_articles\": 134, \"proportion\": 1.0}, {\"term\": 1808, \"total_results\": 155, \"total_articles\": 155, \"proportion\": 1.0}, {\"term\": 1809, \"total_results\": 237, \"total_articles\": 237, \"proportion\": 1.0}, {\"term\": 1810, \"total_results\": 274, \"total_articles\": 274, \"proportion\": 1.0}, {\"term\": 1811, \"total_results\": 228, \"total_articles\": 228, \"proportion\": 1.0}, {\"term\": 1812, \"total_results\": 215, \"total_articles\": 215, \"proportion\": 1.0}, {\"term\": 1813, \"total_results\": 234, \"total_articles\": 234, \"proportion\": 1.0}, {\"term\": 1814, \"total_results\": 239, \"total_articles\": 239, \"proportion\": 1.0}, {\"term\": 1815, \"total_results\": 226, \"total_articles\": 226, \"proportion\": 1.0}, {\"term\": 1816, \"total_results\": 644, \"total_articles\": 644, \"proportion\": 1.0}, {\"term\": 1817, \"total_results\": 1067, \"total_articles\": 1068, \"proportion\": 0.9990636704119851}, {\"term\": 1818, \"total_results\": 1249, \"total_articles\": 1249, \"proportion\": 1.0}, {\"term\": 1819, \"total_results\": 1152, \"total_articles\": 1152, \"proportion\": 1.0}, {\"term\": 1820, \"total_results\": 1275, \"total_articles\": 1275, \"proportion\": 1.0}, {\"term\": 1821, \"total_results\": 1005, \"total_articles\": 1005, \"proportion\": 1.0}, {\"term\": 1822, \"total_results\": 1033, \"total_articles\": 1033, \"proportion\": 1.0}, {\"term\": 1823, \"total_results\": 1156, \"total_articles\": 1156, \"proportion\": 1.0}, {\"term\": 1824, \"total_results\": 1646, \"total_articles\": 1646, \"proportion\": 1.0}, {\"term\": 1825, \"total_results\": 3366, \"total_articles\": 3366, \"proportion\": 1.0}, {\"term\": 1826, \"total_results\": 4595, \"total_articles\": 5201, \"proportion\": 0.8834839453951163}, {\"term\": 1827, \"total_results\": 4587, \"total_articles\": 6618, \"proportion\": 0.6931097008159565}, {\"term\": 1828, \"total_results\": 4207, \"total_articles\": 7335, \"proportion\": 0.5735514655760054}, {\"term\": 1829, \"total_results\": 5034, \"total_articles\": 8935, \"proportion\": 0.5634023503077784}, {\"term\": 1830, \"total_results\": 4715, \"total_articles\": 8977, \"proportion\": 0.5252311462626713}, {\"term\": 1831, \"total_results\": 5344, \"total_articles\": 10989, \"proportion\": 0.4863044863044863}, {\"term\": 1832, \"total_results\": 6242, \"total_articles\": 14814, \"proportion\": 0.42135817469960846}, {\"term\": 1833, \"total_results\": 6993, \"total_articles\": 15622, \"proportion\": 0.44763794648572525}, {\"term\": 1834, \"total_results\": 7822, \"total_articles\": 18704, \"proportion\": 0.4181993156544055}, {\"term\": 1835, \"total_results\": 10079, \"total_articles\": 20389, \"proportion\": 0.4943351807347099}, {\"term\": 1836, \"total_results\": 9371, \"total_articles\": 20553, \"proportion\": 0.4559431713131903}, {\"term\": 1837, \"total_results\": 9663, \"total_articles\": 21805, \"proportion\": 0.4431552396239395}, {\"term\": 1838, \"total_results\": 10827, \"total_articles\": 25911, \"proportion\": 0.4178534213268496}, {\"term\": 1839, \"total_results\": 13919, \"total_articles\": 32876, \"proportion\": 0.423378756539725}, {\"term\": 1840, \"total_results\": 16977, \"total_articles\": 42411, \"proportion\": 0.4002970927353753}, {\"term\": 1841, \"total_results\": 16301, \"total_articles\": 46377, \"proportion\": 0.3514888845764064}, {\"term\": 1842, \"total_results\": 17490, \"total_articles\": 47934, \"proportion\": 0.36487670547002127}, {\"term\": 1843, \"total_results\": 15461, \"total_articles\": 49913, \"proportion\": 0.30975898062628976}, {\"term\": 1844, \"total_results\": 14267, \"total_articles\": 54331, \"proportion\": 0.2625940991330916}, {\"term\": 1845, \"total_results\": 15951, \"total_articles\": 63389, \"proportion\": 0.25163671930461123}, {\"term\": 1846, \"total_results\": 15230, \"total_articles\": 62153, \"proportion\": 0.24504046465979115}, {\"term\": 1847, \"total_results\": 15978, \"total_articles\": 62767, \"proportion\": 0.25456051746937086}, {\"term\": 1848, \"total_results\": 17165, \"total_articles\": 63463, \"proportion\": 0.270472558813797}, {\"term\": 1849, \"total_results\": 20081, \"total_articles\": 62794, \"proportion\": 0.31979169984393413}, {\"term\": 1850, \"total_results\": 21281, \"total_articles\": 81624, \"proportion\": 0.2607198863079486}, {\"term\": 1851, \"total_results\": 23111, \"total_articles\": 96566, \"proportion\": 0.23932854213698404}, {\"term\": 1852, \"total_results\": 21177, \"total_articles\": 79960, \"proportion\": 0.2648449224612306}, {\"term\": 1853, \"total_results\": 26346, \"total_articles\": 96033, \"proportion\": 0.2743431945268814}, {\"term\": 1854, \"total_results\": 30053, \"total_articles\": 118883, \"proportion\": 0.25279476460048955}, {\"term\": 1855, \"total_results\": 34459, \"total_articles\": 159947, \"proportion\": 0.21544011453794068}, {\"term\": 1856, \"total_results\": 37163, \"total_articles\": 168695, \"proportion\": 0.220296985684223}, {\"term\": 1857, \"total_results\": 39085, \"total_articles\": 175343, \"proportion\": 0.2229059614583987}, {\"term\": 1858, \"total_results\": 41241, \"total_articles\": 178765, \"proportion\": 0.2306995217184572}, {\"term\": 1859, \"total_results\": 47499, \"total_articles\": 202996, \"proportion\": 0.23398983231196674}, {\"term\": 1860, \"total_results\": 46600, \"total_articles\": 214554, \"proportion\": 0.21719473885362192}, {\"term\": 1861, \"total_results\": 47009, \"total_articles\": 237976, \"proportion\": 0.1975367263925774}, {\"term\": 1862, \"total_results\": 43544, \"total_articles\": 244694, \"proportion\": 0.17795287175002247}, {\"term\": 1863, \"total_results\": 47375, \"total_articles\": 251706, \"proportion\": 0.18821561663210254}, {\"term\": 1864, \"total_results\": 47636, \"total_articles\": 264282, \"proportion\": 0.18024685752340303}, {\"term\": 1865, \"total_results\": 50562, \"total_articles\": 283302, \"proportion\": 0.1784738547557024}, {\"term\": 1866, \"total_results\": 49534, \"total_articles\": 289342, \"proportion\": 0.17119533285869318}, {\"term\": 1867, \"total_results\": 50044, \"total_articles\": 297136, \"proportion\": 0.1684211943352539}, {\"term\": 1868, \"total_results\": 48929, \"total_articles\": 304914, \"proportion\": 0.16046819759014017}, {\"term\": 1869, \"total_results\": 51071, \"total_articles\": 327192, \"proportion\": 0.1560887796767647}, {\"term\": 1870, \"total_results\": 57119, \"total_articles\": 373286, \"proportion\": 0.1530167217629378}, {\"term\": 1871, \"total_results\": 49769, \"total_articles\": 359781, \"proportion\": 0.1383313738079554}, {\"term\": 1872, \"total_results\": 52343, \"total_articles\": 383784, \"proportion\": 0.13638661330331645}, {\"term\": 1873, \"total_results\": 58912, \"total_articles\": 420951, \"proportion\": 0.1399497803782388}, {\"term\": 1874, \"total_results\": 58905, \"total_articles\": 436788, \"proportion\": 0.13485947416137806}, {\"term\": 1875, \"total_results\": 60267, \"total_articles\": 466891, \"proportion\": 0.1290815200978387}, {\"term\": 1876, \"total_results\": 61748, \"total_articles\": 477911, \"proportion\": 0.12920397312470314}, {\"term\": 1877, \"total_results\": 60858, \"total_articles\": 499944, \"proportion\": 0.12172963371897652}, {\"term\": 1878, \"total_results\": 66986, \"total_articles\": 569203, \"proportion\": 0.11768384917156094}, {\"term\": 1879, \"total_results\": 67002, \"total_articles\": 607530, \"proportion\": 0.11028591180682436}, {\"term\": 1880, \"total_results\": 72023, \"total_articles\": 624376, \"proportion\": 0.1153519674042564}, {\"term\": 1881, \"total_results\": 70072, \"total_articles\": 667930, \"proportion\": 0.10490919707154941}, {\"term\": 1882, \"total_results\": 74996, \"total_articles\": 745914, \"proportion\": 0.10054242178052698}, {\"term\": 1883, \"total_results\": 78288, \"total_articles\": 778865, \"proportion\": 0.1005154936991648}, {\"term\": 1884, \"total_results\": 83140, \"total_articles\": 858553, \"proportion\": 0.0968373530812891}, {\"term\": 1885, \"total_results\": 85033, \"total_articles\": 911515, \"proportion\": 0.09328754875125478}, {\"term\": 1886, \"total_results\": 88217, \"total_articles\": 994828, \"proportion\": 0.08867563036022308}, {\"term\": 1887, \"total_results\": 90595, \"total_articles\": 1043492, \"proportion\": 0.08681906521564133}, {\"term\": 1888, \"total_results\": 97213, \"total_articles\": 1162025, \"proportion\": 0.08365826897011683}, {\"term\": 1889, \"total_results\": 107796, \"total_articles\": 1252262, \"proportion\": 0.08608102777214353}, {\"term\": 1890, \"total_results\": 111188, \"total_articles\": 1357654, \"proportion\": 0.08189715494522168}, {\"term\": 1891, \"total_results\": 109089, \"total_articles\": 1429274, \"proportion\": 0.07632476348131989}, {\"term\": 1892, \"total_results\": 110097, \"total_articles\": 1486705, \"proportion\": 0.07405436855327721}, {\"term\": 1893, \"total_results\": 108298, \"total_articles\": 1510849, \"proportion\": 0.07168022747475095}, {\"term\": 1894, \"total_results\": 106858, \"total_articles\": 1560809, \"proportion\": 0.06846321362831712}, {\"term\": 1895, \"total_results\": 110764, \"total_articles\": 1678554, \"proportion\": 0.06598774897918089}, {\"term\": 1896, \"total_results\": 114251, \"total_articles\": 1798582, \"proportion\": 0.0635228196434747}, {\"term\": 1897, \"total_results\": 116699, \"total_articles\": 1918088, \"proportion\": 0.060841316978157416}, {\"term\": 1898, \"total_results\": 125481, \"total_articles\": 2141840, \"proportion\": 0.05858560863556568}, {\"term\": 1899, \"total_results\": 138664, \"total_articles\": 2308829, \"proportion\": 0.060058150690241674}, {\"term\": 1900, \"total_results\": 158551, \"total_articles\": 2539736, \"proportion\": 0.062428142137608}, {\"term\": 1901, \"total_results\": 158197, \"total_articles\": 2618191, \"proportion\": 0.06042225338029197}, {\"term\": 1902, \"total_results\": 152966, \"total_articles\": 2674429, \"proportion\": 0.05719576029126217}, {\"term\": 1903, \"total_results\": 145254, \"total_articles\": 2713889, \"proportion\": 0.0535224543081902}, {\"term\": 1904, \"total_results\": 143045, \"total_articles\": 2878677, \"proportion\": 0.04969122968641498}, {\"term\": 1905, \"total_results\": 147429, \"total_articles\": 3009076, \"proportion\": 0.04899477447561976}, {\"term\": 1906, \"total_results\": 154438, \"total_articles\": 3057500, \"proportion\": 0.05051120196238757}, {\"term\": 1907, \"total_results\": 169983, \"total_articles\": 3205575, \"proportion\": 0.05302730399382326}, {\"term\": 1908, \"total_results\": 169991, \"total_articles\": 3314151, \"proportion\": 0.05129247279318293}, {\"term\": 1909, \"total_results\": 168614, \"total_articles\": 3379982, \"proportion\": 0.0498860644820002}, {\"term\": 1910, \"total_results\": 179206, \"total_articles\": 3613826, \"proportion\": 0.04958899515361282}, {\"term\": 1911, \"total_results\": 174954, \"total_articles\": 3740413, \"proportion\": 0.04677397923705216}, {\"term\": 1912, \"total_results\": 182838, \"total_articles\": 3890753, \"proportion\": 0.0469929599745859}, {\"term\": 1913, \"total_results\": 183865, \"total_articles\": 3957073, \"proportion\": 0.04646489968721831}, {\"term\": 1914, \"total_results\": 246649, \"total_articles\": 4654167, \"proportion\": 0.05299530506748039}, {\"term\": 1915, \"total_results\": 270277, \"total_articles\": 4751447, \"proportion\": 0.056883092666297236}, {\"term\": 1916, \"total_results\": 246170, \"total_articles\": 4346277, \"proportion\": 0.056639280009074436}, {\"term\": 1917, \"total_results\": 227666, \"total_articles\": 4239223, \"proportion\": 0.053704652951731956}, {\"term\": 1918, \"total_results\": 224754, \"total_articles\": 4016436, \"proportion\": 0.05595856625127352}, {\"term\": 1919, \"total_results\": 183222, \"total_articles\": 3486517, \"proportion\": 0.052551586583401144}, {\"term\": 1920, \"total_results\": 157754, \"total_articles\": 3237978, \"proportion\": 0.048719911006189665}, {\"term\": 1921, \"total_results\": 149936, \"total_articles\": 3435873, \"proportion\": 0.04363839990593366}, {\"term\": 1922, \"total_results\": 151668, \"total_articles\": 3562135, \"proportion\": 0.042577836044956185}, {\"term\": 1923, \"total_results\": 170633, \"total_articles\": 3828641, \"proportion\": 0.04456751103067642}, {\"term\": 1924, \"total_results\": 179522, \"total_articles\": 4125328, \"proportion\": 0.04351702458568143}, {\"term\": 1925, \"total_results\": 178696, \"total_articles\": 4086958, \"proportion\": 0.04372347354682872}, {\"term\": 1926, \"total_results\": 178758, \"total_articles\": 4069533, \"proportion\": 0.04392592466998056}, {\"term\": 1927, \"total_results\": 189449, \"total_articles\": 4126770, \"proportion\": 0.04590733188425815}, {\"term\": 1928, \"total_results\": 190691, \"total_articles\": 4234586, \"proportion\": 0.045031792954494256}, {\"term\": 1929, \"total_results\": 213278, \"total_articles\": 4460722, \"proportion\": 0.04781243933157009}, {\"term\": 1930, \"total_results\": 198349, \"total_articles\": 4315190, \"proportion\": 0.04596529932633325}, {\"term\": 1931, \"total_results\": 175121, \"total_articles\": 3941724, \"proportion\": 0.04442751445814065}, {\"term\": 1932, \"total_results\": 175715, \"total_articles\": 3890710, \"proportion\": 0.04516270809183928}, {\"term\": 1933, \"total_results\": 189702, \"total_articles\": 4069854, \"proportion\": 0.04661150006855283}, {\"term\": 1934, \"total_results\": 201633, \"total_articles\": 4183589, \"proportion\": 0.048196177970637175}, {\"term\": 1935, \"total_results\": 190865, \"total_articles\": 4271382, \"proportion\": 0.04468460090902664}, {\"term\": 1936, \"total_results\": 187576, \"total_articles\": 4377987, \"proportion\": 0.042845261989128795}, {\"term\": 1937, \"total_results\": 182258, \"total_articles\": 4364890, \"proportion\": 0.04175546233696611}, {\"term\": 1938, \"total_results\": 188377, \"total_articles\": 4359675, \"proportion\": 0.04320895479594236}, {\"term\": 1939, \"total_results\": 193220, \"total_articles\": 4133499, \"proportion\": 0.04674490062777323}, {\"term\": 1940, \"total_results\": 178429, \"total_articles\": 3497071, \"proportion\": 0.05102241275627518}, {\"term\": 1941, \"total_results\": 154236, \"total_articles\": 3082401, \"proportion\": 0.05003761677990631}, {\"term\": 1942, \"total_results\": 147766, \"total_articles\": 2470102, \"proportion\": 0.05982182112317629}, {\"term\": 1943, \"total_results\": 132406, \"total_articles\": 2186357, \"proportion\": 0.0605601006605966}, {\"term\": 1944, \"total_results\": 126745, \"total_articles\": 2227377, \"proportion\": 0.05690325436600988}, {\"term\": 1945, \"total_results\": 159991, \"total_articles\": 2412554, \"proportion\": 0.06631602857386819}, {\"term\": 1946, \"total_results\": 138174, \"total_articles\": 2748052, \"proportion\": 0.05028070793420212}, {\"term\": 1947, \"total_results\": 134525, \"total_articles\": 2851317, \"proportion\": 0.0471799522817}, {\"term\": 1948, \"total_results\": 134290, \"total_articles\": 2610229, \"proportion\": 0.051447593295454154}, {\"term\": 1949, \"total_results\": 153636, \"total_articles\": 2867051, \"proportion\": 0.05358676912269785}, {\"term\": 1950, \"total_results\": 165339, \"total_articles\": 2812464, \"proportion\": 0.05878795248579182}, {\"term\": 1951, \"total_results\": 131555, \"total_articles\": 2554766, \"proportion\": 0.05149395287083044}, {\"term\": 1952, \"total_results\": 135417, \"total_articles\": 2608800, \"proportion\": 0.05190777368905244}, {\"term\": 1953, \"total_results\": 138799, \"total_articles\": 2755677, \"proportion\": 0.050368384974000945}, {\"term\": 1954, \"total_results\": 175423, \"total_articles\": 2787339, \"proportion\": 0.06293565296506812}, {\"term\": 1955, \"total_results\": 22806, \"total_articles\": 279909, \"proportion\": 0.08147647985595319}, {\"term\": 1956, \"total_results\": 18326, \"total_articles\": 199416, \"proportion\": 0.09189834316203313}, {\"term\": 1957, \"total_results\": 10072, \"total_articles\": 126201, \"proportion\": 0.07980919327105174}, {\"term\": 1958, \"total_results\": 8615, \"total_articles\": 126115, \"proportion\": 0.0683106688340007}, {\"term\": 1959, \"total_results\": 10530, \"total_articles\": 121259, \"proportion\": 0.08683891504960456}, {\"term\": 1960, \"total_results\": 8524, \"total_articles\": 115050, \"proportion\": 0.07408952629291612}, {\"term\": 1961, \"total_results\": 6943, \"total_articles\": 109239, \"proportion\": 0.06355788683528776}, {\"term\": 1962, \"total_results\": 7333, \"total_articles\": 108575, \"proportion\": 0.0675385678102694}, {\"term\": 1963, \"total_results\": 6201, \"total_articles\": 118702, \"proportion\": 0.052240063351923306}, {\"term\": 1964, \"total_results\": 6479, \"total_articles\": 120392, \"proportion\": 0.05381586816399761}, {\"term\": 1965, \"total_results\": 6715, \"total_articles\": 126432, \"proportion\": 0.05311155403695267}, {\"term\": 1966, \"total_results\": 6834, \"total_articles\": 123452, \"proportion\": 0.05535754787285747}, {\"term\": 1967, \"total_results\": 6543, \"total_articles\": 118503, \"proportion\": 0.05521379205589732}, {\"term\": 1968, \"total_results\": 6397, \"total_articles\": 124919, \"proportion\": 0.051209183550941006}, {\"term\": 1969, \"total_results\": 8896, \"total_articles\": 151689, \"proportion\": 0.05864630922479547}, {\"term\": 1970, \"total_results\": 10276, \"total_articles\": 166785, \"proportion\": 0.06161225529873789}, {\"term\": 1971, \"total_results\": 8859, \"total_articles\": 165077, \"proportion\": 0.05366586502056616}, {\"term\": 1972, \"total_results\": 7285, \"total_articles\": 164687, \"proportion\": 0.04423542841875801}, {\"term\": 1973, \"total_results\": 6720, \"total_articles\": 161814, \"proportion\": 0.04152916311320405}, {\"term\": 1974, \"total_results\": 7045, \"total_articles\": 159025, \"proportion\": 0.04430121050149348}, {\"term\": 1975, \"total_results\": 7437, \"total_articles\": 148204, \"proportion\": 0.05018083182640145}, {\"term\": 1976, \"total_results\": 6266, \"total_articles\": 148816, \"proportion\": 0.04210568756047737}, {\"term\": 1977, \"total_results\": 6424, \"total_articles\": 151194, \"proportion\": 0.04248845853671442}, {\"term\": 1978, \"total_results\": 6143, \"total_articles\": 159199, \"proportion\": 0.038586925797272596}, {\"term\": 1979, \"total_results\": 6682, \"total_articles\": 167101, \"proportion\": 0.03998779181453133}, {\"term\": 1980, \"total_results\": 7631, \"total_articles\": 162269, \"proportion\": 0.047026850476677616}, {\"term\": 1981, \"total_results\": 5953, \"total_articles\": 139980, \"proportion\": 0.04252750392913273}, {\"term\": 1982, \"total_results\": 9735, \"total_articles\": 130222, \"proportion\": 0.07475695351015957}, {\"term\": 1983, \"total_results\": 4970, \"total_articles\": 117491, \"proportion\": 0.0423011124256326}, {\"term\": 1984, \"total_results\": 5428, \"total_articles\": 118615, \"proportion\": 0.04576149728111959}, {\"term\": 1985, \"total_results\": 5802, \"total_articles\": 128586, \"proportion\": 0.04512155289067239}, {\"term\": 1986, \"total_results\": 5803, \"total_articles\": 128312, \"proportion\": 0.045225699856599536}, {\"term\": 1987, \"total_results\": 6322, \"total_articles\": 127591, \"proportion\": 0.049548949377307173}, {\"term\": 1988, \"total_results\": 7830, \"total_articles\": 136944, \"proportion\": 0.05717665615141956}, {\"term\": 1989, \"total_results\": 8523, \"total_articles\": 140206, \"proportion\": 0.06078912457384135}, {\"term\": 1990, \"total_results\": 9462, \"total_articles\": 132596, \"proportion\": 0.07135961869136324}, {\"term\": 1991, \"total_results\": 9248, \"total_articles\": 131729, \"proportion\": 0.07020473851619613}, {\"term\": 1992, \"total_results\": 10163, \"total_articles\": 137422, \"proportion\": 0.07395467974560115}, {\"term\": 1993, \"total_results\": 10509, \"total_articles\": 137978, \"proportion\": 0.07616431605038484}, {\"term\": 1994, \"total_results\": 11296, \"total_articles\": 133420, \"proportion\": 0.08466496777094888}, {\"term\": 1995, \"total_results\": 12252, \"total_articles\": 132425, \"proportion\": 0.09252029450632433}, {\"term\": 1996, \"total_results\": 1120, \"total_articles\": 47101, \"proportion\": 0.023778688350565805}, {\"term\": 1997, \"total_results\": 1393, \"total_articles\": 49822, \"proportion\": 0.02795953594797479}, {\"term\": 1998, \"total_results\": 1785, \"total_articles\": 56947, \"proportion\": 0.03134493476390328}, {\"term\": 1999, \"total_results\": 3028, \"total_articles\": 67005, \"proportion\": 0.04519065741362585}, {\"term\": 2000, \"total_results\": 2422, \"total_articles\": 44770, \"proportion\": 0.05409872682599955}, {\"term\": 2001, \"total_results\": 1887, \"total_articles\": 42081, \"proportion\": 0.04484209025450916}, {\"term\": 2002, \"total_results\": 1952, \"total_articles\": 42631, \"proportion\": 0.04578827613708334}, {\"term\": 2003, \"total_results\": 1702, \"total_articles\": 24075, \"proportion\": 0.0706957424714434}, {\"term\": 2004, \"total_results\": 1150, \"total_articles\": 22072, \"proportion\": 0.052102210945994924}, {\"term\": 2005, \"total_results\": 540, \"total_articles\": 18071, \"proportion\": 0.029882131592053565}, {\"term\": 2006, \"total_results\": 501, \"total_articles\": 17708, \"proportion\": 0.02829229726677208}, {\"term\": 2007, \"total_results\": 589, \"total_articles\": 18719, \"proportion\": 0.03146535605534484}, {\"term\": 2008, \"total_results\": 375, \"total_articles\": 17059, \"proportion\": 0.021982531215194327}, {\"term\": 2009, \"total_results\": 312, \"total_articles\": 5762, \"proportion\": 0.05414786532454009}, {\"term\": 2010, \"total_results\": 305, \"total_articles\": 5794, \"proportion\": 0.0526406627545737}, {\"term\": 2011, \"total_results\": 314, \"total_articles\": 5386, \"proportion\": 0.058299294467137025}, {\"term\": 2012, \"total_results\": 207, \"total_articles\": 4832, \"proportion\": 0.04283940397350993}, {\"term\": 2013, \"total_results\": 265, \"total_articles\": 4904, \"proportion\": 0.05403752039151713}, {\"term\": 2014, \"total_results\": 197, \"total_articles\": 4718, \"proportion\": 0.04175498092412039}, {\"term\": 2015, \"total_results\": 349, \"total_articles\": 4819, \"proportion\": 0.07242166424569413}, {\"term\": 2016, \"total_results\": 20, \"total_articles\": 1422, \"proportion\": 0.014064697609001406}, {\"term\": 2017, \"total_results\": 24, \"total_articles\": 1094, \"proportion\": 0.021937842778793418}, {\"term\": 2018, \"total_results\": 20, \"total_articles\": 1100, \"proportion\": 0.01818181818181818}, {\"term\": 2019, \"total_results\": 32, \"total_articles\": 1153, \"proportion\": 0.027753686036426712}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.VConcatChart(...)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of articles with corrections\n",
    "chart1 = (\n",
    "    alt.Chart(df_merged)\n",
    "    .mark_line(point=True)\n",
    "    .encode(\n",
    "        x=alt.X(\"term:Q\", axis=alt.Axis(format=\"c\", title=\"Year\")),\n",
    "        y=alt.Y(\n",
    "            \"total_results:Q\",\n",
    "            axis=alt.Axis(format=\",d\", title=\"Number of articles with corrections\"),\n",
    "        ),\n",
    "        tooltip=[\n",
    "            alt.Tooltip(\"term:Q\", title=\"Year\"),\n",
    "            alt.Tooltip(\"total_results:Q\", title=\"Articles\", format=\",\"),\n",
    "        ],\n",
    "    )\n",
    "    .properties(width=700, height=250)\n",
    ")\n",
    "\n",
    "# Proportion of articles with corrections\n",
    "chart2 = (\n",
    "    alt.Chart(df_merged)\n",
    "    .mark_line(point=True, color=\"red\")\n",
    "    .encode(\n",
    "        x=alt.X(\"term:Q\", axis=alt.Axis(format=\"c\", title=\"Year\")),\n",
    "        # This time we're showing the proportion (formatted as a percentage) on the Y axis\n",
    "        y=alt.Y(\n",
    "            \"proportion:Q\",\n",
    "            axis=alt.Axis(format=\"%\", title=\"Proportion of articles with corrections\"),\n",
    "        ),\n",
    "        tooltip=[\n",
    "            alt.Tooltip(\"term:Q\", title=\"Year\"),\n",
    "            alt.Tooltip(\"proportion:Q\", title=\"Proportion\", format=\"%\"),\n",
    "        ],\n",
    "        # Make the charts different colors\n",
    "        color=alt.value(\"orange\"),\n",
    "    )\n",
    "    .properties(width=700, height=250)\n",
    ")\n",
    "\n",
    "# This is a shorthand way of stacking the charts on top of each other\n",
    "chart1 & chart2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is really interesting – it seems there's been a deliberate effort to get the earliest newspapers corrected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of corrections by category\n",
    "\n",
    "Let's see how the number of corrections varies across categories. This time we'll use the `category` facet instead of `year`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "params[\"q\"] = \"has:corrections\"\n",
    "params[\"facet\"] = \"category\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_results(params)\n",
    "facets = []\n",
    "for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "    # Get the state and the number of results, and convert it to integers, before adding to our results\n",
    "    facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "df_categories = pd.DataFrame(facets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Article</td>\n",
       "      <td>10385409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Family Notices</td>\n",
       "      <td>1362744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Advertising</td>\n",
       "      <td>1319419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Detailed Lists, Results, Guides</td>\n",
       "      <td>526437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Literature</td>\n",
       "      <td>11335</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              term  total_results\n",
       "0                          Article       10385409\n",
       "1                   Family Notices        1362744\n",
       "2                      Advertising        1319419\n",
       "3  Detailed Lists, Results, Guides         526437\n",
       "4                       Literature          11335"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_categories.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, the raw numbers are probably not all that useful, so let's get the total number of articles in each category and calculate the proportion that have at least one correction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Blank query\n",
    "params[\"q\"] = \" \"\n",
    "data = get_results(params)\n",
    "facets = []\n",
    "for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "    # Get the state and the number of results, and convert it to integers, before adding to our results\n",
    "    facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "df_total_categories = pd.DataFrame(facets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll merge the two corrections by category data with the total articles per category and calculate the proportion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Article</td>\n",
       "      <td>10385409</td>\n",
       "      <td>162296409</td>\n",
       "      <td>0.063990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Family Notices</td>\n",
       "      <td>1362744</td>\n",
       "      <td>1926852</td>\n",
       "      <td>0.707239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Advertising</td>\n",
       "      <td>1319419</td>\n",
       "      <td>43238969</td>\n",
       "      <td>0.030515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Detailed Lists, Results, Guides</td>\n",
       "      <td>526437</td>\n",
       "      <td>26177949</td>\n",
       "      <td>0.020110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Literature</td>\n",
       "      <td>11335</td>\n",
       "      <td>34669</td>\n",
       "      <td>0.326949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Obituaries</td>\n",
       "      <td>9493</td>\n",
       "      <td>10459</td>\n",
       "      <td>0.907639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Humour</td>\n",
       "      <td>8698</td>\n",
       "      <td>26571</td>\n",
       "      <td>0.327349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>News</td>\n",
       "      <td>7983</td>\n",
       "      <td>9928</td>\n",
       "      <td>0.804089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Law, Courts, And Crime</td>\n",
       "      <td>6615</td>\n",
       "      <td>9047</td>\n",
       "      <td>0.731182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Sport And Games</td>\n",
       "      <td>6029</td>\n",
       "      <td>13682</td>\n",
       "      <td>0.440652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Letters</td>\n",
       "      <td>3372</td>\n",
       "      <td>11674</td>\n",
       "      <td>0.288847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Editorial</td>\n",
       "      <td>2228</td>\n",
       "      <td>12908</td>\n",
       "      <td>0.172606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Arts And Culture</td>\n",
       "      <td>2212</td>\n",
       "      <td>3054</td>\n",
       "      <td>0.724296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Puzzles</td>\n",
       "      <td>1616</td>\n",
       "      <td>38716</td>\n",
       "      <td>0.041740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Reviews</td>\n",
       "      <td>1366</td>\n",
       "      <td>1877</td>\n",
       "      <td>0.727757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Shipping Notices</td>\n",
       "      <td>1332</td>\n",
       "      <td>1796</td>\n",
       "      <td>0.741648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Classified Advertisements And Notices</td>\n",
       "      <td>1222</td>\n",
       "      <td>1410</td>\n",
       "      <td>0.866667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Official Appointments And Notices</td>\n",
       "      <td>1019</td>\n",
       "      <td>1046</td>\n",
       "      <td>0.974187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Weather</td>\n",
       "      <td>1011</td>\n",
       "      <td>8625</td>\n",
       "      <td>0.117217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Commerce And Business</td>\n",
       "      <td>1005</td>\n",
       "      <td>2566</td>\n",
       "      <td>0.391660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Display Advertisement</td>\n",
       "      <td>415</td>\n",
       "      <td>456</td>\n",
       "      <td>0.910088</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     term  total_results  total_articles  \\\n",
       "0                                 Article       10385409       162296409   \n",
       "1                          Family Notices        1362744         1926852   \n",
       "2                             Advertising        1319419        43238969   \n",
       "3         Detailed Lists, Results, Guides         526437        26177949   \n",
       "4                              Literature          11335           34669   \n",
       "5                              Obituaries           9493           10459   \n",
       "6                                  Humour           8698           26571   \n",
       "7                                    News           7983            9928   \n",
       "8                  Law, Courts, And Crime           6615            9047   \n",
       "9                         Sport And Games           6029           13682   \n",
       "10                                Letters           3372           11674   \n",
       "11                              Editorial           2228           12908   \n",
       "12                       Arts And Culture           2212            3054   \n",
       "13                                Puzzles           1616           38716   \n",
       "14                                Reviews           1366            1877   \n",
       "15                       Shipping Notices           1332            1796   \n",
       "16  Classified Advertisements And Notices           1222            1410   \n",
       "17      Official Appointments And Notices           1019            1046   \n",
       "18                                Weather           1011            8625   \n",
       "19                  Commerce And Business           1005            2566   \n",
       "20                  Display Advertisement            415             456   \n",
       "\n",
       "    proportion  \n",
       "0     0.063990  \n",
       "1     0.707239  \n",
       "2     0.030515  \n",
       "3     0.020110  \n",
       "4     0.326949  \n",
       "5     0.907639  \n",
       "6     0.327349  \n",
       "7     0.804089  \n",
       "8     0.731182  \n",
       "9     0.440652  \n",
       "10    0.288847  \n",
       "11    0.172606  \n",
       "12    0.724296  \n",
       "13    0.041740  \n",
       "14    0.727757  \n",
       "15    0.741648  \n",
       "16    0.866667  \n",
       "17    0.974187  \n",
       "18    0.117217  \n",
       "19    0.391660  \n",
       "20    0.910088  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_categories_merged = merge_df_with_total(df_categories, df_total_categories)\n",
    "df_categories_merged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lot of the categories have been added recently and don't contain a lot of articles. Some of these have a very high proportion of articles with corrections – 'Obituaries' for example. This suggests users are systematically categorising and correcting certain types of article.\n",
    "\n",
    "Let's focus on the main categories by filtering out those with less than 30,000 articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Article</td>\n",
       "      <td>10385409</td>\n",
       "      <td>162296409</td>\n",
       "      <td>0.063990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Family Notices</td>\n",
       "      <td>1362744</td>\n",
       "      <td>1926852</td>\n",
       "      <td>0.707239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Advertising</td>\n",
       "      <td>1319419</td>\n",
       "      <td>43238969</td>\n",
       "      <td>0.030515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Detailed Lists, Results, Guides</td>\n",
       "      <td>526437</td>\n",
       "      <td>26177949</td>\n",
       "      <td>0.020110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Literature</td>\n",
       "      <td>11335</td>\n",
       "      <td>34669</td>\n",
       "      <td>0.326949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Puzzles</td>\n",
       "      <td>1616</td>\n",
       "      <td>38716</td>\n",
       "      <td>0.041740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               term  total_results  total_articles  proportion\n",
       "0                           Article       10385409       162296409    0.063990\n",
       "1                    Family Notices        1362744         1926852    0.707239\n",
       "2                       Advertising        1319419        43238969    0.030515\n",
       "3   Detailed Lists, Results, Guides         526437        26177949    0.020110\n",
       "4                        Literature          11335           34669    0.326949\n",
       "13                          Puzzles           1616           38716    0.041740"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_categories_filtered = df_categories_merged.loc[\n",
    "    df_categories_merged[\"total_articles\"] > 30000\n",
    "]\n",
    "df_categories_filtered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can visualise the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-cd866ae15cb240ed82ca8c6ab33c0ae8\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-cd866ae15cb240ed82ca8c6ab33c0ae8\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-cd866ae15cb240ed82ca8c6ab33c0ae8\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"hconcat\": [{\"mark\": \"bar\", \"encoding\": {\"x\": {\"type\": \"nominal\", \"field\": \"term\", \"title\": \"Category\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"total_results\", \"title\": \"Articles with corrections\"}}}, {\"mark\": \"bar\", \"encoding\": {\"color\": {\"value\": \"orange\"}, \"x\": {\"type\": \"nominal\", \"field\": \"term\", \"title\": \"Category\"}, \"y\": {\"type\": \"quantitative\", \"axis\": {\"format\": \"%\", \"title\": \"Proportion of articles with corrections\"}, \"field\": \"proportion\"}}}], \"data\": {\"name\": \"data-22a348b04c5ae05b5719bee7592b5c3e\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-22a348b04c5ae05b5719bee7592b5c3e\": [{\"term\": \"Article\", \"total_results\": 10385409, \"total_articles\": 162296409, \"proportion\": 0.063990380711381}, {\"term\": \"Family Notices\", \"total_results\": 1362744, \"total_articles\": 1926852, \"proportion\": 0.7072385424516258}, {\"term\": \"Advertising\", \"total_results\": 1319419, \"total_articles\": 43238969, \"proportion\": 0.0305145804933508}, {\"term\": \"Detailed Lists, Results, Guides\", \"total_results\": 526437, \"total_articles\": 26177949, \"proportion\": 0.020109940622162568}, {\"term\": \"Literature\", \"total_results\": 11335, \"total_articles\": 34669, \"proportion\": 0.32694914765352334}, {\"term\": \"Puzzles\", \"total_results\": 1616, \"total_articles\": 38716, \"proportion\": 0.04173984915797087}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.HConcatChart(...)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_chart1 = (\n",
    "    alt.Chart(df_categories_filtered)\n",
    "    .mark_bar()\n",
    "    .encode(\n",
    "        x=alt.X(\"term:N\", title=\"Category\"),\n",
    "        y=alt.Y(\"total_results:Q\", title=\"Articles with corrections\"),\n",
    "    )\n",
    ")\n",
    "\n",
    "cat_chart2 = (\n",
    "    alt.Chart(df_categories_filtered)\n",
    "    .mark_bar()\n",
    "    .encode(\n",
    "        x=alt.X(\"term:N\", title=\"Category\"),\n",
    "        y=alt.Y(\n",
    "            \"proportion:Q\",\n",
    "            axis=alt.Axis(format=\"%\", title=\"Proportion of articles with corrections\"),\n",
    "        ),\n",
    "        color=alt.value(\"orange\"),\n",
    "    )\n",
    ")\n",
    "\n",
    "cat_chart1 | cat_chart2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the rate of corrections is much higher in the 'Family Notices' category than any other. This probably reflects the work of family historians and others searching for, and correcting, articles containing particular names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of corrections by newspaper\n",
    "\n",
    "How do rates of correction vary across newspapers? We can use the `title` facet to find out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "params[\"q\"] = \"has:corrections\"\n",
    "params[\"facet\"] = \"title\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_results(params)\n",
    "facets = []\n",
    "for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "    # Get the state and the number of results, and convert it to integers, before adding to our results\n",
    "    facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "df_newspapers = pd.DataFrame(facets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term</th>\n",
       "      <th>total_results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35</td>\n",
       "      <td>827605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>780708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11</td>\n",
       "      <td>373062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>347336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>317381</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   term  total_results\n",
       "0    35         827605\n",
       "1    13         780708\n",
       "2    11         373062\n",
       "3    16         347336\n",
       "4    30         317381"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_newspapers.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again we'll calculate the proportion of articles corrected for each newspaper by getting the total number of articles for each newspaper on Trove."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "params[\"q\"] = \" \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_results(params)\n",
    "facets = []\n",
    "for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "    # Get the state and the number of results, and convert it to integers, before adding to our results\n",
    "    facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "df_newspapers_total = pd.DataFrame(facets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_newspapers_merged = merge_df_with_total(\n",
    "    df_newspapers, df_newspapers_total, how=\"right\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_newspapers_merged.sort_values(by=\"proportion\", ascending=False, inplace=True)\n",
    "df_newspapers_merged.rename(columns={\"term\": \"id\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1662</th>\n",
       "      <td>729</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1548</th>\n",
       "      <td>1028</td>\n",
       "      <td>286</td>\n",
       "      <td>286</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1629</th>\n",
       "      <td>1047</td>\n",
       "      <td>38</td>\n",
       "      <td>38</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1617</th>\n",
       "      <td>686</td>\n",
       "      <td>53</td>\n",
       "      <td>53</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1614</th>\n",
       "      <td>118</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  total_results  total_articles  proportion\n",
       "1662   729              3               3         1.0\n",
       "1548  1028            286             286         1.0\n",
       "1629  1047             38              38         1.0\n",
       "1617   686             53              53         1.0\n",
       "1614   118             56              56         1.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_newspapers_merged.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `title` facet only gives us the `id` number for each newspaper, not its title. Let's get all the titles and then merge them with the facet data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all the newspaper titles\n",
    "title_params = {\n",
    "    \"key\": API_KEY,\n",
    "    \"encoding\": \"json\",\n",
    "}\n",
    "\n",
    "title_data = s.get(\n",
    "    \"https://api.trove.nla.gov.au/v2/newspaper/titles\", params=params\n",
    ").json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = []\n",
    "for newspaper in title_data[\"response\"][\"records\"][\"newspaper\"]:\n",
    "    titles.append({\"title\": newspaper[\"title\"], \"id\": int(newspaper[\"id\"])})\n",
    "df_titles = pd.DataFrame(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Canberra Community News (ACT : 1925 - 1927)</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Canberra Illustrated: A Quarterly Magazine (AC...</td>\n",
       "      <td>165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Federal Capital Pioneer (Canberra, ACT : 1924 ...</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Good Neighbour (ACT : 1950 - 1969)</td>\n",
       "      <td>871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Student Notes/Canberra University College Stud...</td>\n",
       "      <td>665</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title   id\n",
       "0        Canberra Community News (ACT : 1925 - 1927)  166\n",
       "1  Canberra Illustrated: A Quarterly Magazine (AC...  165\n",
       "2  Federal Capital Pioneer (Canberra, ACT : 1924 ...   69\n",
       "3                 Good Neighbour (ACT : 1950 - 1969)  871\n",
       "4  Student Notes/Canberra University College Stud...  665"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_titles.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1698, 2)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_titles.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One problem with this list is that it also includes the titles of the Government Gazettes (this seems to be a bug in the API). Let's get the gazette titles and then subtract them from the complete list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get gazette titles\n",
    "gazette_data = s.get(\n",
    "    \"https://api.trove.nla.gov.au/v2/gazette/titles\", params=params\n",
    ").json()\n",
    "gazettes = []\n",
    "for gaz in gazette_data[\"response\"][\"records\"][\"newspaper\"]:\n",
    "    gazettes.append({\"title\": gaz[\"title\"], \"id\": int(gaz[\"id\"])})\n",
    "df_gazettes = pd.DataFrame(gazettes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(38, 2)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gazettes.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subtract the gazettes from the list of titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_titles_not_gazettes = df_titles[~df_titles[\"id\"].isin(df_gazettes[\"id\"])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can merge the newspaper titles with the facet data using the `id` to link the two datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_newspapers_with_titles = (\n",
    "    pd.merge(df_titles_not_gazettes, df_newspapers_merged, how=\"left\", on=\"id\")\n",
    "    .fillna(0)\n",
    "    .sort_values(by=\"proportion\", ascending=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the totals back to integers\n",
    "df_newspapers_with_titles[\n",
    "    [\"total_results\", \"total_articles\"]\n",
    "] = df_newspapers_with_titles[[\"total_results\", \"total_articles\"]].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can display the newspapers with the highest rates of correction. Remember, that a `proportion` of 1.00 means that every available article has at least one correction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>468</th>\n",
       "      <td>The Temora Telegraph and Mining Advocate (NSW ...</td>\n",
       "      <td>729</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>824</th>\n",
       "      <td>Hobart Town Gazette and Van Diemen's Land Adve...</td>\n",
       "      <td>5</td>\n",
       "      <td>1556</td>\n",
       "      <td>1556</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>421</th>\n",
       "      <td>The Satirist and Sporting Chronicle (Sydney, N...</td>\n",
       "      <td>1028</td>\n",
       "      <td>286</td>\n",
       "      <td>286</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>Justice (Narrabri, NSW : 1891)</td>\n",
       "      <td>885</td>\n",
       "      <td>45</td>\n",
       "      <td>45</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>921</th>\n",
       "      <td>Alexandra and Yea Standard, Thornton, Gobur an...</td>\n",
       "      <td>154</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>Suedaustralische Zeitung (Adelaide, SA : 1850 ...</td>\n",
       "      <td>314</td>\n",
       "      <td>47</td>\n",
       "      <td>47</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>The True Sun and New South Wales Independent P...</td>\n",
       "      <td>1038</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>The Branxton Advocate: Greta and Rothbury Reco...</td>\n",
       "      <td>686</td>\n",
       "      <td>53</td>\n",
       "      <td>53</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>The Australian Abo Call (National : 1938)</td>\n",
       "      <td>51</td>\n",
       "      <td>78</td>\n",
       "      <td>78</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>838</th>\n",
       "      <td>Tasmanian and Port Dalrymple Advertiser (Launc...</td>\n",
       "      <td>273</td>\n",
       "      <td>193</td>\n",
       "      <td>193</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>538</th>\n",
       "      <td>Moonta Herald and Northern Territory Gazette (...</td>\n",
       "      <td>118</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>215</th>\n",
       "      <td>Society (Sydney, NSW : 1887)</td>\n",
       "      <td>1042</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>Party (Sydney, NSW : 1942)</td>\n",
       "      <td>1000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>979</th>\n",
       "      <td>Elsternwick Leader and East Brighton, ... (Vic...</td>\n",
       "      <td>201</td>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1472</th>\n",
       "      <td>Swan River Guardian (WA : 1836 - 1838)</td>\n",
       "      <td>1142</td>\n",
       "      <td>437</td>\n",
       "      <td>437</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>The Derwent Star and Van Diemen's Land Intelli...</td>\n",
       "      <td>1046</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td>Logan and Albert Advocate (Qld. : 1893 - 1900)</td>\n",
       "      <td>842</td>\n",
       "      <td>84</td>\n",
       "      <td>84</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>905</th>\n",
       "      <td>The Van Diemen's Land Gazette and General Adve...</td>\n",
       "      <td>1047</td>\n",
       "      <td>38</td>\n",
       "      <td>38</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>The Hobart Town Gazette and Southern Reporter ...</td>\n",
       "      <td>4</td>\n",
       "      <td>1922</td>\n",
       "      <td>1923</td>\n",
       "      <td>0.999480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Federal Capital Pioneer (Canberra, ACT : 1924 ...</td>\n",
       "      <td>69</td>\n",
       "      <td>542</td>\n",
       "      <td>545</td>\n",
       "      <td>0.994495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1231</th>\n",
       "      <td>The Melbourne Advertiser (Vic. : 1838)</td>\n",
       "      <td>935</td>\n",
       "      <td>120</td>\n",
       "      <td>121</td>\n",
       "      <td>0.991736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>South Australian Gazette and Colonial Register...</td>\n",
       "      <td>40</td>\n",
       "      <td>1051</td>\n",
       "      <td>1065</td>\n",
       "      <td>0.986854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>Intelligence (Bowral, NSW : 1884)</td>\n",
       "      <td>624</td>\n",
       "      <td>117</td>\n",
       "      <td>119</td>\n",
       "      <td>0.983193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>892</th>\n",
       "      <td>The People's Horn Boy (Hobart Town, Tas. : 1834)</td>\n",
       "      <td>1240</td>\n",
       "      <td>99</td>\n",
       "      <td>101</td>\n",
       "      <td>0.980198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1657</th>\n",
       "      <td>York Advocate (WA : 1915)</td>\n",
       "      <td>1131</td>\n",
       "      <td>236</td>\n",
       "      <td>241</td>\n",
       "      <td>0.979253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title    id  total_results  \\\n",
       "468   The Temora Telegraph and Mining Advocate (NSW ...   729              3   \n",
       "824   Hobart Town Gazette and Van Diemen's Land Adve...     5           1556   \n",
       "421   The Satirist and Sporting Chronicle (Sydney, N...  1028            286   \n",
       "149                      Justice (Narrabri, NSW : 1891)   885             45   \n",
       "921   Alexandra and Yea Standard, Thornton, Gobur an...   154             21   \n",
       "739   Suedaustralische Zeitung (Adelaide, SA : 1850 ...   314             47   \n",
       "472   The True Sun and New South Wales Independent P...  1038             20   \n",
       "284   The Branxton Advocate: Greta and Rothbury Reco...   686             53   \n",
       "24            The Australian Abo Call (National : 1938)    51             78   \n",
       "838   Tasmanian and Port Dalrymple Advertiser (Launc...   273            193   \n",
       "538   Moonta Herald and Northern Territory Gazette (...   118             56   \n",
       "215                        Society (Sydney, NSW : 1887)  1042             21   \n",
       "195                          Party (Sydney, NSW : 1942)  1000              6   \n",
       "979   Elsternwick Leader and East Brighton, ... (Vic...   201             17   \n",
       "1472             Swan River Guardian (WA : 1836 - 1838)  1142            437   \n",
       "862   The Derwent Star and Van Diemen's Land Intelli...  1046             12   \n",
       "563      Logan and Albert Advocate (Qld. : 1893 - 1900)   842             84   \n",
       "905   The Van Diemen's Land Gazette and General Adve...  1047             38   \n",
       "871   The Hobart Town Gazette and Southern Reporter ...     4           1922   \n",
       "2     Federal Capital Pioneer (Canberra, ACT : 1924 ...    69            542   \n",
       "1231             The Melbourne Advertiser (Vic. : 1838)   935            120   \n",
       "728   South Australian Gazette and Colonial Register...    40           1051   \n",
       "143                   Intelligence (Bowral, NSW : 1884)   624            117   \n",
       "892    The People's Horn Boy (Hobart Town, Tas. : 1834)  1240             99   \n",
       "1657                          York Advocate (WA : 1915)  1131            236   \n",
       "\n",
       "      total_articles  proportion  \n",
       "468                3    1.000000  \n",
       "824             1556    1.000000  \n",
       "421              286    1.000000  \n",
       "149               45    1.000000  \n",
       "921               21    1.000000  \n",
       "739               47    1.000000  \n",
       "472               20    1.000000  \n",
       "284               53    1.000000  \n",
       "24                78    1.000000  \n",
       "838              193    1.000000  \n",
       "538               56    1.000000  \n",
       "215               21    1.000000  \n",
       "195                6    1.000000  \n",
       "979               17    1.000000  \n",
       "1472             437    1.000000  \n",
       "862               12    1.000000  \n",
       "563               84    1.000000  \n",
       "905               38    1.000000  \n",
       "871             1923    0.999480  \n",
       "2                545    0.994495  \n",
       "1231             121    0.991736  \n",
       "728             1065    0.986854  \n",
       "143              119    0.983193  \n",
       "892              101    0.980198  \n",
       "1657             241    0.979253  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_newspapers_with_titles[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the other end, we can see the newspapers with the smallest rates of correction. Note that some newspapers have no corrections at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1247</th>\n",
       "      <td>The Morwell Advocate and Narracan, Boolara and...</td>\n",
       "      <td>1734</td>\n",
       "      <td>0</td>\n",
       "      <td>208</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1559</th>\n",
       "      <td>The Mount Margaret Mercury (WA : 1897)</td>\n",
       "      <td>1641</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1553</th>\n",
       "      <td>The Miner's Right (Perth, WA : 1894)</td>\n",
       "      <td>1729</td>\n",
       "      <td>0</td>\n",
       "      <td>426</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1508</th>\n",
       "      <td>The Derby News (WA : 1887)</td>\n",
       "      <td>1617</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>The Herald of Tasmania (Hobart, Tas. : 1845)</td>\n",
       "      <td>1741</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1246</th>\n",
       "      <td>The Morwell Advocate and Boolara and Mirboo Ch...</td>\n",
       "      <td>1733</td>\n",
       "      <td>0</td>\n",
       "      <td>33</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>874</th>\n",
       "      <td>The Hobart Town Herald and abstinence advocate...</td>\n",
       "      <td>1743</td>\n",
       "      <td>0</td>\n",
       "      <td>434</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1144</th>\n",
       "      <td>Seamen's Strike Bulletin (Melbourne, Vic. : 1919)</td>\n",
       "      <td>1043</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>The Hobart Town Herald (Tas. :  1845)</td>\n",
       "      <td>1740</td>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>875</th>\n",
       "      <td>The Hobart Town Herald, or, Southern reporter ...</td>\n",
       "      <td>1742</td>\n",
       "      <td>0</td>\n",
       "      <td>103</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>837</th>\n",
       "      <td>Saturday Evening Express (Launceston, Tas. : 1...</td>\n",
       "      <td>1747</td>\n",
       "      <td>7</td>\n",
       "      <td>44312</td>\n",
       "      <td>0.000158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>514</th>\n",
       "      <td>Vil'na Dumka = Free Thought (Sydney, NSW : 194...</td>\n",
       "      <td>1593</td>\n",
       "      <td>2</td>\n",
       "      <td>11607</td>\n",
       "      <td>0.000172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>926</th>\n",
       "      <td>Australier Leben = Australian Life (Melbourne,...</td>\n",
       "      <td>1686</td>\n",
       "      <td>1</td>\n",
       "      <td>3816</td>\n",
       "      <td>0.000262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500</th>\n",
       "      <td>To Ethnico Vema = Greek National Tribune (Arnc...</td>\n",
       "      <td>1592</td>\n",
       "      <td>27</td>\n",
       "      <td>62861</td>\n",
       "      <td>0.000430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>Kimba Dispatch (SA. : 1927 - 1941)</td>\n",
       "      <td>1731</td>\n",
       "      <td>17</td>\n",
       "      <td>35136</td>\n",
       "      <td>0.000484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746</th>\n",
       "      <td>The Challenger (Port Lincoln, SA. : 1932 - 1934)</td>\n",
       "      <td>1732</td>\n",
       "      <td>3</td>\n",
       "      <td>5599</td>\n",
       "      <td>0.000536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1273</th>\n",
       "      <td>The Western Port Times and Phillip Island and ...</td>\n",
       "      <td>1365</td>\n",
       "      <td>7</td>\n",
       "      <td>12684</td>\n",
       "      <td>0.000552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>L'Italo-Australiano = The Italo-Australian (Sy...</td>\n",
       "      <td>1597</td>\n",
       "      <td>5</td>\n",
       "      <td>6106</td>\n",
       "      <td>0.000819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>Mu̇sų Pastogė = Our Haven (Sydney, NSW : 195...</td>\n",
       "      <td>1594</td>\n",
       "      <td>9</td>\n",
       "      <td>9060</td>\n",
       "      <td>0.000993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>The Northern Districts Courier (North Adelaide...</td>\n",
       "      <td>1711</td>\n",
       "      <td>1</td>\n",
       "      <td>885</td>\n",
       "      <td>0.001130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1250</th>\n",
       "      <td>The Narracan Shire Advocate and Yallourn Brown...</td>\n",
       "      <td>1735</td>\n",
       "      <td>49</td>\n",
       "      <td>42777</td>\n",
       "      <td>0.001145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>Italo-Australian (Sydney, NSW : 1927 - 1940)</td>\n",
       "      <td>1595</td>\n",
       "      <td>56</td>\n",
       "      <td>38986</td>\n",
       "      <td>0.001436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1535</th>\n",
       "      <td>The Inland Watch (Leonora, WA : 1937 - 1943)</td>\n",
       "      <td>1630</td>\n",
       "      <td>50</td>\n",
       "      <td>28725</td>\n",
       "      <td>0.001741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>The Hobart Town Herald (Tas. : 1880)</td>\n",
       "      <td>1744</td>\n",
       "      <td>1</td>\n",
       "      <td>560</td>\n",
       "      <td>0.001786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>192</th>\n",
       "      <td>Oceania (Sydney, NSW : 1913 - 1915)</td>\n",
       "      <td>1598</td>\n",
       "      <td>4</td>\n",
       "      <td>2167</td>\n",
       "      <td>0.001846</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title    id  total_results  \\\n",
       "1247  The Morwell Advocate and Narracan, Boolara and...  1734              0   \n",
       "1559             The Mount Margaret Mercury (WA : 1897)  1641              0   \n",
       "1553               The Miner's Right (Perth, WA : 1894)  1729              0   \n",
       "1508                         The Derby News (WA : 1887)  1617              0   \n",
       "864        The Herald of Tasmania (Hobart, Tas. : 1845)  1741              0   \n",
       "1246  The Morwell Advocate and Boolara and Mirboo Ch...  1733              0   \n",
       "874   The Hobart Town Herald and abstinence advocate...  1743              0   \n",
       "1144  Seamen's Strike Bulletin (Melbourne, Vic. : 1919)  1043              0   \n",
       "872               The Hobart Town Herald (Tas. :  1845)  1740              0   \n",
       "875   The Hobart Town Herald, or, Southern reporter ...  1742              0   \n",
       "837   Saturday Evening Express (Launceston, Tas. : 1...  1747              7   \n",
       "514   Vil'na Dumka = Free Thought (Sydney, NSW : 194...  1593              2   \n",
       "926   Australier Leben = Australian Life (Melbourne,...  1686              1   \n",
       "500   To Ethnico Vema = Greek National Tribune (Arnc...  1592             27   \n",
       "685                  Kimba Dispatch (SA. : 1927 - 1941)  1731             17   \n",
       "746    The Challenger (Port Lincoln, SA. : 1932 - 1934)  1732              3   \n",
       "1273  The Western Port Times and Phillip Island and ...  1365              7   \n",
       "155   L'Italo-Australiano = The Italo-Australian (Sy...  1597              5   \n",
       "178   Mu̇sų Pastogė = Our Haven (Sydney, NSW : 195...  1594              9   \n",
       "764   The Northern Districts Courier (North Adelaide...  1711              1   \n",
       "1250  The Narracan Shire Advocate and Yallourn Brown...  1735             49   \n",
       "147        Italo-Australian (Sydney, NSW : 1927 - 1940)  1595             56   \n",
       "1535       The Inland Watch (Leonora, WA : 1937 - 1943)  1630             50   \n",
       "873                The Hobart Town Herald (Tas. : 1880)  1744              1   \n",
       "192                 Oceania (Sydney, NSW : 1913 - 1915)  1598              4   \n",
       "\n",
       "      total_articles  proportion  \n",
       "1247             208    0.000000  \n",
       "1559              24    0.000000  \n",
       "1553             426    0.000000  \n",
       "1508               9    0.000000  \n",
       "864               50    0.000000  \n",
       "1246              33    0.000000  \n",
       "874              434    0.000000  \n",
       "1144              14    0.000000  \n",
       "872               57    0.000000  \n",
       "875              103    0.000000  \n",
       "837            44312    0.000158  \n",
       "514            11607    0.000172  \n",
       "926             3816    0.000262  \n",
       "500            62861    0.000430  \n",
       "685            35136    0.000484  \n",
       "746             5599    0.000536  \n",
       "1273           12684    0.000552  \n",
       "155             6106    0.000819  \n",
       "178             9060    0.000993  \n",
       "764              885    0.001130  \n",
       "1250           42777    0.001145  \n",
       "147            38986    0.001436  \n",
       "1535           28725    0.001741  \n",
       "873              560    0.001786  \n",
       "192             2167    0.001846  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_newspapers_with_titles.sort_values(by=\"proportion\")[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll save the full list of newspapers as a CSV file, but first we'll fix up the column headings and add urls for each title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_newspapers_with_titles_csv = df_newspapers_with_titles.copy()\n",
    "df_newspapers_with_titles_csv.rename(\n",
    "    {\"total_results\": \"articles_with_corrections\"}, axis=1, inplace=True\n",
    ")\n",
    "df_newspapers_with_titles_csv[\"percentage_with_corrections\"] = (\n",
    "    df_newspapers_with_titles_csv[\"proportion\"] * 100\n",
    ")\n",
    "df_newspapers_with_titles_csv.sort_values(\n",
    "    by=[\"percentage_with_corrections\"], inplace=True\n",
    ")\n",
    "df_newspapers_with_titles_csv[\n",
    "    [\n",
    "        \"id\",\n",
    "        \"title\",\n",
    "        \"articles_with_corrections\",\n",
    "        \"total_articles\",\n",
    "        \"percentage_with_corrections\",\n",
    "    ]\n",
    "].to_csv(\"titles_corrected.csv\", index=False)\n",
    "df_newspapers_with_titles_csv[\"title_url\"] = df_newspapers_with_titles_csv[\"id\"].apply(\n",
    "    lambda x: f\"http://nla.gov.au/nla.news-title{x}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll save the data as a CSV file and display a link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='titles_corrected.csv' target='_blank'>titles_corrected.csv</a><br>"
      ],
      "text/plain": [
       "/home/tim/Workspace/mycode/glam-workbench/trove-newspapers/notebooks/titles_corrected.csv"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_newspapers_with_titles_csv.to_csv(\"titles_corrected.csv\", index=False)\n",
    "display(FileLink(\"titles_corrected.csv\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neediest newspapers\n",
    "\n",
    "Let's see if we can combine some guesses about OCR error rates with the correction data to find the newspapers most in need of help.\n",
    "\n",
    "To make a guesstimate of error rates, we'll use the occurance of 'tbe' – ie a common OCR error for 'the'. I don't know how valid this is, but it's a place to start!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search for 'tbe' to get an indication of errors by newspaper\n",
    "params[\"q\"] = 'text:\"tbe\"~0'\n",
    "params[\"facet\"] = \"title\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_results(params)\n",
    "facets = []\n",
    "for term in data[\"response\"][\"zone\"][0][\"facets\"][\"facet\"][\"term\"]:\n",
    "    # Get the state and the number of results, and convert it to integers, before adding to our results\n",
    "    facets.append({\"term\": term[\"search\"], \"total_results\": int(term[\"count\"])})\n",
    "df_errors = pd.DataFrame(facets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge the error data with the total articles per newspaper to calculate the proportion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_errors_merged = merge_df_with_total(df_errors, df_newspapers_total, how=\"right\")\n",
    "df_errors_merged.sort_values(by=\"proportion\", ascending=False, inplace=True)\n",
    "df_errors_merged.rename(columns={\"term\": \"id\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1253</th>\n",
       "      <td>1316</td>\n",
       "      <td>2004</td>\n",
       "      <td>2954</td>\n",
       "      <td>0.678402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1046</th>\n",
       "      <td>758</td>\n",
       "      <td>5245</td>\n",
       "      <td>8078</td>\n",
       "      <td>0.649294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>821</th>\n",
       "      <td>927</td>\n",
       "      <td>9438</td>\n",
       "      <td>17227</td>\n",
       "      <td>0.547861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>912</th>\n",
       "      <td>382</td>\n",
       "      <td>6956</td>\n",
       "      <td>12744</td>\n",
       "      <td>0.545825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938</th>\n",
       "      <td>262</td>\n",
       "      <td>6256</td>\n",
       "      <td>11527</td>\n",
       "      <td>0.542726</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  total_results  total_articles  proportion\n",
       "1253  1316           2004            2954    0.678402\n",
       "1046   758           5245            8078    0.649294\n",
       "821    927           9438           17227    0.547861\n",
       "912    382           6956           12744    0.545825\n",
       "938    262           6256           11527    0.542726"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_errors_merged.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the title names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_errors_with_titles = (\n",
    "    pd.merge(df_titles_not_gazettes, df_errors_merged, how=\"left\", on=\"id\")\n",
    "    .fillna(0)\n",
    "    .sort_values(by=\"proportion\", ascending=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is a list of the newspapers with the highest rate of OCR error (by our rather dodgy measure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>487</th>\n",
       "      <td>The Weekly Advance (Granville, NSW : 1892 - 1893)</td>\n",
       "      <td>1316</td>\n",
       "      <td>2004</td>\n",
       "      <td>2954</td>\n",
       "      <td>0.678402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>Dunolly and Betbetshire Express and County of ...</td>\n",
       "      <td>758</td>\n",
       "      <td>5245</td>\n",
       "      <td>8078</td>\n",
       "      <td>0.649294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1021</th>\n",
       "      <td>Hamilton Spectator and Grange District Adverti...</td>\n",
       "      <td>927</td>\n",
       "      <td>9438</td>\n",
       "      <td>17227</td>\n",
       "      <td>0.547861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>519</th>\n",
       "      <td>Wagga Wagga Express and Murrumbidgee District ...</td>\n",
       "      <td>382</td>\n",
       "      <td>6956</td>\n",
       "      <td>12744</td>\n",
       "      <td>0.545825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>The North Australian, Ipswich and General Adve...</td>\n",
       "      <td>262</td>\n",
       "      <td>6256</td>\n",
       "      <td>11527</td>\n",
       "      <td>0.542726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>618</th>\n",
       "      <td>The North Australian (Brisbane, Qld. : 1863 - ...</td>\n",
       "      <td>264</td>\n",
       "      <td>2868</td>\n",
       "      <td>5314</td>\n",
       "      <td>0.539706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>The Herald of Tasmania (Hobart, Tas. : 1845)</td>\n",
       "      <td>1741</td>\n",
       "      <td>26</td>\n",
       "      <td>50</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>The Hay Standard and Advertiser for Balranald,...</td>\n",
       "      <td>725</td>\n",
       "      <td>21671</td>\n",
       "      <td>42068</td>\n",
       "      <td>0.515142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>Robertson Advocate (NSW : 1894 - 1923)</td>\n",
       "      <td>530</td>\n",
       "      <td>36946</td>\n",
       "      <td>72383</td>\n",
       "      <td>0.510424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>Temora Herald and Mining Journal (NSW : 1882 -...</td>\n",
       "      <td>728</td>\n",
       "      <td>639</td>\n",
       "      <td>1253</td>\n",
       "      <td>0.509976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>Sydney Mail (NSW : 1860 - 1871)</td>\n",
       "      <td>697</td>\n",
       "      <td>24539</td>\n",
       "      <td>48535</td>\n",
       "      <td>0.505594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>840</th>\n",
       "      <td>Tasmanian Morning Herald (Hobart, Tas. : 1865 ...</td>\n",
       "      <td>865</td>\n",
       "      <td>5178</td>\n",
       "      <td>10290</td>\n",
       "      <td>0.503207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>835</th>\n",
       "      <td>Morning Star and Commercial Advertiser (Hobart...</td>\n",
       "      <td>1242</td>\n",
       "      <td>850</td>\n",
       "      <td>1703</td>\n",
       "      <td>0.499119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>Molong Argus (NSW : 1896 - 1921)</td>\n",
       "      <td>424</td>\n",
       "      <td>52061</td>\n",
       "      <td>104984</td>\n",
       "      <td>0.495895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1114</th>\n",
       "      <td>Port Phillip Gazette (Vic. : 1851)</td>\n",
       "      <td>1139</td>\n",
       "      <td>241</td>\n",
       "      <td>491</td>\n",
       "      <td>0.490835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1115</th>\n",
       "      <td>Port Phillip Gazette and Settler's Journal (Vi...</td>\n",
       "      <td>1138</td>\n",
       "      <td>5947</td>\n",
       "      <td>12127</td>\n",
       "      <td>0.490393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>846</th>\n",
       "      <td>Telegraph (Hobart Town, Tas. : 1867)</td>\n",
       "      <td>1250</td>\n",
       "      <td>68</td>\n",
       "      <td>140</td>\n",
       "      <td>0.485714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>The Hobart Town Herald (Tas. :  1845)</td>\n",
       "      <td>1740</td>\n",
       "      <td>27</td>\n",
       "      <td>57</td>\n",
       "      <td>0.473684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310</th>\n",
       "      <td>The Cumberland Free Press (Parramatta, NSW : 1...</td>\n",
       "      <td>724</td>\n",
       "      <td>6231</td>\n",
       "      <td>13247</td>\n",
       "      <td>0.470371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>908</th>\n",
       "      <td>Trumpeter General (Hobart, Tas. : 1833 - 1834)</td>\n",
       "      <td>869</td>\n",
       "      <td>693</td>\n",
       "      <td>1482</td>\n",
       "      <td>0.467611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>649</th>\n",
       "      <td>Adelaide Chronicle and South Australian Litera...</td>\n",
       "      <td>986</td>\n",
       "      <td>892</td>\n",
       "      <td>1937</td>\n",
       "      <td>0.460506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>565</th>\n",
       "      <td>Logan Witness (Beenleigh, Qld. : 1878 - 1893)</td>\n",
       "      <td>850</td>\n",
       "      <td>6681</td>\n",
       "      <td>14654</td>\n",
       "      <td>0.455916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>The News, Shoalhaven and Southern Coast Distri...</td>\n",
       "      <td>1588</td>\n",
       "      <td>2466</td>\n",
       "      <td>5495</td>\n",
       "      <td>0.448772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>611</th>\n",
       "      <td>The Darling Downs Gazette and General Advertis...</td>\n",
       "      <td>257</td>\n",
       "      <td>29213</td>\n",
       "      <td>65268</td>\n",
       "      <td>0.447585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>859</th>\n",
       "      <td>The Cornwall Chronicle (Launceston, Tas. : 183...</td>\n",
       "      <td>170</td>\n",
       "      <td>72532</td>\n",
       "      <td>163791</td>\n",
       "      <td>0.442833</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title    id  total_results  \\\n",
       "487   The Weekly Advance (Granville, NSW : 1892 - 1893)  1316           2004   \n",
       "977   Dunolly and Betbetshire Express and County of ...   758           5245   \n",
       "1021  Hamilton Spectator and Grange District Adverti...   927           9438   \n",
       "519   Wagga Wagga Express and Murrumbidgee District ...   382           6956   \n",
       "619   The North Australian, Ipswich and General Adve...   262           6256   \n",
       "618   The North Australian (Brisbane, Qld. : 1863 - ...   264           2868   \n",
       "864        The Herald of Tasmania (Hobart, Tas. : 1845)  1741             26   \n",
       "342   The Hay Standard and Advertiser for Balranald,...   725          21671   \n",
       "209              Robertson Advocate (NSW : 1894 - 1923)   530          36946   \n",
       "235   Temora Herald and Mining Journal (NSW : 1882 -...   728            639   \n",
       "229                     Sydney Mail (NSW : 1860 - 1871)   697          24539   \n",
       "840   Tasmanian Morning Herald (Hobart, Tas. : 1865 ...   865           5178   \n",
       "835   Morning Star and Commercial Advertiser (Hobart...  1242            850   \n",
       "169                    Molong Argus (NSW : 1896 - 1921)   424          52061   \n",
       "1114                 Port Phillip Gazette (Vic. : 1851)  1139            241   \n",
       "1115  Port Phillip Gazette and Settler's Journal (Vi...  1138           5947   \n",
       "846                Telegraph (Hobart Town, Tas. : 1867)  1250             68   \n",
       "872               The Hobart Town Herald (Tas. :  1845)  1740             27   \n",
       "310   The Cumberland Free Press (Parramatta, NSW : 1...   724           6231   \n",
       "908      Trumpeter General (Hobart, Tas. : 1833 - 1834)   869            693   \n",
       "649   Adelaide Chronicle and South Australian Litera...   986            892   \n",
       "565       Logan Witness (Beenleigh, Qld. : 1878 - 1893)   850           6681   \n",
       "392   The News, Shoalhaven and Southern Coast Distri...  1588           2466   \n",
       "611   The Darling Downs Gazette and General Advertis...   257          29213   \n",
       "859   The Cornwall Chronicle (Launceston, Tas. : 183...   170          72532   \n",
       "\n",
       "      total_articles  proportion  \n",
       "487             2954    0.678402  \n",
       "977             8078    0.649294  \n",
       "1021           17227    0.547861  \n",
       "519            12744    0.545825  \n",
       "619            11527    0.542726  \n",
       "618             5314    0.539706  \n",
       "864               50    0.520000  \n",
       "342            42068    0.515142  \n",
       "209            72383    0.510424  \n",
       "235             1253    0.509976  \n",
       "229            48535    0.505594  \n",
       "840            10290    0.503207  \n",
       "835             1703    0.499119  \n",
       "169           104984    0.495895  \n",
       "1114             491    0.490835  \n",
       "1115           12127    0.490393  \n",
       "846              140    0.485714  \n",
       "872               57    0.473684  \n",
       "310            13247    0.470371  \n",
       "908             1482    0.467611  \n",
       "649             1937    0.460506  \n",
       "565            14654    0.455916  \n",
       "392             5495    0.448772  \n",
       "611            65268    0.447585  \n",
       "859           163791    0.442833  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_errors_with_titles[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And those with the lowest rate of errors. Note the number of non-English newspapers in this list – of course our measure of accuracy fails completely in newspapers that don't use the word 'the'!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>id</th>\n",
       "      <th>total_results</th>\n",
       "      <th>total_articles</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>669</th>\n",
       "      <td>Deutsche Zeitung für Sud-Australien = German ...</td>\n",
       "      <td>1577</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>774</th>\n",
       "      <td>The Progressive Times (Largs North, SA : 1949 ...</td>\n",
       "      <td>1307</td>\n",
       "      <td>0</td>\n",
       "      <td>1446</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1506</th>\n",
       "      <td>The Dawn Newsletter (Perth, WA : 1952 - 1954)</td>\n",
       "      <td>1773</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1359</th>\n",
       "      <td>Eco Italiano (Perth, WA : 1958 - 1959)</td>\n",
       "      <td>1387</td>\n",
       "      <td>0</td>\n",
       "      <td>1579</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1208</th>\n",
       "      <td>The Elsternwick Leader and Caulfield and Balac...</td>\n",
       "      <td>200</td>\n",
       "      <td>0</td>\n",
       "      <td>47</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1358</th>\n",
       "      <td>Echo : Polski Tygodnik Niezalezny (Perth, WA :...</td>\n",
       "      <td>1384</td>\n",
       "      <td>0</td>\n",
       "      <td>2601</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Auburn and District News (NSW : 1929)</td>\n",
       "      <td>1320</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1354</th>\n",
       "      <td>Der Australische Spiegel = The Australian Mirr...</td>\n",
       "      <td>1385</td>\n",
       "      <td>0</td>\n",
       "      <td>1455</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>468</th>\n",
       "      <td>The Temora Telegraph and Mining Advocate (NSW ...</td>\n",
       "      <td>729</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>921</th>\n",
       "      <td>Alexandra and Yea Standard, Thornton, Gobur an...</td>\n",
       "      <td>154</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1348</th>\n",
       "      <td>Dampier Despatch (Broome, WA : 1904 - 1905)</td>\n",
       "      <td>1407</td>\n",
       "      <td>0</td>\n",
       "      <td>871</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>The True Sun and New South Wales Independent P...</td>\n",
       "      <td>1038</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Blayney West Macquarie (NSW : 1949)</td>\n",
       "      <td>802</td>\n",
       "      <td>0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>Party (Sydney, NSW : 1942)</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1597</th>\n",
       "      <td>The Southern Cross (Perth, WA : 1893)</td>\n",
       "      <td>1660</td>\n",
       "      <td>0</td>\n",
       "      <td>59</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>905</th>\n",
       "      <td>The Van Diemen's Land Gazette and General Adve...</td>\n",
       "      <td>1047</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193</th>\n",
       "      <td>Out Of Work (Sydney, NSW : 1922)</td>\n",
       "      <td>1008</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>The Derwent Star and Van Diemen's Land Intelli...</td>\n",
       "      <td>1046</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1195</th>\n",
       "      <td>The Chinese Advertiser (Ballarat, Vic. : 1856)</td>\n",
       "      <td>706</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Canberra Illustrated: A Quarterly Magazine (AC...</td>\n",
       "      <td>165</td>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1505</th>\n",
       "      <td>The Dawn News-Sheet (Perth, WA : 1950 - 1952)</td>\n",
       "      <td>1772</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1589</th>\n",
       "      <td>The Possum (Fremantle, WA : 1890)</td>\n",
       "      <td>1201</td>\n",
       "      <td>0</td>\n",
       "      <td>105</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>Citizen Soldier (Sydney, NSW : 1942)</td>\n",
       "      <td>996</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>Clarence and Richmond Examiner (Grafton, NSW :...</td>\n",
       "      <td>104</td>\n",
       "      <td>0</td>\n",
       "      <td>111</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>Party Builder (Sydney, NSW : 1942)</td>\n",
       "      <td>1001</td>\n",
       "      <td>0</td>\n",
       "      <td>160</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title    id  total_results  \\\n",
       "669   Deutsche Zeitung für Sud-Australien = German ...  1577              0   \n",
       "774   The Progressive Times (Largs North, SA : 1949 ...  1307              0   \n",
       "1506      The Dawn Newsletter (Perth, WA : 1952 - 1954)  1773              0   \n",
       "1359             Eco Italiano (Perth, WA : 1958 - 1959)  1387              0   \n",
       "1208  The Elsternwick Leader and Caulfield and Balac...   200              0   \n",
       "1358  Echo : Polski Tygodnik Niezalezny (Perth, WA :...  1384              0   \n",
       "34                Auburn and District News (NSW : 1929)  1320              0   \n",
       "1354  Der Australische Spiegel = The Australian Mirr...  1385              0   \n",
       "468   The Temora Telegraph and Mining Advocate (NSW ...   729              0   \n",
       "921   Alexandra and Yea Standard, Thornton, Gobur an...   154              0   \n",
       "1348        Dampier Despatch (Broome, WA : 1904 - 1905)  1407              0   \n",
       "472   The True Sun and New South Wales Independent P...  1038              0   \n",
       "49                  Blayney West Macquarie (NSW : 1949)   802              0   \n",
       "195                          Party (Sydney, NSW : 1942)  1000              0   \n",
       "1597              The Southern Cross (Perth, WA : 1893)  1660              0   \n",
       "905   The Van Diemen's Land Gazette and General Adve...  1047              0   \n",
       "193                    Out Of Work (Sydney, NSW : 1922)  1008              0   \n",
       "862   The Derwent Star and Van Diemen's Land Intelli...  1046              0   \n",
       "1195     The Chinese Advertiser (Ballarat, Vic. : 1856)   706              0   \n",
       "1     Canberra Illustrated: A Quarterly Magazine (AC...   165              0   \n",
       "1505      The Dawn News-Sheet (Perth, WA : 1950 - 1952)  1772              0   \n",
       "1589                  The Possum (Fremantle, WA : 1890)  1201              0   \n",
       "69                 Citizen Soldier (Sydney, NSW : 1942)   996              0   \n",
       "70    Clarence and Richmond Examiner (Grafton, NSW :...   104              0   \n",
       "196                  Party Builder (Sydney, NSW : 1942)  1001              0   \n",
       "\n",
       "      total_articles  proportion  \n",
       "669               14         0.0  \n",
       "774             1446         0.0  \n",
       "1506              40         0.0  \n",
       "1359            1579         0.0  \n",
       "1208              47         0.0  \n",
       "1358            2601         0.0  \n",
       "34                25         0.0  \n",
       "1354            1455         0.0  \n",
       "468                3         0.0  \n",
       "921               21         0.0  \n",
       "1348             871         0.0  \n",
       "472               20         0.0  \n",
       "49               110         0.0  \n",
       "195                6         0.0  \n",
       "1597              59         0.0  \n",
       "905               38         0.0  \n",
       "193               32         0.0  \n",
       "862               12         0.0  \n",
       "1195              15         0.0  \n",
       "1                 57         0.0  \n",
       "1505              29         0.0  \n",
       "1589             105         0.0  \n",
       "69                60         0.0  \n",
       "70               111         0.0  \n",
       "196              160         0.0  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_errors_with_titles[-25:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's merge the error data with the correction data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrections_errors_merged_df = pd.merge(\n",
    "    df_newspapers_with_titles, df_errors_with_titles, how=\"left\", on=\"id\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title_x</th>\n",
       "      <th>id</th>\n",
       "      <th>total_results_x</th>\n",
       "      <th>total_articles_x</th>\n",
       "      <th>proportion_x</th>\n",
       "      <th>title_y</th>\n",
       "      <th>total_results_y</th>\n",
       "      <th>total_articles_y</th>\n",
       "      <th>proportion_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The Temora Telegraph and Mining Advocate (NSW ...</td>\n",
       "      <td>729</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The Temora Telegraph and Mining Advocate (NSW ...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hobart Town Gazette and Van Diemen's Land Adve...</td>\n",
       "      <td>5</td>\n",
       "      <td>1556</td>\n",
       "      <td>1556</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Hobart Town Gazette and Van Diemen's Land Adve...</td>\n",
       "      <td>40</td>\n",
       "      <td>1556</td>\n",
       "      <td>0.025707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The Satirist and Sporting Chronicle (Sydney, N...</td>\n",
       "      <td>1028</td>\n",
       "      <td>286</td>\n",
       "      <td>286</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The Satirist and Sporting Chronicle (Sydney, N...</td>\n",
       "      <td>0</td>\n",
       "      <td>286</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Justice (Narrabri, NSW : 1891)</td>\n",
       "      <td>885</td>\n",
       "      <td>45</td>\n",
       "      <td>45</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Justice (Narrabri, NSW : 1891)</td>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "      <td>0.022222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alexandra and Yea Standard, Thornton, Gobur an...</td>\n",
       "      <td>154</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Alexandra and Yea Standard, Thornton, Gobur an...</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             title_x    id  total_results_x  \\\n",
       "0  The Temora Telegraph and Mining Advocate (NSW ...   729                3   \n",
       "1  Hobart Town Gazette and Van Diemen's Land Adve...     5             1556   \n",
       "2  The Satirist and Sporting Chronicle (Sydney, N...  1028              286   \n",
       "3                     Justice (Narrabri, NSW : 1891)   885               45   \n",
       "4  Alexandra and Yea Standard, Thornton, Gobur an...   154               21   \n",
       "\n",
       "   total_articles_x  proportion_x  \\\n",
       "0                 3           1.0   \n",
       "1              1556           1.0   \n",
       "2               286           1.0   \n",
       "3                45           1.0   \n",
       "4                21           1.0   \n",
       "\n",
       "                                             title_y  total_results_y  \\\n",
       "0  The Temora Telegraph and Mining Advocate (NSW ...                0   \n",
       "1  Hobart Town Gazette and Van Diemen's Land Adve...               40   \n",
       "2  The Satirist and Sporting Chronicle (Sydney, N...                0   \n",
       "3                     Justice (Narrabri, NSW : 1891)                1   \n",
       "4  Alexandra and Yea Standard, Thornton, Gobur an...                0   \n",
       "\n",
       "   total_articles_y  proportion_y  \n",
       "0                 3      0.000000  \n",
       "1              1556      0.025707  \n",
       "2               286      0.000000  \n",
       "3                45      0.022222  \n",
       "4                21      0.000000  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corrections_errors_merged_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrections_errors_merged_df[\"proportion_uncorrected\"] = corrections_errors_merged_df[\n",
    "    \"proportion_x\"\n",
    "].apply(lambda x: 1 - x)\n",
    "corrections_errors_merged_df.rename(\n",
    "    columns={\n",
    "        \"title_x\": \"title\",\n",
    "        \"proportion_x\": \"proportion_corrected\",\n",
    "        \"proportion_y\": \"proportion_with_errors\",\n",
    "    },\n",
    "    inplace=True,\n",
    ")\n",
    "corrections_errors_merged_df.sort_values(\n",
    "    by=[\"proportion_with_errors\", \"proportion_uncorrected\"],\n",
    "    ascending=False,\n",
    "    inplace=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, for what it's worth, here's a list of the neediest newspapers – those with high error rates and low correction rates! As I've said, this is a pretty dodgy method, but interesting nonetheless."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>proportion_with_errors</th>\n",
       "      <th>proportion_uncorrected</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1193</th>\n",
       "      <td>The Weekly Advance (Granville, NSW : 1892 - 1893)</td>\n",
       "      <td>0.678402</td>\n",
       "      <td>0.970887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>630</th>\n",
       "      <td>Dunolly and Betbetshire Express and County of ...</td>\n",
       "      <td>0.649294</td>\n",
       "      <td>0.929809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>Hamilton Spectator and Grange District Adverti...</td>\n",
       "      <td>0.547861</td>\n",
       "      <td>0.886631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>Wagga Wagga Express and Murrumbidgee District ...</td>\n",
       "      <td>0.545825</td>\n",
       "      <td>0.893597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>189</th>\n",
       "      <td>The North Australian, Ipswich and General Adve...</td>\n",
       "      <td>0.542726</td>\n",
       "      <td>0.758393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>The North Australian (Brisbane, Qld. : 1863 - ...</td>\n",
       "      <td>0.539706</td>\n",
       "      <td>0.827249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1653</th>\n",
       "      <td>The Herald of Tasmania (Hobart, Tas. : 1845)</td>\n",
       "      <td>0.520000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1030</th>\n",
       "      <td>The Hay Standard and Advertiser for Balranald,...</td>\n",
       "      <td>0.515142</td>\n",
       "      <td>0.960397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>Robertson Advocate (NSW : 1894 - 1923)</td>\n",
       "      <td>0.510424</td>\n",
       "      <td>0.949864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td>Temora Herald and Mining Journal (NSW : 1882 -...</td>\n",
       "      <td>0.509976</td>\n",
       "      <td>0.927374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>349</th>\n",
       "      <td>Sydney Mail (NSW : 1860 - 1871)</td>\n",
       "      <td>0.505594</td>\n",
       "      <td>0.871639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>509</th>\n",
       "      <td>Tasmanian Morning Herald (Hobart, Tas. : 1865 ...</td>\n",
       "      <td>0.503207</td>\n",
       "      <td>0.910690</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>Morning Star and Commercial Advertiser (Hobart...</td>\n",
       "      <td>0.499119</td>\n",
       "      <td>0.707575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>691</th>\n",
       "      <td>Molong Argus (NSW : 1896 - 1921)</td>\n",
       "      <td>0.495895</td>\n",
       "      <td>0.935047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>Port Phillip Gazette (Vic. : 1851)</td>\n",
       "      <td>0.490835</td>\n",
       "      <td>0.798371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>Port Phillip Gazette and Settler's Journal (Vi...</td>\n",
       "      <td>0.490393</td>\n",
       "      <td>0.786757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>Telegraph (Hobart Town, Tas. : 1867)</td>\n",
       "      <td>0.485714</td>\n",
       "      <td>0.771429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1654</th>\n",
       "      <td>The Hobart Town Herald (Tas. :  1845)</td>\n",
       "      <td>0.473684</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>361</th>\n",
       "      <td>The Cumberland Free Press (Parramatta, NSW : 1...</td>\n",
       "      <td>0.470371</td>\n",
       "      <td>0.876349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>Trumpeter General (Hobart, Tas. : 1833 - 1834)</td>\n",
       "      <td>0.467611</td>\n",
       "      <td>0.671390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>Adelaide Chronicle and South Australian Litera...</td>\n",
       "      <td>0.460506</td>\n",
       "      <td>0.665978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>Logan Witness (Beenleigh, Qld. : 1878 - 1893)</td>\n",
       "      <td>0.455916</td>\n",
       "      <td>0.828102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1139</th>\n",
       "      <td>The News, Shoalhaven and Southern Coast Distri...</td>\n",
       "      <td>0.448772</td>\n",
       "      <td>0.967789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>278</th>\n",
       "      <td>The Darling Downs Gazette and General Advertis...</td>\n",
       "      <td>0.447585</td>\n",
       "      <td>0.836168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>The Cornwall Chronicle (Launceston, Tas. : 183...</td>\n",
       "      <td>0.442833</td>\n",
       "      <td>0.816089</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title  \\\n",
       "1193  The Weekly Advance (Granville, NSW : 1892 - 1893)   \n",
       "630   Dunolly and Betbetshire Express and County of ...   \n",
       "393   Hamilton Spectator and Grange District Adverti...   \n",
       "419   Wagga Wagga Express and Murrumbidgee District ...   \n",
       "189   The North Australian, Ipswich and General Adve...   \n",
       "259   The North Australian (Brisbane, Qld. : 1863 - ...   \n",
       "1653       The Herald of Tasmania (Hobart, Tas. : 1845)   \n",
       "1030  The Hay Standard and Advertiser for Balranald,...   \n",
       "857              Robertson Advocate (NSW : 1894 - 1923)   \n",
       "615   Temora Herald and Mining Journal (NSW : 1882 -...   \n",
       "349                     Sydney Mail (NSW : 1860 - 1871)   \n",
       "509   Tasmanian Morning Herald (Hobart, Tas. : 1865 ...   \n",
       "144   Morning Star and Commercial Advertiser (Hobart...   \n",
       "691                    Molong Argus (NSW : 1896 - 1921)   \n",
       "220                  Port Phillip Gazette (Vic. : 1851)   \n",
       "204   Port Phillip Gazette and Settler's Journal (Vi...   \n",
       "197                Telegraph (Hobart Town, Tas. : 1867)   \n",
       "1654              The Hobart Town Herald (Tas. :  1845)   \n",
       "361   The Cumberland Free Press (Parramatta, NSW : 1...   \n",
       "134      Trumpeter General (Hobart, Tas. : 1833 - 1834)   \n",
       "127   Adelaide Chronicle and South Australian Litera...   \n",
       "261       Logan Witness (Beenleigh, Qld. : 1878 - 1893)   \n",
       "1139  The News, Shoalhaven and Southern Coast Distri...   \n",
       "278   The Darling Downs Gazette and General Advertis...   \n",
       "243   The Cornwall Chronicle (Launceston, Tas. : 183...   \n",
       "\n",
       "      proportion_with_errors  proportion_uncorrected  \n",
       "1193                0.678402                0.970887  \n",
       "630                 0.649294                0.929809  \n",
       "393                 0.547861                0.886631  \n",
       "419                 0.545825                0.893597  \n",
       "189                 0.542726                0.758393  \n",
       "259                 0.539706                0.827249  \n",
       "1653                0.520000                1.000000  \n",
       "1030                0.515142                0.960397  \n",
       "857                 0.510424                0.949864  \n",
       "615                 0.509976                0.927374  \n",
       "349                 0.505594                0.871639  \n",
       "509                 0.503207                0.910690  \n",
       "144                 0.499119                0.707575  \n",
       "691                 0.495895                0.935047  \n",
       "220                 0.490835                0.798371  \n",
       "204                 0.490393                0.786757  \n",
       "197                 0.485714                0.771429  \n",
       "1654                0.473684                1.000000  \n",
       "361                 0.470371                0.876349  \n",
       "134                 0.467611                0.671390  \n",
       "127                 0.460506                0.665978  \n",
       "261                 0.455916                0.828102  \n",
       "1139                0.448772                0.967789  \n",
       "278                 0.447585                0.836168  \n",
       "243                 0.442833                0.816089  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corrections_errors_merged_df[\n",
    "    [\"title\", \"proportion_with_errors\", \"proportion_uncorrected\"]\n",
    "][:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherratt](https://timsherratt.org/) for the [GLAM Workbench](https://glam-workbench.github.io/).  \n",
    "Support this project by becoming a [GitHub sponsor](https://github.com/sponsors/wragge?o=esb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}