CAS_ToponymExperiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3a6e9b9",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook illustrates how we can use NER to search for placenames in a corpus, and enhance a gazetteer. It uses two datesets to illustrate the concepts.\n",
    "\n",
    "1) [Geograph](https://geograph.org.uk) \n",
    "This site invites users to take pictures in the UK and add descriptions. It has almost 7 million pictures, and the data are licenced using a CC By-SA licence, making them available for research as long as we keep the names of the users, and allow others to have access to any data we might create.\n",
    "\n",
    "2) [Ordnance Survey](https://ordnancesurvey.co.uk/) 50k gazetteer\n",
    "This gazetteer was published under a UK Open Government licence and contains all place name found on 1:50k maps in the UK. It is a legacy product (i.e. not used or updated any more), but it is suitable for our purposes.\n",
    "\n",
    "We are going to look for names found in the Geograph data that don't exist in the gazetteer. Since we know that many names occur multiple times, we will do this locally, to increase the chances that we really find new names.\n",
    "\n",
    "**The first block of our code reads in data and builds a simple spatial index for the gazetteer. We only need to do this once.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5a2314b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import OSGridConverter #To convert from =SGB36 to WGS84\n",
    "import pandas as pd #To use pandas for elegant data handling\n",
    "import spacy #Our NLP tools\n",
    "import matplotlib.pyplot as plt #To plot results\n",
    "\n",
    "\n",
    "import os #We import os to stop a weird kernel crash...\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='TRUE'\n",
    "\n",
    "#Load a language model to do NLP\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c78ff881",
   "metadata": {},
   "outputs": [],
   "source": [
    "#First we read in the geograph data\n",
    "geograph = pd.read_csv('./data/geograph_mini_corpus.csv', encoding='latin-1')\n",
    "print(len(geograph))\n",
    "geograph.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8853829",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read in the Ordnance Survey gazetteer\n",
    "os_50k = pd.read_csv('./data/50kgaz2012.txt',sep=':', encoding='utf8', header=None)\n",
    "os_50k.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd1b8b69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#We need to create locations for each place name in decimal degrees\n",
    "os_50k[20] = os_50k[4] + os_50k[5]/60\n",
    "os_50k[21] = os_50k[6] + os_50k[7]/60\n",
    "os_50k.loc[os_50k[10] == 'W', 21] = -1 * os_50k.loc[os_50k[10] == 'W', 21]\n",
    "\n",
    "#Tidy up the gazetteer by dropping columns we won't use afterwards\n",
    "os_trimmed = os_50k.drop([0,1,3,4,5,6,7,10,11,12,14,15,17,18,19], axis = 1)\n",
    "os_trimmed.head()\n",
    "os_trimmed.columns = ['name','y','x',\n",
    "                     'county','type','lat','lon']\n",
    "os_trimmed.head()\n",
    "\n",
    "#Plot the gazetteer locations as a sanity check\n",
    "os_trimmed.plot.scatter(x = 'x', y = 'y')\n",
    "print(len(os_trimmed))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04dc846e",
   "metadata": {},
   "source": [
    "This block is just to show the NLP results for a single document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3acc01eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#First we demonstrate how the NLP works for a single example document\n",
    "doc = nlp(geograph.text[11])\n",
    "spacy.displacy.render(doc, style=\"ent\")\n",
    "doc_nouns = list(doc.noun_chunks)\n",
    "print(\"nouns:\",doc_nouns)\n",
    "for ent in doc.ents:\n",
    "      print(\"NER:\", ent.text, ent.label_)\n",
    "          "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f300b16f",
   "metadata": {},
   "source": [
    "Here we **draw a random sample of documents** from the Geograph data and perform NER on those data. We can rerun this block to build a new sample. The size of this sample can also be changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11cb825e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now I am going to load a random sample of m Geograph documents and run the NER pipeline\n",
    "m = 200\n",
    "sample = geograph.sample(n = m)\n",
    "docs = list(nlp.pipe(sample.text,n_process=2, batch_size=100))\n",
    "\n",
    "#Now we store the NER results with the original geograph document ids, text and coordinates\n",
    "results = list()\n",
    "for (idxRow, s1), (_, s2) in zip(sample.iterrows(), enumerate(docs)):\n",
    "    try:\n",
    "        g = OSGridConverter.latlong2grid (s1.lat, s1.lon, tag = 'WGS84')\n",
    "        dict = {\n",
    "            \"id\": s1.id,\n",
    "            \"x\": g.E,\n",
    "            \"y\": g.N,\n",
    "            \"entities\": s2.ents,\n",
    "            \"text\": s1.text   \n",
    "        }\n",
    "        results.append(dict)\n",
    "    except ValueError:\n",
    "        print(\"Problem with a document\", s1.text)\n",
    "        #Ignore documents we couldn't parse for some reason"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9a837f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Show the results - these are essentially all the NER tokens found in each document\n",
    "for dict in results:\n",
    "    print(dict.get(\"id\"), dict.get(\"x\"), dict.get(\"y\"), dict.get(\"entities\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c0e2f9",
   "metadata": {},
   "source": [
    "We can change the resolution of our local gazetteer and explore how this changes our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a24cd0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#We build a simple index using rectangular grid cells\n",
    "#Each cell contains all placenames from the gazetteer for that cell and can be used to do a lookup\n",
    "#cellSize determines the resolution in metres\n",
    "\n",
    "cellSize = 5000\n",
    "minx = os_trimmed['x'].min()\n",
    "maxx = os_trimmed['x'].max()\n",
    "miny = os_trimmed['y'].min()\n",
    "maxy = os_trimmed['y'].max()\n",
    "\n",
    "w = maxx - minx\n",
    "h = maxy - miny\n",
    "\n",
    "nc = int(w/cellSize) + 1\n",
    "nr = int(h/cellSize) + 1\n",
    "\n",
    "gaz = pd.DataFrame(index=range(nc),columns=range(nr))\n",
    "\n",
    "#Now we populate the index with names\n",
    "for index, row in os_trimmed.iterrows():\n",
    "    i = int((row['x'] - minx)/cellSize)\n",
    "    j = int((row['y'] - minx)/cellSize)\n",
    "    name = row['name']\n",
    "    \n",
    "    if pd.isnull(gaz.at[i,j]):\n",
    "        gaz.at[i,j] = {name}\n",
    "    else:\n",
    "        names = gaz.at[i,j]\n",
    "        names.add(name)\n",
    "        gaz.at[i,j] = names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6542269",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example output for a single cell - returns nan if no names present\n",
    "gaz.at[20,20]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79585bb5",
   "metadata": {},
   "source": [
    "If we change the sample of names, or the resolution of the gazetteer, then the results of the following **comparison** should change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24b24a99",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now we are going to compare the gazetteer names with those we found\n",
    "\n",
    "data = list()\n",
    "#We iterate through all our results\n",
    "for dict in results:\n",
    "    #First we get the cell indices for the gazetteer\n",
    "    x = dict.get(\"x\")\n",
    "    y = dict.get(\"y\")\n",
    "    i = int((x - minx)/cellSize)\n",
    "    j = int((y - minx)/cellSize)\n",
    "    try:\n",
    "        #Now we find the names in that cell - n.B. we ignore for now the fact that Geograph cell could be at a boundary\n",
    "        gazNames = gaz.at[i,j]\n",
    "        #Deal with a cell having no values in the gazetteer\n",
    "        if (isinstance(gazNames,set) == False): \n",
    "            #print(type(gazNames))\n",
    "            gazNames = {\"NoNamesFound\"}\n",
    "    except KeyError:\n",
    "        gazNames = {\"NoNamesFound\"}\n",
    "    #Get back the named entities for the text        \n",
    "    ents = dict.get(\"entities\")\n",
    "    #Now we iterate through, and find out if each name is already in the local gazetteer\n",
    "    for ent in ents:        \n",
    "        if (ent.text in gazNames):\n",
    "            data.append([dict.get(\"id\"), \"Existing\", ent.text, ent.label_, x, y]) \n",
    "            #print(\"Found existing name:\", ent.text, ent.label_)\n",
    "        else:\n",
    "            #print(\"Potential new name:\" , ent.text, ent.label_)\n",
    "            data.append([dict.get(\"id\"), \"New\", ent.text, ent.label_, x, y]) \n",
    "#Store the results in a dataframe\n",
    "df = pd.DataFrame(data, columns = ['id', 'status','name','type','x','y'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7380072",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Split results into existing and candidate names for reporting\n",
    "new = df.loc[df['status'] == 'New']\n",
    "existing = df.loc[df['status'] == 'Existing']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c87bfdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a bar plot of existing names\n",
    "count = list()\n",
    "values = list()\n",
    "for value in set(existing['type']):\n",
    "    count.append(list(existing['type']).count(value))\n",
    "    values.append(value)\n",
    "y_pos = range(len(values))\n",
    "p = plt.bar(y_pos,count)\n",
    "# Rotation of the bars names\n",
    "plt.xticks(y_pos, values, rotation=270)\n",
    "plt.title('Counts for existing names')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f264911",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a bar plot of new names\n",
    "count = list()\n",
    "values = list()\n",
    "for value in set(new['type']):\n",
    "    count.append(list(new['type']).count(value))\n",
    "    values.append(value)\n",
    "y_pos = range(len(values))\n",
    "p = plt.bar(y_pos,count)\n",
    "# Rotation of the bars names\n",
    "plt.xticks(y_pos, values, rotation=270)\n",
    "plt.title('Counts for candidate new names')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03f1164f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's look at one example NER class in the candidate names\n",
    "names = new.loc[new['type'] == 'PERSON']\n",
    "\n",
    "with pd.option_context('display.max_rows', None,\n",
    "                       'display.max_columns', None,\n",
    "                       'display.precision', 3,\n",
    "                       ):\n",
    "    print(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe46eafe",
   "metadata": {},
   "outputs": [],
   "source": [
    "cn = len(set(new['name']))\n",
    "ce = len(set(existing['name']))\n",
    "print(\"Found\", ce, \"unique existing names and\", cn, \"unique new names.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f27380bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#output all dependencies so that we can reproduce the notebook (we only need this to set things up for Binder)\n",
    "#%load_ext watermark\n",
    "#%watermark --iversions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}