SpatialSearchDemo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9b8ea643",
   "metadata": {},
   "outputs": [],
   "source": [
    "import OSGridConverter #To convert from =SGB36 to WGS84\n",
    "import pandas as pd #To use pandas for elegant data handling\n",
    "import numpy as np\n",
    "import math\n",
    "import plotly.express as px\n",
    "import pickle\n",
    "from pathlib import Path\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf9f511d",
   "metadata": {},
   "source": [
    "First, we need to run the three classes below (`SpatialIndex`, `Postings` and `Gazetteer`). These are required to do all the work.\n",
    "\n",
    "We read in the data first to create our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "24f07eb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The initial corpus contains 138639 documents.\n"
     ]
    }
   ],
   "source": [
    "data_folder = Path('./data')\n",
    "fn1 = data_folder / 'geograph_mini_corpus.csv'\n",
    "\n",
    "geograph = pd.read_csv(fn1, encoding='latin-1')\n",
    "        \n",
    "#sample = geograph.sample(n = 1000) # For testing\n",
    "sample = geograph\n",
    "\n",
    "print(f'The initial corpus contains {len(sample)} documents.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "366a2188",
   "metadata": {},
   "source": [
    "Now we create the three indexes we need:\n",
    "\n",
    "- The spatial index stores documents in square cells and allows us to perform a range query\n",
    "- The postings index indexes terms and the documents in which they are found\n",
    "- The gazetteer indexes toponyms and allows us to look them up\n",
    "\n",
    "The spatial index takes a resolution as a parameter\n",
    "The postings file is restricted to terms from this [paper](https://firstmonday.org/ojs/index.php/fm/article/view/3710/3035) if the argument is true. If false then all terms (including stop words) are indexed.\n",
    "\n",
    "Building these indexes takes time, but only needs to be done once. To make the code run faster in Binder, I use pickles (basically files that store the binary representation of the indexes, after creation). If we change `changes` to be `True` then the indexes are recalculated, but this is **slow**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "65e3018a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I use pickles here to load objects that haven't changed. That is much quicker than rebuilding\n",
    "# the indexes from scratch, especially the postings one (where I do NLP on all of our documents)\n",
    "changes = False\n",
    "data_folder = Path('./data')\n",
    "fn1 = data_folder / 'si.obj'\n",
    "fn2 = data_folder / 'postings.obj'\n",
    "fn3 = data_folder / 'gaz.obj'\n",
    "nlp = spacy.load('en_core_web_md')\n",
    "\n",
    "if changes:\n",
    "    resolution = 10000\n",
    "    limitedTerms = True\n",
    "    si = SpatialIndex(resolution, sample)\n",
    "    postings = Postings(limitedTerms, sample, nlp)\n",
    "    gaz = Gazetteer()\n",
    "    f1 = open(fn1, 'wb')\n",
    "    pickle.dump(si,f1)\n",
    "    f1.close()\n",
    "    f2 = open(fn2, 'wb')\n",
    "    pickle.dump(postings,f2)\n",
    "    f2.close()\n",
    "    f3 = open(fn3, 'wb')\n",
    "    pickle.dump(gaz,f3)\n",
    "    f3.close()\n",
    "else:\n",
    "    f1 = open(fn1, 'rb')\n",
    "    si = pickle.load(f1)\n",
    "    f1.close()\n",
    "    f2 = open(fn2, 'rb')\n",
    "    postings = pickle.load(f2)\n",
    "    f2.close()\n",
    "    f3 = open(fn3, 'rb')\n",
    "    gaz = pickle.load(f3)\n",
    "    f3.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7aee1f8",
   "metadata": {},
   "source": [
    "Demonstrates a look up in the gazetteer. For some toponym types point and bounding box are returned. Bounding box could be used to exclude documents for near queries. Location is returned as a tuple, and the first two elements are used to query the spatial index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "9fe8e929",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['London' 181500 531500 'City of London' 'C']\n",
      "(531500, 181500, 529500, 179500, 533500, 183500)\n"
     ]
    }
   ],
   "source": [
    "name = 'London'\n",
    "location = gaz.getLocation(name)\n",
    "print(location)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b3aa2a",
   "metadata": {},
   "source": [
    "Find all documents within a bounding box \"centred\" around the location returned from the gazetteer. Documents are returned with their distances and ranked according to distance from toponym. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "902beebe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 8084 documents in a 100000m x 100000m around London.\n"
     ]
    }
   ],
   "source": [
    "maxDist = 100000\n",
    "docsSpatial = si.rangeQuery(maxDist, (location[0],location[1]))\n",
    "print(f'Found {len(docsSpatial)} documents in a {maxDist}m x {maxDist}m around {name}.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba34fec",
   "metadata": {},
   "source": [
    "Do a query and return documents ranked by tf-idf scores in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "88612dae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 20489 documents using the query: hill big rough\n"
     ]
    }
   ],
   "source": [
    "query = 'hill big rough'\n",
    "docsThematic = postings.tfIdf(query, nlp)\n",
    "maxThematic = next(iter(docsThematic.items()))[1] # Used to normalise scores\n",
    "print(f'Found {len(docsThematic)} documents using the query: {query}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d03385b6",
   "metadata": {},
   "source": [
    "Find the unique documents from both sets. Typically there will be some duplicates, so the sum of the two previous numbers is the maximum value we can find here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "06b938a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 27965 unique documents merging the thematic and spatial indexes\n"
     ]
    }
   ],
   "source": [
    "#Create a set of unique keys for all the documents retrieved from the thematic and spatial indexes\n",
    "candidates = set(list(docsSpatial.keys()) + list(docsThematic.keys()))\n",
    "\n",
    "print(f'Found {len(candidates)} unique documents merging the thematic and spatial indexes')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86933adb",
   "metadata": {},
   "source": [
    "Rank documents according to spatial (distance) and thematic (tf-idf) scores. `wspatial` determines the **weight** of each component. A **value of 1.0** means that the ranking is purely **spatial**, **0.0** considers only **tf-idf scores**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "959c1660",
   "metadata": {},
   "outputs": [],
   "source": [
    "wspatial = 0.5 # Weight of the spatial score -> weight of 1 means tfidf is ignored\n",
    "\n",
    "scores = dict()\n",
    "for doc in candidates:\n",
    "    st = 0\n",
    "    ss = 0\n",
    "    if doc in docsThematic:\n",
    "        st = docsThematic[doc]/ maxThematic\n",
    "        #print(f'thematic {st}')\n",
    "    if doc in docsSpatial:\n",
    "        ss = 1- docsSpatial[doc]/ maxDist\n",
    "        #print(f'spatial {docsSpatial[doc]} {ss}')\n",
    "    score = ((1-wspatial) * st) + (wspatial * ss)\n",
    "    scores[doc] = score\n",
    "ranked = dict(sorted(scores.items(), key = lambda x: x[1], reverse=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec083a1",
   "metadata": {},
   "source": [
    "This section of code outputs the results in order. It retrieves the document text from the original data frame to display, not that we didn't use this up to now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "0bbbd305",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.67: Hammersmith Town Hall in daylight - Hammersmith Town Hall this time taken on the left of the tree. The bigger building to the left and back of the 1930's version is the soon to be demolished modern main entrance.\n",
      "\n",
      "0.65: Fallen tree by Western Avenue - This tree fell down in big storms the day before. The railway beyond is the Central Line.\n",
      "\n",
      "0.62: Ever hopeful hungry water birds - At Hainault Forest Country Park this collection of water fowl were ever hopeful. Luckily ''Mums and Dads'' and ''Grandparents'' with their ''offspring'' were present with bags of unused or stale bread to replenish food stocks. Why is it small hands always seem to have big pieces to toss towards the eager beaks rather the dainty bits held by their elder stewards?\n",
      "\n",
      "0.60: Pashley Manor oak tree opposite car park - My big regret was not measuring the girth of this tree. If the Sussex post and rail fencing behind it is set at 6 foot centres then the tree could be 5 or 6 feet in diameter. The former would equate to a girth of about 15.75 feet or around 260 years old and the latter over 18.85 feet around or around one hundred years older. But it could be bigger still? See: wbrc.org.uk/.../Estimating%20Age%20of%20Oaks%20-...\n",
      "\n",
      "0.60: A2 Blackheath Hill - The main A2 road climbs a steep hill onto Blackheath in South East London.  This view looks westwards down the hill towards Deptford.\n",
      "\n",
      "0.59: Long Lane - Looking along the lane towards the junction with Squires Lane at the traffic lights in the distance.  The nearest side road with newsagents on the corner is Abingdon Road.  A large advertising hoarding has recently been removed from the side of the shop and the site of a much smaller previous one revealed. At the bottom of the hill is Cromwell Road (behind the red van) and halfway up the hill to Squires Lane is Montpelier Road\n",
      "\n",
      "0.59: Brent Street - Heavy traffic on the hill beside Brent Green for a Sunday lunchtime as shoppers make their way to Brent Cross.  There are probably lots of cars waiting to turn right further down the hill onto Shirehall Lane.  The brightness of the head and tail lights hints at how gloomy the day is.  The roundabout is at the junction with Queen's Road.   The trees on the distant horizon are on Hampstead Heath.\n",
      "\n",
      "0.57: City Thameslink - A First CapitalConnect service for St. Albans drifts down the hill from Blackfriars to City Thameslink station.  View taken from the Apothecary Street footbridge.\n",
      "\n",
      "0.57: St. Andrew's Hill - Named after the church of St. Andrew by the Wardrobe which stands out of sight close to the bottom of the hill.\n",
      "\n",
      "0.57: Ireland Yard - Off St Andrew's Hill\n",
      "\n",
      "0.57: Mill Way - Looking down the hill from adjacent to the junction with Russell Grove.  This street connects the side roads off the A1 with The Broadway at the bottom of the hill.  Note the road surface breaking up in the foreground following the bad winter weather.\n",
      "\n",
      "0.56: Acton Street (2) - Rather grander houses than those lower down the hill.\n",
      "\n",
      "0.56: Fish Street Hill. - The hill down to Old London Bridge.\n",
      "\n",
      "0.56: Archangel Michael - The Protector - In the grounds of St Pancras Church. The plaque inscription reads \" In memory of the victims of the 7th July 2005 bombings and all victims of violence.  'I will lift up my eyes unto the hills' Psalm 121\"\n",
      "\n",
      "0.56: Haymarket - Looking up the hill from pall Mall\n",
      "\n",
      "0.55: Lamp post on Primrose Hill - A hill 256 feet high to the north of Regents Park offering a view over London.  A lamp post appears at more or less the highest point.\n",
      "\n",
      "0.55: View up Primrose Hill to viewpoint - See [[921101]] for a view at the top of the hill.\n",
      "\n",
      "0.55: Show Off on Bread & Cheese Hill - This is a view of the A13 Bread & Cheese Hill Thundersley. The hill rises 54m from the 30m contour line at TQ775844 to the 84m Spot Height at TQ788879 (about 1.5km in road distance). That's a long hill in Essex!. The hill got its name in the Edwardian era when motorcars on the way to Southend-on-Sea would boil over by the time they reached the brow of the hill. Chauffeurs would eat their bread and cheese lunch while the engines cooled. It was not long before a hostelry (which still survives) was opened to take advantage of the trade. There are no prizes for guessing its name!\n",
      "\n",
      "0.55: Primrose Hill in evening sunshine - This path leads up from the northern corner of Primrose Hill in Primrose Hill Road towards the summit of the hill where people can be seen on the left hand horizon. On this fine March evening this was the last of the sunlight.\n",
      "\n",
      "0.55: Path up Primrose Hill - This path leads up from the northern corner of Primrose Hill in Primrose Hill Road towards the summit of the hill where people can be seen admiring the view. On a fine March evening this was the last of the sunlight.\n",
      "\n",
      "0.54: Parliament Hill Fields - This part of the hill has been set aside for the natural growth of wild flowers.\n",
      "\n",
      "0.54: Rosslyn Hill - Looking down the hill towards Belsize Park.  The road on the right is Shepherds Walk.\n",
      "\n",
      "0.54: Kemplay Road - Looking down the hill from Willoughby Road towards Pilgrims Lane.  The multistorey terraced housing is quite varied along this road compared with the neighbouring Carlingford Road.\n",
      "\n",
      "0.54: Waterlow sunflowers - There was a large group of sunflowers near the top of a hill.  This zoom shot picks out some of the brighter ones on this occasion.\n",
      "\n",
      "0.54: Parting of the Ways - Holly Hill and its footpath part company.  The footpath rejoins further down the hill.  It also connects with Holly Walk.\n",
      "\n",
      "0.54: Latchmere Road - Three-storey brick houses (many now flats) on a busy road ascending the hill from the Thames to Clapham Common.\n",
      "\n",
      "0.54: View across London from Hampstead Heath - This is taken from the highest point in London and looks across the centre to the hills beyond.\n",
      "\n",
      "0.54: Road between Shepherd's Hill and Highgate tube station - This unnamed tree lined road leads from Shepherd's Hill to [[1296359]]. Looking towards Shepherd's Hill\n",
      "\n",
      "0.54: Holmdale Road - Looking up the hill towards Mill Lane from close to the junction with Inglewood Road.   The houses here date to around 1880 - being fully mapped by 1894.\n",
      "\n",
      "0.54: West Heath Road - View of the road from West Heath.  A jogger pounds up the hill on the sandy gravel that forms the geology of this area.\n",
      "\n",
      "0.54: Hewitt Road - One of the streets built circa 1880 as part of the Harringay Park development between Green Lanes and the Great northern mainline.  This network of streets is also referred to as the Harringay Ladder because of their ground plan.  This view is looking down the hill towards Green Lanes.\n",
      "\n",
      "0.53: Sumatra Road - Looking up the hill towards Mill Lane.  The housing here dates to around the 1880's.   The pollarded trees would appear to Limes.\n",
      "\n",
      "0.53: St. John's Hill at Night - Looking down the hill towards the Debenhams store from adjacent to The Windsor Castle public house (on the left).\n",
      "\n",
      "0.53: Greenwich Naval Hospital - View across the square of the former Greenwich Naval Hospital with the observatory on the hill in the distance\n",
      "\n",
      "0.53: North End Road - A 210 bus crawls up the hill towards Hampstead with a string of cars in tow.  The steepness of the incline at this point is evidenced by the manner in which the pedestrians are walking.\n",
      "\n",
      "0.53: Greenwich Park and Palace - Taken from up the hill near the observatory.\n",
      "\n",
      "0.53: People enjoying the park - People enjoying a warm spring day in the park.  Looking down the main hill where the Observatory is situated.\n",
      "\n",
      "0.53: Emergency Access - Cascade Avenue is one of three streets near the bottom of Muswell Hill that do not permit access to the hill itself.  All connect with Etheldene Avenue at the other end.\n",
      "\n",
      "0.53: Alexandra Palace Park - By taking this path you can avoid going up the hill upon which sits Alexandra Palace sits however you may come across ravens\n",
      "\n",
      "0.53: Cricklewood Lane - Taken at 13:00 on Christmas day.  There was light traffic in the area and The Tavern on the crest of the hill was quite busy.   The bus shelter has a suitably festive McDonald's advert.\n",
      "\n",
      "0.53: Greenwich Park: Maze Hill gate - One of two pedestrian entrances to the park on the eastern side (the other being the Vanbrugh gate higher up the hill to the south).\n",
      "\n",
      "0.53: Public Road Transport at East Finchley Station - Buses on the 263 and 102 routes wait in the queue of traffic beside the station while a taxi stands on the forecourt.  The traffic jam has been caused by the temporary lights up the hill at the junction of High Road and Fortis Green\n",
      "\n",
      "0.53: View of Westminster and Central London from Canonbie Rd - View looking from the crest of the hill of Canonbie Road. Central London including The Eye can be seen through the morning mist.\n",
      "\n",
      "0.53: St Jude On the Hill - Looking from Hampstead Heath Extension down Heathgate to Church of St Jude On the Hill\n",
      "\n",
      "0.53: Durham Road - Seen at the junction with Huntingdon Road in the county roads part of East Finchley.  Another delivery lorry makes its way up the hill.  The grey veil of cloud on the left is about to bring more rain.\n",
      "\n",
      "0.53: Northway Circus - Looking up the hill towards Central Square from the junction with Litchfield Way.  The leaves are turning in this early November view.\n",
      "\n",
      "0.52: West Norwood station entrance - Knight's Hill\n",
      "\n",
      "0.52: Station Road - This street leads from the centre of Harlesden down the hill to Willesden Junction Station.\n",
      "\n",
      "0.52: Dudley Court - Apartment block set back from the Finchley Road in Temple Fortune.  Parking is for residents only.  The block further down the hill is Montrose Court.\n",
      "\n",
      "0.52: Mountsfield Park - Mountsfield Park is higher along its eastern side and then drops down to the west side. This gives fine views to the west as here. The wooded hill to the left of the tower block is One Tree Hill at Honor Oak.\n",
      "\n",
      "0.52: Mountsfield Park - A fairly steep slope runs north south through the centre of Mountsfield Park. The west side of the park is lower than the east. Shrubs and trees lie along much of the slope as seen to the left of the rotting log. The wooded hill on the horizon to the right of the tower block is Nunhead Cemetery.\n",
      "\n",
      "0.52: Dudden Hill Lane - Looking down the hill from the railway bridge to the rooftops of Neasden.\n",
      "\n",
      "0.52: Avondale Road - Looking up the hill towards Squires Lane.  The road on the left is Montpelier Road.   The London Plane trees and are pollarded every year to keep them a sensible size - left to grow naturally they become very large trees.\n",
      "\n",
      "0.52: Bell Lane - Looking up the hill towards the junction with Green Walk.  An unusual design of bay windows on the semi-detached house on the corner.  I think that the roofline has been altered by a loft conversion.\n",
      "\n",
      "0.52: Brent Green - Three friends stroll down the hill under the Lime trees.\n",
      "\n",
      "0.52: Long Lane - Looking down the hill towards the junction with Squires Lane from Oakfield Road.  The tower blocks in the distance are on Prospect Ring.\n",
      "\n",
      "0.52: Queens Avenue - Looking down the hill to Squires Lane.  The houses here are in four residence terraces.\n",
      "\n",
      "0.52: Crystal Palace Parade - Some of the shops and cafes at the top of the hill.\n",
      "\n",
      "0.52: Woodlands Avenue - Around the corner and looking down the hill to Squires Lane.  Varieties of semi-detached housing on display here.\n",
      "\n",
      "0.52: View of the trees in Broomfield Park - Taken from the balcony of a flat in Alderman's Hill\n",
      "\n",
      "0.52: The way into Maryon Wilson Park - The land now taken up by Maryon Wilson Park was once part of the Hanging Wood which originally stretched from Woolwich Common to Charlton and which provided shelter for the highwaymen of Shooters Hill and Blackheath. Samuel Pepys often told of his dread of Hanging Wood when he was obliged to travel to Woolwich. Although still woodland it is rather more peaceful nowadays. The path followed by the Green Chain Walk and Capital Ring climbs steadily from the Thames through Maryon Park and then Maryon Wilson Park to reach the exit opposite Cemetery Lane. This view looks down the hill from that entrance.\n",
      "\n",
      "0.52: Descending Biggin Hill - There are extensive views over south London on the descent of this hill.\n",
      "\n",
      "0.52: Cement Tanks at Acton - 66620 hauls a train of PCA Cement tanks through platform 4 of Acton Main Line station.   The service is the 6M91 from Lafarge's facility at Theale to Earles Sidings at Hope.   The double amber signal and the angled white lights above indicates to the driver that the train is turning off at Acton East Junction and will climb up to Acton Wells Junction to pick up the Dudden hill line to Cricklewood.\n",
      "\n",
      "0.52: Still Racing Cabbies - An AEC Routemaster converted to open top configuration climbs the hill from Hendon Central Circus in mid-evening traffic.  The vehicle was originally a Greenline coach bearing the fleet number RCL2250.\n",
      "\n",
      "0.51: Between The Carriageways - Part of a minor subway network at the junction of the A41 Watford Way with the A504.  The ramp I am standing on leads up to a small garden area between the carriageways of the A41.  The tunnel through which the young lady is walking takes you to The Burroughs and Middlesex University.  Going in the direction that the lady is walking will take you to Station Road and a walk down the hill to Hendon Station on the St.Pancras to Leicester railway line.  Both the university and the station are pedestrian signposted through the subway.\n",
      "\n",
      "0.51: Dollis Lane Finchley - Looking SE up the hill towards Ballards Rd and Finchley Central tube station.\n",
      "\n",
      "0.51: Brunswick Park Road - Looking up the hill towards Betstyle Circle.  A 382 bus passes the stop for Marne Avenue on its way to Southgate from Mill Hill East.\n",
      "\n",
      "0.51: The Arts Depot - Located on the short section of Nether Street between The Great North Road and Ballards Lane - This photo is taken from the Ballards Lane end.   The area occupied by the Foyer and the shops further down the hill once formed part of North Finchley bus station.   The bus station now resides underneath the development adjacent to Kingsway in the area originally occupied by the Gaumont cinema.\n",
      "\n",
      "0.51: Grange Road - Down hill from Beulah Hill with Grangewood Park in the distance.\n",
      "\n",
      "0.51: Auckland Road - In the distance is the transmitter mast at the southern end of the Crystal Palace hill (not the one in the Park).\n",
      "\n",
      "0.51: Footpath junction in Beaulieu Heights - A path leads left towards Auckland Road (through the woodland on the hill) or straight on towards South Norwood Hill (road) via an open-space area.\n",
      "\n",
      "0.51: Beaulieu Heights woodland - This patch of woodland and open-space area is on South Norwood hill.\n",
      "\n",
      "0.51: Auckland Road - Typical late 19th century villas of the sort that predominate around the Crystal Palace / Upper Norwood heights: between them is a drive to some lock-up garages and in the gap one can just make out the large Sylvan Road estate on the hill behind.\n",
      "\n",
      "0.51: Brooks Road - A group of semidetached houses at the Wellesley Road end of Brooks Road.   A ubiquitous Fed Ex van storms up the hill on its delivery rounds.\n",
      "\n",
      "0.51: Steps in Gunnersbury Park - Leading down the hill from the mansions to a small walled garden.\n",
      "\n",
      "0.51: Pollards Hill - Pollards Hill occupies the highest point in Norbury and on a clear day the view stretches to Windsor Castle. It was originally part of the Great North Wood and the name 'Pollard' derives from this connection. The road layout of Pollards Hill was shown on maps of 1868 with two avenues of trees. Pollards Hill had been left as an open space as the surrounding estate developed and it was given to Croydon Council in 1913 by Sir Frederick Ederidge. A pedestal was erected on the summit of the hill to show the direction of various local landmarks but the direction pointer on the top has disappeared. The trig point seen here is close by. The view takes in Croydon town centre on the extreme right of the photograph. The houses are in Pollards Crescent and Pollards Hill West.\n",
      "\n",
      "0.51: High Street Wimbledon - This is Wimbledon Village as opposed to Wimbledon down the hill. The traffic is unusually light in this view.\n",
      "\n",
      "0.50: Alverstone Avenue - Seen from the corner at the southern end of this street.  Semi-detached houses march up the hill.   New street lighting has recently been installed - something being done throughout the Borough of Barnet in 2009/2010 - and the street light on the right has been decommissioned.  It awaits the final removal of the pole to complete the job.\n",
      "\n",
      "0.50: Totteridge Lane - Looking up the hill towards Whetstone past Totteridge and Whetstone Station (on the left).  Barnet's council offices at 1255 High Road are visible on the right.\n",
      "\n",
      "0.50: White Lodge from near Pen Ponds - Looking up the hill from a path that traverses the slope towards the car park.\n",
      "\n",
      "0.50: Track to Warren Farm - This track (also classified as a bridleway) leads around the large circular earthworks (the remains of an Iron Age hill fort known as Caesar's Camp) within Wimbledon Common.\n",
      "\n",
      "0.50: The Grand Union Canal leaves  the Thames at Brentford - The Grand Union Canal leaves the River Thames at Brentford and climbs over fifty locks up into the Chiltern hills. Brentford was once an important trading route but is now used mainly by pleasure boaters.\n",
      "\n",
      "0.50: Tramlink across South Norwood Country Park - South Norwood Country Park was opened in 1989 and the Tramlink route between Arena and Harrington Road cut across the south-west side of the park in 2000. It actually fits into the landscape very well and the colourful trams add interest to the scene. This view from the foot crossing to the south of the hill in the park looking up the line shows the steepness of some of the gradients the trams have to cope with.\n",
      "\n",
      "0.50: Tramlink crossing - South Norwood Country Park was opened in 1989 and the Tramlink route between Arena and Harrington Road cut across the south-west side of the park in 2000. It actually fits into the landscape very well and the colourful trams add interest to the scene. This view looks across the foot crossing to the south of the hill in the park as a West Croydon bound train heads towards the Arena stop.\n",
      "\n",
      "0.50: Acid grassland on Winns Common - The underlying soils on Winns Common are poor and gravelly and so the grasses tend to stay short. Gorse is also typical of these soils. The extensive views northwards extend well beyond the Thames to the Essex hills - places such as Havering-atte-Bower.\n",
      "\n",
      "0.50: Capel Road - Looking down the hill from the junction with Cranbrook Road towards Church Hill Road and the valley of Pymmes Brook.\n",
      "\n",
      "0.50: Glendaruel North Mobile Phone Towers - These 2towers are quite well hidden out of the way in this Grid Square. Not like the big BT-Radio Tower high on the opposite side of the valley. The Tower in the front is a joint venture with Vodafone and O2 when the one behind is for Airwave a police installation. The dishes on both towers link to the big BT-Tower on the hill opposite.\n",
      "\n",
      "0.50: Mullach Sron na h-Uamhaidh. - The 834m summit of a hill that does not feature in any of the big lists and is scarcely visited other than by locals. An easy climb with a big view of great quality. Well worth a visit. \n",
      "\n",
      "0.50: Lane from Holly Lodge to Sawyer's Hill - Lane from Holly Lodge (renamed from Bog Lodge as shown on OS) to Sawyer's Hill\n",
      "\n",
      "0.50: Watling Avenue - Looking up the hill towards the A5 from the bottom of the Silk Stream valley.\n",
      "\n",
      "0.50: Burnt Oak Broadway - Looking north towards the hills in Stanmore on the horizon. The main shopping parade and station are in the road to the right of the lights.\n",
      "\n",
      "0.50: View from Winns Common - The eastern part of Plumstead Common is known as Winns Common and its northern side is characterised by steep slopes down to lower Plumstead and fine views. This viewpoint is between Riverdale Road and Purrett Road and looks over Belmarsh Prison (the complex of white buildings) to the hills of Essex.\n",
      "\n",
      "0.50: Office blocks at Limeburner Lane - Looking north from Ludgate Hill; the curved block on the left wraps around the straight block on the right.\n",
      "\n",
      "0.50: Ludgate Hill & Fleet Street - View down Ludgate Hill and along Fleet Street\n",
      "\n",
      "0.50: Ludgate Circus - Buses and cycle couriers pass between Fleet Street and Ludgate Hill with Ludgate House as a backdrop.\n",
      "\n",
      "0.50: Danger! Low Tree! Buses diverted - In the depths of rural Shropshire? No! In Gray's Inn Road near the junction with High Holborn in central London. London Plane trees grow ever larger.\n",
      "\n",
      "0.50: City of London: Kings Bench Walk - Within the Inner Temple area of London are these pleasant surroundings which bring to mind the centre of a market town.\n",
      "\n",
      "0.50: Ludgate Hill - Looking down Ludgate Hill from St Paul's Cathedral.\n",
      "\n",
      "0.50: Ludgate Hill - View to St.Paul's Cathedral up Ludgate Hill\n",
      "\n",
      "0.50: Ludgate Hill looking towards Creed Lane - Rather confusingly this narrow street is also called Ludgate Hill as well as the better known west approach to St. Paul's cathedral.\n",
      "\n",
      "0.50: Sewer ventilation pipe - This Sewer ventilation pipe sits between Paradise Road and Lichfield Terrace. The road going down hill nest to the grand Victorian house is Lichfield Gardens.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "n = 100\n",
    "i=0\n",
    "for key in ranked:\n",
    "    score = ranked[key]\n",
    "    row = sample.loc[sample['id'] == key]\n",
    "    title = row.iloc[0]['title']\n",
    "    text = row.iloc[0]['text']\n",
    "    print(f'{score:.2f}: {title} - {text}\\n')\n",
    "    if i == n:\n",
    "        break\n",
    "    i = i+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "0c6ed970",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>score</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.491779</td>\n",
       "      <td>-0.233868</td>\n",
       "      <td>0.667730</td>\n",
       "      <td>Hammersmith Town Hall in daylight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.530867</td>\n",
       "      <td>-0.298820</td>\n",
       "      <td>0.646966</td>\n",
       "      <td>Fallen tree by Western Avenue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.613480</td>\n",
       "      <td>0.127329</td>\n",
       "      <td>0.617152</td>\n",
       "      <td>Ever hopeful hungry water birds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.036166</td>\n",
       "      <td>0.433847</td>\n",
       "      <td>0.601278</td>\n",
       "      <td>Pashley Manor oak tree opposite car park</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.472709</td>\n",
       "      <td>-0.008968</td>\n",
       "      <td>0.601217</td>\n",
       "      <td>A2 Blackheath Hill</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.513209</td>\n",
       "      <td>-0.108808</td>\n",
       "      <td>0.497640</td>\n",
       "      <td>City of London: Kings Bench Walk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.513715</td>\n",
       "      <td>-0.101149</td>\n",
       "      <td>0.497422</td>\n",
       "      <td>Ludgate Hill</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.513713</td>\n",
       "      <td>-0.101005</td>\n",
       "      <td>0.497387</td>\n",
       "      <td>Ludgate Hill</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.513268</td>\n",
       "      <td>-0.101311</td>\n",
       "      <td>0.497266</td>\n",
       "      <td>Ludgate Hill looking towards Creed Lane</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51.460837</td>\n",
       "      <td>-0.300273</td>\n",
       "      <td>0.497216</td>\n",
       "      <td>Sewer ventilation pipe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>101 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          lat       lon     score                                     title\n",
       "0   51.491779 -0.233868  0.667730         Hammersmith Town Hall in daylight\n",
       "0   51.530867 -0.298820  0.646966             Fallen tree by Western Avenue\n",
       "0   51.613480  0.127329  0.617152           Ever hopeful hungry water birds\n",
       "0   51.036166  0.433847  0.601278  Pashley Manor oak tree opposite car park\n",
       "0   51.472709 -0.008968  0.601217                        A2 Blackheath Hill\n",
       "..        ...       ...       ...                                       ...\n",
       "0   51.513209 -0.108808  0.497640          City of London: Kings Bench Walk\n",
       "0   51.513715 -0.101149  0.497422                              Ludgate Hill\n",
       "0   51.513713 -0.101005  0.497387                              Ludgate Hill\n",
       "0   51.513268 -0.101311  0.497266   Ludgate Hill looking towards Creed Lane\n",
       "0   51.460837 -0.300273  0.497216                    Sewer ventilation pipe\n",
       "\n",
       "[101 rows x 4 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i=0\n",
    "\n",
    "\n",
    "df = pd.DataFrame()\n",
    "\n",
    "for key in ranked:\n",
    "    row = sample.loc[sample['id'] == key]\n",
    "    t = pd.DataFrame(\n",
    "        {\n",
    "            'lat': row.iloc[0]['lat'],\n",
    "            'lon': [row.iloc[0]['lon']],\n",
    "            'score': [ranked[key]],\n",
    "            'title': [row.iloc[0]['title']]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    df = pd.concat([df, t])\n",
    "\n",
    "    if i == n:\n",
    "        break\n",
    "    i = i+1\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "db5c2e5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "<b>%{hovertext}</b><br><br>lat=%{lat}<br>lon=%{lon}<br>score=%{marker.color}<extra></extra>",
         "hovertext": [
          "Hammersmith Town Hall in daylight",
          "Fallen tree by Western Avenue",
          "Ever hopeful hungry water birds",
          "Pashley Manor oak tree opposite car park",
          "A2 Blackheath Hill",
          "Long Lane",
          "Brent Street",
          "City Thameslink",
          "St. Andrew's Hill",
          "Ireland Yard",
          "Mill Way",
          "Acton Street (2)",
          "Fish Street Hill.",
          "Archangel Michael - The Protector",
          "Haymarket",
          "Lamp post on Primrose Hill",
          "View up Primrose Hill to viewpoint",
          "Show Off on Bread & Cheese Hill",
          "Primrose Hill in evening sunshine",
          "Path up Primrose Hill",
          "Parliament Hill Fields",
          "Rosslyn Hill",
          "Kemplay Road",
          "Waterlow sunflowers",
          "Parting of the Ways",
          "Latchmere Road",
          "View across London from Hampstead Heath",
          "Road between Shepherd's Hill and Highgate tube station",
          "Holmdale Road",
          "West Heath Road",
          "Hewitt Road",
          "Sumatra Road",
          "St. John's Hill at Night",
          "Greenwich Naval Hospital",
          "North End Road",
          "Greenwich Park and Palace",
          "People enjoying the park",
          "Emergency Access",
          "Alexandra Palace Park",
          "Cricklewood Lane",
          "Greenwich Park: Maze Hill gate",
          "Public Road Transport at East Finchley Station",
          "View of Westminster and Central London from Canonbie Rd",
          "St Jude On the Hill",
          "Durham Road",
          "Northway Circus",
          "West Norwood station entrance",
          "Station Road",
          "Dudley Court",
          "Mountsfield Park",
          "Mountsfield Park",
          "Dudden Hill Lane",
          "Avondale Road",
          "Bell Lane",
          "Brent Green",
          "Long Lane",
          "Queens Avenue",
          "Crystal Palace Parade",
          "Woodlands Avenue",
          "View of the trees in Broomfield Park",
          "The way into Maryon Wilson Park",
          "Descending Biggin Hill",
          "Cement Tanks at Acton",
          "Still Racing Cabbies",
          "Between The Carriageways",
          "Dollis Lane Finchley",
          "Brunswick Park Road",
          "The Arts Depot",
          "Grange Road",
          "Auckland Road",
          "Footpath junction in Beaulieu Heights",
          "Beaulieu Heights woodland",
          "Auckland Road",
          "Brooks Road",
          "Steps in Gunnersbury Park",
          "Pollards Hill",
          "High Street Wimbledon",
          "Alverstone Avenue",
          "Totteridge Lane",
          "White Lodge from near Pen Ponds",
          "Track to Warren Farm",
          "The Grand Union Canal leaves  the Thames at Brentford",
          "Tramlink across South Norwood Country Park",
          "Tramlink crossing",
          "Acid grassland on Winns Common",
          "Capel Road",
          "Glendaruel North Mobile Phone Towers",
          "Mullach Sron na h-Uamhaidh.",
          "Lane from Holly Lodge to Sawyer's Hill",
          "Watling Avenue",
          "Burnt Oak Broadway",
          "View from Winns Common",
          "Office blocks at Limeburner Lane",
          "Ludgate Hill & Fleet Street",
          "Ludgate Circus",
          "Danger! Low Tree! Buses diverted",
          "City of London: Kings Bench Walk",
          "Ludgate Hill",
          "Ludgate Hill",
          "Ludgate Hill looking towards Creed Lane",
          "Sewer ventilation pipe"
         ],
         "lat": [
          51.491779,
          51.530867,
          51.61348,
          51.036166,
          51.472709,
          51.599141,
          51.584614,
          51.512669,
          51.512819,
          51.512644,
          51.617846,
          51.52855,
          51.510769,
          51.526863,
          51.508452,
          51.537361,
          51.539485,
          51.562627,
          51.540761,
          51.540853,
          51.558062,
          51.555039,
          51.556126,
          51.569828,
          51.557757,
          51.465943,
          51.562042,
          51.577352,
          51.551336,
          51.562012,
          51.582529,
          51.55048,
          51.463187,
          51.483445,
          51.569453,
          51.480538,
          51.481577,
          51.589197,
          51.592354,
          51.56045,
          51.479669,
          51.587822,
          51.444289,
          51.579578,
          51.592706,
          51.584901,
          51.431373,
          51.53471,
          51.588008,
          51.446831,
          51.446821,
          51.555515,
          51.600566,
          51.587205,
          51.584343,
          51.60135,
          51.603612,
          51.420805,
          51.603814,
          51.616207,
          51.482932,
          51.416738,
          51.517097,
          51.583496,
          51.586321,
          51.604641,
          51.619508,
          51.613114,
          51.410359,
          51.4096,
          51.409307,
          51.40878,
          51.407238,
          51.489598,
          51.499993,
          51.403162,
          51.425061,
          51.633911,
          51.630235,
          51.445295,
          51.426767,
          51.482951,
          51.393784,
          51.393316,
          51.483902,
          51.639995,
          56.104325,
          57.046051,
          51.449608,
          51.601068,
          51.600633,
          51.483275,
          51.514925,
          51.51405,
          51.514147,
          51.519367,
          51.513209,
          51.513715,
          51.513713,
          51.513268,
          51.460837
         ],
         "legendgroup": "",
         "lon": [
          -0.233868,
          -0.29882,
          0.127329,
          0.433847,
          -0.008968,
          -0.180469,
          -0.21829,
          -0.10321,
          -0.10133,
          -0.101626,
          -0.251512,
          -0.117688,
          -0.085851,
          -0.130299,
          -0.131774,
          -0.157267,
          -0.160786,
          0.573831,
          -0.161888,
          -0.162029,
          -0.15845,
          -0.172131,
          -0.172665,
          -0.146142,
          -0.179235,
          -0.161873,
          -0.177189,
          -0.144248,
          -0.194349,
          -0.181085,
          -0.102467,
          -0.197123,
          -0.169614,
          -0.005764,
          -0.185406,
          -0.004018,
          -0.001668,
          -0.13698,
          -0.126312,
          -0.208126,
          0.004729,
          -0.163887,
          -0.051649,
          -0.1889,
          -0.159936,
          -0.184213,
          -0.102982,
          -0.248356,
          -0.199102,
          -0.010238,
          -0.009662,
          -0.244527,
          -0.179545,
          -0.21126,
          -0.218156,
          -0.183701,
          -0.178846,
          -0.078106,
          -0.180282,
          -0.116519,
          0.044912,
          -0.104451,
          -0.267336,
          -0.227572,
          -0.23006,
          -0.198877,
          -0.143107,
          -0.176877,
          -0.088178,
          -0.085765,
          -0.08434,
          -0.085081,
          -0.078818,
          -0.280766,
          -0.284409,
          -0.12701,
          -0.216448,
          -0.161159,
          -0.179225,
          -0.264443,
          -0.240116,
          -0.299739,
          -0.059114,
          -0.057984,
          0.09191,
          -0.16496,
          -5.09854,
          -4.276302,
          -0.282557,
          -0.266163,
          -0.267191,
          0.096778,
          -0.103693,
          -0.10517,
          -0.104489,
          -0.111436,
          -0.108808,
          -0.101149,
          -0.101005,
          -0.101311,
          -0.300273
         ],
         "marker": {
          "color": [
           0.6677302943333453,
           0.6469661396706158,
           0.6171519467828916,
           0.6012784455738461,
           0.6012165410104877,
           0.5906561951257233,
           0.5890028649855374,
           0.5688379543185886,
           0.5685944865623263,
           0.5685704990337628,
           0.5677062960639441,
           0.5640164201108027,
           0.5636301624290105,
           0.5615570832807071,
           0.5614144668132208,
           0.5505258055998483,
           0.5488597124148196,
           0.548854336882822,
           0.5481475703344907,
           0.5480784608361505,
           0.542415386239796,
           0.5404118752367997,
           0.5398607237016667,
           0.5390960898227763,
           0.5375713727684295,
           0.5371086078592782,
           0.5364396511508805,
           0.5355150549990262,
           0.5355010324704685,
           0.5354930590716667,
           0.5351198373900992,
           0.534923944645683,
           0.5343048903599986,
           0.5319474246653514,
           0.5314955031919721,
           0.5306314231372958,
           0.5301974350566699,
           0.5300374267064298,
           0.5290997623595153,
           0.5287313105908844,
           0.5277430775746469,
           0.5274006630142143,
           0.5268023963378787,
           0.5264715697105791,
           0.5255379088325629,
           0.5251180716263986,
           0.5238136066538058,
           0.5212343395200457,
           0.5206078676504683,
           0.5201520833617682,
           0.5200147971813489,
           0.5190022616709904,
           0.5186006120770035,
           0.5181648285699989,
           0.5176232594036763,
           0.5175157333165356,
           0.5172193421405985,
           0.5170687788361158,
           0.5168874507735074,
           0.5162942620543662,
           0.5157231105679754,
           0.5156840592756757,
           0.5155989930530437,
           0.5155335523629501,
           0.5138464905680133,
           0.5131961824962706,
           0.5131540203112155,
           0.5127759370180653,
           0.5118098796373095,
           0.5112970856335515,
           0.5110757300743204,
           0.5108163017752311,
           0.5096673973029895,
           0.509021280043976,
           0.5089382754807729,
           0.5077248232246359,
           0.5075762893769518,
           0.5038288875049588,
           0.5037060238424255,
           0.5035742461835581,
           0.5030381277136955,
           0.501698082001551,
           0.5009973148016267,
           0.5006520835775744,
           0.5003097057147115,
           0.5002090314808516,
           0.5,
           0.5,
           0.49967727514526317,
           0.4990256580926423,
           0.4989035159054227,
           0.49859139578062905,
           0.4985029946559882,
           0.4982535750803427,
           0.4982431562391607,
           0.4977906958108943,
           0.49763969811252884,
           0.4974223023839092,
           0.49738723805140994,
           0.4972664126134327,
           0.4972163217058483
          ],
          "coloraxis": "coloraxis"
         },
         "mode": "markers",
         "name": "",
         "showlegend": false,
         "subplot": "mapbox",
         "type": "scattermapbox"
        }
       ],
       "layout": {
        "coloraxis": {
         "colorbar": {
          "title": {
           "text": "score"
          }
         },
         "colorscale": [
          [
           0,
           "#0d0887"
          ],
          [
           0.1111111111111111,
           "#46039f"
          ],
          [
           0.2222222222222222,
           "#7201a8"
          ],
          [
           0.3333333333333333,
           "#9c179e"
          ],
          [
           0.4444444444444444,
           "#bd3786"
          ],
          [
           0.5555555555555556,
           "#d8576b"
          ],
          [
           0.6666666666666666,
           "#ed7953"
          ],
          [
           0.7777777777777778,
           "#fb9f3a"
          ],
          [
           0.8888888888888888,
           "#fdca26"
          ],
          [
           1,
           "#f0f921"
          ]
         ]
        },
        "height": 600,
        "legend": {
         "tracegroupgap": 0
        },
        "mapbox": {
         "center": {
          "lat": 51.620976029702966,
          "lon": -0.22179181188118813
         },
         "domain": {
          "x": [
           0,
           1
          ],
          "y": [
           0,
           1
          ]
         },
         "style": "open-street-map",
         "zoom": 6
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"25bf5e13-71ad-474b-b734-4abd4c5d8765\" class=\"plotly-graph-div\" style=\"height:600px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"25bf5e13-71ad-474b-b734-4abd4c5d8765\")) {                    Plotly.newPlot(                        \"25bf5e13-71ad-474b-b734-4abd4c5d8765\",                        [{\"hovertemplate\":\"<b>%{hovertext}</b><br><br>lat=%{lat}<br>lon=%{lon}<br>score=%{marker.color}<extra></extra>\",\"hovertext\":[\"Hammersmith Town Hall in daylight\",\"Fallen tree by Western Avenue\",\"Ever hopeful hungry water birds\",\"Pashley Manor oak tree opposite car park\",\"A2 Blackheath Hill\",\"Long Lane\",\"Brent Street\",\"City Thameslink\",\"St. Andrew's Hill\",\"Ireland Yard\",\"Mill Way\",\"Acton Street (2)\",\"Fish Street Hill.\",\"Archangel Michael - The Protector\",\"Haymarket\",\"Lamp post on Primrose Hill\",\"View up Primrose Hill to viewpoint\",\"Show Off on Bread & Cheese Hill\",\"Primrose Hill in evening sunshine\",\"Path up Primrose Hill\",\"Parliament Hill Fields\",\"Rosslyn Hill\",\"Kemplay Road\",\"Waterlow sunflowers\",\"Parting of the Ways\",\"Latchmere Road\",\"View across London from Hampstead Heath\",\"Road between Shepherd's Hill and Highgate tube station\",\"Holmdale Road\",\"West Heath Road\",\"Hewitt Road\",\"Sumatra Road\",\"St. John's Hill at Night\",\"Greenwich Naval Hospital\",\"North End Road\",\"Greenwich Park and Palace\",\"People enjoying the park\",\"Emergency Access\",\"Alexandra Palace Park\",\"Cricklewood Lane\",\"Greenwich Park: Maze Hill gate\",\"Public Road Transport at East Finchley Station\",\"View of Westminster and Central London from Canonbie Rd\",\"St Jude On the Hill\",\"Durham Road\",\"Northway Circus\",\"West Norwood station entrance\",\"Station Road\",\"Dudley Court\",\"Mountsfield Park\",\"Mountsfield Park\",\"Dudden Hill Lane\",\"Avondale Road\",\"Bell Lane\",\"Brent Green\",\"Long Lane\",\"Queens Avenue\",\"Crystal Palace Parade\",\"Woodlands Avenue\",\"View of the trees in Broomfield Park\",\"The way into Maryon Wilson Park\",\"Descending Biggin Hill\",\"Cement Tanks at Acton\",\"Still Racing Cabbies\",\"Between The Carriageways\",\"Dollis Lane Finchley\",\"Brunswick Park Road\",\"The Arts Depot\",\"Grange Road\",\"Auckland Road\",\"Footpath junction in Beaulieu Heights\",\"Beaulieu Heights woodland\",\"Auckland Road\",\"Brooks Road\",\"Steps in Gunnersbury Park\",\"Pollards Hill\",\"High Street Wimbledon\",\"Alverstone Avenue\",\"Totteridge Lane\",\"White Lodge from near Pen Ponds\",\"Track to Warren Farm\",\"The Grand Union Canal leaves  the Thames at Brentford\",\"Tramlink across South Norwood Country Park\",\"Tramlink crossing\",\"Acid grassland on Winns Common\",\"Capel Road\",\"Glendaruel North Mobile Phone Towers\",\"Mullach Sron na h-Uamhaidh.\",\"Lane from Holly Lodge to Sawyer's Hill\",\"Watling Avenue\",\"Burnt Oak Broadway\",\"View from Winns Common\",\"Office blocks at Limeburner Lane\",\"Ludgate Hill & Fleet Street\",\"Ludgate Circus\",\"Danger! Low Tree! Buses diverted\",\"City of London: Kings Bench Walk\",\"Ludgate Hill\",\"Ludgate Hill\",\"Ludgate Hill looking towards Creed Lane\",\"Sewer ventilation pipe\"],\"lat\":[51.491779,51.530867,51.61348,51.036166,51.472709,51.599141,51.584614,51.512669,51.512819,51.512644,51.617846,51.52855,51.510769,51.526863,51.508452,51.537361,51.539485,51.562627,51.540761,51.540853,51.558062,51.555039,51.556126,51.569828,51.557757,51.465943,51.562042,51.577352,51.551336,51.562012,51.582529,51.55048,51.463187,51.483445,51.569453,51.480538,51.481577,51.589197,51.592354,51.56045,51.479669,51.587822,51.444289,51.579578,51.592706,51.584901,51.431373,51.53471,51.588008,51.446831,51.446821,51.555515,51.600566,51.587205,51.584343,51.60135,51.603612,51.420805,51.603814,51.616207,51.482932,51.416738,51.517097,51.583496,51.586321,51.604641,51.619508,51.613114,51.410359,51.4096,51.409307,51.40878,51.407238,51.489598,51.499993,51.403162,51.425061,51.633911,51.630235,51.445295,51.426767,51.482951,51.393784,51.393316,51.483902,51.639995,56.104325,57.046051,51.449608,51.601068,51.600633,51.483275,51.514925,51.51405,51.514147,51.519367,51.513209,51.513715,51.513713,51.513268,51.460837],\"legendgroup\":\"\",\"lon\":[-0.233868,-0.29882,0.127329,0.433847,-0.008968,-0.180469,-0.21829,-0.10321,-0.10133,-0.101626,-0.251512,-0.117688,-0.085851,-0.130299,-0.131774,-0.157267,-0.160786,0.573831,-0.161888,-0.162029,-0.15845,-0.172131,-0.172665,-0.146142,-0.179235,-0.161873,-0.177189,-0.144248,-0.194349,-0.181085,-0.102467,-0.197123,-0.169614,-0.005764,-0.185406,-0.004018,-0.001668,-0.13698,-0.126312,-0.208126,0.004729,-0.163887,-0.051649,-0.1889,-0.159936,-0.184213,-0.102982,-0.248356,-0.199102,-0.010238,-0.009662,-0.244527,-0.179545,-0.21126,-0.218156,-0.183701,-0.178846,-0.078106,-0.180282,-0.116519,0.044912,-0.104451,-0.267336,-0.227572,-0.23006,-0.198877,-0.143107,-0.176877,-0.088178,-0.085765,-0.08434,-0.085081,-0.078818,-0.280766,-0.284409,-0.12701,-0.216448,-0.161159,-0.179225,-0.264443,-0.240116,-0.299739,-0.059114,-0.057984,0.09191,-0.16496,-5.09854,-4.276302,-0.282557,-0.266163,-0.267191,0.096778,-0.103693,-0.10517,-0.104489,-0.111436,-0.108808,-0.101149,-0.101005,-0.101311,-0.300273],\"marker\":{\"color\":[0.6677302943333453,0.6469661396706158,0.6171519467828916,0.6012784455738461,0.6012165410104877,0.5906561951257233,0.5890028649855374,0.5688379543185886,0.5685944865623263,0.5685704990337628,0.5677062960639441,0.5640164201108027,0.5636301624290105,0.5615570832807071,0.5614144668132208,0.5505258055998483,0.5488597124148196,0.548854336882822,0.5481475703344907,0.5480784608361505,0.542415386239796,0.5404118752367997,0.5398607237016667,0.5390960898227763,0.5375713727684295,0.5371086078592782,0.5364396511508805,0.5355150549990262,0.5355010324704685,0.5354930590716667,0.5351198373900992,0.534923944645683,0.5343048903599986,0.5319474246653514,0.5314955031919721,0.5306314231372958,0.5301974350566699,0.5300374267064298,0.5290997623595153,0.5287313105908844,0.5277430775746469,0.5274006630142143,0.5268023963378787,0.5264715697105791,0.5255379088325629,0.5251180716263986,0.5238136066538058,0.5212343395200457,0.5206078676504683,0.5201520833617682,0.5200147971813489,0.5190022616709904,0.5186006120770035,0.5181648285699989,0.5176232594036763,0.5175157333165356,0.5172193421405985,0.5170687788361158,0.5168874507735074,0.5162942620543662,0.5157231105679754,0.5156840592756757,0.5155989930530437,0.5155335523629501,0.5138464905680133,0.5131961824962706,0.5131540203112155,0.5127759370180653,0.5118098796373095,0.5112970856335515,0.5110757300743204,0.5108163017752311,0.5096673973029895,0.509021280043976,0.5089382754807729,0.5077248232246359,0.5075762893769518,0.5038288875049588,0.5037060238424255,0.5035742461835581,0.5030381277136955,0.501698082001551,0.5009973148016267,0.5006520835775744,0.5003097057147115,0.5002090314808516,0.5,0.5,0.49967727514526317,0.4990256580926423,0.4989035159054227,0.49859139578062905,0.4985029946559882,0.4982535750803427,0.4982431562391607,0.4977906958108943,0.49763969811252884,0.4974223023839092,0.49738723805140994,0.4972664126134327,0.4972163217058483],\"coloraxis\":\"coloraxis\"},\"mode\":\"markers\",\"name\":\"\",\"showlegend\":false,\"subplot\":\"mapbox\",\"type\":\"scattermapbox\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"mapbox\":{\"domain\":{\"x\":[0.0,1.0],\"y\":[0.0,1.0]},\"center\":{\"lat\":51.620976029702966,\"lon\":-0.22179181188118813},\"zoom\":6,\"style\":\"open-street-map\"},\"coloraxis\":{\"colorbar\":{\"title\":{\"text\":\"score\"}},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"legend\":{\"tracegroupgap\":0},\"margin\":{\"t\":60},\"height\":600},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('25bf5e13-71ad-474b-b734-4abd4c5d8765');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.scatter_mapbox(df, lat=\"lat\", \n",
    "                     lon=\"lon\",\n",
    "                     color=\"score\", # which column to use to set the color of markers\n",
    "                     hover_name=\"title\", # column added to hover information\n",
    "                     zoom=4,\n",
    "                     height=600,\n",
    "                    )\n",
    "fig.update_layout(mapbox_style=\"open-street-map\")\n",
    "fig.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ba2ffbe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SpatialIndex:\n",
    "    \n",
    "    def __init__(self, resolution, sample):\n",
    "        \n",
    "        sample.dropna() # Get rid of problematic rows with nas\n",
    "        \n",
    "        for i in sample.index:\n",
    "            try:\n",
    "                g = OSGridConverter.latlong2grid (sample.at[i, 'lat'], sample.at[i, 'lon'], tag = 'WGS84')\n",
    "                sample.at[i, 'x'] = g.E\n",
    "                sample.at[i, 'y'] = g.N\n",
    "            except ValueError:\n",
    "                #print(\"Problem with a document\", sample.at[i,'id'])\n",
    "                sample = sample.drop(i)\n",
    "\n",
    "        # Now we can set up the parameters for our index        \n",
    "        self.resolution = resolution\n",
    "\n",
    "        self.minx = sample['x'].min()\n",
    "        self.maxx = sample['x'].max()\n",
    "        self.miny = sample['y'].min()\n",
    "        self.maxy = sample['y'].max()\n",
    "\n",
    "        w = self.maxx - self.minx\n",
    "        h = self.maxy - self.miny\n",
    "\n",
    "        nc = int(w/self.resolution) + 1\n",
    "        nr = int(h/self.resolution) + 1\n",
    "\n",
    "        #print(maxx, minx, maxy, miny)\n",
    "        #print(nr, nc)\n",
    "\n",
    "        #Build the spatial index now\n",
    "        self.spatialIndex = pd.DataFrame(index=range(nc),columns=range(nr))\n",
    "\n",
    "        #Now we populate the index with document ids\n",
    "        for index, row in sample.iterrows():\n",
    "            i = int((row['x'] - self.minx)/self.resolution)\n",
    "            j = int((row['y'] - self.miny)/self.resolution)\n",
    "            id = row['id']\n",
    "    \n",
    "            #print(row['id'])\n",
    "            #print(row['x'],row['y'],i,j)\n",
    "            if pd.isnull(self.spatialIndex.at[i,j]):\n",
    "                self.spatialIndex.at[i,j] = {id:(row['x'],row['y'])}\n",
    "            else:\n",
    "                names = self.spatialIndex.at[i,j]\n",
    "                names.update({id:(row['x'],row['y'])})\n",
    "                self.spatialIndex.at[i,j] = names\n",
    "\n",
    "        \n",
    "    def rangeQuery(self, dist, point):\n",
    "        x1 = point[0] - dist/2\n",
    "        x2 = point[0] + dist/2\n",
    "        y1 = point[1] - dist/2\n",
    "        y2 = point[1] + dist/2\n",
    "    \n",
    "        i1 = int((x1 - self.minx)/self.resolution)\n",
    "        j1 = int((y1 - self.miny)/self.resolution)\n",
    "        i2 = int((x2 - self.minx)/self.resolution) + 1\n",
    "        j2 = int((y2 - self.miny)/self.resolution) + 1\n",
    "\n",
    "        # Retrieve only the relevant part of the index\n",
    "        result = self.spatialIndex.iloc[i1:i2, j1:j2]\n",
    "        # Turn the data frame into a 1d list\n",
    "        tlist = result.values.flatten()\n",
    "        # Remove all the nans\n",
    "        filtered = filter(lambda i:not(type(i) is float), tlist)\n",
    "        \n",
    "        #Rank by distance\n",
    "        ranked = {}\n",
    "        for item in filtered:\n",
    "            for key in item:\n",
    "                d = si.dist(point, item[key])\n",
    "                #print(key, item[key], dist)\n",
    "                ranked[key] = d    \n",
    "        ranked = dict(sorted(ranked.items(), key = lambda x: x[1], reverse=False))\n",
    "                \n",
    "        return ranked\n",
    "    \n",
    "    def dist(self, p1, p2):\n",
    "        #print(p1[0], p1[1], p2[0], p2[1])\n",
    "        dist = (((p1[0] - p2[0]) ** 2) + ((p1[1] - p2[1]) ** 2)) ** 0.5\n",
    "        #print(dist)\n",
    "        return dist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92390576",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9c7fe898",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Postings:\n",
    "    \n",
    "    def __init__(self, firstMondayTerms, sample, nlp):\n",
    "        #Load a language model to do NLP\n",
    "        self.ndocs = len(sample)\n",
    "                \n",
    "        # firstMonday works like an inverse stop list, and we only use words in these lists for our posting file\n",
    "        if firstMondayTerms:\n",
    "            #We do this so that the serialisation of the Pickle works propery\n",
    "            data_folder = Path('./data')\n",
    "            fn1 = data_folder / 'elements.txt'\n",
    "            fn2 = data_folder / 'qualities.txt'\n",
    "            fn3 = data_folder / 'activities.txt'\n",
    "            \n",
    "            elements = set(pd.read_csv(fn1, header=None)[0])\n",
    "            qualities = set(pd.read_csv(fn2, header=None)[0])\n",
    "            activities = set(pd.read_csv(fn3, header=None)[0])\n",
    "\n",
    "            terms = elements.union(qualities).union(activities)\n",
    "            lemmas = ' '.join(str(e) for e in terms)\n",
    "\n",
    "            doc = nlp(lemmas)\n",
    "            terms = set()\n",
    "            for token in doc:\n",
    "                terms.add(token.lemma_)\n",
    "                \n",
    "            # Now we process our corpus and create a postings file\n",
    "            docs = nlp.pipe(sample.text,n_process=2, batch_size=100)\n",
    "\n",
    "            self.postings = dict()\n",
    "\n",
    "            for (idxRow, s1), (_, s2) in zip(sample.iterrows(), enumerate(docs)):\n",
    "                id = s1.id\n",
    "                for token in s2:\n",
    "                    lemma = token.lemma_\n",
    "                    if lemma in terms:\n",
    "\n",
    "                        if lemma in self.postings:\n",
    "                            tf = self.postings[lemma]\n",
    "                            if id in tf:\n",
    "                                tf[id] = tf[id] + 1\n",
    "                            else:\n",
    "                                tf[id] = 1\n",
    "                        else:\n",
    "                            tf = {id: 1}\n",
    "                        self.postings[lemma] = tf\n",
    "                        \n",
    "    def tfIdf(self, query,nlp):\n",
    "        results = {}\n",
    "        qdoc = nlp(query)\n",
    "        for token in qdoc:\n",
    "            qt = token.lemma_\n",
    "            if qt in self.postings:\n",
    "                dc = len(self.postings[qt])\n",
    "                idf = math.log10(self.ndocs/(dc + 1))\n",
    "                for doc in self.postings[qt]:\n",
    "                    tf = self.postings[qt][doc]\n",
    "                    tfidf = tf * idf\n",
    "                    if doc in results:\n",
    "                        score = results[doc]\n",
    "                        results[doc] = tfidf + score\n",
    "                    else:\n",
    "                        results[doc] = tfidf\n",
    "        results = dict(sorted(results.items(), key = lambda x: x[1], reverse=True))\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "622ab43c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature codes in gazetteer are as follows:\n",
    "# A Antiquity (non-Roman)\n",
    "# F Forest or wood\n",
    "# FM Farm\n",
    "# H Hill or mountain\n",
    "# R Antiquity (Roman)\n",
    "# C City\n",
    "# T Town\n",
    "# O Other\n",
    "# W Water feature\n",
    "# X All other features\n",
    "\n",
    "class Gazetteer:\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.gaz = dict()\n",
    "        self.offset = {'C': 2000, 'T':500, 'H':250, 'F':500}\n",
    "        # Read in gazetteer data\n",
    "        data_folder = Path('./data')\n",
    "        fn1 = data_folder / '50kgaz2012.txt'\n",
    "        os_50k = pd.read_csv(fn1,sep=':', encoding='utf8', header=None)\n",
    "        os_trimmed = os_50k.drop([0,1,3,4,5,6,7,10,11,12,15,16,17,18,19], axis = 1)\n",
    "        os_trimmed.columns = ['name','y','x','county','type']\n",
    "        for index, row in os_trimmed.iterrows():\n",
    "            name = row['name']\n",
    "            entry = os_trimmed.iloc[index].values \n",
    "            # Store gazetteer in a dictionary of unique names\n",
    "            if name in self.gaz:\n",
    "                entries = self.gaz[name]\n",
    "                entries.append(entry)\n",
    "                self.gaz[name] = entries\n",
    "            else:\n",
    "                self.gaz[name] = [entry]\n",
    "            \n",
    "    def getLocation(self, name):\n",
    "        if (name in self.gaz) == False:\n",
    "            return('Name not found in gazetteer')\n",
    "\n",
    "        if len(self.gaz[name]) > 1:\n",
    "            # We let the user disambiguate\n",
    "            i = 0\n",
    "            print(\"This place name is ambiguous - choose an entry\")\n",
    "            for entry in self.gaz[name]:\n",
    "                print(f'{i}: {name}, {entry[3]}')\n",
    "                i = i + 1\n",
    "            index = int(input(\"Choose a value:\"))\n",
    "            entry = self.gaz[name][index]\n",
    "        else:\n",
    "            entry = self.gaz[name][0]\n",
    "            \n",
    "        print(entry)\n",
    "        x = entry[2]\n",
    "        y = entry[1]\n",
    "            \n",
    "        if entry[4] in self.offset:\n",
    "            diff = self.offset[entry[4]]\n",
    "            return (x,y,x-diff, y-diff, x + diff, y + diff)\n",
    "        else:\n",
    "            return(x,y)\n",
    "                                    \n",
    "    def gazDump(self):\n",
    "        for name in self.gaz:\n",
    "            print(name)\n",
    "            print(self.gaz[name])\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "84017327",
   "metadata": {},
   "outputs": [],
   "source": [
    "#output all dependencies so that we can reproduce the notebook (we only need this to set things up for Binder)\n",
    "#%load_ext watermark\n",
    "#%watermark --iversions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbdbce06",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "145f066d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}