geocodingDemos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started With Geocoding\n",
    "\n",
    "In this notebook, you will learn how to geocode different sorts of location data by making requests to several online APIs (*Application Programming Interface*) for latitude and longitude co-ordinates associated with those locations.\n",
    "\n",
    "The aim of the notebook is *not* to teach you formal approaches for working with APIs or the data that is returned from them. Instead, it's something to whet your curiosity. Something to show you how, with a few lines of Python code, you can start to work with live, third-party datasources and online services to perform real-world programming tasks.\n",
    "\n",
    "If something doesn't work: *DON'T PANIC*. You won't break your computer and you won't break the internet. And you won't fail the module if you just move on!\n",
    "\n",
    "The location data we will consider includes:\n",
    "\n",
    "- <a href=\"#postcodes\">postcodes</a>\n",
    "- <a href=\"#addresses\">addresses</a>\n",
    "- <a href=\"#IPaddresses\">IP addresses</a>\n",
    "- <a href=\"#cellTowers\">cell towers</a>\n",
    "- <a href=\"#wifiHotspits\">wifi access points</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#The requests library makes it easy to call URLs using Python\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"postcodes\"></a>\n",
    "## Postcodes\n",
    "\n",
    "Postcodes are a widely used form of location data, typically capable of identifying a location to a resolution of a few hundred square metres.\n",
    "\n",
    "There are several online services that will return geolocation information given a postcode.\n",
    "\n",
    "To call the service, we construct a URL as defined for a particular API and make a request to that URL using the *python* `requests` package.\n",
    "\n",
    "Data is often returned from webservices using the JSON (Javascript Object Notation) data format, although some APIs allow you to specify other formats such as XML.\n",
    "\n",
    "(One advantage of the JSON response is that it can be immediately consumed by a Javascript script called from inside a webpage.)\n",
    "\n",
    "JSON and XML both allow data to be represented in a structured, *tree* based hierarchical format. The first API we will use, published via the *postcodes.io* website, structures its response data in the following way:\n",
    "\n",
    "![Hierarchical structure of postocdes api data, showire results tree with latitude, longitude and codes children, and codes showing admin_district and parish district](images/postodes_io_struct.png)\n",
    "\n",
    "<!--- http://blockdiag.com/en/blockdiag/demo.html\n",
    "{\n",
    "\n",
    "  \"result\" -> \"postcode\";\n",
    "\"result\" -> \"country\";\n",
    "\"result\" -> \"longitude\";\n",
    "\"result\" -> \"latitude\";\n",
    "\"dots\" [shape = \"dots\"];\n",
    "\"result\" -> \"dots\" [style = \"none\"];\n",
    "\"result\" -> \"codes\";\n",
    "\"codes\" -> \"admin_district\";\n",
    "\"dots2\" [shape = \"dots\"];\n",
    "\"codes\" -> \"dots2\" [style = \"none\"];\n",
    "\"codes\" -> \"parish\";\n",
    "\n",
    "})\n",
    "\n",
    "--->\n",
    "\n",
    "The `result` node is at the top of the tree with children `postcode`, `latitude`, `longitude` and so on. The `codes` child has further children, such as: `admin` and `parish`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In *python*, data structures of this form can be represented using the `dict` (\"dictionary\") structure, which you will meet elsewhere in the course.\n",
    "\n",
    "The *python* `requests` library has a method that parses a correctly formed JSON response as a *python* `dict`, or more generally, as a set of *nested dicts*. In this case, one `dict` structure may be nested inside another to support child, grandchild, great grandchild, and so on, levels of structure.\n",
    "\n",
    "![Hierarchical structure of postocdes api data, showire results tree with latitude, longitude and codes children, and codes showing admin_district and parish district](images/postodes_io_struct_data.png)\n",
    "\n",
    "The contents of different levels of the nested `dict` data structure can be accessed by using a form of associative, relative addressing. For example, if the variable `mypostcode` is set to the `dict` shown above, we could access the contents of the main `result` part of the data structure by writing: `mypostcode[\"result\"]`.\n",
    "\n",
    "To obtain the value of items in deeper nested parts of the data structure, we simply add further levels of relative addressing. To fetch the value of the `postcode`, we need to specify the path to it via the `result` node: `mypostcode[\"result\"][\"postcode\"]`. To obtain the value of the `parish` in the `code` part of the data structure, we specify the path to it as `mypostcode[\"result\"][\"code\"][\"parish\"]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to call the `postcodes.io` API with a particular postcode.\n",
    "\n",
    "See if you can make sense of the result that is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'result': {'admin_county': None,\n",
       "  'admin_district': 'Milton Keynes',\n",
       "  'admin_ward': 'Monkston',\n",
       "  'ccg': 'NHS Milton Keynes',\n",
       "  'codes': {'admin_county': 'E99999999',\n",
       "   'admin_district': 'E06000042',\n",
       "   'admin_ward': 'E05009415',\n",
       "   'ccg': 'E38000107',\n",
       "   'nuts': 'UKJ12',\n",
       "   'parish': 'E04001275'},\n",
       "  'country': 'England',\n",
       "  'eastings': 488625,\n",
       "  'european_electoral_region': 'South East',\n",
       "  'incode': '6AA',\n",
       "  'latitude': 52.0249147315159,\n",
       "  'longitude': -0.709747474196332,\n",
       "  'lsoa': 'Milton Keynes 017C',\n",
       "  'msoa': 'Milton Keynes 017',\n",
       "  'nhs_ha': 'South Central',\n",
       "  'northings': 237063,\n",
       "  'nuts': 'Milton Keynes',\n",
       "  'outcode': 'MK7',\n",
       "  'parish': 'Walton',\n",
       "  'parliamentary_constituency': 'Milton Keynes South',\n",
       "  'postcode': 'MK7 6AA',\n",
       "  'primary_care_trust': 'Milton Keynes',\n",
       "  'quality': 1,\n",
       "  'region': 'South East'},\n",
       " 'status': 200}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "postcode = 'MK7 6AA'\n",
    "r=requests.get('https://api.postcodes.io/postcodes/{PC}'.format(PC=postcode))\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try rerunning the previous cell using different postcodes - can the service locate your home postcode?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing the `postcodes.io` JSON data\n",
    "\n",
    "Once we have retrieved the data from the API, and cast it as a *python* data object, we can look inside it programmatically.\n",
    "\n",
    "For example, we can find the latitude and longitude values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52.0249147315159 -0.709747474196332\n"
     ]
    }
   ],
   "source": [
    "#Obtain the lat/long of a postcode\n",
    "lat=r.json()['result']['latitude']\n",
    "lon=r.json()['result']['longitude']\n",
    "\n",
    "#Display the result\n",
    "print(lat,lon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having access to the latitude and longitude means we can start to make use of that information, for example by plotting it on a map.\n",
    "\n",
    "You may recall how we previously used the `folium` package to generate interactive maps from *python* code within a notebook.\n",
    "\n",
    "We can do a similar thing again here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><iframe src=\"data:text/html;base64,PCFET0NUWVBFIGh0bWw+CjxoZWFkPiAgICAKICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgogICAgPHNjcmlwdD5MX1BSRUZFUl9DQU5WQVMgPSBmYWxzZTsgTF9OT19UT1VDSCA9IGZhbHNlOyBMX0RJU0FCTEVfM0QgPSBmYWxzZTs8L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL3VucGtnLmNvbS9sZWFmbGV0QDEuMC4xL2Rpc3QvbGVhZmxldC5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9hamF4Lmdvb2dsZWFwaXMuY29tL2FqYXgvbGlicy9qcXVlcnkvMS4xMS4xL2pxdWVyeS5taW4uanMiPjwvc2NyaXB0PgogICAgPHNjcmlwdCBzcmM9Imh0dHBzOi8vbWF4Y2RuLmJvb3RzdHJhcGNkbi5jb20vYm9vdHN0cmFwLzMuMi4wL2pzL2Jvb3RzdHJhcC5taW4uanMiPjwvc2NyaXB0PgogICAgPHNjcmlwdCBzcmM9Imh0dHBzOi8vY2RuanMuY2xvdWRmbGFyZS5jb20vYWpheC9saWJzL0xlYWZsZXQuYXdlc29tZS1tYXJrZXJzLzIuMC4yL2xlYWZsZXQuYXdlc29tZS1tYXJrZXJzLmpzIj48L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2NkbmpzLmNsb3VkZmxhcmUuY29tL2FqYXgvbGlicy9sZWFmbGV0Lm1hcmtlcmNsdXN0ZXIvMS4wLjAvbGVhZmxldC5tYXJrZXJjbHVzdGVyLXNyYy5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvbGVhZmxldC5tYXJrZXJjbHVzdGVyLzEuMC4wL2xlYWZsZXQubWFya2VyY2x1c3Rlci5qcyI+PC9zY3JpcHQ+CiAgICA8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9Imh0dHBzOi8vdW5wa2cuY29tL2xlYWZsZXRAMS4wLjEvZGlzdC9sZWFmbGV0LmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvY3NzL2Jvb3RzdHJhcC5taW4uY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL21heGNkbi5ib290c3RyYXBjZG4uY29tL2Jvb3RzdHJhcC8zLjIuMC9jc3MvYm9vdHN0cmFwLXRoZW1lLm1pbi5jc3MiIC8+CiAgICA8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9Imh0dHBzOi8vbWF4Y2RuLmJvb3RzdHJhcGNkbi5jb20vZm9udC1hd2Vzb21lLzQuNi4zL2Nzcy9mb250LWF3ZXNvbWUubWluLmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL2NkbmpzLmNsb3VkZmxhcmUuY29tL2FqYXgvbGlicy9sZWFmbGV0Lm1hcmtlcmNsdXN0ZXIvMS4wLjAvTWFya2VyQ2x1c3Rlci5EZWZhdWx0LmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvbGVhZmxldC5tYXJrZXJjbHVzdGVyLzEuMC4wL01hcmtlckNsdXN0ZXIuY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL3Jhd2dpdC5jb20vcHl0aG9uLXZpc3VhbGl6YXRpb24vZm9saXVtL21hc3Rlci9mb2xpdW0vdGVtcGxhdGVzL2xlYWZsZXQuYXdlc29tZS5yb3RhdGUuY3NzIiAvPgogICAgPHN0eWxlPmh0bWwsIGJvZHkge3dpZHRoOiAxMDAlO2hlaWdodDogMTAwJTttYXJnaW46IDA7cGFkZGluZzogMDt9PC9zdHlsZT4KICAgIDxzdHlsZT4jbWFwIHtwb3NpdGlvbjphYnNvbHV0ZTt0b3A6MDtib3R0b206MDtyaWdodDowO2xlZnQ6MDt9PC9zdHlsZT4KICAgIAogICAgICAgICAgICA8c3R5bGU+ICNtYXBfOWQzMzM4YmNhYTE0NGYwNmFiNTU1ODMyMzRhZGIzZmUgewogICAgICAgICAgICAgICAgcG9zaXRpb24gOiByZWxhdGl2ZTsKICAgICAgICAgICAgICAgIHdpZHRoIDogMTAwLjAlOwogICAgICAgICAgICAgICAgaGVpZ2h0OiAxMDAuMCU7CiAgICAgICAgICAgICAgICBsZWZ0OiAwLjAlOwogICAgICAgICAgICAgICAgdG9wOiAwLjAlOwogICAgICAgICAgICAgICAgfQogICAgICAgICAgICA8L3N0eWxlPgogICAgICAgIAo8L2hlYWQ+Cjxib2R5PiAgICAKICAgIAogICAgICAgICAgICA8ZGl2IGNsYXNzPSJmb2xpdW0tbWFwIiBpZD0ibWFwXzlkMzMzOGJjYWExNDRmMDZhYjU1NTgzMjM0YWRiM2ZlIiA+PC9kaXY+CiAgICAgICAgCjwvYm9keT4KPHNjcmlwdD4gICAgCiAgICAKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIHNvdXRoV2VzdCA9IEwubGF0TG5nKC05MCwgLTE4MCk7CiAgICAgICAgICAgICAgICB2YXIgbm9ydGhFYXN0ID0gTC5sYXRMbmcoOTAsIDE4MCk7CiAgICAgICAgICAgICAgICB2YXIgYm91bmRzID0gTC5sYXRMbmdCb3VuZHMoc291dGhXZXN0LCBub3J0aEVhc3QpOwogICAgICAgICAgICAKCiAgICAgICAgICAgIHZhciBtYXBfOWQzMzM4YmNhYTE0NGYwNmFiNTU1ODMyMzRhZGIzZmUgPSBMLm1hcCgKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdtYXBfOWQzMzM4YmNhYTE0NGYwNmFiNTU1ODMyMzRhZGIzZmUnLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAge2NlbnRlcjogWzUyLjAyNDkxNDczMTUxNTksLTAuNzA5NzQ3NDc0MTk2MzMyXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHpvb206IDE1LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4Qm91bmRzOiBib3VuZHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsYXllcnM6IFtdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd29ybGRDb3B5SnVtcDogZmFsc2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjcnM6IEwuQ1JTLkVQU0czODU3CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH0pOwogICAgICAgICAgICAKICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgdGlsZV9sYXllcl8zZmM4NTYwMjQ1ODc0Y2U2YTMzM2YxNTVkODhhMmRmMCA9IEwudGlsZUxheWVyKAogICAgICAgICAgICAgICAgJ2h0dHBzOi8ve3N9LnRpbGUub3BlbnN0cmVldG1hcC5vcmcve3p9L3t4fS97eX0ucG5nJywKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBtYXhab29tOiAxOCwKICAgICAgICAgICAgICAgICAgICBtaW5ab29tOiAxLAogICAgICAgICAgICAgICAgICAgIGNvbnRpbnVvdXNXb3JsZDogZmFsc2UsCiAgICAgICAgICAgICAgICAgICAgbm9XcmFwOiBmYWxzZSwKICAgICAgICAgICAgICAgICAgICBhdHRyaWJ1dGlvbjogJ0RhdGEgYnkgPGEgaHJlZj0iaHR0cDovL29wZW5zdHJlZXRtYXAub3JnIj5PcGVuU3RyZWV0TWFwPC9hPiwgdW5kZXIgPGEgaHJlZj0iaHR0cDovL3d3dy5vcGVuc3RyZWV0bWFwLm9yZy9jb3B5cmlnaHQiPk9EYkw8L2E+LicsCiAgICAgICAgICAgICAgICAgICAgZGV0ZWN0UmV0aW5hOiBmYWxzZQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkuYWRkVG8obWFwXzlkMzMzOGJjYWExNDRmMDZhYjU1NTgzMjM0YWRiM2ZlKTsKCiAgICAgICAgCiAgICAKCiAgICAgICAgICAgIHZhciBtYXJrZXJfMTE4MTExYjE3YTM0NGNjYzhjODEwNWM5MDExOWEyNWQgPSBMLm1hcmtlcigKICAgICAgICAgICAgICAgIFs1Mi4wMjM5LC0wLjcwNzJdLAogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgIGljb246IG5ldyBMLkljb24uRGVmYXVsdCgpCiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgLmFkZFRvKG1hcF85ZDMzMzhiY2FhMTQ0ZjA2YWI1NTU4MzIzNGFkYjNmZSk7CiAgICAgICAgICAgIAogICAgCiAgICAgICAgICAgIHZhciBwb3B1cF9mNjVjYjBiNjM2NjE0YzgzYjBjZDllYTQwMzYzYmQ4YiA9IEwucG9wdXAoe21heFdpZHRoOiAnMzAwJ30pOwoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgaHRtbF80MjJlOWU0ZWEwMmE0OWEwYTczZGYzZGYzZTg2NTRiOSA9ICQoJzxkaXYgaWQ9Imh0bWxfNDIyZTllNGVhMDJhNDlhMGE3M2RmM2RmM2U4NjU0YjkiIHN0eWxlPSJ3aWR0aDogMTAwLjAlOyBoZWlnaHQ6IDEwMC4wJTsiPkxvY2F0aW9uIG9mIE1LNyA2QUE6ICg1Mi4wMjQ5MTQ3MzE1MTU5LC0wLjcwOTc0NzQ3NDE5NjMzMik8L2Rpdj4nKVswXTsKICAgICAgICAgICAgICAgIHBvcHVwX2Y2NWNiMGI2MzY2MTRjODNiMGNkOWVhNDAzNjNiZDhiLnNldENvbnRlbnQoaHRtbF80MjJlOWU0ZWEwMmE0OWEwYTczZGYzZGYzZTg2NTRiOSk7CiAgICAgICAgICAgIAoKICAgICAgICAgICAgbWFya2VyXzExODExMWIxN2EzNDRjY2M4YzgxMDVjOTAxMTlhMjVkLmJpbmRQb3B1cChwb3B1cF9mNjVjYjBiNjM2NjE0YzgzYjBjZDllYTQwMzYzYmQ4Yik7CgogICAgICAgICAgICAKICAgICAgICAKPC9zY3JpcHQ+\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;\"></iframe></div></div>"
      ],
      "text/plain": [
       "<folium.folium.Map at 0x1080e1ba8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Plot the lat long of a postcode on a map\n",
    "\n",
    "#We need to import the following packages to access the maps\n",
    "import folium\n",
    "\n",
    "#Create a map centered on the postcode location at a particular zoom level\n",
    "mymap = folium.Map(location=[lat, lon], zoom_start=15)\n",
    "\n",
    "#Create a popup message using Python string formatting to create the label based on variable values\n",
    "popupstr = 'Location of {PC}: ({lat},{lon})'.format(PC=postcode, lat=lat,lon=lon)\n",
    "\n",
    "#Display a marker for the location\n",
    "folium.Marker([52.0239, -0.7072], popup=popupstr).add_to(mymap)\n",
    "mymap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"addresses\"></a>\n",
    "## Addresses\n",
    "\n",
    "As well as geolocating postcodes, we can also goecode complete (or partial) addresses. One API that supports address based geocoding is the Google Maps geocoding API.\n",
    "\n",
    "Once again, we need to construct a URL according to a pattern defined by the API documentation. Then we can make a request to that URL and hopefully get the geocoded data back as a response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'results': [{'address_components': [{'long_name': 'Walton Hall',\n",
       "     'short_name': 'Walton Hall',\n",
       "     'types': ['establishment', 'point_of_interest']},\n",
       "    {'long_name': 'Kents Hill',\n",
       "     'short_name': 'Kents Hill',\n",
       "     'types': ['locality', 'political']},\n",
       "    {'long_name': 'Milton Keynes',\n",
       "     'short_name': 'Milton Keynes',\n",
       "     'types': ['postal_town']},\n",
       "    {'long_name': 'Milton Keynes',\n",
       "     'short_name': 'Milton Keynes',\n",
       "     'types': ['administrative_area_level_2', 'political']},\n",
       "    {'long_name': 'England',\n",
       "     'short_name': 'England',\n",
       "     'types': ['administrative_area_level_1', 'political']},\n",
       "    {'long_name': 'United Kingdom',\n",
       "     'short_name': 'GB',\n",
       "     'types': ['country', 'political']},\n",
       "    {'long_name': 'MK7 6BH',\n",
       "     'short_name': 'MK7 6BH',\n",
       "     'types': ['postal_code']}],\n",
       "   'formatted_address': 'Walton Hall, Kents Hill, Milton Keynes MK7 6BH, UK',\n",
       "   'geometry': {'location': {'lat': 52.02462269999999, 'lng': -0.7107079},\n",
       "    'location_type': 'APPROXIMATE',\n",
       "    'viewport': {'northeast': {'lat': 52.02597168029149,\n",
       "      'lng': -0.709358919708498},\n",
       "     'southwest': {'lat': 52.02327371970849, 'lng': -0.712056880291502}}},\n",
       "   'place_id': 'ChIJW3FMFVuhd0gRVUSpS2HG-ps',\n",
       "   'types': ['establishment', 'point_of_interest']}],\n",
       " 'status': 'OK'}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "address='Open University, Walton Hall, Milton Keynes, MK7 6AA, UK'\n",
    "r= requests.get(\"https://maps.googleapis.com/maps/api/geocode/json\", params={'address': address, 'sensor': \"false\"})\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try rerunning the previous cell with an address that is familiar to you. Does the API find it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional Activities\n",
    "\n",
    "- see if you can write a loop that will look up the geolocations of several postcodes, one at a time. To be nice to the API import the *python* `time` library and add the statement `time.sleep(1)` inside the loop to pause its execution for one second during each iteration.\n",
    "- create a new `folium` map object to display several markers, one for each of your (looped) postcodes. Inside the postcode loop add a corresponding marker to the map. Don't forget to render the map from the last line of code in the cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"IPaddresses\"></a>\n",
    "## IP Addresses\n",
    "\n",
    "As well as looking up geolocation data for a *postal* address, we can also try to look up a location based on the IP address of a computer. There are seveal websites that allow you to lookup the IP address of the device you are using to connect to the internet, and several webservices too.\n",
    "\n",
    "I'm going to use a simple service from Amazon web services that returns an IP address terminated by an end of line (`\\n`) character. By using the `requests` library, I can call the URL, access the data response (`text`) and then strip (`.strip()`)) the end-of-line whitespace character from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'109.157.179.177'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myIPaddress=requests.get('http://checkip.amazonaws.com/').text.strip()\n",
    "myIPaddress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://freegeoip.net/json/109.157.179.177'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We can construct a URL based around the IP address of the machine making the request as follows:\n",
    "url='https://freegeoip.net/json/{IP}'.format(IP=myIPaddress)\n",
    "url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': 'Sandown',\n",
       " 'country_code': 'GB',\n",
       " 'country_name': 'United Kingdom',\n",
       " 'ip': '109.157.179.177',\n",
       " 'latitude': 50.6674,\n",
       " 'longitude': -1.186,\n",
       " 'metro_code': 0,\n",
       " 'region_code': 'ENG',\n",
       " 'region_name': 'England',\n",
       " 'time_zone': 'Europe/London',\n",
       " 'zip_code': 'PO36'}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r=requests.get(url)\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result may surprise you, for example if the notebook and the python process associated with it is running on a server hosted in the cloud. In this case, try looking up the IP address associated with computer you are using to access the internet. You can find this IP by visiting the link: [http://checkip.amazonaws.com/](http://checkip.amazonaws.com/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"cellTowers\"></a>\n",
    "## Cell Tower Lookup\n",
    "\n",
    "The Google geolocation API can be used to look-up the geographical locations (latitue and longitude co-ordinates) of cell towers and wifi hotsposts based on their unique IDs.\n",
    "\n",
    "To call the Google webservice to look up the geographical locations of cell towers or wifi hotspots from their IDs, you will need to get a Google Geolocation API token: visit [https://developers.google.com/maps/documentation/geocoding/get-api-key](https://developers.google.com/maps/documentation/geocoding/get-api-key) and follow the instructions on how to get a key for the geolocation API.\n",
    "\n",
    "When you have obtained your key, use it to set the `googleMapsAPIkey` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "googleMapsAPIkey=\"AIzaSyAnpCrSlBn72gHzcxrX5EHKxeeKOiOuBVg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have set your Google API key, run the following cell to look up the details of a particular cell tower:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Add your cell tower details here.\n",
    "#You can find them using an app such as the OpenSignal app\n",
    "\n",
    "postjson = {\n",
    "  \"cellTowers\": [\n",
    "    {\n",
    "        \"mobileCountryCode\": 234,\n",
    "        \"mobileNetworkCode\": 15,\n",
    "        \"locationAreaCode\": 714,#979,\n",
    "        \"cellId\": 1671#42333969\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'cellTowers': [{'cellId': 1671, 'mobileNetworkCode': 15, 'mobileCountryCode': 234, 'locationAreaCode': 714}]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 2588.0, 'location': {'lat': 50.6544242, 'lng': -1.200891}}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url='https://www.googleapis.com/geolocation/v1/geolocate?key={}'.format(googleMapsAPIkey)\n",
    "\n",
    "print(postjson)\n",
    "r = requests.post(url, json=postjson)\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"wifiHotspots\"></a>\n",
    "## WiFi Hotspot MAC Addresses\n",
    "\n",
    "As well as services that provide access to directories that try to associate IP addresses with physical locations, there are also databases that also try to associate MAC addresses of wifi routers with physical locations.\n",
    "\n",
    "If your computer has a wifi enabled, you will use access a low level command on your computer that identifies in-range wifi routers and provides adminstrative information about them.\n",
    "\n",
    "**STILL NEEDS TESTING & REFINING - TO DO** \n",
    "\n",
    "Note that to call the Google webservice to look up the geographical locations of cell towers or wifi hotspots from their IDs, you will need to get a Google Geolocation API token: visit [https://developers.google.com/maps/documentation/geocoding/get-api-key](https://developers.google.com/maps/documentation/geocoding/get-api-key) and follow the instructions on how to get a key for the geolocation API.\n",
    "\n",
    "When you have obtained your key, use it to set the `googleMapsAPIkey` variable below.\n",
    "\n",
    "Also note that the code may look a little bit involved. But `DON'T PANIC`, you don't need to be able to write, or even read, this sort of code for the purposes of this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "googleMapsAPIkey=\"YOUR_KEY_HERE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import requests\n",
    "\n",
    "#http://stackoverflow.com/a/9859202/454773\n",
    "def isInt_str(v):\n",
    "    v = str(v).strip()\n",
    "    return v=='0' or (v if v.find('..') > -1 else v.lstrip('-+').rstrip('0').rstrip('.')).isdigit()\n",
    "\n",
    "\n",
    "#/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport\n",
    "import subprocess\n",
    "def getWifiMacAddresses():\n",
    "    #autodetect platform and then report based on this? \n",
    "    print(sys.platform)\n",
    "    \n",
    "    macAddr={}\n",
    "    \n",
    "    #For Mac:\n",
    "    if sys.platform=='darwin':\n",
    "            results = subprocess.check_output([\"/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport\", \"-s\"])\n",
    "            results = results.decode(\"utf-8\").split(\"\\n\")\n",
    "            for l in [x.strip() for x in results[1:] if x.strip()!='']:\n",
    "                ll=l.split(' ')\n",
    "                #We could use a regular expression - or we can construct our parser a step at a time...\n",
    "                macAddress=l.strip().split(' ')[1]\n",
    "                strength=l.strip().split(' ')[2]\n",
    "                if isInt_str(strength):\n",
    "                    macAddr[l.strip().split(' ')[0]]={'macAddress':macAddress,\n",
    "                                                      'signalStrength':int(strength)}\n",
    "                \n",
    "    elif 'win' in sys.platform:\n",
    "        results = subprocess.check_output([\"netsh\", \"wlan\", \"show\", \"network\", \"mode=bssid\"])\n",
    "        results = results.replace(\"\\r\",\"\").split(\"\\n\")\n",
    "        macAddress='UNKNOWN'\n",
    "        for l in results[4:]:\n",
    "            if l.startswith('SSID'):\n",
    "                macAddress=':'.join(l.split(':')[1:]).strip()\n",
    "            if 'BSSID' in l:\n",
    "                macAddr[macAddress]=l.split(':')[1].strip()\n",
    "                macAddress='UNKNOWN'\n",
    "\n",
    "    elif 'linux' in sys.platform:\n",
    "        #linux?\n",
    "        #! apt-get -y install wireless-tools\n",
    "        #results = subprocess.check_output([\"iwlist\",\"scanning\"])    \n",
    "        #via PP - linux text - TO DO\n",
    "        # apt-get -y install wireless-tools then run iwlist scanning to display the details of wireless access points your computer can see.\n",
    "        #apt-get -y install wireless-tools gave me \"Could not open the lock file ...\"\n",
    "        #However when I checked in the Ubuntu Software Centre wireless-tools was already installed. I think non-expert users may use the Software Centre to install additional applications.\n",
    "        #iwlist just give you a not very helpful usage list. What works directly is:\n",
    "        #iwlist wlan0 scan\n",
    "        pass\n",
    "\n",
    "    return macAddr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "postjson={'wifiAccessPoints':[]}\n",
    "hotspots=getWifiMacAddresses()\n",
    "\n",
    "for h in hotspots:\n",
    "    postjson['wifiAccessPoints'].append(hotspots[h])\n",
    "    print(h,hotspots[h])\n",
    "    \n",
    "print('JSON posted to Google service: ',postjson)\n",
    "url='https://www.googleapis.com/geolocation/v1/geolocate?key={}'.format(googleMapsAPIkey)\n",
    "\n",
    "\n",
    "r = requests.post(url, json=postjson)\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook, you have learned how to geocode several different sorts of location identifer - postcodes, postal addresses, IP addresses and maybe even the MAC address of any WiFI routers in view of your computer.\n",
    "\n",
    "You have also seen how we can take the JSON data returned from the geolocation services and parse it as python dict that we can then start to work *as data*, for example, by plotting markers associated with identified locations on an interactive map."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}