23 - Regular expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regular expressions\n",
    "Regular expressions are string patterns. The `re` module - included in the standard library - allow to define, search for, isolate and operate on string patterns within strings of text. For example: \n",
    "\n",
    "- `FRA`, `USA` and `ITA` are all 3-letter, uppercase strings\n",
    "- `+33`, `+44`, `+91` are all strings starting with a + sign followed by 2 digits\n",
    "- `lady.gaga@gmail.com` and `bradley-cooper@aol.net` are both email address: they are sequences of characters (A through Z along with some special characters), followed by an ampersand (`@`), followed by a domain-name and finished by a top-level domain (`.net`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#import the module\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Defining and finding patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(13, 18), match='crude'>\n"
     ]
    }
   ],
   "source": [
    "#create a pattern using a string prefixed with the r character\n",
    "#the 'r' means 'raw'\n",
    "pattern = r'crude'\n",
    "\n",
    "#can we find the pattern in the text?\n",
    "text = \"The price of crude increased by 7.5% in the week of 7 January 2019\"\n",
    "\n",
    "#let's try! \n",
    "match = re.search(pattern, text)\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n",
      "18\n",
      "(13, 18)\n",
      "crude\n"
     ]
    }
   ],
   "source": [
    "#We found our string!\n",
    "print(match.start())\n",
    "print(match.end())\n",
    "print(match.span())\n",
    "print(match.group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(7, 14), match='bullish'>\n"
     ]
    }
   ],
   "source": [
    "#either one pattern or another\n",
    "match = re.search(r'bullish|bearish|neutral', \"we are bullish on Apple and Microsoft\")\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(13, 18), match='color'>\n"
     ]
    }
   ],
   "source": [
    "#optional character (0 or 1 occurence)\n",
    "match = re.search(r'colou?r', 'My favourite color is blue')\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(18, 21), match='AAA'>\n"
     ]
    }
   ],
   "source": [
    "#one or more occurences\n",
    "match = re.search(r'A+', 'The bond is rated AAA')\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(0, 4), match='9999'>\n"
     ]
    }
   ],
   "source": [
    "#any number of times (including 0)\n",
    "match = re.search(r'9*', '9999')\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(9, 13), match='9999'>\n"
     ]
    }
   ],
   "source": [
    "#between 3 and 5 times\n",
    "match = re.search(r'9{3,5}','019374564999953578934532453459834058345089')\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(0, 0), match=''>\n"
     ]
    }
   ],
   "source": [
    "#careful, the search method will stop as soon as it can\n",
    "#here, the pattern is any number of 9s, including none\n",
    "#as the first character is not a 9, this is akin to 0 x 9s. \n",
    "match = re.search(r\"9*\", \"123987\")\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "price of crude\n"
     ]
    }
   ],
   "source": [
    "#any character a through z (dash), one or several times (+)\n",
    "match = re.search(r'price of [a-zA-Z]+', text)\n",
    "\n",
    "print(match.group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(None, <_sre.SRE_Match object; span=(0, 2), match='NY'>)\n"
     ]
    }
   ],
   "source": [
    "#at the start of a string\n",
    "matches = re.search(r'^NY', 'I live in NY'), re.search(r'^NY', 'NY is awesome')\n",
    "\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(21, 29), match='sentence'>\n"
     ]
    }
   ],
   "source": [
    "#at the end of a string\n",
    "match = re.search(r'[A-Za-z]+$', 'The last word of the sentence')\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(304481, 304487), match='040995'>\n"
     ]
    }
   ],
   "source": [
    "#your turn to play! \n",
    "#can you spot your birthday in the first million digits of pi? \n",
    "with open('data/pi.txt', 'r') as source: \n",
    "    pi = source.read().strip()\n",
    "\n",
    "match = re.search('040995', pi)\n",
    "\n",
    "print(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2. Meta-characters\n",
    "Meta-characters allow to refer to particular patterns.\n",
    "\n",
    "1. `\\d` matches any decimal digit; this is equivalent to the class `[0-9]`.\n",
    "2. `\\D` matches any non-digit character; this is equivalent to the class `[^0-9]`\n",
    "3. `\\s` matches any whitespace character; this is equivalent to the class `[ \\t\\n\\r\\f\\v]`\n",
    "4. `\\S` matches any non-whitespace character; this is equivalent to the class `[^ \\t\\n\\r\\f\\v]`\n",
    "5. `\\w` matches any alphanumeric character; this is equivalent to the class `[a-zA-Z0-9_]`\n",
    "6. `\\W` matches any non-alphanumeric character; this is equivalent to the class `[^a-zA-Z0-9_]`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Address is valid: 341 Roosevelt Avenue, London EC34 2C, UNITED KINGDOM\n"
     ]
    }
   ],
   "source": [
    "#let's check this is a valid address\n",
    "address = \"341 Roosevelt Avenue, London EC34 2C, UNITED KINGDOM\"\n",
    "\n",
    "match = re.search(r'^\\d{1,4},? (\\w|\\s)+, \\w+ [A-Z0-9 ]+,[A-Z ]+$',address)\n",
    "\n",
    "if match: \n",
    "    print(\"Address is valid:\", match.group())\n",
    "else:\n",
    "    print(\"Address is invalid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3. Capturing\n",
    "Capturing allows to isolate sequences in string patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Brackets 1 : 12\n",
      "Brackets 2 : Feb\n",
      "Brackets 3 : 2019\n"
     ]
    }
   ],
   "source": [
    "#create groups with brackets\n",
    "match = re.search(r'(\\d{2})-([A-Za-z]{3})-(\\d{2,4})', '12-Feb-2019')\n",
    "\n",
    "for i, group in enumerate(match.groups()):\n",
    "    print(\"Brackets\", i+1,\":\",group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Brackets 1 : 341\n",
      "Brackets 2 : Roosevelt Avenue\n",
      "Brackets 3 : London\n",
      "Brackets 4 : EC34 2C\n",
      "Brackets 5 : UNITED KINGDOM\n"
     ]
    }
   ],
   "source": [
    "#You can make non-capturing brackets with (?:XXX) where XXX is your pattern\n",
    "address = \"341 Roosevelt Avenue, London EC34 2C, UNITED KINGDOM\"\n",
    "\n",
    "match = re.search(r'^(\\d{1,4}),? ((?:\\w|\\s)+), (\\w+) ([A-Z0-9 ]+), ([A-Z ]+)$',address)\n",
    "\n",
    "for i, group in enumerate(match.groups()):\n",
    "    print(\"Brackets\", i+1, ':', group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Day:\t23\n",
      "Month:\tMarch\n",
      "Year:\t2019\n"
     ]
    }
   ],
   "source": [
    "#You can name groups by using ?P<name> within the brackets\n",
    "date = \"23 March 2019\"\n",
    "\n",
    "match = re.search(r'(?P<day>\\d{1,2}) (?P<month>[A-Z][a-z]+) (?P<year>\\d{2,4})', date)\n",
    "\n",
    "print(\"Day:\", match.group(\"day\"), sep=\"\\t\")\n",
    "print(\"Month:\", match.group(\"month\"), sep=\"\\t\")\n",
    "print(\"Year:\", match.group(\"year\"), sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 4. Splitting and replacing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "World and U.S Supply and Use for Grains\n",
      "Million Metric Tons\n",
      "\n",
      "Total Grains 4/          \n",
      "    2017/18                  2616.60    3413.89     415.07    2601.51     812.39\n",
      "    2018/19 (Est.)           2625.43    3437.81     427.50    2636.02     801.79\n",
      "    2019/20 (Proj.)          2664.79    3465.70     436.11    2678.54     787.16\n",
      "    \n",
      "Wheat                    \n",
      "    2017/18                   761.87    1023.94     182.04     742.77     281.18\n",
      "    2018/19 (Est.)            730.55    1011.72     174.16     736.23     275.49\n",
      "    2019/20 (Proj.)           771.46    1046.61     183.11     760.15     286.46\n",
      "    \n",
      "Coarse Grains 5/         \n",
      "    2017/18                  1359.87    1745.48     185.91    1376.51     368.97\n",
      "    2018/19 (Est.)           1396.26    1765.22     207.80    1410.22     355.00\n",
      "    2019/20 (Proj.)          1395.51    1750.36     206.09    1422.32     328.05\n",
      "    \n",
      "Rice, milled             \n",
      "    2017/18                   494.86     644.48      47.13     482.23     162.25\n",
      "    2018/19 (Est.)            498.62     660.87      45.55     489.57     171.30\n",
      "    2019/20 (Proj.)           497.82     668.73      46.91     496.08     172.65\n"
     ]
    }
   ],
   "source": [
    "#let's try to parse some data provided by the US Department of Agriculture\n",
    "with open(\"data/WASDE.txt\", \"r\") as f: \n",
    "    data = f.read().strip()\n",
    "    \n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title World and U.S Supply and Use for Grains\n",
      "Units Million Metric Tons\n",
      "Total Grains,2017/18,(Final),2616.6,3413.89,415.07,2601.51,812.39\n",
      "Total Grains,2018/19,(Est.),2625.43,3437.81,427.5,2636.02,801.79\n",
      "Total Grains,2019/20,(Proj.),2664.79,3465.7,436.11,2678.54,787.16\n",
      "Wheat,2017/18,(Final),761.87,1023.94,182.04,742.77,281.18\n",
      "Wheat,2018/19,(Est.),730.55,1011.72,174.16,736.23,275.49\n",
      "Wheat,2019/20,(Proj.),771.46,1046.61,183.11,760.15,286.46\n",
      "Coarse Grains,2017/18,(Final),1359.87,1745.48,185.91,1376.51,368.97\n",
      "Coarse Grains,2018/19,(Est.),1396.26,1765.22,207.8,1410.22,355.0\n",
      "Coarse Grains,2019/20,(Proj.),1395.51,1750.36,206.09,1422.32,328.05\n",
      "Rice, milled,2017/18,(Final),494.86,644.48,47.13,482.23,162.25\n",
      "Rice, milled,2018/19,(Est.),498.62,660.87,45.55,489.57,171.3\n",
      "Rice, milled,2019/20,(Proj.),497.82,668.73,46.91,496.08,172.65\n"
     ]
    }
   ],
   "source": [
    "table = []\n",
    "\n",
    "for i, line in enumerate(data.split(\"\\n\")):\n",
    "    if i == 0: \n",
    "        title = line.strip()\n",
    "        continue\n",
    "        \n",
    "    if i == 1: \n",
    "        units = line.strip()\n",
    "        continue\n",
    "    \n",
    "    #is it an empty line?\n",
    "    match = re.search(r\"^\\s*$\", line)\n",
    "    if match: \n",
    "        continue\n",
    "        \n",
    "    #try to match it with a data line\n",
    "    match = re.search(\"\\s+(?P<year>\\d{4}/\\d{2}) (?P<status>\\((Est.|Proj.)\\))? (?P<data>(\\d| |.)+)$\", line)\n",
    "    if match:\n",
    "        year   = match.group(\"year\")\n",
    "        status = match.group(\"status\") or \"(Final)\"\n",
    "        points = re.split(\"\\s+\", match.group(\"data\").strip())  #we are splitting on blank characters\n",
    "        table.append([section, year, status]+[float(point) for point in points])\n",
    "        continue\n",
    "        \n",
    "    #else it must be section title\n",
    "    match = re.search(\"([A-Za-z ,]+)\", line)\n",
    "    section = match.groups()[0].strip()\n",
    "\n",
    "print(\"Title\", title)\n",
    "print(\"Units\", units)\n",
    "for row in table: \n",
    "    print(*row, sep=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 5. Your turn to play! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Exercise 1\n",
    "Create a function to verify that a phone number is valid. Phone numbers can start with a country code (e.g. '+44') or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Exercise 2\n",
    "In finance, futures on commodity markets generally follow a 3-part naming convention: \n",
    "1. Commodity: 1 to 3 letters\n",
    "2. Month: one of FGHJKMNQUVXZ\n",
    "3. Year: one, two or four digits. In the case of single-digits, assume the year is either the current year or the first year ending with that digit. \n",
    "\n",
    "For example\n",
    "- `CLZ9` refers to the December 2019 crude oil future\n",
    "- `NGX20` refers to the November 2020 natural gas future\n",
    "- `WF2018` refers to the January 18 wheat future\n",
    "\n",
    "Create a function that accepts a ticker (e.g. `CLZ9`) are return a 3-element tuple: commodity, month and year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}