python_solutions_4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "python-solutions-4.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "549BxVjhWyfM",
        "colab_type": "text"
      },
      "source": [
        "As was mentioned in the lecture, one of the great strengths of Python is the ease with which it allows you to assemble tools to solve a problem instead of having to reinvent all the techniques yourself.\n",
        "\n",
        "For example, consider the following doctor's note:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AyYDqo2PWwDX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "note = \"\"\"\n",
        "John is 32 years old. Sarah is 29 years old. Their child, Sam is 2 years old.\n",
        "All members of the family were healthy at Sam's last checkup. Today, however,\n",
        "while the mother appeared healthy -- Sarah's temperature was 37 degrees, she\n",
        "said -- she also said that the father couldn't make Sam's appointment today\n",
        "because John's temperature was 40 degrees, so he went to go see his physician\n",
        "instead. During the visit, we measured that Sam's temperature was 38 degrees,\n",
        "a low grade fever. Sarah mentioned her mother, Jean was diagnosed with mprbble\n",
        "recently, but after further discussion they hadn't seen Jean in over two weeks\n",
        "and given the usual incubation time, it was improbable that this was also\n",
        "afflicting little Sam. Several tests were ordered. When the results were in,\n",
        "Sam was diagnosed with strep.\n",
        "\"\"\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7SAtixrJcwBv",
        "colab_type": "text"
      },
      "source": [
        "We know how to do a little bit of analysis on the `note` already. For example, as a very crude approximation of the level of concern, one might wonder how many words are written in various doctor's notes, and see if e.g. notes about pneumonia are longer than those about the common cold.\n",
        "Using what you've already learned, how many words are in this `note`?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xTqnKrxuX7ZM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "64431ee7-0400-4cfa-ec05-a38927c8a497"
      },
      "source": [
        "len(note.split())"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "137"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xxnWFI9CdK-L",
        "colab_type": "text"
      },
      "source": [
        "Does your count include numbers (e.g. \"32\")? How can you prove this to yourself? (It's always important to test things to make sure that they work the way you think they do.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T4v57dSEdzAo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3a36e7f2-7def-442c-ac55-69caf9a2230e"
      },
      "source": [
        "'32' in note.split()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7eX__svd0EQ",
        "colab_type": "text"
      },
      "source": [
        "Let's say that we wanted to answer a more complicated question: **how many people are mentioned in this note?** (Maybe you're testing a hypothesis that some diseases can be better understood in the context of the community.)\n",
        "\n",
        "You could count, and it wouldn't take that long, but that is impractical if you have a million notes to analyze.\n",
        "\n",
        "Go back and look at this note. See if you notice any recurring patterns that could indicate a name."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGHQ_h5kfF2D",
        "colab_type": "text"
      },
      "source": [
        "```\n",
        "___ is ___ years old\n",
        "\n",
        "___'s temperature was __ degrees\n",
        "\n",
        "___ was diagnosed with ____\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUxMbuzhfZqD",
        "colab_type": "text"
      },
      "source": [
        "Pattern recognition is a pretty basic concept. You might think to yourself that people have probably tried to do pattern recognition in strings on the computer before, and you'd be right. Try a quick search for e.g. `python pattern recognition string` to see if you can find a module (library) that can help."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "phpjQ4FsgjzG",
        "colab_type": "text"
      },
      "source": [
        "The **re** module"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjy8Miutgn1d",
        "colab_type": "text"
      },
      "source": [
        "Good job. You probably found that there's a concept called regular expressions that provides a formalized way of describing patterns in strings and that there's a module `re` for working with regular expressions. Go ahead and `import` the `re` module:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kEa6YsTWfEeN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSpKydgmhaDw",
        "colab_type": "text"
      },
      "source": [
        "Let's find out what functions are in the module by doing a `dir`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cczx8iE5hfOg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d11df2e8-bb65-412c-af1a-97bf48045e91"
      },
      "source": [
        "dir(re)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['A',\n",
              " 'ASCII',\n",
              " 'DEBUG',\n",
              " 'DOTALL',\n",
              " 'I',\n",
              " 'IGNORECASE',\n",
              " 'L',\n",
              " 'LOCALE',\n",
              " 'M',\n",
              " 'MULTILINE',\n",
              " 'RegexFlag',\n",
              " 'S',\n",
              " 'Scanner',\n",
              " 'T',\n",
              " 'TEMPLATE',\n",
              " 'U',\n",
              " 'UNICODE',\n",
              " 'VERBOSE',\n",
              " 'X',\n",
              " '_MAXCACHE',\n",
              " '__all__',\n",
              " '__builtins__',\n",
              " '__cached__',\n",
              " '__doc__',\n",
              " '__file__',\n",
              " '__loader__',\n",
              " '__name__',\n",
              " '__package__',\n",
              " '__spec__',\n",
              " '__version__',\n",
              " '_alphanum_bytes',\n",
              " '_alphanum_str',\n",
              " '_cache',\n",
              " '_compile',\n",
              " '_compile_repl',\n",
              " '_expand',\n",
              " '_locale',\n",
              " '_pattern_type',\n",
              " '_pickle',\n",
              " '_subx',\n",
              " 'compile',\n",
              " 'copyreg',\n",
              " 'enum',\n",
              " 'error',\n",
              " 'escape',\n",
              " 'findall',\n",
              " 'finditer',\n",
              " 'fullmatch',\n",
              " 'functools',\n",
              " 'match',\n",
              " 'purge',\n",
              " 'search',\n",
              " 'split',\n",
              " 'sre_compile',\n",
              " 'sre_parse',\n",
              " 'sub',\n",
              " 'subn',\n",
              " 'template']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H23Sh3E9pBAQ",
        "colab_type": "text"
      },
      "source": [
        "`findall` sounds promising. Ask for `help` on `findall`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aNLncuvApF3x",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        },
        "outputId": "d981e2df-8b6d-47fe-9037-a099b71dbbbf"
      },
      "source": [
        "help(re.findall)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Help on function findall in module re:\n",
            "\n",
            "findall(pattern, string, flags=0)\n",
            "    Return a list of all non-overlapping matches in the string.\n",
            "    \n",
            "    If one or more capturing groups are present in the pattern, return\n",
            "    a list of groups; this will be a list of tuples if the pattern\n",
            "    has more than one group.\n",
            "    \n",
            "    Empty matches are included in the result.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9aa716AShF3B",
        "colab_type": "text"
      },
      "source": [
        "So we've learned `findall` takes a pattern and a string and returns a list of matches. The simplest pattern is of course just a string:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wMsOUpLvrsLY",
        "colab_type": "code",
        "outputId": "527e9e1e-21e5-4556-c810-0a548a4897e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "re.findall(\"years\", note)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['years', 'years', 'years']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mfaeXrafrxrv",
        "colab_type": "text"
      },
      "source": [
        "So \"years\" occurs three times. That's good to know, but we'd really like to know how many years.\n",
        "\n",
        "The goal of this exercise is to get experience with calling library functions not regular expressions, but here is an introduction to a tiny but useful subset of regular expression pattern specification:\n",
        "\n",
        "    [] indicate options, e.g. [ae] means a or e, ranges are allowed, e.g. [a-z] or even [a-zA-Z]\n",
        "    + means the previous item repeats 1 or more times; e.g. [0-9]+ means one or more repeating digits\n",
        "    () indicates grouping; if no group is specified, the whole match is returned\n",
        "\n",
        "For reasons that don't matter with this subset, you'll often find regular expression patterns written using a raw-string which is like a regular string but with the letter `r` written in front, e.g. `r\"the\"`; we'll use this for all of our examples even though it's not strictly necessary:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pYi8IZEttTLI",
        "colab_type": "text"
      },
      "source": [
        "For example, suppose we want to find all the words that occur after \"the\". That's the letters \"the\", a space, and then one or more letters. If we do:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hnLrIU3hhD-t",
        "colab_type": "code",
        "outputId": "64ff8a1e-db6d-476c-d9be-726d64d19468",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "re.findall(r\"the [a-zA-Z]+\", note)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['the family',\n",
              " 'the mother',\n",
              " 'the father',\n",
              " 'the visit',\n",
              " 'the usual',\n",
              " 'the results']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dVpJNjgLwKQj",
        "colab_type": "text"
      },
      "source": [
        "Compare to the text of the `note` to convince yourself this result is correct.\n",
        "\n",
        "If we want to get only specific data, we can use `()` around the piece or pieces that we're interested in:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qOVD81bwjU1H",
        "colab_type": "code",
        "outputId": "9d5fe34f-b370-4c40-eb69-a5714bbe584b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "re.findall(r\"the ([a-zA-Z]+)\", note)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['family', 'mother', 'father', 'visit', 'usual', 'results']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlzKrbz4wioU",
        "colab_type": "text"
      },
      "source": [
        "Adapt this to find the various ages. (Check the `note` to see the pattern for mentioning ages used by this author. You can imagine that there are other ways of indicating age, but we'll ignore that for now.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBcjOUngwYod",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "00ed05f1-de34-48c2-8ff6-65dc56a8fd3c"
      },
      "source": [
        "re.findall(r\"([0-9]+) years old\", note)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['32', '29', '2']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VogsOGP0wm2O",
        "colab_type": "text"
      },
      "source": [
        "Now modify your expression (you'll need to add a second set of parentheses) to find out who is each age:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ss0uZnS1wgcE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "79878c24-7402-4d8f-cc39-668d793bc77f"
      },
      "source": [
        "re.findall(r\"([a-zA-Z]+) is ([0-9]+) years old\", note)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('John', '32'), ('Sarah', '29'), ('Sam', '2')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_wlYd2rxEQy",
        "colab_type": "text"
      },
      "source": [
        "Extend this slightly by writing a function that takes a string (e.g. `note`) and indicates in a human-readable way who is each age. Make your own test and then run it with `note`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NLOAfNR4xh2U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def show_ages(text):\n",
        "  for name, age in re.findall(r\"([a-zA-Z]+) is ([0-9]+) years old\", text):\n",
        "    print(f\"{name} is {age}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8GYI2XMHrpob",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "0fc6c1c6-158e-4ffb-c6dc-6fc9a4f21b60"
      },
      "source": [
        "show_ages(\"Tim is 42 years old and Sarah is 37 years old.\")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tim is 42\n",
            "Sarah is 37\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZpZbwtxxruhh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "a5755c04-25dd-4f58-ec1d-f4239e492a66"
      },
      "source": [
        "show_ages(note)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "John is 32\n",
            "Sarah is 29\n",
            "Sam is 2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Quo4wsTlxiZc",
        "colab_type": "text"
      },
      "source": [
        "Modify your function to report whether or not each person is an adult (age 18 or older) in addition to their age. You will want to use the `int` function to convert a string into a number. Run it with the `note`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VvrO3lDwyV5-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def show_ages2(text):\n",
        "  for name, age in re.findall(r\"([a-zA-Z]+) is ([0-9]+) years old\", text):\n",
        "    if int(age) >= 18:\n",
        "      adultness = f'{name} is an adult.'\n",
        "    else:\n",
        "      adultness = f'{name} is not an adult.'\n",
        "    print(f\"{name} is {age}. {adultness}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YpnjLuoWsAZB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "outputId": "fdb4cb34-65d2-43ba-eced-bd4332c0859f"
      },
      "source": [
        "show_ages2(note)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "John is 32. John is an adult.\n",
            "Sarah is 29. Sarah is an adult.\n",
            "Sam is 2. Sam is not an adult.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XyRg2prmyflk",
        "colab_type": "text"
      },
      "source": [
        "Find a similar pattern for indicating who has been diagnosed with what, and use that to get a *set* of the people who have diagnoses mentioned in the note:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AlXHVbMHytwK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a6f49a48-41fc-4e0e-a4cd-2d0f0995ad0a"
      },
      "source": [
        "people_with_diagnoses = set(re.findall(r\"([a-zA-Z]+) was diagnosed with [a-zA-Z]+\", note))\n",
        "people_with_diagnoses"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Jean', 'Sam'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHpZc66vyylM",
        "colab_type": "text"
      },
      "source": [
        "Assume that everyone who is mentioned in a `note` either has an age or a diagnosis. Using your two regular expression patterns, get a *set* of all the people mentioned in the note. (Hint: you might want to use a set's `union` method to combine two sets.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lsFAuY0CzfiM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "17d24038-efeb-4bce-f163-6fbd95baaba0"
      },
      "source": [
        "people_with_ages = re.findall(r\"([a-zA-Z]+) is [0-9]+ years old\", note)\n",
        "all_people = people_with_diagnoses.union(people_with_ages)\n",
        "all_people"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Jean', 'John', 'Sam', 'Sarah'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKtlzgRXzhp-",
        "colab_type": "text"
      },
      "source": [
        "Think about why we'd want to use a *set* instead of a *list*.\n",
        "\n",
        "**We do this to avoid double counting the same person**\n",
        "\n",
        "Finally, have the computer answer the original question: how many people are mentioned in this note?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8uhtQYOzoqG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "98f66016-2ea0-48b7-907a-0f00e0612657"
      },
      "source": [
        "len(all_people)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    }
  ]
}