1_Pragmatic_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pragmatic Python\n",
    "Python is a multi-paradigm language. What that means is that Python takes the best of many styles of programming and puts them together in 1 language with ~~a bowtie on top~~ batteries included. Some major features with first-class support are Object-Oriented Programming and functional programming. You can do everything with objects; you can do everything with functions. You get a dynamically-typed language that also offers gradual typing (with type hints). You get a flexibility of a dynamically typed language that you can run in a REPL but also the high-performance of multiprocessing, multithreading, and event-driven frameworks. Whatever you can imagine, Python can (probably) do.\n",
    "\n",
    "So what does it mean to write \"Pythonic\" code or idiomatic Python? \"Pragmatic\" Python is not a technical term--it is not a coding style, per se. I use \"Pragmatic\" Python to mean idiomatic Python or \"Pythonic\" style--what Python code should look like. This notebook will give you an introduction to all the major tips and tricks you can first master before you delve deeper into a specify style. This is like a sampler plate for Python--try 'em all! 😃"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warmups\n",
    "If you are fairly new to Python (or coming from a different language), here are some very cool features (which may be different than how you are used to in your previous language).\n",
    "\n",
    "### Enumerate the Possibilities\n",
    "For people starting to learn Python, they often use C or Java syntax.\n",
    "```python\n",
    "simple_string = \"asdf\"\n",
    "for i in range(len(simple_string)):\n",
    "    print(simple_string[i])\n",
    "```\n",
    "When they figure out you can iterate through containers (list, tuple, set, dict, string, etc.), they are amazed.\n",
    "```python\n",
    "simple_string = \"asdf\"\n",
    "for char in simple_string:\n",
    "    print(char)\n",
    "```\n",
    "But then they want the index (such as for filtering purposes) <i>and</i> the character, so they go back to the original syntax:\n",
    "```python\n",
    "simple_string = \"asdf\"\n",
    "for i in range(len(simple_string)):\n",
    "    print(simple_string[i])\n",
    "    if i == ...: do_something\n",
    "```\n",
    "However, you can use enumerate instead to get the index. `enumerate()` gives you a tuple--(index, element). By default, index starts at 0. E enumerate can take a 2nd argument, which is the index you __actually__ want to start at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 a\n",
      "1 s\n",
      "2 d\n",
      "3 f\n",
      "\n",
      "1 a\n",
      "2 s\n",
      "3 d\n",
      "4 f\n"
     ]
    }
   ],
   "source": [
    "for index, char in enumerate(\"asdf\"): \n",
    "    print(index, char)\n",
    "    \n",
    "print()\n",
    "\n",
    "for index, char in enumerate(\"asdf\", 1):\n",
    "    print(index, char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `zip` it up!\n",
    "If you want to pair 2 iterables together, use `zip()`, which stops at the shorter list. If you want to zip to the longer list, use `itertools.zip_longest()`. `zip()` can also take more than 2 arguments--you can have an arbitrary number of iterables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a 0\n",
      "s 1\n",
      "d 2\n",
      "f 3\n",
      "g 4\n",
      "h 5\n",
      "j 6\n",
      "k 7\n",
      "l 8\n",
      "\n",
      "0 100 200\n",
      "1 101 201\n",
      "2 102 202\n",
      "3 103 203\n",
      "4 104 204\n"
     ]
    }
   ],
   "source": [
    "for char, num in zip(\"asdfghjkl\", range(42)):\n",
    "    print(char, num)\n",
    "    \n",
    "print()\n",
    "\n",
    "for char, num1, num2 in zip(range(5), range(100, 105), range(200, 205)):\n",
    "    print(char, num1, num2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both `enumerate()` and `zip()` are lazy (which will be explained below) in Python 3, so they are extremely memory efficient.  \n",
    "CAVEAT: `zip` is not lazy in Python 2, so you can instead use `itertools.izip`, which is lazy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dict-tutorial: Key to Understanding Your Values\n",
    "A dictionary is so incredibly powerful and useful. A dict stores key-value pairs where the keys are unique and hashable and the values can be anything (often non-unique and unhashable).  \n",
    "Note: a set is basically a value-less dictionary that only has keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hey': 'bye'}\n",
      "{'eeny': 'meeny', 'miny': 'moe'}\n",
      "{'enny': 'meeny', 'miny': 'moe'}\n",
      "{0: '0', 1: '1', 2: '2'}\n",
      "{0: 'my_favorite_value', 1: 'my_favorite_value', 2: 'my_favorite_value'}\n",
      "{0: 'zero', 1: 'one', 2: 'two'}\n",
      "{0: 'zero', 1: 'one', 2: 'two'}\n"
     ]
    }
   ],
   "source": [
    "# There are many ways to create a dict\n",
    "print({\"hey\": \"bye\"})  # dict literal\n",
    "print(dict([(\"eeny\", \"meeny\"), (\"miny\", \"moe\")]))  # dict constructor with a collection of key-value pairs\n",
    "print(dict(enny=\"meeny\", miny=\"moe\"))  # dict constructor with key arguments\n",
    "print({i: str(i) for i in range(3)})  # dict comprehension\n",
    "print(dict.fromkeys(range(3), \"my_favorite_value\"))  # dict method\n",
    "\n",
    "print(dict(enumerate([\"zero\", \"one\", \"two\"])))  # using the lesson above\n",
    "print(dict(zip([0, 1, 2], [\"zero\", \"one\", \"two\"])))  # using the lesson above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other useful dict methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'non-existent key'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-4bc2d57e4419>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mmarco_polo\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;34m\"hide\"\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;34m\"seek\"\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mmarco_polo\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"hide\"\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;31m# just accessing the value\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mmarco_polo\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"non-existent key\"\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;31m# raises KeyError for non-existent key\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m: 'non-existent key'"
     ]
    }
   ],
   "source": [
    "marco_polo = {\"hide\": \"seek\"}\n",
    "marco_polo[\"hide\"]  # just accessing the value\n",
    "marco_polo[\"non-existent key\"]  # raises KeyError for non-existent key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n",
      "Silk Road\n",
      "\n",
      "seek\n",
      "Silk Road\n",
      "\n",
      "('Pizarro', 'Peru')\n"
     ]
    }
   ],
   "source": [
    "print(marco_polo.get(\"non-existent key\"))  # always safe, returns None if key is not found so never raises KeyError\n",
    "print(marco_polo.get(\"non-existent key\", \"Silk Road\"))  # can set a default value if key is not found\n",
    "print()\n",
    "\n",
    "print(marco_polo.pop(\"hide\"))  # returns value and deletes key\n",
    "print(marco_polo.pop(\"hide\", \"Silk Road\"))  # if key doesn't exist, then return default value. If default value not set, then raise KeyError\n",
    "print()\n",
    "\n",
    "explorers = {\"Columbus\": \"America\", \"Cortes\": \"Mexico\", \"Pizarro\": \"Peru\"}\n",
    "print(explorers.popitem())  # pop random key-value pair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'sweet': 'apple', 'sour': 'lemon', 'bitter': 'matcha', 'salty': 'pastrami', 'umami': 'tomato'}\n"
     ]
    }
   ],
   "source": [
    "flavors_n_foods = {\"sweet\": \"watermelon\", \"sour\": \"lemon\", \"bitter\": \"matcha\"}\n",
    "more_flavors_n_foods = {\"salty\": \"pastrami\", \"sweet\": \"apple\", \"umami\": \"tomato\"}\n",
    "flavors_n_foods.update(more_flavors_n_foods)  # update will add new key (and its value) and overwrite existing key with new value\n",
    "print(flavors_n_foods)  # \"watermelon\" overwritten by \"apple, (\"umami\", \"tomato\") added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'apple': 'sweet', 'lemon': 'sour', 'matcha': 'bitter', 'pastrami': 'salty', 'tomato': 'umami'}\n",
      "{'apple': 'sweet', 'lemon': 'sour', 'matcha': 'bitter', 'pastrami': 'salty', 'tomato': 'umami'}\n"
     ]
    }
   ],
   "source": [
    "print(dict((value, key) for key, value in flavors_n_foods.items()))  # \"inverting\" a dict\n",
    "print(dict(reversed(tup) for tup in flavors_n_foods.items()))  # another way to \"invert\" a dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# checking if a key exists in dict is very fast.\n",
    "print(\"sweet\" in flavors_n_foods)  # I personally use this style\n",
    "print(\"sweet\" in flavors_n_foods.keys())  # also valid, fast in Python 3 but slow in Python 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sweet apple\n",
      "sour lemon\n",
      "bitter matcha\n",
      "salty pastrami\n",
      "umami tomato\n",
      "\n",
      "sweet apple\n",
      "sour lemon\n",
      "bitter matcha\n",
      "salty pastrami\n",
      "umami tomato\n"
     ]
    }
   ],
   "source": [
    "# When iterating over the keys and values, don't do this\n",
    "for key in flavors_n_foods:\n",
    "    value = flavors_n_foods[key]\n",
    "    print(key, value)\n",
    "\n",
    "print()\n",
    "\n",
    "# do this\n",
    "for key, value in flavors_n_foods.items():\n",
    "    print(key, value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operator, Do You Like This Condition?\n",
    "Often you just need to pick between 2 things based on a condition. This is called a __ternary operator__ (AKA __conditional expression__). Ternary operators are in many languages you have seen before.\n",
    "\n",
    "```C\n",
    "x = condition ? value_if_true : value_if_false;  // C syntax\n",
    "```\n",
    "\n",
    "```R\n",
    "x <- if (a < b) value_if_true else value_if_false  # R syntax\n",
    "x <- ifelse (a < b, value_if_true, value_if_false)  # vectorized ternary operator\n",
    "```\n",
    "\n",
    "In Python, you COULD write:\n",
    "```python\n",
    "if condition:\n",
    "    x = value_if_true\n",
    "else:\n",
    "    x = value_if_false\n",
    "```\n",
    "But you are a pro!\n",
    "```python\n",
    "x = value_if_true if condition else value_if_false\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On Sunday, I am a nun\n",
      "On all other days, I am a sinner\n"
     ]
    }
   ],
   "source": [
    "day_of_week = \"Sunday\"\n",
    "who_am_i = \"nun\" if day_of_week == \"Sunday\" else \"sinner\"\n",
    "print(\"On {}, I am a {}\".format(day_of_week, who_am_i))\n",
    "\n",
    "day_of_week = \"all other days\"\n",
    "who_am_i = \"nun\" if day_of_week == \"Sunday\" else \"sinner\"\n",
    "print(\"On {}, I am a {}\".format(day_of_week, who_am_i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ¿Comprendes? List Comprehensions to the Rescue\n",
    "List comprehensions are a very succinct way to do `for` loops. Instead of appending to a list like in other languages, just create the same, outputted list in just 1 line. The syntax is: ```[some_expression_here for x in some_iterable_here]```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\n",
      "[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\n"
     ]
    }
   ],
   "source": [
    "# boring, don't do\n",
    "l = []\n",
    "for x in range(10):\n",
    "    l.append(x * 2)\n",
    "print(l)\n",
    "\n",
    "print([x * 2 for x in range(10)])  # better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can put a function/method or use the element's method as the expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3, 2, 1, 0, 1, 2]\n",
      "['A', 'B', 'C', 'D', 'E', 'F']\n",
      "['A', 'B', 'C', 'D', 'E', 'F']\n"
     ]
    }
   ],
   "source": [
    "print([abs(x) for x in range(-3, 3)])  # using function\n",
    "\n",
    "print([str.upper(char) for char in \"abcdef\"])  # method call\n",
    "print([char.upper() for char in \"abcdef\"])  # method call of the element itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can iterate over 2 (or more) iterables simultaneously and have the 2 (or more) looping elements interact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10, 12, 14, 16, 18, 20, 22, 24, 26, 28]\n",
      "[10, 12, 14, 16, 18, 20, 22, 24, 26, 28]\n"
     ]
    }
   ],
   "source": [
    "print([i + j for i, j in zip(range(10), range(10, 20))])\n",
    "print([i + j for i, j in enumerate(range(10, 20))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also filter elements out, so they don't end up in the list. The syntax is: `[x for x in some_iterable_here if some_condition_here]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 3, 5, 7, 9]\n",
      "[1, 3, 5, 7, 9]\n"
     ]
    }
   ],
   "source": [
    "# boo!\n",
    "l = []\n",
    "for x in range(10):\n",
    "    if x % 2:\n",
    "        l.append(x)\n",
    "print(l)\n",
    "\n",
    "print([x for x in range(10) if x % 2])  # booyah!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even a stick a ternary operator (AKA conditional expression) into a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[\"odd\" if x % 2 else \"even\" for x in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can even, even stick a ternary operator into a list comprehension as a filtering condition.  (Not that you would ever want to do this)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 3, 5, 7, 9]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x for x in range(10) if (\"odd\" if x % 2 else \"even\") == \"odd\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do nested comprehension. The syntax is: `[[some_expression_here(x) for x in inner_list] for inner_list in outer_list]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[2, 4, 6], [8, 10, 12], [14, 16, 18]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[[l * 2 for l in l] for l in [[1, 2, 3], [4, 5, 6], [7, 8, 9]]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also, also do a \"double\" list comprehension. The syntax is: `[some_expression_here(x) for inner_list in outer_list for x in inner_list]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 4, 6, 8, 10, 12, 14, 16, 18]\n",
      "[2, 4, 6, 8, 10, 12, 14, 16, 18]\n"
     ]
    }
   ],
   "source": [
    "# newbie\n",
    "final = []\n",
    "for l in [[1, 2, 3], [4, 5, 6], [7, 8, 9]]:\n",
    "    for x in l:\n",
    "        final.append(x * 2)\n",
    "print(final)\n",
    "\n",
    "# this following is one way to flatten nested lists (though itertools.chain.from_iterable is better)\n",
    "print([x * 2 for l in [[1, 2, 3], [4, 5, 6], [7, 8, 9]] for x in l]) # like a pro!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you also, also, also do a double list comprehension with a nested list comprehension that has multiple iterables with a filtering condition and has an expression that is a ternary operator? Yes, let me show you:  \n",
    "```<Insanity detected, the existing example was (ex)terminated with extreme prejudice!>```  \n",
    "<p align=\"center\"><img src=\"images/daleks-exterminate.jpg\" width=400></p><br>\n",
    "In practice, the most you'll do in a list comprehension is an expression and a filtering condition.\n",
    "\n",
    "\n",
    "#### Other Comprehensions\n",
    "There are comprehensions other than for creating lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'a', 1: 'e', 2: 'i', 3: 'o', 4: 'u'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{key: value for key, value in enumerate(\"aeiou\")}  # dictionary comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a', 'e', 'i', 'o', 'u'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{x for x in \"aaaeeeiiiooouuu\"}  # set comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 2, 4, 6, 8, 10, 12, 14, 16, 18)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuple(x * 2 for x in range(10))  # tuple comprehension, really a generator forced to manifest by the tuple() constructor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object <genexpr> at 0x00000195C4B31678>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(x * 2 for x in range(10))  # not a tuple comprehension, it is a generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try and Catch Me if You Can: `try`/`except`\n",
    "If you are from languages where exceptions are rare and not-to-be expected in normal runtime (like C or R), then you are more cautious in programming. Well in Python, you can throw caution to the wind!  \n",
    "In Python, there are 2 paradigms that you can learn towards for different use cases:\n",
    "* Look before you leap (LBYL): be a bit more defensive and check if the conditions are right.\n",
    "* Easier to ask forgiveness than permission (EAFP): just try it out. YOLO.\n",
    "\n",
    "Let's take a look at the second. Make fatal exceptions a thing of the past. In some languages, division by zero terminates your program. That is also true in Python...unless you put a `try`/`except` block. Basically, exceptions are common in Python and not to be feared. For more advanced users, the full syntax is `try`/`except`/`else`/`finally`; learn more about `try`/`except` in `2_Procedural_Python.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "ZeroDivisionError",
     "evalue": "division by zero",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mZeroDivisionError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-13-cb6775e0c57f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;36m1\u001b[0m \u001b[1;33m/\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Done\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mZeroDivisionError\u001b[0m: division by zero"
     ]
    }
   ],
   "source": [
    "1 / 0\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Woops, what I meant was ...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    1 / 0\n",
    "except ZeroDivisionError:\n",
    "    print(\"Woops, what I meant was ...\")\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### /End of Warmups\n",
    "Hopefully that was mostly a review of things you already now. Let's get into juicer stuff.\n",
    "\n",
    "Advanced | Python | Now!\n",
    ":------------------------------:|:---------------------------------:|:-----------------------------|\n",
    "![](images/advanced-python.jpg) | ![](images/python-word-cloud.jpg) | ![](images/calling-python.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical ~~oops~~ OOP\n",
    "Everybody talks about Object Oriented Programming. Here's some actual useful functionality.\n",
    "\n",
    "#### Lingo:\n",
    "* `class`: blueprints or definitions for creating an object. Another synonym for class is `type`.\n",
    "* `object` (also called an instance): an actual, living, breathing creation of a `class`--the manifestation of building out what was in your blueprint. The process of creating an instance is called `instantiation`. A secondary, more open definition of object is anything that is storable/assignable is an object--hence even a function is an object.\n",
    "* `method`: a function you put inside a class.\n",
    "* `attribute`: a variable inside an instance (also called instance variable) or class (also called class variable). Attributes hold data about the instance or class.\n",
    "\n",
    "In short, a Python object/instance has 4 things: a type/class, attributes (data), methods (functions), and a unique identity (which can be found by calling `id()`).\n",
    "\n",
    "#### 3 Types (What an OOP pun!) of Methods:\n",
    "* `instance method`: Most common (vast majority of time). Do something with the instance, ie update the instance's attributes or return something from that instance. Call from the instance.\n",
    "* `class method`: uncommon (maybe 10% of the time). Do something with the class, ie update the class's attributes or return something from that class. Call from the class or an instance.\n",
    "* `static method`: very uncommon (<5% of the time). Use no information about an instance or class, ie knows <i>nothing</i> about the instance or class. Just a regular function but you attached to a class (because it might be helpful). Call from the class or an instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyClass:\n",
    "    my_class_attribute = 42  # notice it looks like a regular assignment within a class\n",
    "    \n",
    "    def my_instance_method(self, my_instance_attribute):\n",
    "        self.my_instance_attribute = my_instance_attribute\n",
    "        return self.my_instance_attribute    \n",
    "    \n",
    "    @classmethod  # notice this decorator\n",
    "    def my_class_method(cls):  # notice `cls`, not `self`\n",
    "        return cls.my_class_attribute  # notice `cls`, not `self`\n",
    "    \n",
    "    @staticmethod  # notice this decorator\n",
    "    def my_static_method(x, y):  # no `self` or `cls`\n",
    "        return x + y "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "my_instance_method() missing 1 required positional argument: 'my_instance_attribute'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-2-26e591635784>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mmy_instance\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mMyClass\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_instance\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmy_instance_method\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# instance method call on instance works\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mMyClass\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmy_instance_method\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# instance method call on class does not work\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m: my_instance_method() missing 1 required positional argument: 'my_instance_attribute'"
     ]
    }
   ],
   "source": [
    "# instance method\n",
    "my_instance = MyClass()\n",
    "print(my_instance.my_instance_method(10))  # instance method call on instance works\n",
    "print(MyClass.my_instance_method(10))  # instance method call on class does not work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# Each instance has its own \"batch\" of data/attributes.\n",
    "# Each instance's attributes are indepedent of other instances' attributes.\n",
    "my_instance.my_instance_method(1)\n",
    "\n",
    "my_instance_2 = MyClass()\n",
    "my_instance_2.my_instance_method(2)\n",
    "\n",
    "print(my_instance.my_instance_attribute)\n",
    "print(my_instance_2.my_instance_attribute)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42\n",
      "42\n",
      "42\n"
     ]
    }
   ],
   "source": [
    "# class method\n",
    "print(my_instance.my_class_method())  # class method call on instance works\n",
    "print(MyClass.my_class_method())  # class method call on class works, though this notation is probably \n",
    "# preferred to denote that it is class method\n",
    "\n",
    "\n",
    "# class attributes are visible to all instances. There is only 1 \"copy\" of the class attribute\n",
    "my_instance_2 = MyClass()\n",
    "print(my_instance_2.my_class_method())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "43\n",
      "43\n",
      "43\n"
     ]
    }
   ],
   "source": [
    "# class attributes can be updated, which then all instances can see\n",
    "MyClass.my_class_attribute += 1\n",
    "print(my_instance.my_class_method())\n",
    "print(my_instance_2.my_class_method())\n",
    "print(MyClass.my_class_method())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# static method\n",
    "print(my_instance.my_static_method(1, 2))  # static method call on instance works\n",
    "print(MyClass.my_static_method(1, 2))  # static method call on class works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical Functional Programming\n",
    "Suppose you live in a world where you don't use classes (gasp!). OOP uses attributes to keep statefulness, but sometimes statefulness hinders more than helps. Functional Programming (often abbreviated as FP) helps solve problems by avoiding mutating state. Hence, FP tends to do everything as a function call, often using `map()`, `filter()`, amd `reduce()`.\n",
    "* `map`: apply a function onto each element of an iterable such as a list. Can be done with a list comprehension.\n",
    "* `filter`: keep only elements in an iterable that meet a certain condition. Can be done with a list comprehension.\n",
    "* `reduce`: compress an iterable down to 1 element/result. <i>Cannot</i> be done with a list comprehension. Can be done with a `for` loop and an accumulator.\n",
    "\n",
    "### I Need Help: Can you `lambda` hand?  \n",
    "A __lambda function__ (AKA anonymous function) is a quick way to define a small (usually 1 line) function. Lambda functions are for the most part equivalent to regular `def` functions. The only small difference is that a lambda function has to be a single expression--statements are not allowed (such as assignment).\n",
    "\n",
    "Common use cases for lambda functions is in functional programming (`map()`, `filter()`, `reduce()`) and DAG-based transformations like Spark (which uses functional programming). However, lambda functions are not exclusive to FP--you can use lambdas in `sorted()`'s key argument and pandas DataFrame `.apply()` and generally anywhere you want to define a simple function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(3, 'x'), (2, 'y'), (1, 'z')]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted([(1, \"z\"), (2, \"y\"), (3, \"x\")], key=lambda tup: tup[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     4\n",
       "1    16\n",
       "2    36\n",
       "3    64\n",
       "Name: a, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({\"a\": range(2, 10, 2)})\n",
    "df[\"a\"].apply(lambda x: x ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from functools import reduce\n",
    "\n",
    "reduce(\n",
    "    lambda x, y: x + y, \n",
    "    filter(\n",
    "        lambda x: x % 2, \n",
    "        map(\n",
    "            lambda x: x * 3, \n",
    "            range(10)\n",
    "        )\n",
    "    )\n",
    ")  # multiple all the numbers by 3, filter and keep numbers that are odd, add them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the previous example is a bit contrived since list comprehensions (and generators) can perform both map and filter, and a for-loop is recommended over a reduce function\n",
    "summation = 0\n",
    "map_and_filter = (element * 3 for element in range(10) if element * 3 % 2)\n",
    "for element in map_and_filter:\n",
    "    summation += element\n",
    "summation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "43\n",
      "3\n",
      "45\n"
     ]
    }
   ],
   "source": [
    "# Technically you can \"name\" your lambda function and also have default values. \n",
    "# Lambda truly is like a `def` function with a 1-line body.\n",
    "summer = lambda x, y=42: x + y\n",
    "\n",
    "print(summer(1))\n",
    "print(summer(1, 2))\n",
    "print(reduce(summer, range(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can have only 1 expression, so you can't do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-6-ce15678b0692>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-6-ce15678b0692>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    (lambda x, y: x + y; x * y)\u001b[0m\n\u001b[1;37m                       ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "(lambda x, y: x + y; x * y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BUT you have make those 2 expressions in 1 \"composite\" expression by making it return a list/tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3, 2]\n"
     ]
    }
   ],
   "source": [
    "silly_func = (lambda x, y: [x + y, x * y])\n",
    "print(silly_func(1, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OMG: You are so Lazy!\n",
    "A nice thing about functional programming is that you can have lazy transformations using `map()` and `filter()` (`reduce()` is not lazy, since it eagerly/immediately executes).\n",
    "**Lazy evaluation**: the object is evaluated when it is needed, not when it is created. Defer the computation until you actually need the result. You get 2 benefits:\n",
    "* save memory since you are not reifying/materializing the entire object in memory.\n",
    "* defer runtime (until you want computation executed), so creating a lazy object creation is effectively instantenous--takes 0 seconds.\n",
    "\n",
    "For example, in Python 3 `range()` is a lazy iterable--it only stores 4 things in memory: start value, end value, skip value, and current value. Hence, `range()` has constant memory requirements. The equivalent in Python 2 is `xrange()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48\n",
      "48\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "print(sys.getsizeof(range(0, 10, 2)))  # gets the size of the object in memory\n",
    "print(sys.getsizeof(range(0, 100000, 2000)))  # much more elements to iterate over much, but same memory footprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When possible, do not materialize/reify/execute lazy objects into its full memory footprint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45112\n"
     ]
    }
   ],
   "source": [
    "print(sys.getsizeof(list(range(0, 100000, 20))))  # making the list will take more memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`map` and `filter` are lazy. They also take constant memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<map object at 0x00000178EC8C5518>\n",
      "56\n"
     ]
    }
   ],
   "source": [
    "doubled = map(lambda x: 2 * x, range(10))\n",
    "print(doubled)\n",
    "print(sys.getsizeof(doubled))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't need to consume all the elements/results of `map` or `filter` in 1 shot. You can iterate over 1 element at a time by using the `next()` function. When the lazy object is exhausted, `next()` will raise StopIteration. The added benefit is that you only \"consume\" what you need. For example, you don't need to create a large list. Suppose you had reified a lazy object into a large list but you only consume a few elements before a function returns; however instantiating the large list and then destroying/garbage collecting it takes RAM and CPU, which was unnecessary.\n",
    "\n",
    "You could have used a lazy object instead and consume only the necessary elements. A lazy generator will defer runtime to when you actually need the next element. Hence, you are able to \"backload\" the runtime in a lazy generator whereas if you use a list comprehension, you are \"frontloading\" the runtime. Again, a lazy object \"backloads\" the runtime and is helpful if you don't intend to iterate through the entire list <i>or</i> you want to \"amoritize\"/split the full runtime into smaller runtime pieces when you compute the next element 1 at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-11-89ff2cbe26e1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdoubled\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdoubled\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdoubled\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# when generator is exhausted of all elements, then generator will raise StopIteration\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "doubled = map(lambda x: 2 * x, range(5))\n",
    "print(next(doubled))\n",
    "print(next(doubled))\n",
    "print(next(doubled))\n",
    "print(next(doubled))\n",
    "print(next(doubled))\n",
    "print(next(doubled))  # when generator is exhausted of all elements, then generator will raise StopIteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also chain lazy objects together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<map object at 0x00000178EC8D1A58>\n",
      "56\n",
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# instantenous runtime\n",
    "doubled = map(lambda x: 2 * x, range(1000000))\n",
    "divisible_by_3 = filter(lambda x: x % 3 == 0, doubled)\n",
    "divisible_by_6 = map(lambda x: 2 * x, divisible_by_3)\n",
    "print(divisible_by_6)  # still a lazy object\n",
    "print(sys.getsizeof(divisible_by_6))  # low memory footprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<map at 0x178ec8d1400>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(lambda x: 2 * x, filter(lambda x: x % 3 == 0, map(lambda x: 2 * x, range(10)))) # logically equivalent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could have materialized/reified all the elements at each transformation, but it is not necesary since you would definitely take more memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 245 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# reifying to a list is often not necessary\n",
    "doubled_list = [2 * x for x in  range(1000000)]  # this intermediate list still takes memory\n",
    "divisible_by_3_list = [x for x in doubled_list if x % 3 == 0]  # this intermediate list also takes memory\n",
    "divisible_by_6_list = [2 * x for x in divisible_by_3_list]  # but you might only want the final list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, the first \"Wall time\" actually corresponds to `%time` (runtime of 1 line) and the second \"Wall time\" correspond to `%%time` (runtime of entire cell).\n",
    "\n",
    "For the cell with \"lazy iterables\", you see that running 1 line and 1 cell is essentially the same runtime since creating lazy iterables is virtually instanteous. The 1 line runtime in lazy iterables takes longer than the 1 line runtime in \"reified iterables\" since the lazy version has to do all the intermediate transformations whereas the reifed iterables just have to add numbers from an already manifested list.\n",
    "\n",
    "The lesson learned is to defer reification/evaluation of a lazy object until the end and <i>only if</i> you actually need a a reified object like a list. Do not reify iterables (i.e. into a list/tuple) for a `reduce()` operation since you want to keep the low memory footprint (which is constant memory O(1))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "666667333332\n",
      "Wall time: 348 ms\n",
      "Wall time: 349 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# lazy iterables\n",
    "from functools import reduce\n",
    "\n",
    "doubled = map(lambda x: 2 * x, range(1000000))\n",
    "divisible_by_3 = filter(lambda x: x % 3 == 0, doubled)\n",
    "divisible_by_6 = map(lambda x: 2 * x, divisible_by_3)\n",
    "\n",
    "%time print(reduce(lambda x, y: x + y, divisible_by_6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "666667333332\n",
      "Wall time: 50.8 ms\n",
      "Wall time: 286 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# reified iterables\n",
    "doubled_list = [2 * x for x in  range(1000000)]\n",
    "divisible_by_3_list = [x for x in doubled_list if x % 3 == 0]\n",
    "divisible_by_6_list = [2 * x for x in divisible_by_3_list]\n",
    "\n",
    "%time print(reduce(lambda x, y: x + y, divisible_by_6_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have only `map` transformations chained together, that is equivalent to <i>function composition</i> (`f(g(x))`) over all the elements of an iterable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 6, 12, 18, 24]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doubled = map(lambda x: 2 * x, range(1000000))\n",
    "tripled = map(lambda x: 3 * x, doubled)\n",
    "\n",
    "[x for x, y in zip(tripled, range(5))]  # trick to extract only a few elements as zip() stops at the shorter at iterable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators: Your Personal Power Plant\n",
    "Generators and generator functions are not Functional Programming topics, but they are also ways to create lazy objects that save memory and defer computation. Lazy iterators are an optimization since you create only what you need <i>Just In Time</i>, not before.\n",
    "\n",
    "### Generator (Expressions)\n",
    "`map` and `filter` can be written as comprehensions. Below is a generator (also called generator expressions since they can be assigned to a variable) equivalent to the `map` and `filter` above. As before, you can chain generators together--that's actually 1 of their strengths. When you iterate over a chained generator, then you are getting 1 element and applying all the transformations first. This is the opposite of chaining list comprehensions in that all the elements go through 1 transformation first before they go through the second transformation.\n",
    "\n",
    "Note: generators (and `map` and `filter`) are lazy <i>iterators</i>, not <i>iterables</i>. In the last section, I used these 2 words loosely synonymously, but there are both technical terms with specific meanings. The difference is not too important, so you don't have to really worry about it.\n",
    "* `Iterator`: An object you can call `next()` on. Calling `iter()` on an iterator just returns itself. For example: `map`, `filter`, `zip`, `enumerate`, generators, file handles\n",
    "* `Iterable`: An object you can call `iter()` on to create an iterator. For example: list, tuple, string, dict, set, string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n",
      "CPU times: user 0 ns, sys: 0 ns, total: 0 ns\n",
      "Wall time: 1.02 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import sys\n",
    "\n",
    "# instantaneous runtime\n",
    "doubled = (x * 2 for x in  range(1000000))\n",
    "divisible_by_3 = (x for x in doubled if x % 3 == 0)  # chained generator\n",
    "divisible_by_6 = (2 * x for x in divisible_by_3)  # chained generator\n",
    "print(sys.getsizeof(divisible_by_6))  # low memory footprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference between list comprehension and a generator is `[]` and `()`. Be careful, this `()` syntax is NOT a tuple comprehension. For tuple comprehension, you have to explicitly call `tuple()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object <genexpr> at 0x7f734047ab88>\n",
      "<generator object <genexpr> at 0x7f734047ab88>\n",
      "(0, 2, 4, 6, 8, 10, 12, 14, 16, 18)\n"
     ]
    }
   ],
   "source": [
    "print((x * 2 for x in  range(10)))  # generator\n",
    "print(x * 2 for x in  range(10))  # generator: parentheses are not required when parentheses are available due to something else such as print()\n",
    "print(tuple(x * 2 for x in  range(10)))  # tuple comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "90\n",
      "90\n"
     ]
    }
   ],
   "source": [
    "print(sum((x * 2 for x in  range(10))))  # extraneous parentheses for generator expression\n",
    "print(sum(x * 2 for x in  range(10)))  # preferred style, no extraneous parentheses\n",
    "generator = (x * 2 for x in  range(10))  # parentheses only needed when it is a stand-alone generator expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49999995000000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Do not reify a generator expression into a list if you don't have to.\n",
    "# ie, don't use a list comprehension in a reducer when a generator will suffice.\n",
    "sum(x for x in range(10000000))  # preferred syntax as sum() is effectively a reduce() operation\n",
    "# sum([x for x in range(10000000)])  # not preferred syntax. Takes lots of memorry! Avoid!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator Functions\n",
    "Generator comprehensions are useful, but generators don't have to be comprehensions. What if your logic <i>cannot</i> be encapsulated by `map` and/or `filter`? If you want to write a more sophisticated generator that still gives you the benefit of constant memory usage and deferred execution, you write a generator <i>function</i> that when called will create a <i>generator</i> (also called <i>generator object</i> but no longer a generator expression).\n",
    "\n",
    "A generator function is like a factory that creates more generators. A generator function uses the keyword `yield` (not `return`) to denote when to pause the function. Do not use `return` in a generator function (as `return` raises an exception inside a generator function). If a function has the word `yield` anywhere in the body, then the function is a generator function, and calling the function returns a generator.\n",
    "\n",
    "How do you iterate over a generator, ie get the next element? You call the `next()` on the generator. The generator automatically \"pauses\" at the `yield` statement. Normal/regular functions have no memory of the previous time you called the function. However a generator is \"paused\" everytime you call `next()`; the generator will start from where it was paused in the generator function (ie the line after the `yield` statement) and end when it hits the next `yield` statement. Hence, a generator function gains its laziness by knowing where to start and stop between function calls--the generator function will <i>not</i> start from the beginning (of its body of code) every time `next()` is called on the generator object. However, the generator function <i>will</i> create another generator object every time the generator function is called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<function simple_generator at 0x0000023530A804C8>\n",
      "<class 'function'>\n"
     ]
    }
   ],
   "source": [
    "def simple_generator():\n",
    "    yield 2\n",
    "    yield 3\n",
    "    yield 5\n",
    "\n",
    "print(simple_generator)\n",
    "print(type(simple_generator))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object simple_generator at 0x0000023530A6EA48>\n",
      "<class 'generator'>\n"
     ]
    }
   ],
   "source": [
    "get_prime = simple_generator()  # call the generator function to create generator object\n",
    "print(get_prime)  # a generator object\n",
    "print(type(get_prime))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "3\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "print(next(get_prime))  # next() is keyword to extract next element from generator\n",
    "print(next(get_prime))\n",
    "print(next(get_prime))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-8-8454382336f4>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_prime\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# when generator is empty, it will raise StopIteration exception\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "print(next(get_prime))  # when generator is empty, it will raise StopIteration exception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a more complex generator function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<function create_generator at 0x0000023530B1A4C8>\n",
      "<class 'function'>\n"
     ]
    }
   ],
   "source": [
    "def create_generator(x):  # generator functions can take argument(s)\n",
    "    print(\"start\")\n",
    "    for x in range(x):\n",
    "        if x % 2:\n",
    "            print(\"hi\")\n",
    "            yield x  # yield is the keyword to pause/maintain state of execution\n",
    "            \n",
    "print(create_generator)\n",
    "print(type(create_generator))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object create_generator at 0x0000023530B19DC8>\n",
      "<class 'generator'>\n"
     ]
    }
   ],
   "source": [
    "get_odds = create_generator(10)  # call the generator function to create generator object\n",
    "print(get_odds)  # a generator object\n",
    "print(type(get_odds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n",
      "hi\n",
      "1\n",
      "hi\n",
      "3\n",
      "hi\n",
      "5\n",
      "hi\n",
      "7\n",
      "hi\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "print(next(get_odds))  # prints \"start\" 1 time\n",
    "print(next(get_odds))  # next() is keyword to extract next element from generator\n",
    "print(next(get_odds))\n",
    "print(next(get_odds))\n",
    "print(next(get_odds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-12-dd7f08c5c4aa>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_odds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# when generator is empty, it will raise StopIteration exception\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "print(next(get_odds))  # when generator is empty, it will raise StopIteration exception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Infinite Generators: To Infinity and Beyond\n",
    "A generator does not even have to terminate. A generator can go on forever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "def create_generator():\n",
    "    current_element = 0\n",
    "    while True:\n",
    "        yield current_element\n",
    "        current_element += 1\n",
    "        \n",
    "infinite_generator = create_generator()\n",
    "print(next(infinite_generator))\n",
    "print(next(infinite_generator))\n",
    "print(next(infinite_generator))\n",
    "print(next(infinite_generator))\n",
    "print(next(infinite_generator))\n",
    "print(next(infinite_generator))\n",
    "# sum(infinite_generator) # do not execute this line as it will run forever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "3\n",
      "5\n",
      "7\n",
      "11\n",
      "13\n",
      "17\n",
      "19\n",
      "23\n"
     ]
    }
   ],
   "source": [
    "def create_prime_generator():\n",
    "    def is_prime(x):\n",
    "        for _x in range(2, int(x ** 0.5) + 1):\n",
    "            if x % _x == 0:\n",
    "                return False\n",
    "        else:\n",
    "            return True\n",
    "\n",
    "    yield 2\n",
    "    yield 3\n",
    "    current_number = 4\n",
    "    while True:\n",
    "        if is_prime(current_number):\n",
    "            yield current_number\n",
    "        current_number += 1\n",
    "\n",
    "primes = create_prime_generator()\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))\n",
    "print(next(primes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there are cases when you don't want to use a generator.\n",
    "* You can only move forward in a generator; you cannot stay in the same place. For example, `my_list[0]` called twice will give you the same element. This is not true if you call `next(my_generator)` twice.\n",
    "* You cannot slice a generator or jump directly to the nth element. Generators create elements Just In Time, so you cannot slice/jump to the nth element. You have to create elements 1 at a time. For example, you cannot do `generator[5:10]` or even `generator[1]`.\n",
    "* Once a generator is exhausted, then the state is lost. You cannot iterate over a generator a second time (which you can with a list/tuple/dict/set/materialized data structure). A corollary is you cannot go backwards in a generator. To iterate over the elements again, you have to create another generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Key in Getting into Arguments\n",
    "Often times, you see `*args` and `**kwargs`, and you always were curious about what it does but were too afraid to ask. Be careful: where you put `*args` affects how it works.\n",
    "\n",
    "To put it more concretely, if you see `*args` in the function **definition/signature** (`def my_function(*args)`), then it is gathering all the arguments into 1 tuple stored in a variable called `args`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2, 3, 4, 4)\n",
      "<class 'tuple'>\n"
     ]
    }
   ],
   "source": [
    "def silly_function(*args):\n",
    "    print(args)\n",
    "    print(type(args))\n",
    "\n",
    "silly_function(1, 2, 3, 4, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if you see `*args` in the function **call**, then it is separating the iterable into multiple arguments: 1 argument becomes multiple arguments. The order of the separated arguments is retained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "\n",
      "x\n",
      "y\n",
      "z\n"
     ]
    }
   ],
   "source": [
    "def silly_function(a, b, c):\n",
    "    print(a)\n",
    "    print(b)\n",
    "    print(c)\n",
    "\n",
    "silly_function(*(1, 2, 3))  # this function call passes in 1 argument, which is a tuple.\n",
    "# logically equivalent to:\n",
    "# silly_function(1, 2, 3)\n",
    "print()\n",
    "silly_function(*\"xyz\")  # this function call passes in a string, which is technically iterable.\n",
    "# logically equivalent to:\n",
    "# silly_function(\"x\", \"y\", \"z\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`**kwargs` stands for keyword arguments. It means that you are explicitly passing in an argument name and argument value. Again, the placement of `**kwargs` changes the behavior. `**kwargs` in the function **definition/signature** means that all named arguments will be placed in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 1, 'b': 2, 'c': 3}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "def silly_function(**kwargs):\n",
    "    print(kwargs)\n",
    "    print(type(kwargs))\n",
    "    \n",
    "silly_function(a=1, b=2, c=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`**kwargs` in the function **call** means separate the dictionary into named arguments. This is useful if you want to guarantee that the variables are explicitly given an argument name and argument value when you cannot rely on the order of the arguments (such as in `*args`) to match the named arguments in a function signature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "def silly_function(a, b, c):\n",
    "    print(a)\n",
    "    print(b)\n",
    "    print(c)\n",
    "    \n",
    "silly_function(**{\"c\": 3, \"b\": 2, \"a\": 1})  # equivalent to silly_function(c=3, b=2, a=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `*args` and `**kwargs` together. Also, the word `args` or `kwargs` are not special. It's just a Python convention. You can name it anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2, 3)\n",
      "{'d': 4, 'e': 5, 'f': 6}\n"
     ]
    }
   ],
   "source": [
    "def silly_function(*my_silly_args, **my_silly_kwargs):  # * in the function definition\n",
    "    print(my_silly_args)\n",
    "    print(my_silly_kwargs)\n",
    "    \n",
    "silly_function(1, 2, 3, d=4, e=5, f=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2 3 4 5 6\n"
     ]
    }
   ],
   "source": [
    "def silly_function(a, b, c, d, e, f):\n",
    "    print(a, b, c, d, e, f)\n",
    "    \n",
    "silly_function(\n",
    "    *[1, 2, 3],  # * in the function call\n",
    "    **{\"f\": 6, \"e\": 5, \"d\": 4}\n",
    ")\n",
    "# logically equivalent to:\n",
    "# silly_function(1, 2, 3, f=6, e=5, d=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2, 3)\n",
      "{'d': 4, 'e': 5, 'f': 6}\n"
     ]
    }
   ],
   "source": [
    "# stars everywhere: it's a constellation!\n",
    "def silly_function(*my_silly_args, **my_silly_kwargs):\n",
    "    print(my_silly_args)\n",
    "    print(my_silly_kwargs)\n",
    "    \n",
    "silly_function(\n",
    "    *[1, 2, 3],\n",
    "    **{\"d\": 4, \"e\": 5, \"f\": 6}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often times, you will see a function called a decorator that has both `*args` and `**kwargs` and the purpose is to give the full expressiveness of the original function (without loss of generality). Whatever you could do with the original function, you can do with the decorated function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def identity_decorator(func):  # does nothing special\n",
    "    def inner(*args, **kwargs):\n",
    "        return func(*args, **kwargs)\n",
    "    return inner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, \\*args (or \\*\\*kwargs) inside the function **signature** means to <i>group together</i> all the arguments into 1 tuple (or dict). \\*args (or \\*\\*kwargs) inside a function **call** means to separate a tuple/list/iterable (or dict) into multiple arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decorate like a Boss!\n",
    "Everything you wanted to know about **decorators** but were too afraid to ask.\n",
    "\n",
    "Decorators sounds like an ornament you put on a Christmas tree. Then you put your programming hat on, and you think it must be very complicated. BUT it's actually a very simple concept.\n",
    "* Boring, academic definition: decorators are higher-order functions that either take in a function and/or return a function to compose more sophisticated functions.\n",
    "* Actual practical importance: You have a function that you want to change some behavior before or after the function call.\n",
    "\n",
    "Now you may be wondering: why would you want a decorator if you can just change the original function (if all you are going to do is change something before and/or after the original function call). The reason could be:\n",
    "* You can't change the original function because it is too complicated to understand.\n",
    "* You can't change the original function because you don't have access to change it. For example, many numpy functions are written in C, so you literally could not change the source code.\n",
    "* You can't change the original function because it is used everywhere and you don't know which ones you want to change.\n",
    "* You just wanna be ~~cool~~ a pro!\n",
    "\n",
    "*Nota bene*: subtitle stolen/appropriated/borrowed/adapted from an article called \"Everything You Wanted to Know about the Kernel Trick\n",
    "(But Were Too Afraid to Ask)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_my_function_polite(func):\n",
    "    def inner(arg):\n",
    "        print(\"hello!\")\n",
    "        result = func(arg)  # notice I'm storing results if I want to do something after my function call\n",
    "        print(\"bye!\")\n",
    "        return result\n",
    "    return inner  # notice I am returning a function back"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84\n"
     ]
    }
   ],
   "source": [
    "def double(x):\n",
    "    return x * 2\n",
    "print(double(42))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.make_my_function_polite.<locals>.inner>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_my_function_polite(double)  # notice that a function is returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello!\n",
      "bye!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "84"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_my_function_polite(double)(42)  # apply decorator 1 time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello!\n",
      "bye!\n",
      "2\n",
      "\n",
      "hello!\n",
      "bye!\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "# however, often times, you want to make the effects of the decorator \"permanent\" to all function calls\n",
    "double = make_my_function_polite(double)\n",
    "print(double(1))\n",
    "print()\n",
    "print(double(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello!\n",
      "bye!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "84"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# however, the previous syntax is ugly--nobody uses it. Here's the decorator syntax you see in the real-world\n",
    "@make_my_function_polite  # @ syntax is equivalent to double = make_my_function_polite(double)\n",
    "def double(x):\n",
    "    return x * 2\n",
    "\n",
    "double(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you may be thinking, what are some practical things you want to change before you your original function call: change the arguments or check for valid types. Some practical things you want after the original function call: logging action to disk, closing a connection to a database, return the output of the original function call and an extra flag depending on the output.\n",
    "\n",
    "The decorator example below is a simple to way to make Python enforce type on the arguments. You can think of it as making Python (pseudo) statically typed. It is not real static typing (because Python does not have a compilation step) and because the invalid argument types are only caught at runtime (when you call the function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Not a valid argument",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-7-e9bcd36c2dec>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     11\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m*\u001b[0m \u001b[1;36m2\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mdouble\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"42\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-7-e9bcd36c2dec>\u001b[0m in \u001b[0;36minner\u001b[1;34m(arg)\u001b[0m\n\u001b[0;32m      2\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0minner\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mfloat\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mint\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Not a valid argument\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Not a valid argument"
     ]
    }
   ],
   "source": [
    "def penalize_type(func):\n",
    "    def inner(arg):\n",
    "        if not isinstance(arg, (float, int)):\n",
    "            raise ValueError(\"Not a valid argument\")\n",
    "        else:\n",
    "            return func(arg)\n",
    "    return inner\n",
    "\n",
    "@penalize_type\n",
    "def double(x):\n",
    "    return x * 2\n",
    "\n",
    "double(\"42\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "84.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def accommodate_type(func):\n",
    "    def inner(arg):\n",
    "        if isinstance(arg, str):\n",
    "            arg = float(arg)\n",
    "        return func(arg)\n",
    "    return inner\n",
    "\n",
    "@accommodate_type\n",
    "def double(x):\n",
    "    return x * 2\n",
    "\n",
    "double(\"42\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is more content about decorators in `4_Functional_Python.ipynb`. But for now, are there any more questions? 🧐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bug Smashers\n",
    "Practical debugging tricks\n",
    "\n",
    "#### `pdb` or `ipdb`\n",
    "When your back is against the wall, use the debugger! I tend to use `print()` when I hit a bug. However, when the problem is so deep that I cannot easily think it through, then it's time to use the trusty debugger.  \n",
    "Put `pdb.set_trace()` (or `breakpoint()` in Python 3.7+) right before the line you get an error message--you can even use it in the notebook!  \n",
    "2 important keys:\n",
    " * n - run next line\n",
    " * q - quit the debugger\n",
    "While inside the debugger, all your variables are alive, so you can literally see what state your variables/objects are in. You can run arbitrary functions, expressions, and statements in the debugger. For example, you can overwrite a variable with a different value to see how your function would work under a different condition.\n",
    "\n",
    "`ipdb` is a fancier debugger that needs to be installed, but it offers tab completion and nicer printouts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting script_to_debug.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile script_to_debug.py\n",
    "import pdb\n",
    "\n",
    "def flawed_function():\n",
    "    x = 1\n",
    "    y = 0\n",
    "    pdb.set_trace()\n",
    "    return x / y\n",
    "\n",
    "flawed_function()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run `python script_to_debug.py` in the terminal, the Python interpreter will stop where you put `set_trace()` and the variables will be alive. You will be in the interactive mode in the Python REPL.\n",
    "\n",
    "You can also run `pdb` in the notebook itself!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> \u001b[1;32mc:\\users\\eugene\\appdata\\local\\temp\\ipykernel_12452\\3917449600.py\u001b[0m(7)\u001b[0;36mflawed_function\u001b[1;34m()\u001b[0m\n",
      "\n",
      "ipdb> n\n",
      "ZeroDivisionError: division by zero\n",
      "> \u001b[1;32mc:\\users\\eugene\\appdata\\local\\temp\\ipykernel_12452\\3917449600.py\u001b[0m(7)\u001b[0;36mflawed_function\u001b[1;34m()\u001b[0m\n",
      "\n",
      "ipdb> x\n",
      "1\n",
      "ipdb> y\n",
      "0\n",
      "ipdb> q\n"
     ]
    },
    {
     "ename": "BdbQuit",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mBdbQuit\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_12452\\3917449600.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m/\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[0mflawed_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_12452\\3917449600.py\u001b[0m in \u001b[0;36mflawed_function\u001b[1;34m()\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m     \u001b[0mpdb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_trace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m/\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[0mflawed_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\bdb.py\u001b[0m in \u001b[0;36mtrace_dispatch\u001b[1;34m(self, frame, event, arg)\u001b[0m\n\u001b[0;32m     92\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch_return\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     93\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'exception'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 94\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     95\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'c_call'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     96\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrace_dispatch\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\bdb.py\u001b[0m in \u001b[0;36mdispatch_exception\u001b[1;34m(self, frame, arg)\u001b[0m\n\u001b[0;32m    172\u001b[0m                     and arg[0] is StopIteration and arg[2] is None):\n\u001b[0;32m    173\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0muser_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 174\u001b[1;33m                 \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mquitting\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32mraise\u001b[0m \u001b[0mBdbQuit\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    175\u001b[0m         \u001b[1;31m# Stop at the StopIteration or GeneratorExit exception when the user\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    176\u001b[0m         \u001b[1;31m# has set stopframe in a generator by issuing a return command, or a\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mBdbQuit\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import pdb\n",
    "\n",
    "def flawed_function():\n",
    "    x = 1\n",
    "    y = 0\n",
    "    pdb.set_trace()\n",
    "    return x / y\n",
    "\n",
    "flawed_function()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you can't easily edit the source code of the function that failed (ie can't put in `pdb.set_trace()`), then use `pdb.post_mortem()`/`ipdb.post_mortem()` to stay within the frame of failing function/code. Remember to do `try/except` to first catch the exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "ZeroDivisionError",
     "evalue": "division by zero",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mZeroDivisionError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_12452\\2530605904.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mflawed_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_12452\\2530605904.py\u001b[0m in \u001b[0;36mflawed_function\u001b[1;34m()\u001b[0m\n\u001b[0;32m      2\u001b[0m     \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m     \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m/\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mZeroDivisionError\u001b[0m: division by zero"
     ]
    }
   ],
   "source": [
    "def flawed_function():\n",
    "    x = 1\n",
    "    y = 0\n",
    "    return x / y\n",
    "\n",
    "\n",
    "flawed_function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> \u001b[1;32mc:\\users\\eugene\\appdata\\local\\temp\\ipykernel_12452\\2530605904.py\u001b[0m(4)\u001b[0;36mflawed_function\u001b[1;34m()\u001b[0m\n",
      "\n",
      "ipdb> x\n",
      "1\n",
      "ipdb> y\n",
      "0\n",
      "ipdb> q\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    flawed_function()\n",
    "except:\n",
    "    pdb.post_mortem()  # you stay inside flawed_function() call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `inspect/dir`\n",
    "Another equally handy tool is the `inspect` library. Specifically, `getsource()` extracts the source code for modules, classes, functions, and methods----the main exceptions are built-ins and Cython code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def flawed_function():\n",
      "    x = 1\n",
      "    y = 0\n",
      "    return x / y\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import inspect\n",
    "\n",
    "print(inspect.getsource(flawed_function))  # get the source code--you can do this inside the debugger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    @Substitution(\"\")\n",
      "    @Appender(_merge_doc, indents=2)\n",
      "    def merge(\n",
      "        self,\n",
      "        right: FrameOrSeriesUnion,\n",
      "        how: str = \"inner\",\n",
      "        on: IndexLabel | None = None,\n",
      "        left_on: IndexLabel | None = None,\n",
      "        right_on: IndexLabel | None = None,\n",
      "        left_index: bool = False,\n",
      "        right_index: bool = False,\n",
      "        sort: bool = False,\n",
      "        suffixes: Suffixes = (\"_x\", \"_y\"),\n",
      "        copy: bool = True,\n",
      "        indicator: bool = False,\n",
      "        validate: str | None = None,\n",
      "    ) -> DataFrame:\n",
      "        from pandas.core.reshape.merge import merge\n",
      "\n",
      "        return merge(\n",
      "            self,\n",
      "            right,\n",
      "            how=how,\n",
      "            on=on,\n",
      "            left_on=left_on,\n",
      "            right_on=right_on,\n",
      "            left_index=left_index,\n",
      "            right_index=right_index,\n",
      "            sort=sort,\n",
      "            suffixes=suffixes,\n",
      "            copy=copy,\n",
      "            indicator=indicator,\n",
      "            validate=validate,\n",
      "        )\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "print(inspect.getsource(pd.DataFrame.merge))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dir()` is a builtin-in function (not a module) that allows you the inspect the modules, classes, and objects. I use it primarily to see what attributes/methods an object has. Once I know what attributes it has, then I can go inspect and see what values the attributes contain.  \n",
    "You can use a combination of `dir()` and `inspect.getsource()` to get to know an object better--get a little more intimate ;)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__bool__',\n",
       " '__class__',\n",
       " '__contains__',\n",
       " '__delattr__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__getitem__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__iter__',\n",
       " '__le__',\n",
       " '__len__',\n",
       " '__lt__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__reversed__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " 'count',\n",
       " 'index',\n",
       " 'start',\n",
       " 'step',\n",
       " 'stop']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = range(10)\n",
    "dir(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helpful Libraries\n",
    "#### `collections`:\n",
    "* Marketing docstring: This module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers: dict, list, set, and tuple.\n",
    "* What it actually does: Whatever types you like, they can be even cooler!\n",
    "* Useful classes: `Counter` and `defaultdict`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 7, 'b': 5, 'c': 6, 'd': 4}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# most basic way to count letters. Body of `for` loop is 4 lines\n",
    "letters = \"abcdcacdacadacdabbbabc\"\n",
    "\n",
    "letter_count = {}\n",
    "for letter in letters:\n",
    "    if letter in letter_count:\n",
    "        letter_count[letter] += 1\n",
    "    else:\n",
    "        letter_count[letter] = 1\n",
    "letter_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 7, 'b': 5, 'c': 6, 'd': 4}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use `dict.get`, which is a safe operator for unknown keys. Body of `for` loop is 1 line\n",
    "letters = \"abcdcacdacadacdabbbabc\"\n",
    "\n",
    "letter_count = {}\n",
    "for letter in letters:\n",
    "    letter_count[letter] = letter_count.get(letter, 0) + 1\n",
    "letter_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'a': 7, 'c': 6, 'b': 5, 'd': 4})\n",
      "[('a', 7), ('c', 6), ('b', 5), ('d', 4)]\n",
      "4\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "# use Counter instead! Counter is really an upgraded dictionary: it counts! No `for` loop!\n",
    "from collections import Counter\n",
    "\n",
    "letters = \"abcdcacdacadacdabbbabc\"\n",
    "counter = Counter(letters)  # just put your iterable here and Counter will do the rest\n",
    "print(counter)\n",
    "print(counter.most_common())\n",
    "print(counter['d'])  # key is inside dict\n",
    "print(counter['z'])  # key is not inside dict, so will output 0. Hence you don't need to counter.get('z', 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n",
      "defaultdict(<class 'list'>, {'a': [1]})\n"
     ]
    }
   ],
   "source": [
    "# defaultdict takes in a function, and that will be your default value if the key doesn't exist yet\n",
    "from collections import defaultdict\n",
    "\n",
    "dict_key_always_has_value = defaultdict(list)  # put a function here\n",
    "print(dict_key_always_has_value['a'])  # the key doesn't exist in your dictionary but the value is already available\n",
    "dict_key_always_has_value['a'].append(1)\n",
    "print(dict_key_always_has_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<class 'int'>, {'a': 7, 'b': 5, 'c': 6, 'd': 4})\n"
     ]
    }
   ],
   "source": [
    "# Here is how you do letter counts using a defaultdict.\n",
    "# If you think about it, a Counter is just a defaultdict using int, since int() returns 0.\n",
    "letters = \"abcdcacdacadacdabbbabc\"\n",
    "\n",
    "dict_key_always_has_value = defaultdict(int)  # because int() returns 0\n",
    "for letter in letters:\n",
    "    dict_key_always_has_value[letter] = dict_key_always_has_value[letter] + 1  # equivalent to dict.get(letter, function_here())\n",
    "print(dict_key_always_has_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[None, None, None, None, None, None, None, None, None, None]\n",
      "defaultdict(<function <lambda> at 0x00000276BB924C80>, {'a': [None, None, None, None, None, None, None, None, None, None]})\n",
      "False\n",
      "defaultdict(<function <lambda> at 0x00000276BB924C80>, {'a': [None, None, None, None, None, None, None, None, None, None]})\n"
     ]
    }
   ],
   "source": [
    "# since the defaultdict argument is just a function, you can create whatever default values you like\n",
    "dict_key_always_has_value = defaultdict(lambda: [None] * 10)\n",
    "print(dict_key_always_has_value['a'])  # the value by default is automatically [None] * 10\n",
    "print(dict_key_always_has_value)  # notice that once you lookup ANY key, the key-value pair now exists in your dictionary\n",
    "print('b' in dict_key_always_has_value)  # use this notation if you just want to check membership, but not add key-value pair to dict\n",
    "print(dict_key_always_has_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<class 'list'>, {})\n",
      "[]\n",
      "defaultdict(<function <lambda> at 0x00000276BB9248C8>, {'a': defaultdict(<class 'list'>, {'a': [42]})})\n"
     ]
    }
   ],
   "source": [
    "# can nest defaultdicts for interesting data structure. I have actually used dict in dict before \n",
    "nested_defaultdict = defaultdict(lambda: defaultdict(list))  # each argument of a defaultdict must be a function (or callable)\n",
    "print(nested_defaultdict['a'])  # gives you the inner defaultdict back\n",
    "print(nested_defaultdict['a']['a'])  # gives you the nested list\n",
    "nested_defaultdict['a']['a'].append(42)\n",
    "print(nested_defaultdict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `tqdm`:\n",
    "* Marketing docstring: A Fast, Extensible Progress Bar for Python and CLI.\n",
    "* What it actually does: put a timer everywhere! If you think a piece of your code is slow, time it to find out!\n",
    "* Fun fact: tqdm means \"progress\" in Arabic (taqadum, تقدّم) and is an abbreviation for \"I love you so much\" in Spanish (te quiero demasiado)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:10<00:00,  1.00s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm  # tqdm is a great library to show progress bar\n",
    "import time\n",
    "\n",
    "for i in tqdm(range(10)):  # just wrap tqdm() around your iterable; useful Time Lapsed and Estimated Time of Completion for loops\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `os`:\n",
    "Instead  of directory_name + \"/\" + filename, use this instead since it doesn't require adding '/' to directory_name and is safer between operation systems and also won't add an unnecessary \"/\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "directory_name\\file_name\n",
      "directory_name/file_name\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "print(os.path.join('directory_name', 'file_name'))\n",
    "print(os.path.join('directory_name/', 'file_name'))  # adding an extra \"/\" will not break it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`os` also has other useful utility functions such `os.path.isdir()`, `os.listdir()`, `os.mkdir()`, `os.rmdir()`, `os.getcwd()`, `os.cpu_count()`. If you are extra fancy, you can use a library called `pathlib` instead of `os`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pythonic Syntax\n",
    "#### String Interpolation\n",
    "There are 3 ways to do string interpolation: %-formatting (inherited from C syntax), str.format(), f-strings (new in Python 3.6). I tend to use str.format() and f-strings, but this is up to your personal preference.\n",
    "\n",
    "Here's an example of str.format() where I put all the variables at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, my name is Peter Pan\n",
      "Last name: Pan; first name: Peter. Again my name is Peter Pan\n"
     ]
    }
   ],
   "source": [
    "first_name = 'Peter'\n",
    "last_name = 'Pan'\n",
    "print(\"Hello, my name is {} {}\".format(first_name, last_name))\n",
    "# You can change the order using named arguments and also repeat the arguments\n",
    "print(\n",
    "    \"Last name: {last_name}; first name: {first_name}. \"\n",
    "    \"Again my name is {first_name} {last_name}\"\n",
    "    .format(first_name=first_name, last_name=last_name)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f\\*\\*\\*\\* it! You can use the new f-string syntax that came out in Python 3.6. The only downside is that you cannot use this syntax in earlier versions of Python. But you should be using Python 3.6+ anyways 😉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, my name is Peter Pan\n",
      "I am 13 years old!\n"
     ]
    }
   ],
   "source": [
    "print(f\"Hello, my name is {first_name} {last_name}\")\n",
    "# have you ever considered nested string formatting? ;-)\n",
    "print(f\"{f'I am {42 -29} years old!'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Context Manager\n",
    "Context manager (`with open() as f:`) syntax guarantees that the file will be closed after everything in the code block is run. It saves one line (don't have to close file manually) and also make the code block (for what you want to do the file) very obvious due to the indentation. Context managers exist for other things like connecting to a database or creating a Spark context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      " \"cells\": [\n",
      "  {\n",
      "   \"cell_type\": \"markdown\",\n",
      "   \"metadata\": {},\n",
      "   \"source\": [\n",
      "    \"## Name Binding\n"
     ]
    }
   ],
   "source": [
    "with open(\"6_Pedantic_Python_Tricks.ipynb\") as f:\n",
    "    print(f.read()[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Implicit Line Continuation and Implicit String Concatenation\n",
    "Instead of `\\` to denote line continuation, I use parentheses. Good for function calls with lots of arguments. Line continuation also works inside brackets []."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is my super looooooooooooooooooooooooooooooong string and it ends here!\n",
      "True\n",
      "This is my super long string that never ends because I do not want to stop typing for some reason until I'm out of breath!\n",
      "I like to count to big numbers. I start at 1 and finally end up at 10000000000\n"
     ]
    }
   ],
   "source": [
    "sum([1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5,\n",
    "     6, 7, 8, 9, 0])\n",
    "print(\"This is my super looooooooooooooooooooooooooooooong string \"\n",
    "     \"and it ends here!\")\n",
    "\n",
    "# Implicit string concatenation: notice that string literals are automatically appended to each other without '+' operator\n",
    "print(\"asdf\" \"asdf\" == 'asdfasdf')\n",
    "# Hence, parentheses and long strings work well together.\n",
    "my_long_string = ('This is my super long string that never ends because I do not want to stop '\n",
    "                  \"typing for some reason until I'm out of breath!\")\n",
    "my_long_string_2 = ('I like to count to big numbers. I start at {} '\n",
    "                    'and finally end up at {}'\n",
    "                    .format(1, int(1e10)))\n",
    "print(my_long_string)\n",
    "print(my_long_string_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Trailing Comma\n",
    "You can have a trailing comma in Python collections (list, tuple, set, dict) and functions. The reason is for  ease of updating the collection (or function) if it is multi-line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = [\n",
    "    1,\n",
    "    2,\n",
    "    3,\n",
    "]\n",
    "def silly(\n",
    "    billy,\n",
    "    \n",
    "    hilly,\n",
    "): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### PEP8: Python's Style Guide\n",
    "PEP8 is a nice style guide for readability. I try my best, but even I don't get it right every time.\n",
    "If you are very fancy, you can have Python automatically format your code to conform to pep8 if you type this in the Terminal\n",
    "```\n",
    "autopep8 your_python_script_here.py\n",
    "```\n",
    "This will print to Terminal the correctly formatted script but doesn't save it. If you want to save the results back into your script, you can use the argument `--in-place`\n",
    "```\n",
    "autopep8 --in-place your_python_script_here.py\n",
    "```\n",
    "You can read more about Python's style guide at\n",
    "* https://www.python.org/dev/peps/pep-0008/ (the official Python style guide)\n",
    "* http://pymbook.readthedocs.io/en/latest/pep8.html (focus on the main points)\n",
    "* http://google.github.io/styleguide/pyguide.html (very comprehensive style guide)\n",
    "\n",
    "NOTE: now there's a new kid on the block created by a Python core developer: `black` (https://github.com/psf/black). If you install `black` and run `black {source_file_or_directory}`, it will automatically format your code to be PEP8 compliant. No fuss, no muss, just only 1 way to do it. `black` is more popular than `autopep8`.\n",
    "\n",
    "I usually use triple hash sign `###` when I need to make an important comment. Usually to mention something is hard coded. This is not a Python PEP8 style. It's just a personal preference to make note that this is not a regular comment.\n",
    "```python\n",
    "PI = 3.14159265 ### this is hard coded\n",
    "```\n",
    "\n",
    "As a personal preference, I prefer to write pure functions with no side effects where if you put in the same input, you always get the same output. When practical, I try to avoid functions that mutate whatever is inputted, though this is not always possible or practical. Pure functions are easier to debug. Sometimes impure/mutative functions are just easier to write or run faster. In that case, you can add to the function's docstring that this function mutates the underlying object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Resources\n",
    "* https://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-python.html: This article inspired me to make this series of notebooks. The author is Nick Coghlan, who is a core Python dev, showed me the universe of possibilities using Python.\n",
    "* Advanced Python (https://www.youtube.com/watch?v=uOzdG3lwcB4): Thomas Wouters, a core Python dev, gave a great talk on Python that focused more on Python syntax and tricks. It is amazing how timeless his lessons are, as he gave this talk in 2007 before Python 3 existed. He was working with Python 2.5 at the time. I learned quite a lot from his talks, as he tells you how Python *really* works with the magic methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concluding Remarks\n",
    "Since software is nothing but knowledge representation; it is the answer to the question: \"What is the best way to represent my problem?\" You can choose a specific programming paradigm that acts like a framework on how to approach a problem. However, Python is not strictly OOP like Java nor is it pure functional like Haskell. You choose the parts you like and put them together. Python takes a multi-paradigm approach and it is constantly evolving. Your style should evolve too to fit the problem you are trying to solve. And also don't take paradigms _too_ seriously. The reason is that you can combine stylistic elements to formulate a more comprehensive solution. For example, there's a phrase called \"thesis, antithesis, synthesis\". You have an idea, somebody has an opposite idea, then you try to think of a path that reconciles the two. For example, OOP is not the opposite of functional; for example, Spark uses both elements of OOP and functional. Don't let the paradigm be the bounding box of your imagination.\n",
    "\n",
    "Also, I recommend you take notes when you learn Python. Once upon a time, I was a R programmer, and I was pretty good at it; I made a 2-player snake game that plays itself. A student smarter told me to learn Python, but at the time I had no motivation to do so. However, once I jumped on the Python bandwagon (ie was forced to learn it), then it opened my eyes to a whole universe of possibilities. As a student of Python, I recommend that you keep taking notes of Python, so you can look back at it for useful tips but also see your improvement as your notes will become more advanced over time. Your programming notes act like a diary as it documents your evolution through problem solving."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}