Python_Generators_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Generators Example (iterators & iterables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iterators and Iterables: Below code does not do any thing but iterating on list of numbers and multiplying with itself, but if you are working on realtime data analysis on hugde data or specifically evaluating each and every item in list, then its not advisable to use list comprehension, since they are not efficient in terms of memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = range(100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "333328333350000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([n**2 for n in numbers])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without using array or without iteration/loop manipulating each and every item is possible though"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "333328333350000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(n**2 for n in numbers) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Iterator & Iterables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iterator is object that manages a single pass over a sequence and Iterable are objects that we can iterate, like list, tuple & dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its an object like iterator, even internally implemented as iterator internally in python (subclass of iterator), but its lazily evaludated, unlike iterators, which return a whole array, a generator yields one value at a time (using yield keyword). This requires less memory.\n",
    "\n",
    "Any function which has yield keyword is called generator function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why it matters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generally use it since we dont have to worry about where in records we have to call next and cache next with try/except, for ex. consider below code fragement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fragment1: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default for loop takes care of tracking which is next record to be printed and when to stop printing (internally its implemented using iterators (subclass of iterator, which is iterable))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "generator = (word + '!' for word in 'baby let me iterate ya'.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baby!\n",
      "let!\n",
      "me!\n",
      "iterate!\n",
      "ya!\n"
     ]
    }
   ],
   "source": [
    "for val in generator:\n",
    "    print(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fragment@: Iteration Protocol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The iteration protocol, if next is not available in list then we have to manually try and except for caching StopIteration exception, which for loop takes care of by default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = iter([1, 2, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<list_iterator at 0x1ded095d198>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-36-4dcbf4f77c89>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-36-4dcbf4f77c89>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    next(x) // As we can see it throws error\u001b[0m\n\u001b[1;37m                   ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "next(x) // As we can see it throws error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fragment3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below fragment tell us how we can tweak iterator protocol by catching StopIteration exception at any point in time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "4\n",
      "27\n",
      "256\n",
      "3125\n",
      "46656\n",
      "823543\n",
      "16777216\n",
      "387420489\n",
      "10000000000\n",
      "285311670611\n",
      "8916100448256\n",
      "302875106592253\n",
      "11112006825558016\n"
     ]
    }
   ],
   "source": [
    "# my_iterator_protocol.py\n",
    "\n",
    "class my_iterator_protocol:\n",
    "    \n",
    "   def __init__(self):\n",
    "      self.x = 0\n",
    "\n",
    "   def __next__(self):\n",
    "      self.x += 1\n",
    "      \n",
    "      if self.x > 14:\n",
    "         raise StopIteration\n",
    "\n",
    "      return self.x ** self.x\n",
    "\n",
    "   def __iter__(self):\n",
    "      return self\n",
    "\n",
    "\n",
    "numbers = my_iterator_protocol()\n",
    "\n",
    "for n in numbers:\n",
    "   print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fragment 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fibonacci Generator: Checkout yield b and followed by a,b reassignment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "1\n",
      "2\n",
      "3\n",
      "5\n",
      "8\n",
      "13\n",
      "21\n",
      "34\n",
      "55\n",
      "Calculation stopped\n"
     ]
    }
   ],
   "source": [
    "# fibonacci_gen.py\n",
    "\n",
    "import time\n",
    "\n",
    "def fib():\n",
    "    \n",
    "   a, b = 0, 1\n",
    "\n",
    "   while True:\n",
    "      yield b\n",
    "      \n",
    "      a, b = b, a + b\n",
    "\n",
    "g = fib()\n",
    "try:\n",
    "   for e in g:\n",
    "      print(e)\n",
    "      \n",
    "      time.sleep(1)\n",
    "            \n",
    "except KeyboardInterrupt:\n",
    "   print(\"Calculation stopped\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fragment 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Generating expressions using generator (better form of comprehension), since  it returns one element in list at a time not entire list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = (e for e in range(50000000) if not e % 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "3\n",
      "6\n",
      "9\n",
      "12\n",
      "15\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-45-fb1ad3bd0caf>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m5\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "\n",
    "for e in n:\n",
    "    print(e)\n",
    "    \n",
    "    i += 1\n",
    "    \n",
    "    if i > 5:\n",
    "        raise StopIteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}