Lecture-24-Parallel-Computing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 24:  Parallel Computing\n",
    "----\n",
    "\n",
    "Start the engines and load other auxillary librarys for function access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys,time\n",
    "import numpy\n",
    "\n",
    "from ipyparallel import Client \n",
    "rc = Client()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dview = rc[:]\n",
    "e0 = rc[0]\n",
    "e1 = rc[1]\n",
    "e2 = rc[2]\n",
    "\n",
    "dview.block = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We are using Python to demonstrate concepts.  This is by no means the only way to program parallel.  My simulations are written using an entirely different library.  I'm using this environment because it is interactive and can clarify the types of things we need to think about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are 4 engines available for computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[15384, 9800, 17248, 15136]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.apply_sync(os.getpid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scatter and Gather\n",
    "\n",
    "Lots of parallel computations involve partitioning data onto processes.\n",
    "\n",
    "DirectViews have `scatter()` and `gather()` methods, to help with this.\n",
    "\n",
    "Pass any container or numpy array, and IPython will partition the object onto the engines wih `scatter`,\n",
    "or reconstruct the full object in the Client with `gather()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `range()` function creates an array of integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(16))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We will now `scatter` those numbers across our engines (processors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[range(0, 4), range(4, 8), range(8, 12), range(12, 16)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.scatter('a',range(16))\n",
    "dview['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can examine engine `0` and see what data is present there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 5, 6, 7]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(e1['a'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8, 9, 10, 11]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(e2['a'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let us now re-assemble the data on our client machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[range(0, 4), range(4, 8), range(8, 12), range(12, 16)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.gather('a')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What happened?\n",
    "\n",
    "We distributed our data across processors.  When we do this we are programming (interactively) in the MIMD paradigm.  \n",
    "\n",
    "We have to manage the data to/from the processors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Executing Instructions\n",
    "\n",
    "Now let us do something interesting with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[range(0, 4), range(4, 8), range(8, 12), range(12, 16)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.scatter('a',range(16))\n",
    "dview['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[6, 22, 38, 54]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview.scatter('a',range(16))\n",
    "dview.execute(\"asum = sum(a)\")\n",
    "dview.gather('asum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# With Great Power...\n",
    "\n",
    "We are responsible for managing the data to/from the processors.  Our summation happened on each processor - now it is up to us to continue the summation and get the result we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summed_parts = dview.gather('asum')\n",
    "sum(summed_parts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Multiplexing Exercise - Monte Carlo π\n",
    "\n",
    "Modified from https://github.com/minrk/IPython.parallel.tutorial\n",
    "\n",
    "-   **Min Ragan-Kelley**, *UC Berkeley Applied Science & Technology*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple toy problem to get a handle on multiple engines is a Monte\n",
    "Carlo approximation of π.\n",
    "\n",
    "Let's say we have a dartboard with a round target inscribed on a square\n",
    "board. If you threw darts randomly, and they land evenly distributed on\n",
    "the square board, how many darts would you expect to hit the target?\n",
    "\n",
    "<img src=\"./images/darts.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{A_c}{A_{sq}} = \\frac{\\pi r^2}{(2r)^2} = \\frac{\\pi}{4}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import random\n",
    "from math import pi\n",
    "\n",
    "def mcpi(nsamples):\n",
    "    s = 0\n",
    "    for i in range(nsamples):\n",
    "        x = random()\n",
    "        y = random()\n",
    "        if x*x + y*y <= 1:\n",
    "            s+=1\n",
    "    return 4.*s/nsamples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      10\n",
      "3.60000\n",
      "3.60000\n",
      "2.80000\n",
      "     100\n",
      "3.12000\n",
      "3.36000\n",
      "3.12000\n",
      "    1000\n",
      "3.13200\n",
      "3.14000\n",
      "3.12000\n",
      "   10000\n",
      "3.16560\n",
      "3.14200\n",
      "3.14720\n",
      "  100000\n",
      "3.13920\n",
      "3.14644\n",
      "3.13848\n",
      " 1000000\n",
      "3.14206\n",
      "3.14102\n",
      "3.14246\n"
     ]
    }
   ],
   "source": [
    "for n in [10, 100, 1000, 10000, 100000, 1000000]:\n",
    "    print(\"%8i\" % n)\n",
    "    for i in range(3):\n",
    "        print(\"%.5f\" % mcpi(n),)\n",
    "    #print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Parallel MC Estimation of $\\pi$\n",
    "\n",
    "We can now write a parallel version of our algorithm.  In this case we simply partition groups of the random numbers to different processors and re-assemble the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def mcpi(nsamples):\n",
    "    from random import random\n",
    "    s = 0\n",
    "    for i in range(nsamples):\n",
    "        x = random()\n",
    "        y = random()\n",
    "        if x*x + y*y <= 1:\n",
    "            s+=1\n",
    "    return 4.*s/nsamples\n",
    "    \n",
    "def multi_mcpi(dview, nsamples):\n",
    "    p = len(dview.targets)\n",
    "    if nsamples % p:\n",
    "        # ensure even divisibility\n",
    "        nsamples += p - (nsamples%p)\n",
    "    \n",
    "    subsamples = nsamples/p\n",
    "   \n",
    "    ar = dview.apply_async(mcpi, subsamples)\n",
    "    return ar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.29122138023376465\n"
     ]
    }
   ],
   "source": [
    "t0 = time.time()\n",
    "mcpi(1000000)\n",
    "t1 = time.time()\n",
    "\n",
    "print(t1-t0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0019919872283935547\n"
     ]
    }
   ],
   "source": [
    "t0 = time.time()\n",
    "ar = dview.apply_async(mcpi, 1000000)\n",
    "t1 = time.time()\n",
    "\n",
    "print(t1-t0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.ready()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3.142828, 3.142756, 3.140668, 3.141052]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar.get()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.141826"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(ar.get())/4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}