lab0_student.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#![Spark Logo](http://spark-mooc.github.io/web-assets/images/ta_Spark-logo-small.png) + ![Python Logo](http://spark-mooc.github.io/web-assets/images/python-logo-master-v3-TM-flattened_small.png)\n",
    "# **First Notebook: Virtual machine test and assignment submission**\n",
    "#### This notebook will test that the virtual machine (VM) is functioning properly and will show you how to submit an assignment to the autograder.  To move through the notebook just run each of the cells.  You will not need to solve any problems to complete this lab.  You can run a cell by pressing \"shift-enter\", which will compute the current cell and advance to the next cell, or by clicking in a cell and pressing \"control-enter\", which will compute the current cell and remain in that cell.  At the end of the notebook you will export / download the notebook and submit it to the autograder.\n",
    "#### ** This notebook covers: **\n",
    "#### *Part 1:* Test Spark functionality\n",
    "#### *Part 2:* Check class testing library\n",
    "#### *Part 3:* Check plotting\n",
    "#### *Part 4:* Check MathJax formulas\n",
    "#### *Part 5:* Export / download and submit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 1: Test Spark functionality **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1a) Parallelize, filter, and reduce **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check that Spark is working\n",
    "largeRange = sc.parallelize(xrange(100000))\n",
    "reduceTest = largeRange.reduce(lambda a, b: a + b)\n",
    "filterReduceTest = largeRange.filter(lambda x: x % 7 == 0).sum()\n",
    "\n",
    "print reduceTest\n",
    "print filterReduceTest\n",
    "\n",
    "# If the Spark jobs don't work properly these will raise an AssertionError\n",
    "assert reduceTest == 4999950000\n",
    "assert filterReduceTest == 714264285"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (1b) Loading a text file **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check loading data with sc.textFile\n",
    "import os.path\n",
    "baseDir = os.path.join('data')\n",
    "inputPath = os.path.join('cs100', 'lab1', 'shakespeare.txt')\n",
    "fileName = os.path.join(baseDir, inputPath)\n",
    "\n",
    "rawData = sc.textFile(fileName)\n",
    "shakespeareCount = rawData.count()\n",
    "\n",
    "print shakespeareCount\n",
    "\n",
    "# If the text file didn't load properly an AssertionError will be raised\n",
    "assert shakespeareCount == 122395"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 2: Check class testing library **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (2a) Compare with hash **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# TEST Compare with hash (2a)\n",
    "# Check our testing library/package\n",
    "# This should print '1 test passed.' on two lines\n",
    "from test_helper import Test\n",
    "\n",
    "twelve = 12\n",
    "Test.assertEquals(twelve, 12, 'twelve should equal 12')\n",
    "Test.assertEqualsHashed(twelve, '7b52009b64fd0a2a49e6d8a939753077792b0554',\n",
    "                        'twelve, once hashed, should equal the hashed value of 12')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (2b) Compare lists **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# TEST Compare lists (2b)\n",
    "# This should print '1 test passed.'\n",
    "unsortedList = [(5, 'b'), (5, 'a'), (4, 'c'), (3, 'a')]\n",
    "Test.assertEquals(sorted(unsortedList), [(3, 'a'), (4, 'c'), (5, 'a'), (5, 'b')],\n",
    "                  'unsortedList does not sort properly')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 3: Check plotting **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (3a) Our first plot **\n",
    "#### After executing the code cell below, you should see a plot with 50 blue circles.  The circles should start at the bottom left and end at the top right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check matplotlib plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "from math import log\n",
    "\n",
    "# function for generating plot layout\n",
    "def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999', gridWidth=1.0):\n",
    "    plt.close()\n",
    "    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')\n",
    "    ax.axes.tick_params(labelcolor='#999999', labelsize='10')\n",
    "    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:\n",
    "        axis.set_ticks_position('none')\n",
    "        axis.set_ticks(ticks)\n",
    "        axis.label.set_color('#999999')\n",
    "        if hideLabels: axis.set_ticklabels([])\n",
    "    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')\n",
    "    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])\n",
    "    return fig, ax\n",
    "\n",
    "# generate layout and plot data\n",
    "x = range(1, 50)\n",
    "y = [log(x1 ** 2) for x1 in x]\n",
    "fig, ax = preparePlot(range(5, 60, 10), range(0, 12, 1))\n",
    "plt.scatter(x, y, s=14**2, c='#d6ebf2', edgecolors='#8cbfd0', alpha=0.75)\n",
    "ax.set_xlabel(r'$range(1, 50)$'), ax.set_ylabel(r'$\\log_e(x^2)$')\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 4: Check MathJax Formulas **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (4a) Gradient descent formula **\n",
    "#### You should see a formula on the line below this one: $$ \\scriptsize \\mathbf{w}_{i+1} = \\mathbf{w}_i - \\alpha_i \\sum_j (\\mathbf{w}_i^\\top\\mathbf{x}_j  - y_j) \\mathbf{x}_j \\,.$$\n",
    " \n",
    "#### This formula is included inline with the text and is $ \\scriptsize (\\mathbf{w}^\\top \\mathbf{x} - y) \\mathbf{x} $."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (4b) Log loss formula **\n",
    "#### This formula shows log loss for single point. Log loss is defined as: $$  \\begin{align} \\scriptsize \\ell_{log}(p, y) = \\begin{cases} -\\log (p) & \\text{if } y = 1 \\\\\\ -\\log(1-p) & \\text{if } y = 0 \\end{cases} \\end{align} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ** Part 5: Export / download and submit **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ** (5a) Time to submit **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You have completed the lab.  To submit the lab for grading you will need to download it from your IPython Notebook environment.  You can do this by clicking on \"File\", then hovering your mouse over \"Download as\", and then clicking on \"Python (.py)\".  This will export your IPython Notebook as a .py file to your computer.\n",
    "#### To upload this file to the course autograder, go to the edX website and find the page for submitting this assignment.  Click \"Choose file\", then navigate to and click on the downloaded .py file.  Now click the \"Open\" button and then the \"Check\" button.  Your submission will be graded shortly and will be available on the page where you submitted.  Note that when submission volumes are high, it may take as long as an hour to receive results."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 0
}