0-overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Science and Machine Learning at Scale\n",
    "\n",
    "<img src=\"images/logo-dask-matrix.jpeg\"\n",
    "     width=\"50%\"\n",
    "     alt=\"Dask logo\">\n",
    "     \n",
    "## Materials & setup\n",
    "\n",
    "- Tutorial materials available at https://github.com/coiled/data-science-at-scale\n",
    "- Two ways to go through the tutorial:\n",
    "    1. Run locally on your laptop\n",
    "    2. Run using [Binder](https://mybinder.org/v2/gh/coiled/data-science-at-scale/master?urlpath=lab) (no setup required)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "Dask is a flexible, open source library for parallel computing in Python\n",
    "\n",
    "- Documentation: https://docs.dask.org\n",
    "- GitHub: https://github.com/dask/dask\n",
    "\n",
    "From a high-level Dask:\n",
    "\n",
    "- Enables parallel and larger-than-memory computations\n",
    "- Scales the existing Python ecosystem\n",
    "    - Uses familiar APIs you're used to from projects like NumPy, pandas, and scikit-learn\n",
    "    - Allows you to scale existing workflows with minimal code changes\n",
    "- Dask works on your laptop, but also scales out to large clusters\n",
    "- Offers great built-in diagnosic tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/dask-overview.png\"\n",
    "     width=\"75%\"\n",
    "     alt=\"Dask components\\\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dask Schedulers, Workers, and Beyond\n",
    "\n",
    "Work (Python code) is performed on a cluster, which consists of\n",
    "\n",
    "* a scheduler (which manages and sends the work / tasks to the workers)\n",
    "* workers, which compute the tasks.\n",
    "\n",
    "The client is \"the user-facing entry point for cluster users.\" What this means is that the client lives wherever you are writing your Python code and the client talks to the scheduler, passing it the tasks.\n",
    "\n",
    "<img src=\"images/dask-cluster.png\"\n",
    "     width=\"75%\"\n",
    "     alt=\"Dask components\\\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dask in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets up Dask's distributed scheduler\n",
    "from dask.distributed import Client\n",
    "\n",
    "client = Client()\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download data\n",
    "%run prep.py -d flights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Pandas-like operations\n",
    "import os\n",
    "import dask.dataframe as dd\n",
    "\n",
    "df = dd.read_csv(os.path.join(\"data\", \"nycflights\", \"*.csv\"),\n",
    "                 parse_dates={\"Date\": [0, 1, 2]},\n",
    "                 dtype={\"TailNum\": str,\n",
    "                        \"CRSElapsedTime\": float,\n",
    "                        \"Cancelled\": bool})\n",
    "\n",
    "df.groupby(\"Origin\").DepDelay.mean().compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial goals\n",
    "\n",
    "The goal for this tutorial is to cover the basics of Dask. Attendees should walk away with an understanding of what\n",
    "Dask offers, how it works, and ideas of how Dask can help them effectively scale their own data intensive workloads.\n",
    "\n",
    "The tutorial consists of several Jupyter notebooks which contain explanatory material on how Dask works. Specifically, the notebooks presented cover the following topics:\n",
    "\n",
    "- [Dask Delayed](1-delayed.ipynb)\n",
    "- [Dask DataFrame](2-dataframe.ipynb)\n",
    "- [Machine Learning](3-machine-learning.ipynb)\n",
    "\n",
    "Each notebook also contains hands-on exercises to illustrate the concepts being presented. Let's look at our first example to get a sense for how they work.\n",
    "\n",
    "### Exercise: Print `\"Hello world!\"`\n",
    "\n",
    "Use Python to print the string \"Hello world!\" to the screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to see a solution\n",
    "%load solutions/overview.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that several of the examples here have been adapted from the Dask tutorial at https://tutorial.dask.org."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Work directly from the cloud with Coiled \n",
    "\n",
    "<br>\n",
    "<img src=\"images/Coiled_Social-Templates_sand.png\" alt=\"Coiled logo\" width=50%/>\n",
    "<br>\n",
    "\n",
    "Here, I'll spin up a cluster on Coiled to show you just how easy it can be. Note that to do so, I've also signed into the [Coiled Cloud](https://cloud.coiled.io/), pip/conda installed `coiled`, and authenticated. You can do the same!\n",
    "\n",
    "(Note: Coiled will be already installed if you followed the instructions for local setup, but you still need to authenticate)\n",
    "\n",
    "You can also spin up [this hosted Coiled notebook](https://cloud.coiled.io/jobs/coiled/quickstart), which means you don't have to do anything locally.\n",
    "\n",
    "The plan:\n",
    "\n",
    "* use Coiled to load in **all** of the NYC taxi dataset from 10+ CSVs (8+ GBs) on an AWS cluster, \n",
    "* massage the data, \n",
    "* engineer a feature, and\n",
    "* compute the average tip as a function of the number of passengers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import coiled\n",
    "from dask.distributed import LocalCluster, Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = coiled.Cluster(n_workers=10)\n",
    "client = Client(cluster)\n",
    "print('Dashboard:', client.dashboard_link)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe as dd\n",
    "\n",
    "# Read data into a Dask DataFrame\n",
    "df = dd.read_csv(\n",
    "    \"s3://nyc-tlc/trip data/yellow_tripdata_2019-*.csv\", \n",
    "    parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"],\n",
    "    dtype={\n",
    "        'RatecodeID': 'float64',\n",
    "       'VendorID': 'float64',\n",
    "       'passenger_count': 'float64',\n",
    "       'payment_type': 'float64'\n",
    "    },\n",
    "    storage_options={\"anon\":True}\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Prepare to compute the average tip \n",
    "# as a function of the number of passengers\n",
    "mean_amount = df.groupby(\"passenger_count\").tip_amount.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Compute the average tip \n",
    "# as a function of the number of passengers\n",
    "mean_amount.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "We have:\n",
    "\n",
    "* used Coiled to load in **all** of the NYC taxi dataset from 10+ CSVs (10 GBs) on an AWS cluster,\n",
    "* computed the average tip as a function of the number of passengers, and \n",
    "* learned a bunch about using Dask on cloud-based clusters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}