0-overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/dask_horizontal.svg\"\n",
    "     width=\"45%\"\n",
    "     alt=\"Dask logo\\\">\n",
    "     \n",
    "# Scaling your data work with Dask\n",
    "\n",
    "## PyData Global 2020\n",
    "\n",
    "### Materials & setup\n",
    "\n",
    "- Tutorial materials available at https://github.com/coiled/pydata-global-dask\n",
    "- Two ways to go through the tutorial:\n",
    "    1. Run locally on your laptop\n",
    "    2. Run using Binder (no setup required)\n",
    "\n",
    "### About the speakers\n",
    "\n",
    "- **[James Bourbeau](https://www.jamesbourbeau.com/)**: Dask maintainer and Software Engineer at [Coiled](https://coiled.io/).\n",
    "- **[Hugo Bowne-Anderson](http://hugobowne.github.io/)**: Head of Data Science Evangelism and Marketing at [Coiled](https://coiled.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Overview\n",
    "\n",
    "Dask is a flexible, open source library for parallel computing in Python\n",
    "\n",
    "- Documentation: https://docs.dask.org\n",
    "- GitHub: https://github.com/dask/dask\n",
    "\n",
    "From a high-level Dask:\n",
    "\n",
    "- Enables parallel and larger-than-memory computations\n",
    "- Scales the existing Python ecosystem\n",
    "    - Uses familiar APIs you're used to from projects like NumPy, Pandas, and scikit-learn\n",
    "    - Allows you to scale existing workflows with minimal code changes\n",
    "- Dask works on your laptop, but also scales out to large clusters\n",
    "- Offers great built-in diagnosic tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/dask-components.svg\"\n",
    "     width=\"85%\"\n",
    "     alt=\"Dask components\\\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dask in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets up Dask's distributed scheduler\n",
    "from dask.distributed import Client\n",
    "\n",
    "client = Client()\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to download NYC flight dataset\n",
    "%run prep.py -d flights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Pandas-like operations\n",
    "import os\n",
    "import dask.dataframe as dd\n",
    "\n",
    "df = dd.read_csv(os.path.join(\"data\", \"nycflights\", \"*.csv\"),\n",
    "                 parse_dates={\"Date\": [0, 1, 2]},\n",
    "                 dtype={\"TailNum\": str,\n",
    "                        \"CRSElapsedTime\": float,\n",
    "                        \"Cancelled\": bool})\n",
    "\n",
    "df.groupby(\"Origin\").DepDelay.mean().compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial goals\n",
    "\n",
    "The goal for this tutorial is to cover the basics of Dask. Attendees should walk away with an understanding of what\n",
    "Dask offers, how it works, and ideas of how Dask can help them effectively scale their own data intensive workloads.\n",
    "\n",
    "The tutorial consists of several Jupyter notebooks which contain explanatory material on how Dask works. Specifically, the notebooks presented cover the following topics:\n",
    "\n",
    "- [Dask Delayed](1-delayed.ipynb)\n",
    "- [Dask DataFrame](2-dataframe.ipynb)\n",
    "- [Schedulers](3-schedulers.ipynb)\n",
    "\n",
    "Each notebook also contains hands-on exercises to illustrate the concepts being presented. Let's look at our first example to get a sense for how they work.\n",
    "\n",
    "### Exercise: Print `\"Hello world!\"`\n",
    "\n",
    "Use Python to print the string \"Hello world!\" to the screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to see a solution\n",
    "%load solutions/overview.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that several of the examples here have been adapted from the Dask tutorial at https://tutorial.dask.org."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next step\n",
    "\n",
    "Let's start by covering our first Dask collection, the `dask.delayed` interface, in the [Dask delayed notebook](1-delayed.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}