04-GUI-Solution-Revisited.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dask GUIs: Monitoring Workers, Tasks, and Memory\n",
    "\n",
    "Here we'll revisit that lab solution one more time.\n",
    "\n",
    "But this time, we'll focus less on getting the answers, and more on seeing what Dask is doing.\n",
    "\n",
    "Specifically, we'll look at some of the elements of the Dask Dashboard GUI.\n",
    "\n",
    "Once again, we'll start by creating that client with 2 workers, 1 thread, and 1GB of RAM each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import Client\n",
    "\n",
    "client = Client(n_workers=2, threads_per_worker=1, memory_limit='1GB')\n",
    "\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we have widgets integrated into Jupyter Lab, those are just for convenience. The \"real\" dashboard is at the URL above (though it may require some tweaking to work for Binder-hosted containers.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we __Read data__ you'll notice that nothing happens in the Dask GUI widgets, because these operations are just setting up a compute graph which will be executed later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe\n",
    "\n",
    "ddf = dask.dataframe.read_csv('data/pageviews_small.csv', sep=' ', blocksize=10e6)\n",
    "\n",
    "ddf.columns = ['project', 'page', 'requests', 'x']\n",
    "\n",
    "ddf2 = ddf.drop('x', axis=1)\n",
    "\n",
    "ddf3 = ddf2[ddf2.project == 'en']\n",
    "ddf3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we __Count__ (and `.compute()`) all the records, we'll see tasks get scheduled. __Before__ running this command, note memory, CPU, etc. in the GUI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf2.count().compute() #all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The GUI tells us quite a lot about what's happened. If you really want to see how the computation was decomposed by Dask, you can render a task graph before executing (although you won't normally need to do this):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf2.count().visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That explains a bit about the hover labels in, e.g., the Task Stream display. It's pretty much what we would expect for a lazy, data-parallel count, where the partitions are Pandas dataframes.\n",
    "\n",
    "How about the *English-only* count? That one is a bit more complicated because of the filtering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf3.count().visualize() #English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf3.count().compute() #English"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's look at Dask's Profile View\n",
    "\n",
    "Note: almost all of Dask's dashboard views update in realtime. The Profile View __does not__. Although Dask is collecting perf data behind the scenes, the profiler timeline doesn't update until you click the \"Update\" button. \n",
    "\n",
    "At that point you can select a time period from the refreshed timeline, and Dask will render a flame graph from that selected period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf2.set_index('project')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab Exercise\n",
    "\n",
    "With the Wikimedia data, find the total number of pageviews for each project, and create a report with the top 20. \n",
    "\n",
    "Hints:\n",
    "* Use groupby / sum to aggregate\n",
    "* Use nlargest to report the top 20\n",
    "\n",
    "The goal is to explore some of these performance tools (GUI widgets and graph visualization), moreso than creating code.\n",
    "\n",
    "<!--\n",
    "bigger hint: ddf2.groupby('project').sum().nlargest(20, 'requests').compute()\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}