03-big-data-visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "72daec7f-addf-4640-a7f9-6f3f4d90db75",
   "metadata": {},
   "source": [
    "<img src=\"https://hvplot.holoviz.org/_static/logo_horizontal.svg\" width=\"25%\" align=\"right\"/>\n",
    "\n",
    "# Big data visualization with Dask and hvPlot\n",
    "\n",
    "In this notebook, we'll continue to explore the dataset, but with visuals! We will learn to use `hvplot` with Dask to create some quick interactive visualizations.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "724f8bcc-770b-483e-be86-e08321aa4eeb",
   "metadata": {},
   "source": [
    "## What is hvPlot?\n",
    "\n",
    "hvPlot a familiar and high-level API for data exploration and visualization. \n",
    "\n",
    "<img src=\"https://hvplot.holoviz.org/assets/diagram.svg\" width=\"70%\"/>\n",
    "\n",
    " \n",
    "It is a powerful and interactive version of the pandas' `.plot()` API.\n",
    "**By replacing .plot() with .hvplot() you get an interactive figure.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c466ac2c-9c8c-4ee0-a6d1-f0e4bc0cf9e8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Ignore some DeprecationWarnings\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e329369-2c9f-4ab2-afb1-ee3677bb4dbc",
   "metadata": {},
   "source": [
    "## Reconnect to our Dask Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f3eae75-0829-4bca-bebc-6d49c8ba0050",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dask_gateway\n",
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9265dde-8387-4c27-8702-1cb9015045c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gateway = dask_gateway.Gateway()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f07602-4398-40e1-a7e0-c19bc5ff96eb",
   "metadata": {},
   "source": [
    "You can connect to a running cluster (that we created in the previous notebook), and note that you may need to refresh your dashboard page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "168b1ded-076a-4894-a3ad-64dceefbe3c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(running_clusters := gateway.list_clusters())>0:\n",
    "    cluster = gateway.connect(running_clusters[0].name)\n",
    "else:\n",
    "    cluster = gateway.new_cluster(conda_environment=\"global/global-pydata-global-2023\", profile=\"Medium Worker\")\n",
    "    cluster.adapt(5,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f895739f-ff0f-41e8-a933-6057b138b212",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993ca5bf-59d8-472d-aac3-85ea367e175c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client = cluster.get_client()\n",
    "\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4d8612-d103-406f-bb21-5774e9af4129",
   "metadata": {},
   "source": [
    "## Load a subset of flights data\n",
    "\n",
    "We can do all of the following computations and visualizations on the full dataset with the power of Dask and hvplot. \n",
    "However, in order to do so, we'd need a larger compute pool and there are quite a few of you. So we'll grab a subset for\n",
    "demonstration purposes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f27e8e6-0066-46c3-819c-209d22cb065f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "columns = [\n",
    "    'MONTH', 'DAY_OF_MONTH', 'DAY_OF_WEEK', 'FL_DATE', 'OP_CARRIER', \n",
    "    'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'CRS_DEP_TIME', \n",
    "    'DEP_TIME', 'DEP_DELAY', 'ARR_TIME', 'ARR_DELAY', 'CANCELLED', \n",
    "    'CANCELLATION_CODE', 'DIVERTED', 'AIR_TIME', 'FLIGHTS', 'DISTANCE',\n",
    "    'CARRIER_DELAY', 'WEATHER_DELAY', 'NAS_DELAY', 'SECURITY_DELAY', \n",
    "    'LATE_AIRCRAFT_DELAY', 'DIV_ARR_DELAY',\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32157df-66ed-4dac-b8bb-705f95b9d944",
   "metadata": {},
   "source": [
    "Even with 10 compute notes, the visualization in this notebook will take some time to execute if we use the full dataset.\n",
    "\n",
    "We'll look at the data since 2020 and limit the number of carriers for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ece664-9d56-4fb1-8607-bcf9739026f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "    f\"gcs://quansight-datasets/airline-ontime-performance/sorted/full_dataset.parquet\", \n",
    "    columns=columns,\n",
    "    filters=[('YEAR', '>', 2020)],\n",
    ")\n",
    "flights_subset = flights[flights.OP_CARRIER.isin(['AA', 'UA', 'WN', 'DL'])]\n",
    "\n",
    "flights_subset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd2e0df0-db55-4273-9d38-78efbfef14a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Our subset dataset has {len(flights_subset)/1e6:2} million rows!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eae9fb4-3a10-40ff-8395-c1e485a37d6d",
   "metadata": {
    "tags": []
   },
   "source": [
    "Persist the data on the cluster so we don't need to reread it with every computation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8d6226f-af2e-4269-83fa-add27b8c9e22",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de8c1c82-53e5-4163-87ba-db873a8cbc70",
   "metadata": {},
   "source": [
    "## hvPlot + Dask\n",
    "\n",
    "To use hvPlot's build in Dask integration, we need to import the Dask accessor for hvplot:\n",
    "\n",
    "`import hvplot.dask` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657583d5-91f2-4926-9bea-6d40dda3c45a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import hvplot.dask\n",
    "hvplot.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5a95e4-5af7-4dc5-918f-ec74e63f4b38",
   "metadata": {},
   "source": [
    "### Plot the departure delay per day for the entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37290761-6df5-485b-81a2-8012d3a91a0d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('FL_DATE')['DEP_DELAY'].count().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64f15fc-d30c-45b3-a1ba-e1caa8b3ab17",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Visualize the weekly distribution of the mean of any variable in the datasets\n",
    "\n",
    "You can any plot type from the [hvPlot Gallery](https://hvplot.holoviz.org/reference/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a0542e3-5571-4221-b183-b21c79f186ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots below for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "232e750f-7270-47d4-9453-e1cddd5763d3",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('DAY_OF_WEEK')['ARR_DELAY'].mean().hvplot.scatter(x=\"DAY_OF_WEEK\", y='ARR_DELAY')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae45126f-b7a6-4540-a864-b0ee521c91ae",
   "metadata": {
    "tags": []
   },
   "source": [
    "## More interactivity with quick widgets\n",
    "\n",
    "Zoom, pan, and hover are just the tip of the iceberg for interactivity, widgets open up a whole new world of interaction. Some examples of widgets are dropdown selectors, range/date/color selectors, radio buttons, text fields, etc.\n",
    "\n",
    "hvPlot automatically includes the best widgets for your visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7405fd4a-df16-4263-a834-73c4c00d939a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.hvplot.hist('DEP_DELAY', groupby='OP_CARRIER', bins=20, bin_range=(-20, 100), width=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e497dc-4918-46e8-9e49-228d8a2227ea",
   "metadata": {},
   "source": [
    "## Compose and overlay plots \n",
    "\n",
    "With hvPlot, you can compose and overlay your plots easily with the `+` or `*` operations, respectively.\n",
    "\n",
    "Let's plot the minimum, maximum, and mean departure delays for each carrier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6835c344-5f7f-4cbc-988c-7353b79b1b73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e9982e3-c7d5-429e-b377-1a34d708ac82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Caution: reset_index() and sort_values() are compute intensive but we do this because Dask Dataframe API doesn't yet understand multi-indices\n",
    "\n",
    "delays = flights_subset.groupby(['DAY_OF_WEEK', 'OP_CARRIER'])['DEP_DELAY'].agg([np.min, np.mean, np.max]).reset_index().sort_values(by='DAY_OF_WEEK')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab87ab68-8070-4928-8aff-935325ca4115",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "delays = delays.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d77b6fd1-350f-47eb-9f9f-e209ce54217f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot = delays.hvplot.area(x='DAY_OF_WEEK', y='min', y2='max', alpha=0.2, groupby=\"OP_CARRIER\", stacked=False, legend=\"bottom_right\", hover=False)\n",
    "min_max_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a241f52c-1b1a-45af-a0c1-b6acdee5a31f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mean_plot = delays.hvplot.scatter(x='DAY_OF_WEEK', y=\"mean\", groupby=\"OP_CARRIER\", color='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62828b6f-d59c-486d-a37c-1765043c4a2a",
   "metadata": {},
   "source": [
    "The + operation creates a layout, displaying the plots side-by-side:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9181d6a-56c0-466b-8f73-65bcf2e25f26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot + mean_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3fa6bf-a999-4914-b7be-a6fbfb47bf53",
   "metadata": {},
   "source": [
    "The * operation overlays one plot on top of the other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b491df-de54-419c-af48-ad50e8b21d42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "min_max_plot * mean_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa954704-bbcc-48e0-87fe-5dc484d6faea",
   "metadata": {},
   "source": [
    "## Explorer\n",
    "\n",
    "For creating all of our previous plots, we needed some preliminary knowledge of the dataset.\n",
    "\n",
    "What if you want to explore a dataset visually from scratch? hvPlot's data explorer can help you explore and create interactive visualizations using a graphical UI.\n",
    "\n",
    "Note: We're using pandas DataFrame here to demonstrate the Explorer, because it's the most useful & performant with a small subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93dd59d1-0d22-4204-86aa-9b0526543d67",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset_pandas = flights_subset.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed02cf8d-070f-451b-87ec-90c5040930e4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explorer = hvplot.explorer(flights_subset_pandas)\n",
    "explorer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b0de41d-d6d2-4a8d-a15e-a2de62f4b71a",
   "metadata": {},
   "source": [
    "You can use the above GUI to create a plot you want!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9c083c-846b-49f5-a0ef-7777bbc51ffd",
   "metadata": {},
   "source": [
    "### Save your plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76f0d4b-3bd3-4ca0-b6c1-fd0c86f48fb7",
   "metadata": {},
   "source": [
    "You can then save the selected visualization using `save()`, or generate the code to create the specific viz using `plot_code`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b5014f-5c4f-497e-809f-3456071b6148",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "explorer.save(filename='hvplot_explorer.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9df050f5-8d7d-459d-968f-ecc6a07db65e",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Use the explorer to plot the flights cancellations per day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d6e567-15a5-4dc9-b08a-84326f48484f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots for the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffc70dff-d9cc-417c-b2ad-f363d4d613a2",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights_subset.groupby('FL_DATE')['CANCELLED'].count().hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc002c6-be3d-4334-a488-6be260f144d3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-08T05:23:34.526542Z",
     "iopub.status.busy": "2023-04-08T05:23:34.525475Z",
     "iopub.status.idle": "2023-04-08T05:23:34.530800Z",
     "shell.execute_reply": "2023-04-08T05:23:34.529489Z",
     "shell.execute_reply.started": "2023-04-08T05:23:34.526507Z"
    },
    "tags": []
   },
   "source": [
    "## Geographic plots\n",
    "\n",
    "To plot data on geographic maps, we need the latitude and longitude values. `ip2location` has created a list of lat/lon values for US airports here: https://github.com/ip2location/ip2location-iata-icao\n",
    "\n",
    "We'll use this information to plot the departure delays on a world map!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ba547f9-ea4b-4165-b207-ecd420f41b0e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports = dd.read_csv('https://raw.githubusercontent.com/ip2location/ip2location-iata-icao/master/iata-icao.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "443d8e40-f4dc-477b-bf25-f073cff6b538",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports = airports.set_index('iata')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a774d66-0418-434d-87a4-4a15e66c5a6c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33128334-e926-4885-9885-2e283528631c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays = flights.groupby('ORIGIN')['DEP_DELAY'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8149bdbc-7d5f-47e1-9890-f58e675f9868",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays = dd.merge(airports, airport_delays, left_index=True, right_index=True).persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dff55e2-15b7-49a4-9f06-5e47c243f8bb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fae8feb-cf6e-4b9d-ad19-8b65f7211f3c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airport_delays.hvplot.points('longitude', 'latitude', geo=True, c='DEP_DELAY', alpha=1, xlim=(-180, -30), ylim=(0, 72), tiles='ESRI')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f7d953b-8bfa-4b04-9f3e-e25a07c60250",
   "metadata": {},
   "source": [
    "## Plotting large datasets\n",
    "\n",
    "In the above visualization of daily counts we saw a bunch of compute happening before we saw the plot appear. But after it was generated, panning and zooming did not cause any new Dask computes.\n",
    "\n",
    "This is because the final dataset after the groupby is only about `20 years * 365 days` long, so it fits completely in memory.\n",
    "\n",
    "Now let's look at the entire dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "584117ea-d8ae-4a6f-addf-5729437a1521",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "    f\"gcs://quansight-datasets/airline-ontime-performance/sorted/full_dataset.parquet\", \n",
    "    columns=columns,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c312a4-34f7-4591-9cfd-955d4c754732",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Reminder, the full dataset has {len(flights)/1e6:2} million rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98673fe0-af60-4d1e-92fa-4d0a47017dfe",
   "metadata": {},
   "source": [
    "If we try and send these many data points to the browser for visualization in a plot, the *browser* would run out of memory and crash.\n",
    "\n",
    "<a href=\"https://datashader.org/\"><img src=\"images/datashader.svg\" width=\"30%\" align=\"right\"></a>\n",
    "\n",
    "The solution for this is to take advantage of the fact that the output plot has a fixed resolution in terms of number of pixels. A 600x400 image has 240,000 pixels. This means that if we plotted 125 million points on the these pixels, most would overlay each other and not be visible. Instead, we pre-render or rasterize the data and shade in a manner that maintains an accurate the distribution of your data. \n",
    "\n",
    "We do this via the hvPlot integration with [**Datashader**](https://datashader.org/).\n",
    "\n",
    "We will use a smaller dataset for the next few examples for quick outputs. These examples will work with the full dataset, but will take a bit longer to run with the 10 compute nodes we are currently using for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06227e51-764e-4071-9348-817e86b80466",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "        f\"gcs://quansight-datasets/airline-ontime-performance/sorted/parquet_by_year\", \n",
    "        filters=[('YEAR', '>', 2017)],\n",
    "        columns=columns,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ffdc289-c85f-48f7-9e3b-1979b08c6b6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"The smaller dataset has {len(flights)/1e6} million rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7474b7ff-eda1-4305-a368-13c76e133539",
   "metadata": {
    "tags": []
   },
   "source": [
    "In these next two visualizations, Datashader data is displayed on the plots. \n",
    "As we pan and zoom, Datashader recomputes the appropriate pixel shades using Dask.\n",
    "\n",
    "This allows us to easily look at the entire 30 million row dataset, but still\n",
    "zoom into a single point, without requiring downsampling or decimation of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb333fe-8ffb-439e-ba77-52d1a8312a6d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights.hvplot.line(x='FL_DATE', y='DEP_DELAY', datashade=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c2d360-a548-44f0-87e5-488b59d7996e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights[['ARR_DELAY', 'DISTANCE']].hvplot.scatter(x='ARR_DELAY', y='DISTANCE', datashade=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3238086d-31dd-4d5b-8db2-e43905f4c2b6",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Datashade options on hvplot\n",
    "\n",
    "```\n",
    "aggregator (default=None):\n",
    "    Aggregator to use when applying rasterize or datashade operation\n",
    "    (valid options include 'mean', 'count', 'min', 'max' and more, and\n",
    "    datashader reduction objects)\n",
    "dynamic (default=True):\n",
    "    Whether to return a dynamic plot which sends updates on widget and\n",
    "    zoom/pan events or whether all the data should be embedded\n",
    "    (warning: for large groupby operations embedded data can become\n",
    "    very large if dynamic=False)\n",
    "datashade (default=False):\n",
    "    Whether to apply rasterization and shading using datashader\n",
    "    library returning an RGB object\n",
    "dynspread (default=False):\n",
    "    Allows plots generated with datashade=True to increase the point\n",
    "    size to make sparse regions more visible\n",
    "rasterize (default=False):\n",
    "    Whether to apply rasterization using the datashader library\n",
    "    returning an aggregated Image\n",
    "x_sampling/y_sampling (default=None):\n",
    "    Specifies the smallest allowed sampling interval along the x/y axis.\n",
    "    \n",
    "```\n",
    "\n",
    "Note: `rasterize()` returns an `hv.Image`, while `datashade()` return an `hv.RGB`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25399415-fc40-42f8-8220-f8588c368719",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights[['ARR_DELAY', 'DISTANCE']].hvplot.scatter(x='ARR_DELAY', y='DISTANCE', datashade=True, aggregator='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f25b9ca0-86bf-442c-8672-4add978ec001",
   "metadata": {},
   "source": [
    "## Shutdown the cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abd28a3-003e-4512-9b81-b577d824f945",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12a5a268-68ca-4714-81f9-f4c274d1f6be",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Next →\n",
    "\n",
    "[Conclusion](./04-conclusion.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "global-global-pydata-global-2023",
   "language": "python",
   "name": "conda-env-global-global-pydata-global-2023-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}