09-Best-Practices-Wrapup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Best Practices and Wrapup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Dask docs collect a number of best practices:\n",
    "* Dataframe: https://docs.dask.org/en/latest/dataframe-best-practices.html\n",
    "* Array: https://docs.dask.org/en/latest/array-best-practices.html\n",
    "* Delayed: https://docs.dask.org/en/latest/delayed-best-practices.html \n",
    "* Overall: https://docs.dask.org/en/latest/best-practices.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Partitions/Chunks and Tasks\n",
    "\n",
    "Remember that Dask is a scheduler for regular Python functions operating on (and producing) regular Python objects.\n",
    "\n",
    "Your partitions, chunks, or data segments should be small enough to comfortably fit in RAM for each worker thread/core.\n",
    "\n",
    "That is...\n",
    "* if you have a 1GB worker with 1 core, want to keep your partitions below 1GB\n",
    "* with 2 x 1 GB workers with 1 cores, we still want partitions below 1GB\n",
    "* with n x 4 GB workers with 2 cores per worker, we want partitions below 2 GB\n",
    "\n",
    "It's also good to take into account that more memory may be used for operations than the data chunk size itself, and that it's helpful to have a few chunks of data available to keep Dask's worker cores busy. \n",
    "\n",
    "So we might want to take those numbers above and make them 2-4x smaller (or, equivalently, create 2-4x as many partitions).\n",
    "\n",
    "Generally speaking, a lot of tasks is not a bad thing. Scheduling overhead for each additional task is typically less than 1 millisecond, and can be a lot less.\n",
    "\n",
    "That said, if you have, say, a billion tasks, those milliseconds will add up to minutes. In that case you may want to simplify your task graph or use larger (and hence fewer) partitions/chunks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Caching (Persistence)\n",
    "\n",
    "The results of computations can be cached in the cluster memory, so that they are available for reuse, or for use to derive subsequent results.\n",
    "\n",
    "(See: `persist` which is available on `Client`, `Bag`, `Array`, `Dataframe`, etc.)\n",
    "\n",
    "Use caching wisely (not indiscriminately) and monitor memory usage using the `Workers` and `Memory` dashboard panes.\n",
    "\n",
    "### Data Formats and Compression\n",
    "\n",
    "Use compression schemes which are *splittable* and allow random access, so that processing your files in parallel is more flexible, e.g., Snappy, LZ4 instead of gzip.\n",
    "\n",
    "For datasets, consider files (and collections of files) in Parquet, ORC, HDF5, etc.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}