2-Chunking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Chunking in HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "> Objectives:\n",
    "> * Explain the concept of data chunking\n",
    "> * Show how to create and read datasets that are chunked\n",
    "> * Learn how to choose reasonable chunk sizes for your datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The HDF5 library supports several layouts so as to store datasets.\n",
    "\n",
    "* Continuous layout:\n",
    "  ![Continuous](img/dset_contiguous4x4.jpg)\n",
    "  More compact, and usually it can be read faster.  Typically used for small datasets (< 1 MB).\n",
    "  \n",
    "* Chunked layout:\n",
    "  ![Chunked](img/dset_chunked4x4.jpg)\n",
    "  Datasets can be enlarged and compressed.  Can be read fast using a fast decompressor. Typically used for large datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Creating chunked datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "data_dir = \"chunking\"\n",
    "if os.path.exists(data_dir):\n",
    "    shutil.rmtree(data_dir)\n",
    "os.mkdir(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def create_files(size, chunksize):\n",
    "    data = np.arange(size, dtype=np.int64)\n",
    "\n",
    "    # Contiguous array\n",
    "    with h5py.File(os.path.join(data_dir, \"continuous.h5\"), \"w\") as f:\n",
    "        f.create_dataset(data=data, name=\"data\", dtype=np.int64)\n",
    "\n",
    "    # Simple chunking\n",
    "    with h5py.File(os.path.join(data_dir, \"chunked.h5\"), \"w\") as f:\n",
    "        dset = f.create_dataset(\"data\", (size,), chunks=(chunksize,), dtype=np.int64)\n",
    "        dset[:] = data\n",
    "\n",
    "    # Automatic chunking and unlimited resizing\n",
    "    with h5py.File(os.path.join(data_dir, \"automatic.h5\"), \"w\") as f:\n",
    "        dset = f.create_dataset(\"data\", (0,), chunks=True, maxshape=(None,), dtype=np.int64)\n",
    "        dset.resize((size,))\n",
    "        dset[:] = data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "create_files(size=1000, chunksize=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"chunking/chunked.h5\" with sec2 driver.\r\n",
      "data                     Dataset {1000/1000}\r\n",
      "    Location:  1:800\r\n",
      "    Links:     1\r\n",
      "    Chunks:    {100} 800 bytes\r\n",
      "    Storage:   8000 logical bytes, 8000 allocated bytes, 100.00% utilization\r\n",
      "    Type:      native long\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -v {data_dir}/chunked.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 72\r\n",
      "-rw-r--r--  1 faltet  staff  11688 May 20 13:03 automatic.h5\r\n",
      "-rw-r--r--  1 faltet  staff  11496 May 20 13:03 chunked.h5\r\n",
      "-rw-r--r--  1 faltet  staff  10144 May 20 13:03 continuous.h5\r\n"
     ]
    }
   ],
   "source": [
    "%ls -l chunking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the example above, set the `chunksize` parameter to 99 and re-run it.  How the sizes of the different files changes?  Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 72\r\n",
      "-rw-r--r--  1 faltet  staff  11688 May 20 13:03 automatic.h5\r\n",
      "-rw-r--r--  1 faltet  staff  12208 May 20 13:03 chunked.h5\r\n",
      "-rw-r--r--  1 faltet  staff  10144 May 20 13:03 continuous.h5\r\n"
     ]
    }
   ],
   "source": [
    "create_files(size=1000, chunksize=99)\n",
    "%ls -l chunking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Opened \"chunking/chunked.h5\" with sec2 driver.\r\n",
      "data                     Dataset {1000/1000}\r\n",
      "    Location:  1:800\r\n",
      "    Links:     1\r\n",
      "    Chunks:    {99} 792 bytes\r\n",
      "    Storage:   8000 logical bytes, 8712 allocated bytes, 91.83% utilization\r\n",
      "    Type:      native long\r\n"
     ]
    }
   ],
   "source": [
    "!h5ls -v {data_dir}/chunked.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Reading chunked datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reading continuous.h5...\n",
      "The slowest run took 4.04 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "1000 loops, best of 3: 451 µs per loop\n",
      "reading chunked.h5...\n",
      "1000 loops, best of 3: 496 µs per loop\n",
      "reading automatic.h5...\n",
      "1000 loops, best of 3: 476 µs per loop\n"
     ]
    }
   ],
   "source": [
    "for h5file in (\"continuous.h5\", \"chunked.h5\", \"automatic.h5\"):\n",
    "    print(\"reading %s...\" % h5file)\n",
    "    %timeit h5py.File(os.path.join(data_dir, h5file))['data'][:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Exercise 2\n",
    "\n",
    "In the example above, set the `size` to 10 millions and choose a minimal `chunksize` that offers a reasonable filesize and read speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--  1 faltet  staff    77M May 20 13:03 chunking/automatic.h5\r\n",
      "-rw-r--r--  1 faltet  staff    76M May 20 13:03 chunking/chunked.h5\r\n",
      "-rw-r--r--  1 faltet  staff    76M May 20 13:03 chunking/continuous.h5\r\n"
     ]
    }
   ],
   "source": [
    "create_files(size=1000*1000*10, chunksize=100000)\n",
    "%ls -lh chunking/*.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reading continuous.h5...\n",
      "10 loops, best of 3: 37.2 ms per loop\n",
      "reading chunked.h5...\n",
      "10 loops, best of 3: 41.1 ms per loop\n",
      "reading automatic.h5...\n",
      "10 loops, best of 3: 124 ms per loop\n"
     ]
    }
   ],
   "source": [
    "for h5file in (\"continuous.h5\", \"chunked.h5\", \"automatic.h5\"):\n",
    "    print(\"reading %s...\" % h5file)\n",
    "    %timeit h5py.File(os.path.join(data_dir, h5file))['data'][:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Exercise 3\n",
    "\n",
    "Using the 10 million datasets above, retrieve just a small slice (say [10000:30000]) for each and time the time it takes to read.  Do you think that the whole dataset needs to be read in any case?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reading continuous.h5...\n",
      "1000 loops, best of 3: 482 µs per loop\n",
      "reading chunked.h5...\n",
      "1000 loops, best of 3: 656 µs per loop\n",
      "reading automatic.h5...\n",
      "1000 loops, best of 3: 590 µs per loop\n"
     ]
    }
   ],
   "source": [
    "for h5file in (\"continuous.h5\", \"chunked.h5\", \"automatic.h5\"):\n",
    "    print(\"reading %s...\" % h5file)\n",
    "    %timeit h5py.File(os.path.join(data_dir, h5file))['data'][10000:30000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}