index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "615e53a0",
   "metadata": {},
   "source": [
    "# StaticFrame from the Ground Up: Getting Started with Immutable DataFrames\n",
    "### Christopher Ariza"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306c07db",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Back in 2017 I found myself frequently asking: \"is Pandas a suitable foundation for production library code?\" While Pandas is well-known for its utility in data science, I consistently found its flexibility and implicit behaviors a detriment in building library code for production systems.\n",
    "\n",
    "This led me to create StaticFrame, an alternative dataframe library built on an immutable data model. After years of development and use, I am confident that StaticFrame reduces opportunities for error and leads to more maintainable code. While not yet always more efficient than Pandas, in many areas StaticFrame offers very significant improvements in run-time and memory usage. Beyond common functionality, StaticFrame offers a more explicit and consistent API, novel multi-Frame containers and processors, and support for high-performance serialization through the NPZ format.\n",
    "\n",
    "This notebook is designed as to provide a rapid, breadth-first survey of StaticFrame. How StaticFrame relates to Pandas is highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5a8754",
   "metadata": {},
   "source": [
    "# What is a DataFrame?\n",
    "* A 2D table with labelled axis (rows, columns)\n",
    "    * Labels stay with data after selection\n",
    "    * Operations align on labels\n",
    "    * Can reindex axis based on labels\n",
    "* Distinct from a simple 2D array\n",
    "    * Labels can be any (hashable) type\n",
    "    * Types can be hetergenous by column\n",
    "* Just like a 2D array, a dataframe supports binary operators and broadcasting\n",
    "    * Can multiply a dataframe by a constant, 1D, or 2D container\n",
    "    * All operations align on labels, not order\n",
    "* A high-level language (Python) can be used to implement dataframe functionality over a low-level, high-performance array library (NumPy)\n",
    "    * A dataframe manages underlying arrays segmented by dtype\n",
    "    * Index objects assigned to axis translate labels to array positions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfda586",
   "metadata": {},
   "source": [
    "# A Brief History of DataFrames\n",
    "\n",
    "* 1991: earliest implementation of a dataframe in the S language\n",
    "* 2009: Pandas 0.1 released\n",
    "* 2018: StaticFrame 0.1 released\n",
    "* 2022: StaticFrame 1.0 released\n",
    "* There are presently a number of dataframe libraries in Python and other languages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aba08641",
   "metadata": {},
   "source": [
    "# Why Not Just Use Pandas?\n",
    "\n",
    "* Pandas supports in-place mutation\n",
    "* Pandas API has many inconsistencies and non-orthogonal parameters\n",
    "* Pandas only optionally supports unique indices (`verify_integrity` defaults to `False`)\n",
    "* Pandas does not support all NumPy types (Unicode, `datetime64`)\n",
    "* Pandas removed multi-frame containers (i.e., the `pd.Panel`)\n",
    "\n",
    "* See also: https://dev.to/flexatone/ten-reasons-to-use-staticframe-instead-of-pandas-4aad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc7e0ffa",
   "metadata": {},
   "source": [
    "# Learning StaticFrame from Pandas\n",
    "\n",
    "* Nearly everything you can do with Pandas you can do with StaticFrame\n",
    "* There are few things Pandas does that StaticFrame does not do\n",
    "    * No internal graphing / plotting support\n",
    "    * Few internal implementations of calculations available elsewhere (NumPy, SciPy)\n",
    "* Much of what you already know from Pandas will directly translate\n",
    "    * Many interfaces and methods are identical\n",
    "    * StaticFrame has more numerous, more narrow interfaces with keyword only arguments\n",
    "    * StaticFrame follows hierarchical naming\n",
    "* You can go back and forth\n",
    "    * `sf.Frame.to_pandas()`\n",
    "    * `sf.Frame.from_pandas()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2506be0b",
   "metadata": {},
   "source": [
    "# Learning StaticFrame from Examples\n",
    "* Examples used here are intentionally compact\n",
    "* Examples mostly on `sf.Frame`: interfaces on `sf.Series` are often identical\n",
    "* Browse and search the API with examples: https://www.staticframe.dev"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e7a879d",
   "metadata": {},
   "source": [
    "# StaticFrame Development\n",
    "\n",
    "* Development\n",
    "    * Code contributions from a small pool of developers\n",
    "    * Feature and design contributions from multiple internal teams\n",
    "    * New contributors are welcome!\n",
    "* Quality & Test\n",
    "    * 100% test coverage\n",
    "    * Robust CI/CD with MyPy, Pylint, and multiplatform test\n",
    "* Documentation\n",
    "    * Fully code-generated API documentation (https://static-frame.readthedocs.io)\n",
    "    * Thousands of examples for most of the API\n",
    "    * Every object exposes API via `interface` attribute\n",
    "* Very Few Core Dependencies\n",
    "    * NumPy\n",
    "    * Team-maintained CPython extension libraries: `automap`, `arraykit`\n",
    "    * Additional, optional dependencies provide support for different serialization formats\n",
    "* Releases\n",
    "    * Stable API\n",
    "    * Semantic versioning used since 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8576f3d2",
   "metadata": {},
   "source": [
    "# Installing & Importing\n",
    "\n",
    "* Available via pip, conda-forge\n",
    "* `import static_frame as sf`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2ff0937c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import static_frame as sf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3e4956",
   "metadata": {},
   "source": [
    "# The `sf.Frame` & the `sf.Series`\n",
    "* A `sf.Series` is a 1D array (of a single dtype) with labels \n",
    "* A `sf.Frame` is a 2D container (of one or more columnar dtypes) with row and column labels\n",
    "* When extracting a row or column from a `sf.Frame`, we get a `sf.Series`.\n",
    "* Support for higher-dimensional data\n",
    "    * Use hierarchical indices on a 2D container (i.e., the `sf.IndexHierarchy`)\n",
    "    * Use multi-`sf.Frame` containers (i.e., the `sf.Bus`)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449c5ed7",
   "metadata": {},
   "source": [
    "# Anatomy of a Frame\n",
    "\n",
    "* A `sf.Frame` wraps 1D and 2D NumPy arrays\n",
    "* NumPy dtype are partitioned by column\n",
    "* Each axis is labelled with an `sf.Index` (or subclass)\n",
    "    * Row labels available via `sf.Frame.index`\n",
    "    * Column labels available via `sf.Frame.columns`\n",
    "    * An `sf.IndexAutoFactory` can be used to create integer labels\n",
    "* Hashable metadata via `name` attributes on all containers\n",
    "    * `sf.Frame.name` (StaticFrame only)\n",
    "    * `sf.Frame.index.name`\n",
    "    * `sf.Frame.columns.name`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2194971",
   "metadata": {},
   "source": [
    "# Getting Data In & Out: Constructors & Exporters\n",
    "\n",
    "* Constructors always live on containers (i.e., `sf.Frame`)\n",
    "    * `pd.read_csv()`, `pd.DataFrame.from_records()`\n",
    "    * `sf.Frame.from_csv()`, `sf.Frame.from_records()`\n",
    "* Explicit constructors with narrow functionality\n",
    "    * `pd.DataFrame()` supports a single element, or a column of elements\n",
    "    * `sf.Frame.from_element()`, `sf.Frame.from_elements()`\n",
    "* Support for common serialization formats\n",
    "    * `pd.read_excel()`, `pd.read_csv()`, `pd.read_parquet()`\n",
    "    * `sf.Frame.from_xlsx()`, `sf.Frame.from_csv()`, `sf.Frame.from_parquet()`\n",
    "* Serialization methods exclusive to StaticFrame\n",
    "    * NPZ and NPY formats faster than Parquet with comparable file sizes\n",
    "    * Encodes all `sf.Frame` characteristics\n",
    "    * NPY supports memory mapping out-of-core data\n",
    "    * `sf.Frame.to_npz()`, `sf.Frame.from_npz()`\n",
    "    * More here: https://youtu.be/HLH5AwF-jx4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9065f03e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index> 0      1       2          <int64>\n",
      "<Index>\n",
      "0       True   20      1954-11-02\n",
      "1       False  30      2020-04-28\n",
      "<int64> <bool> <int64> <<U10>\n"
     ]
    }
   ],
   "source": [
    "# Creating a Frame from row iterables\n",
    "f = sf.Frame.from_records(((True, 20, '1954-11-02'), \n",
    "                           (False, 30, '2020-04-28')))\n",
    "# Force a string representation \n",
    "print(str(f))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8882a2",
   "metadata": {},
   "source": [
    "# String Representations\n",
    "\n",
    "* `sf.Frame.__repr__()` provides more information than `pd.DataFrame.__repr__()`\n",
    "* Shows types and `name` of `Frame`, `.index`, and `.columns`\n",
    "* Shows NumPy dtypes for each column, `.index`, and `.columns`\n",
    "* In terminal environments can use colors for types, dtypes\n",
    "* Completely configurable with `sf.DisplayConfig`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4e00ff8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FrameGO: records>\n",
      "<IndexGO>          A      B       C       <<U1>\n",
      "<IndexYear: year>\n",
      "1954               True   20      1954-11\n",
      "2020               False  30      2020-04\n",
      "<datetime64[Y]>    <bool> <int64> <<U7>\n"
     ]
    }
   ],
   "source": [
    "# Creating a Frame with Frame subclass, Index subclasses, name attributes\n",
    "f = sf.FrameGO.from_records(((True, 20, '1954-11'), (False, 30, '2020-04')), \n",
    "        index=sf.IndexYear(('1954', '2020'), name='year'),\n",
    "        columns=('A', 'B', 'C'),\n",
    "        name='records', \n",
    "        )\n",
    "print(str(f))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3773c2",
   "metadata": {},
   "source": [
    "# Representation in Jupyter Notebooks\n",
    "\n",
    "* Default is an HTML table repsentation\n",
    "* `name` attributes, type, and dtype information are hidden by default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b12d073c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1 = sf.Frame.from_records(((True, 20, '1954-11-02'), (False, 30, '2020-04-28')), \n",
    "                            index=tuple('xy'), columns=tuple('ABC'))\n",
    "f1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06a66d64",
   "metadata": {},
   "source": [
    "# Finding All Constructors\n",
    "\n",
    "* Every StaticFrame container has an `.interface` attribute\n",
    "* `.interface` returns a `sf.Frame` of the complete API\n",
    "* The same representation is used to populate API overview: https://static-frame.readthedocs.io/en/latest/api_overview/frame.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "96c3fa66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>cls_name</th><th>group</th><th>doc</th></tr></thead><tbody><tr><th>__init__(data, *, index, columns, ...)</th><td>Frame</td><td>Constructor</td><td>Initializer. Args: data: Default Frame initialization requires typed data such a...</td></tr><tr><th>from_arrow(value, *, index_depth, index_name_depth_level, ...)</th><td>Frame</td><td>Constructor</td><td>Realize a Frame from an Arrow Table. Args: value: A pyarrow.Table instance. inde...</td></tr><tr><th>from_clipboard(*, delimiter, index_depth, index_column_first, ...)</th><td>Frame</td><td>Constructor</td><td>Create a Frame from the contents of the clipboard (assuming a table is stored as...</td></tr><tr><th>from_concat(frames, *, axis, union, ...)</th><td>Frame</td><td>Constructor</td><td>Concatenate multiple Frames into a new Frame. If index or columns are provided a...</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame: Frame>\n",
       "<Index>                              cls_name group       doc                  <<U18>\n",
       "<Index: signature>\n",
       "__init__(data, *, index, columns,... Frame    Constructor Initializer. Args...\n",
       "from_arrow(value, *, index_depth,... Frame    Constructor Realize a Frame f...\n",
       "from_clipboard(*, delimiter, inde... Frame    Constructor Create a Frame fr...\n",
       "from_concat(frames, *, axis, unio... Frame    Constructor Concatenate multi...\n",
       "<<U94>                               <<U5>    <<U27>      <<U83>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using the interface attribute to show the signature of all constructors\n",
    "f = sf.Frame.interface\n",
    "f.loc[f['group'] == 'Constructor'].head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879f734d",
   "metadata": {},
   "source": [
    "# Constructors Are Class Methods\n",
    "* Pandas places some constructors on the `pd` name space\n",
    "* All StaticFrame constructors are class methods on classes\n",
    "* Creating a Frame from concatenation \n",
    "    * Pandas: `pd.concat()`\n",
    "    * StaticFrame: `sf.Frame.from_concat()`, `sf.Frame.from_concat_items()`\n",
    "* Creating a Frame from other `sf.Frame`s by overlaying on missing values\n",
    "    * Pandas offers an instance method: `pd.DataFrame.combine_first()`\n",
    "    * StaticFrame offers a class method for combining one or more `sf.Frame`: `sf.Frame.from_overlay()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a7d685",
   "metadata": {},
   "source": [
    "# `sf.Frame` and `sf.Series` Are Like Python `dict`\n",
    "* Both containers have `items()` and `keys()` methods\n",
    "* Both containers have a `values` attribute (not a method like on `dict`)\n",
    "* `dict`-like Interfaces are almost the same in Pandas and StaticFrame\n",
    "* `sf.Frame`:\n",
    "    * `keys()` returns column labels\n",
    "    * `items()` returns pairs of label, column `sf.Series`\n",
    "    * `values` returns a homogenized array (often requiring dtype casting)\n",
    "    * `iter(sf.Frame)` iterates `keys()`\n",
    "* `sf.Series`:\n",
    "    * `keys()` returns labels\n",
    "    * `items()` returns pairs of label, element values\n",
    "    * `values` returns the 1D immutable array (a no-copy operation)\n",
    "    * `iter(sf.Series)` iterates `keys()`\n",
    "* Deviation from Pandas:\n",
    "    * With Pandas, `iter(pd.Series)` iterates `values`, not `keys()`\n",
    "    * Pandas deviates from the expected `dict`-like interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dbe1e3cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "('A', 'B', 'C')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sf.Frame.keys() iterates column labels\n",
    "display(f1)\n",
    "tuple(f1.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0ba7a30a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(('A',\n",
       "  <Series: A>\n",
       "  <Index>\n",
       "  x           True\n",
       "  y           False\n",
       "  <<U1>       <bool>),\n",
       " ('B',\n",
       "  <Series: B>\n",
       "  <Index>\n",
       "  x           20\n",
       "  y           30\n",
       "  <<U1>       <int64>),\n",
       " ('C',\n",
       "  <Series: C>\n",
       "  <Index>\n",
       "  x           1954-11-02\n",
       "  y           2020-04-28\n",
       "  <<U1>       <<U10>))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sf.Frame.items() iterates pairs of label, sf.Series\n",
    "tuple(f1.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2b678277",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[True, 20, '1954-11-02'],\n",
       "       [False, 30, '2020-04-28']], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sf.Frame.values is an array representation of the sf.Frame\n",
    "f1.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f9cfba",
   "metadata": {},
   "source": [
    "# Selection Interfaces\n",
    "* StaticFrame implements all NumPy and Pandas-style selection routines\n",
    "* Selection interfaces\n",
    "    * `loc[]`: use labels\n",
    "    * `iloc[]`: use integer position (from zero)\n",
    "    * `bloc[]`: use Boolean indicator (StaticFrame only)\n",
    "* NumPy-style per-axis selection values \n",
    "    * A single label (a tuple is a single label)\n",
    "    * A list of labels (must be a list to distinguish from a tuple label)\n",
    "    * A slice of labels\n",
    "    * A 1D Boolean array\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1902e921",
   "metadata": {},
   "source": [
    "# Selection Interfaces on `sf.Frame`\n",
    "    \n",
    "* `sf.Frame[]`: root `__getitem__()` selection \n",
    "    * `pd.DataFrame[]` selects by column labels, or row and column labels, or by 2D Boolean array\n",
    "    * `sf.Frame[]` is exclusively column selection\n",
    "* `sf.Frame.loc[]`: select rows, optionally columns, by label (same as Pandas)\n",
    "* `sf.Frame.iloc[]`: select rows, optionally columns, by integer position (same as Pandas)\n",
    "* `sf.Frame.bloc[]`: select with a 2D Boolean array (StaticFrame only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c7f951d7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>20</td></tr><tr><th>y</th><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: B>\n",
       "<Index>\n",
       "x           20\n",
       "y           30\n",
       "<<U1>       <int64>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1 = sf.Frame.from_records(((True, 20, '1954-11-02'), (False, 30, '2020-04-28')), \n",
    "                            index=tuple('xy'), columns=tuple('ABC'))\n",
    "display(f1)\n",
    "f1['B'] # Select a column with a single label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8afeb820",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>C</th></tr></thead><tbody><tr><th>x</th><td>1954-11-02</td></tr><tr><th>y</th><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> C          <<U1>\n",
       "<Index>\n",
       "x       1954-11-02\n",
       "y       2020-04-28\n",
       "<<U1>   <<U10>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1.columns == 'C')\n",
    "# Select columns with a Boolean indicator\n",
    "f1[f1.columns == 'C'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "aa41f482",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>C</th></tr></thead><tbody><tr><th>y</th><td>False</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      C          <<U1>\n",
       "<Index>\n",
       "y       False  2020-04-28\n",
       "<<U1>   <bool> <<U10>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1.loc['y':, ['A', 'C']] # Select a row with a slice and list of labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "823374cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2020-04-28'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1.iloc[-1, -1] # Select an element with iloc labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fb1fc00e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>False</td><td>True</td><td>False</td></tr><tr><th>y</th><td>False</td><td>False</td><td>True</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B      C      <<U1>\n",
       "<Index>\n",
       "x       False  True   False\n",
       "y       False  False  True\n",
       "<<U1>   <bool> <bool> <bool>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>(&#x27;x&#x27;, &#x27;B&#x27;)</th><td>20</td></tr><tr><th>(&#x27;y&#x27;, &#x27;C&#x27;)</th><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "('x', 'B') 20\n",
       "('y', 'C') 2020-04-28\n",
       "<object>   <object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1.isin([20, '2020-04-28']))\n",
    "f1.bloc[f1.isin([20, '2020-04-28'])] # Selcting non-contiguous values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf113ec8",
   "metadata": {},
   "source": [
    "# Mixing `loc` and `iloc` Selection\n",
    "\n",
    "* `sf.ILoc` (StaticFrame only) permits embedding `iloc`-style selection in a `loc` selection\n",
    "* There exists a related interface for embedding hierarchical selection in `loc` selection\n",
    "    * `sf.HLoc` (similar to `pd.IndexSlice`)\n",
    "    * For use on axis with `sf.IndexHierarchy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b429d48f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>False</td></tr><tr><th>C</th><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: y>\n",
       "<Index>\n",
       "A           False\n",
       "C           2020-04-28\n",
       "<<U1>       <object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1)\n",
    "f1.loc[sf.ILoc[-1], ['A', 'C']] # Get the last row, columns A and C"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeeebe8c",
   "metadata": {},
   "source": [
    "# Dropping\n",
    "* The inverse of selection is dropping \n",
    "* `sf.Frame.drop` interface exposes the same selection interfaces as attributes\n",
    "    * `sf.Frame.drop[]`\n",
    "    * `sf.Frame.drop.loc[]`\n",
    "    * `sf.Frame.drop.iloc[]`\n",
    "* This approach of interfaces that expose sub-component interfaces is common in StaticFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5b197031",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>B</th></tr></thead><tbody><tr><th>x</th><td>20</td></tr><tr><th>y</th><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> B       <<U1>\n",
       "<Index>\n",
       "x       20\n",
       "y       30\n",
       "<<U1>   <int64>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop two columns\n",
    "display(f1)\n",
    "f1.drop[['A', 'C']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "994db8d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>20</td><td>1954-11-02</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> B       C          <<U1>\n",
       "<Index>\n",
       "x       20      1954-11-02\n",
       "<<U1>   <int64> <<U10>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the last row and drop column \"A\"\n",
    "f1.drop.loc[sf.ILoc[-1], 'A']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a50e5eca",
   "metadata": {},
   "source": [
    "# Handling Missing Values\n",
    "* Missing values are `None` and `np.nan` (same as Pandas)\n",
    "* Boolean indicators (same as Pandas)\n",
    "    * `sf.Frame.isna()`\n",
    "    * `sf.Frame.notna()`\n",
    "* Replacing missing values with new containers (same as Pandas)\n",
    "    * `sf.Frame.dropna()`\n",
    "    * `sf.Frame.fillna()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc7ce05",
   "metadata": {},
   "source": [
    "# Handling Falsy Values\n",
    "* Sometimes we want to treat `0` or `''` or `()` as missing\n",
    "* Functions corresponding to `*na` functions (StaticFrame only)\n",
    "    * `sf.Frame.isfalsy()`\n",
    "    * `sf.Frame.notfalsy()`\n",
    "    * `sf.Frame.dropfalsy()`\n",
    "    * `sf.Frame.fillfalsy()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12826b3",
   "metadata": {},
   "source": [
    "# Fill Missing Values Along an Axis\n",
    "* Fill the first or last non-missing observation up to the `limit` parameter.\n",
    "    * Related functionaliy provided in `pd.DataFrame.fillna()`\n",
    "    * `sf.Frame.fillna_forward()`\n",
    "    * `sf.Frame.fillna_backward()`\n",
    "* Fill the leading or trailing missing values with a provided value\n",
    "    * StaticFrame only\n",
    "    * `sf.Frame.fillna_leading()`\n",
    "    * `sf.Frame.fillna_trailing()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3122deb9",
   "metadata": {},
   "source": [
    "# Fill Falsy Values Along an Axis\n",
    "* StaticFrame only\n",
    "* Fill the first or last non-missing observation up to the `limit` parameter.\n",
    "    * `sf.Frame.fillfalsy_forward()`\n",
    "    * `sf.Frame.fillfalsy_backward()`\n",
    "* Fill the leading or trailing missing values with a provided value\n",
    "    * `sf.Frame.fillfalsy_leading()`\n",
    "    * `sf.Frame.fillfalsy_trailing()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae82f296",
   "metadata": {},
   "source": [
    "# Immutability and \"No-Copy\" Operations\n",
    "* Immutability reduces opportunities for errors \n",
    "* NumPy provides no-copy \"views\" of array data when possible\n",
    "* With immutabile arrays, we can pass around views without making defensive copies\n",
    "* Examples:\n",
    "    * Renaming an `sf.Frame` is no-copy\n",
    "    * Relabelling `index` or `columns` does not copy underlying arrays\n",
    "    * Horizontal concatenation of same-index components is no-copy\n",
    "* Pandas support for mutation, combined with NumPy views, leads to the commonly observed Pandas `SettingWithCopyWarning`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d48d6dd",
   "metadata": {},
   "source": [
    "# Assignment with Immutable Frames\n",
    "* Pandas permits in-place assignment and mutation to all types of selections\n",
    "    * `pd.DataFrame.loc['x', 'B':] = 1.0`\n",
    "* StaticFrame offers an `assign` interface that defines a selection that is then called with a value to assign\n",
    "* The value to assign can be an element or labelled data (i.e., `sf.Series`, `sf.Frame`)\n",
    "* Example: `sf.Frame.assign.loc['x', 'B':](1.0)`\n",
    "    * Returns a new container\n",
    "    * Unchanged columns will be views and re-used (no-copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "965eee6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>-1</td><td>-1</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   -1      -1\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assigning a value to a slice in a single row\n",
    "display(f1)\n",
    "f1.assign.loc['x', 'B':](-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "81483bfd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>x</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>y</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B     C          <<U1>\n",
       "<Index>\n",
       "x       True   x     1954-11-02\n",
       "y       False  y     2020-04-28\n",
       "<<U1>   <bool> <<U1> <<U10>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assigning a Series to a column, matching on label\n",
    "f1.assign['B'](sf.Series(('y', 'x'), index=('y', 'x')))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12997dc",
   "metadata": {},
   "source": [
    "# Grow-Only Mutation\n",
    "* Pandas permits growing a DataFrame by columns (efficient) and rows (very inefficient)\n",
    "* The `sf.FrameGO` permits grow-only column addition or whole-`sf.Frame` extension\n",
    "* While the `sf.Frame.GO` container is mutable, underlying array data always remains immutable\n",
    "    * Going from an `sf.Frame` to an `sf.FrameGO` is a no-copy operations\n",
    "    * `sf.FrameGO` are often used within a narrow scope (i.e., a single function)\n",
    "* Growing rows is never permitted (use `sf.Frame.from_concat()` with collected rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "60222c21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td><td>34</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td><td>87</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A      B       C          D       <<U1>\n",
       "<Index>\n",
       "x         True   20      1954-11-02 34\n",
       "y         False  30      2020-04-28 87\n",
       "<<U1>     <bool> <int64> <<U10>     <int64>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Adding a column to a FrameGO; column arrays in f1 are not copied\n",
    "display(f1)\n",
    "f2 = f1.to_frame_go()\n",
    "f2['D'] = (34, 87)\n",
    "f2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a077bfdc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th><th>a</th><th>b</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td><td>34</td><td>100</td><td>2000</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td><td>87</td><td>0</td><td>3000</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A      B       C          D       a       b       <<U1>\n",
       "<Index>\n",
       "x         True   20      1954-11-02 34      100     2000\n",
       "y         False  30      2020-04-28 87      0       3000\n",
       "<<U1>     <bool> <int64> <<U10>     <int64> <int64> <int64>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extending a FrameGO with another Frame\n",
    "# On aligned indices this is a no-copy operation from f2, f3\n",
    "f3 = (f1[['A', 'B']] * 100).relabel(columns=lambda l: l.lower())\n",
    "f2.extend(f3)\n",
    "f2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dbf69c7",
   "metadata": {},
   "source": [
    "# A Family of `sf.Frame`\n",
    "\n",
    "* Pandas has only one `DataFrame` class\n",
    "* StaticFrame has a family\n",
    "    * `sf.Frame`\n",
    "    * `sf.FrameGO`: a grow-only `sf.Frame`\n",
    "    * `sf.FrameHE`: a hashable `sf.Frame`\n",
    "        * HE for `__hash__` and `__eq__`, the methods implemented to support hashability\n",
    "        * Some hashing scenarios may require a full values comparison for lookup\n",
    "* Methods exist to easily convert between all three (always a no-copy operation)\n",
    "    * `sf.Frame.to_frame_go()`\n",
    "    * `sf.Frame.to_frame_he()`\n",
    "    * `sf.FrameGO.to_frame()`\n",
    "    * `sf.FrameGO.to_frame_he()`\n",
    "    * `sf.FrameHE.to_frame()`\n",
    "    * `sf.FrameHE.to_frame_go()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7afe9160",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A FrameHE as a key in a dictionary\n",
    "f = sf.Frame(np.arange(4).reshape(2, 2)).to_frame_he()\n",
    "d = {f: True} \n",
    "f in d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9d2a36",
   "metadata": {},
   "source": [
    "# Changing Columnar dtypes\n",
    "\n",
    "* `sf.Frame.astype()` can be used to retype an entire Frame (same as Pandas)\n",
    "* Can use column selection to isolate targets\n",
    "    * `sf.Frame.astype[['A', 'B']](float)`\n",
    "    * Similar to `sf.Frame.drop`, `sf.Frame.assign` interfaces\n",
    "* Returns a new `sf.Frame`\n",
    "* Unaffected columns will not be copied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c4251e14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>1.0</td><td>20.0</td><td>1954-11-02</td></tr><tr><th>y</th><td>0.0</td><td>30.0</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A         B         C          <<U1>\n",
       "<Index>\n",
       "x       1.0       20.0      1954-11-02\n",
       "y       0.0       30.0      2020-04-28\n",
       "<<U1>   <float64> <float64> <<U10>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converting two columns to a float dtype\n",
    "display(f1)\n",
    "f1.astype[['A', 'B']](float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dac9586",
   "metadata": {},
   "source": [
    "# Full Support for All NumPy dtypes\n",
    "* NumPy is the foundation of StaticFrame and Pandas\n",
    "* Pandas only supports a subset of NumPy dtypes; StaticFrame supports all\n",
    "* NumPy's fixed-size Unicode arrays\n",
    "    * Optimal when string are diverse and of similar size\n",
    "    * Pandas always converts these to object arrays of Python strings\n",
    "* NumPy's `datetime64` type\n",
    "    * Fast datetime representation with units defining resolution (from year to attosecond)\n",
    "    * Pandas coerces any `datetime64` to nanosecond units\n",
    "    * StaticFrame permits using year, date, or any `datetime64` unit\n",
    "    * See also: https://www.youtube.com/watch?v=jdnr7sgxCQI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "09f1bb29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index> A      B       C          <<U1>\n",
      "<Index>\n",
      "x       True   20      1954-11-02\n",
      "y       False  30      2020-04-28\n",
      "<<U1>   <bool> <int64> <<U10>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>bool</td></tr><tr><th>B</th><td>int64</td></tr><tr><th>C</th><td>&lt;U10</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "A        bool\n",
       "B        int64\n",
       "C        <U10\n",
       "<<U1>    <object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# By default, StaticFrame always shows all types and dtypes\n",
    "print(str(f1))\n",
    "# Can get a Series by column label\n",
    "f1.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7628a4bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index> A      B       C          <<U1>\n",
      "<Index>\n",
      "x       True   20      1954-11-02\n",
      "y       False  30      2020-04-28\n",
      "<<U1>   <bool> <int64> <object>\n"
     ]
    }
   ],
   "source": [
    "# Can convert Unicode dtypes to Python string object (as Pandas)\n",
    "print(str(f1.astype['C'](object)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "82602f81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index> A      B       C               <<U1>\n",
      "<Index>\n",
      "x       True   20      1954-11-02\n",
      "y       False  30      2020-04-28\n",
      "<<U1>   <bool> <int64> <datetime64[D]>\n"
     ]
    }
   ],
   "source": [
    "# Can convert strings to NumPy datetime64 date objects\n",
    "print(str(f1.astype['C'](np.datetime64)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5746e817",
   "metadata": {},
   "source": [
    "# A Family of `sf.Index`\n",
    "\n",
    "* To use `datetime64` as index labels, use a `datetime64` `sf.Index` subclass\n",
    "    * `sf.IndexDate`, `sf.IndexYearMonth`, `sf.IndexYear`, etc.\n",
    "    * Provides robust translation from Python date / datetime objects\n",
    "    * Provides partial selection with less granular date units\n",
    "    * Provides alternative constructors to build date ranges\n",
    "* Hierarchical indices with `sf.IndexHierarchy`\n",
    "* Many interfaces expose `index_constructor` arguments to specify what kind of index to make.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c3787644",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index>         A      B       <<U1>\n",
      "<IndexDate: C>\n",
      "1954-11-02      True   20\n",
      "2020-04-28      False  30\n",
      "<datetime64[D]> <bool> <int64>\n"
     ]
    }
   ],
   "source": [
    "# Transfer a column to an index\n",
    "f4 = f1.set_index('C', drop=True, index_constructor=sf.IndexDate)\n",
    "print(str(f4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6618eac7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th></tr></thead><tbody><tr><th>2020-04-28</th><td>False</td><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index>         A      B       <<U1>\n",
       "<IndexDate: C>\n",
       "2020-04-28      False  30\n",
       "<datetime64[D]> <bool> <int64>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Selection with a less granular unit (year)\n",
    "f4.loc['2020']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c8fe80fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>True</td></tr><tr><th>B</th><td>20</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: 1954-11-02>\n",
       "<Index>\n",
       "A                    True\n",
       "B                    20\n",
       "<<U1>                <object>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sf.IndexDate understands Python datetime / date objects\n",
    "import datetime\n",
    "f4.loc[datetime.date(1954, 11, 2)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "08b62fc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame>\n",
      "<Index> C               A      B       <<U1>\n",
      "<Index>\n",
      "0       1954-11-02      True   20\n",
      "1       2020-04-28      False  30\n",
      "<int64> <datetime64[D]> <bool> <int64>\n"
     ]
    }
   ],
   "source": [
    "# Removing an index, similar to Pandas pd.DataFrame.reset_index()\n",
    "print(str(f4.unset_index()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c60da4a",
   "metadata": {},
   "source": [
    "# Rename, Reindex, Relabel\n",
    "* Changing \"outer\" attributes of a `sf.Frame`\n",
    "* Will always try to reuse as much data as possible\n",
    "* `rename()` sets the `name` attribute on all containers\n",
    "    * `pd.DataFrame.rename()` relabels the axis, `pd.Series.rename()` sets the name of the container\n",
    "    * `sf.Frame.rename()`, `sf.Series.rename()` all do the same thing\n",
    "    * Always a no-copy operation\n",
    "* `reindex()` applies new index, aligning to the previous index\n",
    "    * Can transform the shape of the `sf.Frame`\n",
    "    * Similar to `pd.DataFrame.reindex()`\n",
    "    * Conform the old index to the new index, where matching labels retain their data\n",
    "    * New labels will introduce missing values (provided with a `fill_value`)\n",
    "* `relabel()` applies a new index, regardless of alignment to previous index\n",
    "    * Cannot change the shape of the `sf.Frame`\n",
    "    * Three values accepted per axis\n",
    "        * Can map old to new with `dict`\n",
    "        * Can process old to new with a function\n",
    "        * Can replace with a new `sf.Index` or iterable\n",
    "    * Always a no-copy operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d9b57add",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Frame: p>\n",
      "<Index: r> A      B       C          <<U1>\n",
      "<Index: q>\n",
      "x          True   20      1954-11-02\n",
      "y          False  30      2020-04-28\n",
      "<<U1>      <bool> <int64> <<U10>\n"
     ]
    }
   ],
   "source": [
    "# Renaming the sf.Frame, the index, and the columns\n",
    "print(f1.rename('p', index='q', columns='r'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "16b157b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>y</th><td>False</td><td>2020-04-28</td><td></td></tr><tr><th>z</th><td></td><td></td><td></td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A        C          D     <<U1>\n",
       "<Index>\n",
       "y       False    2020-04-28\n",
       "z\n",
       "<<U1>   <object> <<U10>     <<U1>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reindexing index and columns, filling new values with \"\"\n",
    "f1.reindex(index=tuple('yz'), columns=tuple('ACD'), fill_value='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f4c16a5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr></thead><tbody><tr><th>-1</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>-2</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> a      b       c          <<U1>\n",
       "<Index>\n",
       "-1      True   20      1954-11-02\n",
       "-2      False  30      2020-04-28\n",
       "<int64> <bool> <int64> <<U10>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Relabelling index by assignment, columns by function (can also use a mapping)\n",
    "f1.relabel(index=(-1, -2), columns=lambda l: l.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04bc4083",
   "metadata": {},
   "source": [
    "# Iterating Components of an `sf.Frame`\n",
    "* Iterating elements: `Frame.iter_elements()`\n",
    "* Iterating rows or columns:\n",
    "    * Specify axis, 1 for by row, 0 for column\n",
    "    * Distinct methods determine what container to return\n",
    "        * `Frame.iter_series()`\n",
    "        * `Frame.iter_tuple()`\n",
    "        * `Frame.iter_array()`\n",
    "    * The \"lighter\" the container, the better the performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f0cb1b59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "1         3       4       5       b\n",
       "2         6       7       8       b\n",
       "3         9       10      11      a\n",
       "4         12      13      14      c\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an sf.FrameGO and add a column\n",
    "f5 = sf.FrameGO(np.arange(18).reshape(6,3), columns=tuple('ABC'))\n",
    "f5['D'] = tuple('abbaca')\n",
    "f5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f9c861e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1, 2, 'a', 3, 4, 5, 'b', 6, 7, 8, 'b', 9, 10, 11, 'a', 12, 13)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Iteration of elements proceed row-wise (by default)\n",
    "tuple(f5.iter_element())[:18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "60e49e72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>0</td></tr><tr><th>B</th><td>1</td></tr><tr><th>C</th><td>2</td></tr><tr><th>D</th><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: 0>\n",
       "<Index>\n",
       "A           0\n",
       "B           1\n",
       "C           2\n",
       "D           a\n",
       "<<U1>       <object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 'a'], dtype=object)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Iterating sf.Series or array by axis 1 iterates rows; next() gets the first\n",
    "display(next(iter(f5.iter_series(axis=1))))\n",
    "next(iter(f5.iter_array(axis=1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "db44de28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>0</th><td>0</td></tr><tr><th>1</th><td>3</td></tr><tr><th>2</th><td>6</td></tr><tr><th>3</th><td>9</td></tr><tr><th>4</th><td>12</td></tr><tr><th>5</th><td>15</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: A>\n",
       "<Index>\n",
       "0           0\n",
       "1           3\n",
       "2           6\n",
       "3           9\n",
       "4           12\n",
       "5           15\n",
       "<int64>     <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([ 0,  3,  6,  9, 12, 15])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Iterating Series or array by axis 0 iterates columns, next() gets the first\n",
    "display(next(iter(f5.iter_series(axis=0))))\n",
    "next(iter(f5.iter_array(axis=0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0e6fb11",
   "metadata": {},
   "source": [
    "# Applying Functions & Maps\n",
    "* Function application implies iteration\n",
    "* Choose what you want to iterate on and call `apply()`\n",
    "* Can multi-process / thread with `apply_pool()`\n",
    "* Can iterate through results with `apply_iter()`\n",
    "* Can apply a map (i.e, `dict` or `sf.Series`) instead of function\n",
    "    * `map_all()`: if value not mappable, raise\n",
    "    * `map_any()`: map what you can, leave the rest unchanged\n",
    "    * `map_fill()`: map what you can, provide `fill_value` for others"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "53b91aaa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>--0--</td><td>--1--</td><td>--2--</td><td>--a--</td></tr><tr><th>1</th><td>--3--</td><td>--4--</td><td>--5--</td><td>--b--</td></tr><tr><th>2</th><td>--6--</td><td>--7--</td><td>--8--</td><td>--b--</td></tr><tr><th>3</th><td>--9--</td><td>--10--</td><td>--11--</td><td>--a--</td></tr><tr><th>4</th><td>--12--</td><td>--13--</td><td>--14--</td><td>--c--</td></tr><tr><th>5</th><td>--15--</td><td>--16--</td><td>--17--</td><td>--a--</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A      B      C      D     <<U1>\n",
       "<Index>\n",
       "0         --0--  --1--  --2--  --a--\n",
       "1         --3--  --4--  --5--  --b--\n",
       "2         --6--  --7--  --8--  --b--\n",
       "3         --9--  --10-- --11-- --a--\n",
       "4         --12-- --13-- --14-- --c--\n",
       "5         --15-- --16-- --17-- --a--\n",
       "<int64>   <<U6>  <<U6>  <<U6>  <<U5>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using iter_element.apply(), we get back a same-shaped container\n",
    "f5.iter_element().apply(lambda e: f'--{e}--')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "46658855",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td></td><td>1</td><td>2</td><td></td></tr><tr><th>1</th><td>3</td><td></td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td></td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td></td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td></td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A        B        C        D     <<U1>\n",
       "<Index>\n",
       "0                  1        2\n",
       "1         3                 5        b\n",
       "2         6        7                 b\n",
       "3         9        10       11\n",
       "4         12       13       14       c\n",
       "5         15       16       17\n",
       "<int64>   <object> <object> <object> <<U1>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replacing values with map_any()\n",
    "f5.iter_element().map_any({0:'', 4:'', 8:'', 'a':''})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "06db2a29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "1         3       4       5       b\n",
       "2         6       7       8       b\n",
       "3         9       10      11      a\n",
       "4         12      13      14      c\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>0</th><td>0.0</td></tr><tr><th>1</th><td>0.6</td></tr><tr><th>2</th><td>0.75</td></tr><tr><th>3</th><td>0.8181818181818182</td></tr><tr><th>4</th><td>0.8571428571428571</td></tr><tr><th>5</th><td>0.8823529411764706</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "0        0.0\n",
       "1        0.6\n",
       "2        0.75\n",
       "3        0.8181818181818182\n",
       "4        0.8571428571428571\n",
       "5        0.8823529411764706\n",
       "<int64>  <float64>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply a sf.Series-consuming fuction to each row\n",
    "# Returns a sf.Series labeled by index\n",
    "display(f5)\n",
    "f5[:'C'].iter_series(axis=1).apply(lambda s: s['A'] / s['C'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "933fe0b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "1         3       4       5       b\n",
       "2         6       7       8       b\n",
       "3         9       10      11      a\n",
       "4         12      13      14      c\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>0.6</td></tr><tr><th>B</th><td>0.625</td></tr><tr><th>C</th><td>0.6470588235294118</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "A        0.6\n",
       "B        0.625\n",
       "C        0.6470588235294118\n",
       "<<U1>    <float64>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply a sf.Series-consuming fuction to each column\n",
    "# Returns a sf.Series labeled by columns\n",
    "display(f5)\n",
    "f5[:'C'].iter_series(axis=0).apply(lambda s: s[3] / s[5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "0e085b97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>A</th><td>0.6</td></tr><tr><th>B</th><td>0.625</td></tr><tr><th>C</th><td>0.6470588235294118</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "A        0.6\n",
       "B        0.625\n",
       "C        0.6470588235294118\n",
       "<<U1>    <float64>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Choose the container to iterate depending on your needs\n",
    "# Iterating arrays is always faster\n",
    "f5[:'C'].iter_array(axis=0).apply(lambda a: a[3] / a[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f12f12",
   "metadata": {},
   "source": [
    "# Grouping & Windowing\n",
    "\n",
    "* Grouping and windowing are two different ways of collecting sub-`sf.Frame`s from an `sf.Frame`\n",
    "* Just another type of `sf.Frame` iterator\n",
    "* `sf.Frame.iter_group()`\n",
    "    * Group by unique values in one or more columns (axis 0) or rows (axis 1)\n",
    "    * Can use `apply()` if reducing to an `sf.Series`\n",
    "    * `sf.Frame.iter_group_items()` returns pairs of group label, sub-`sf.Frame`\n",
    "    * Can use an `sf.Batch` for performing operations on sub-Frames like `pd.DataFrameGroupBy`\n",
    "* `sf.Frame.iter_window()`\n",
    "    * `sf.Frame.iter_window_items()` returns pairs of window label, sub-`sf.Frame`\n",
    "    * Can use an `sf.Batch` for performing operations on sub Frames like `pd.Rolling`\n",
    "    * `sf.Frame.iter_window_array()` available when only array data is needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c3577ef2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "1         3       4       5       b\n",
       "2         6       7       8       b\n",
       "3         9       10      11      a\n",
       "4         12      13      14      c\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3c69ccae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "3         9       10      11      a\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "1         3       4       5       b\n",
       "2         6       7       8       b\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "4         12      13      14      c\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Iterate the first group by unique values found in column 'D'\n",
    "it = iter(f5.iter_group('D'))\n",
    "display(next(it))\n",
    "display(next(it))\n",
    "display(next(it))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "4adbb93e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>a</th><td>9</td></tr><tr><th>b</th><td>6</td></tr><tr><th>c</th><td>3</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index: D>\n",
       "a          9\n",
       "b          6\n",
       "c          3\n",
       "<<U1>      <int64>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Applying a function returns a sf.Series labelled by group \n",
    "# drop removes the column or columns used in grouping\n",
    "f5.iter_group('D', drop=True).apply(lambda f: f.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "ec76a912",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>None</th></tr></thead><tbody><tr><th>a</th><td>32</td></tr><tr><th>b</th><td>14</td></tr><tr><th>c</th><td>26</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> None    <object>\n",
       "<Index>\n",
       "a       32\n",
       "b       14\n",
       "c       26\n",
       "<<U1>   <int64>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can do operations on all groups (each sf.Frame) with a sf.Batch\n",
    "# Need to deliver to sf.Batch pairs of label, sf.Frame\n",
    "sf.Batch(\n",
    "    f5.iter_group_items('D')\n",
    ").loc[sf.ILoc[-1], ['A', 'C']].sum().to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6cdf15ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "0         0       1       2       a\n",
       "1         3       4       5       b\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "2         6       7       8       b\n",
       "3         9       10      11      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       D     <<U1>\n",
       "<Index>\n",
       "4         12      13      14      c\n",
       "5         15      16      17      a\n",
       "<int64>   <int64> <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Windowing axis 0 collects rows; many parameters to configure window shape, size, and step\n",
    "it = iter(f5.iter_window(size=2, step=2, axis=0))\n",
    "display(next(it))\n",
    "display(next(it))\n",
    "display(next(it))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "e717d57e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>0</th><td>0</td><td>1</td><td>2</td></tr><tr><th>1</th><td>3</td><td>4</td><td>5</td></tr><tr><th>2</th><td>6</td><td>7</td><td>8</td></tr><tr><th>3</th><td>9</td><td>10</td><td>11</td></tr><tr><th>4</th><td>12</td><td>13</td><td>14</td></tr><tr><th>5</th><td>15</td><td>16</td><td>17</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> A       B       C       <<U1>\n",
       "<Index>\n",
       "0         0       1       2\n",
       "1         3       4       5\n",
       "2         6       7       8\n",
       "3         9       10      11\n",
       "4         12      13      14\n",
       "5         15      16      17\n",
       "<int64>   <int64> <int64> <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><th>0</th><td>1</td><td>2</td><td>a</td></tr><tr><th>1</th><td>4</td><td>5</td><td>b</td></tr><tr><th>2</th><td>7</td><td>8</td><td>b</td></tr><tr><th>3</th><td>10</td><td>11</td><td>a</td></tr><tr><th>4</th><td>13</td><td>14</td><td>c</td></tr><tr><th>5</th><td>16</td><td>17</td><td>a</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<FrameGO>\n",
       "<IndexGO> B       C       D     <<U1>\n",
       "<Index>\n",
       "0         1       2       a\n",
       "1         4       5       b\n",
       "2         7       8       b\n",
       "3         10      11      a\n",
       "4         13      14      c\n",
       "5         16      17      a\n",
       "<int64>   <int64> <int64> <<U1>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Windowing axis 1 collects columns\n",
    "it = iter(f5.iter_window(size=3, step=1, axis=1))\n",
    "display(next(it))\n",
    "display(next(it))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "258f2c09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2, 'a'],\n",
       "       [3, 4, 5, 'b']], dtype=object)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([[6, 7, 8, 'b'],\n",
       "       [9, 10, 11, 'a']], dtype=object)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([[12, 13, 14, 'c'],\n",
       "       [15, 16, 17, 'a']], dtype=object)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Better performance available by iterating arrays\n",
    "it = iter(f5.iter_window_array(size=2, step=2))\n",
    "display(next(it))\n",
    "display(next(it))\n",
    "display(next(it))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "269fb418",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>C</th></tr></thead><tbody><tr><th>1</th><td>1.5</td><td>3.5</td></tr><tr><th>3</th><td>7.5</td><td>9.5</td></tr><tr><th>5</th><td>13.5</td><td>15.5</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A         C         <<U1>\n",
       "<Index>\n",
       "1       1.5       3.5\n",
       "3       7.5       9.5\n",
       "5       13.5      15.5\n",
       "<int64> <float64> <float64>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Processing window items with an sf.Batch\n",
    "# Need to deliver to sf.Batch pairs of label, sf.Frame\n",
    "sf.Batch(f5.iter_window_items(size=2, step=2))[['A', 'C']].mean().to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f31908",
   "metadata": {},
   "source": [
    "# Processing Collections of Frames\n",
    "* `sf.Batch` is a general-purpose interface for processing collections of `sf.Frame`\n",
    "* Consumes an iterator of label, `sf.Frame`\n",
    "* Methods and selection called on `sf.Batch` are called on each `sf.Frame` in the iterator\n",
    "* Chained operations permit function pipelining\n",
    "* Each step can use mutli-processing / threading to process each `sf.Frame` \n",
    "* More here: https://static-frame.readthedocs.io/en/latest/articles/uhoc.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d837089",
   "metadata": {},
   "source": [
    "# Working with Collections of Frames\n",
    "* Pandas deprecated the `pd.Panel` for 3D data\n",
    "* Hierarchical indices incur overhead and force loading all data at once\n",
    "* The `sf.Bus` provide a novel alternative\n",
    "    * Offers a `sf.Series`-like interface to collections of `sf.Frame`s\n",
    "    * Can read to and write from multi-table storage formats\n",
    "        * XLSX, HDF5, SQLite\n",
    "            * XLSX authoring similar to Pandas `pd.ExcelWriter`\n",
    "            * HDF5 authoring similar to Pandas `pd.HDFStore`\n",
    "        * Zipped archives of CSV, TSV, Parquet, and NPZ\n",
    "    * When reading from disk, loads lazily, only when data is accesed\n",
    "    * Optionally unloads eagerly with `max_persist` argument\n",
    "* A family of higher-order containers\n",
    "    * The `sf.Yarn` lazily links `sf.Bus`\n",
    "    * The `sf.Quilt` is a virtual concatenation of the contents of a `sf.Bus`\n",
    "* More here: https://static-frame.readthedocs.io/en/latest/articles/uhoc.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "e5aad46e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Bus>\n",
       "<Index>\n",
       "f1      Frame\n",
       "f3      Frame\n",
       "f5      Frame\n",
       "<<U2>   <object>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a sf.Bus from an iterable of label, sf.Frame pairs\n",
    "b = sf.Bus.from_items((('f1', f1), ('f3', f3), ('f5', f5.to_frame())))\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ea2c6bd8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>loaded</th><th>size</th><th>nbytes</th><th>shape</th></tr></thead><tbody><tr><th>f1</th><td>True</td><td>6.0</td><td>98.0</td><td>(2, 3)</td></tr><tr><th>f3</th><td>True</td><td>4.0</td><td>32.0</td><td>(2, 2)</td></tr><tr><th>f5</th><td>True</td><td>24.0</td><td>168.0</td><td>(6, 4)</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> loaded size      nbytes    shape    <<U6>\n",
       "<Index>\n",
       "f1      True   6.0       98.0      (2, 3)\n",
       "f3      True   4.0       32.0      (2, 2)\n",
       "f5      True   24.0      168.0     (6, 4)\n",
       "<<U2>   <bool> <float64> <float64> <object>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When reading from a file store, loading is lazy\n",
    "# When creating a sf.Bus from in-memory sf.Frame, all are loaded\n",
    "b.status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "024ae13a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>a</th><th>b</th></tr></thead><tbody><tr><th>x</th><td>100</td><td>2000</td></tr><tr><th>y</th><td>0</td><td>3000</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> a       b       <<U1>\n",
       "<Index>\n",
       "x       100     2000\n",
       "y       0       3000\n",
       "<<U1>   <int64> <int64>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Accessing a single element provides an sf.Frame\n",
    "b['f3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "68623447",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th></th><th>A</th><th>B</th><th>C</th><th>D</th><th>a</th><th>b</th></tr></thead><tbody><tr><th>f1</th><th>x</th><td>True</td><td>20</td><td>1954-11-02</td><td></td><td></td><td></td></tr><tr><th>f1</th><th>y</th><td>False</td><td>30</td><td>2020-04-28</td><td></td><td></td><td></td></tr><tr><th>f3</th><th>x</th><td></td><td></td><td></td><td></td><td>100</td><td>2000</td></tr><tr><th>f3</th><th>y</th><td></td><td></td><td></td><td></td><td>0</td><td>3000</td></tr><tr><th>f5</th><th>0</th><td>0</td><td>1</td><td>2</td><td>a</td><td></td><td></td></tr><tr><th>f5</th><th>1</th><td>3</td><td>4</td><td>5</td><td>b</td><td></td><td></td></tr><tr><th>f5</th><th>2</th><td>6</td><td>7</td><td>8</td><td>b</td><td></td><td></td></tr><tr><th>f5</th><th>3</th><td>9</td><td>10</td><td>11</td><td>a</td><td></td><td></td></tr><tr><th>f5</th><th>4</th><td>12</td><td>13</td><td>14</td><td>c</td><td></td><td></td></tr><tr><th>f5</th><th>5</th><td>15</td><td>16</td><td>17</td><td>a</td><td></td><td></td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index>                   A        B        C          D     a        b        <<U1>\n",
       "<IndexHierarchy>\n",
       "f1               x        True     20       1954-11-02\n",
       "f1               y        False    30       2020-04-28\n",
       "f3               x                                           100      2000\n",
       "f3               y                                           0        3000\n",
       "f5               0        0        1        2          a\n",
       "f5               1        3        4        5          b\n",
       "f5               2        6        7        8          b\n",
       "f5               3        9        10       11         a\n",
       "f5               4        12       13       14         c\n",
       "f5               5        15       16       17         a\n",
       "<<U2>            <object> <object> <object> <object>   <<U1> <object> <object>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using a sf.Batch, all sf.Frame in a sf.Bus can be processed and/or combined\n",
    "sf.Batch(b.items()).to_frame(fill_value='')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41518c2f",
   "metadata": {},
   "source": [
    "# By Way of `via`\n",
    "\n",
    "* Alternate interfaces for \"viewing\" a container (or its elements) differently\n",
    "* Provides a hierarchical interface\n",
    "* Available on `sf.Frame`, `sf.Series`, and `sf.Index`\n",
    "* Some have corresponding Pandas interfaces, some are StaticFrame only"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a49c1f",
   "metadata": {},
   "source": [
    "# Interfaces for Working with Strings\n",
    "* `sf.Frame.via_str`, similar to `pd.Series.str`\n",
    "* Exposes Python string interfaces for application to all elements\n",
    "* https://static-frame.readthedocs.io/en/latest/api_overview/frame.html#frame-accessor-string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "9b42925e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>TRUE</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>FALSE</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A     B      C          <<U1>\n",
       "<Index>\n",
       "x       TRUE  20     1954-11-02\n",
       "y       FALSE 30     2020-04-28\n",
       "<<U1>   <<U5> <<U21> <<U10>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1)\n",
    "f1.via_str.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "b5bf9745",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>2+</td><td>1954-11-+2</td></tr><tr><th>y</th><td>False</td><td>3+</td><td>2+2+-+4-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A     B     C          <<U1>\n",
       "<Index>\n",
       "x       True  2+    1954-11-+2\n",
       "y       False 3+    2+2+-+4-28\n",
       "<<U1>   <<U5> <<U2> <<U10>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1.via_str.replace('0', '+')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edecbd46",
   "metadata": {},
   "source": [
    "# Interfaces for Working with Dates\n",
    "* `sf.Frame.via_dt`, similar to `pd.Series.dt`\n",
    "* Exposes Python `date`, `datetime` interface for application on `np.datetime64` types\n",
    "* https://static-frame.readthedocs.io/en/latest/api_overview/frame.html#frame-accessor-datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "adfe3671",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>11</td></tr><tr><th>y</th><td>4</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: C>\n",
       "<Index>\n",
       "x           11\n",
       "y           4\n",
       "<<U1>       <int64>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1)\n",
    "f1['C'].astype(np.datetime64).via_dt.month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b366d47e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>1954</td></tr><tr><th>y</th><td>2020</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: C>\n",
       "<Index>\n",
       "x           1954\n",
       "y           2020\n",
       "<<U1>       <int64>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1['C'].astype(np.datetime64).via_dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "639bd2e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>1</td></tr><tr><th>y</th><td>1</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: C>\n",
       "<Index>\n",
       "x           1\n",
       "y           1\n",
       "<<U1>       <int64>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1['C'].astype(np.datetime64).via_dt.weekday()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076f6b06",
   "metadata": {},
   "source": [
    "# Interfaces for Applying Regular Expressions\n",
    "* `sf.Frame.via_re` \n",
    "* Similar to `pd.Series.str.extract()`, but provides full interface from `re` module\n",
    "* https://static-frame.readthedocs.io/en/latest/api_overview/frame.html#frame-accessor-regular-expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "494e8db3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>True</td><td>20</td><td>1954-11-02</td></tr><tr><th>y</th><td>False</td><td>30</td><td>2020-04-28</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B       C          <<U1>\n",
       "<Index>\n",
       "x       True   20      1954-11-02\n",
       "y       False  30      2020-04-28\n",
       "<<U1>   <bool> <int64> <<U10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><th>x</th><td>False</td><td>True</td><td>True</td></tr><tr><th>y</th><td>True</td><td>False</td><td>True</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> A      B      C      <<U1>\n",
       "<Index>\n",
       "x       False  True   True\n",
       "y       True   False  True\n",
       "<<U1>   <bool> <bool> <bool>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1)\n",
    "# Match any element with \"2\" or \"a\"\n",
    "f1.via_re('[2a]').search()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0060c33f",
   "metadata": {},
   "source": [
    "# Configuring `fill_value` in Operator Application\n",
    "\n",
    "* Binary operators on labelled containers may force reindexing\n",
    "* By default, no way to provide fill value\n",
    "* `sf.Frame.via_fill_value()` permits providing a fill value\n",
    "* Pandas offers related functionality with `pd.DataFrame.add()`, `pd.DataFrame.sub()`, `pd.DataFrame.mul()`, etc., methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f4f417bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>20</td></tr><tr><th>y</th><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: B>\n",
       "<Index>\n",
       "x           20\n",
       "y           30\n",
       "<<U1>       <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>0.02</td></tr><tr><th>y</th><td>30.0</td></tr><tr><th>z</th><td>nan</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "x        0.02\n",
       "y        30.0\n",
       "z        nan\n",
       "<<U1>    <float64>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(f1['B'])\n",
    "# Default binary operator application takes the union index and uses `nan` as a fill value\n",
    "f1['B'] * sf.Series((1000, 1, .001), index=tuple('zyx'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7b8d520e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>0.02</td></tr><tr><th>y</th><td>30.0</td></tr><tr><th>z</th><td>0.0</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series>\n",
       "<Index>\n",
       "x        0.02\n",
       "y        30.0\n",
       "z        0.0\n",
       "<<U1>    <float64>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using via_fill_value() a fill value can be specified\n",
    "f1['B'].via_fill_value(0) * sf.Series((1000, 1, .001), index=tuple('zyx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b929bb6c",
   "metadata": {},
   "source": [
    "# Virtual Transposition in Operator Application\n",
    "* Applying a 1D container on a 2D container applies to rows\n",
    "* `sf.Frame.via_T` presents 2D containers \"virtually\" transposed\n",
    "* Useful for applying a 1D container to the columns (not rows) of a 2D container\n",
    "* Pandas offers related functionality with `pd.DataFrame.add()`, `pd.DataFrame.sub()`, `pd.DataFrame.mul()`, etc., methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "a1cf44e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><th>x</th><td>0</td><td>1</td><td>2</td><td>3</td></tr><tr><th>y</th><td>4</td><td>5</td><td>6</td><td>7</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> 0       1       2       3       <int64>\n",
       "<Index>\n",
       "x       0       1       2       3\n",
       "y       4       5       6       7\n",
       "<<U1>   <int64> <int64> <int64> <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>20</td></tr><tr><th>y</th><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: B>\n",
       "<Index>\n",
       "x           20\n",
       "y           30\n",
       "<<U1>       <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th><th>y</th><th>x</th></tr></thead><tbody><tr><th>x</th><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><th>y</th><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> 0         1         2         3         y         x         <object>\n",
       "<Index>\n",
       "x       nan       nan       nan       nan       nan       nan\n",
       "y       nan       nan       nan       nan       nan       nan\n",
       "<<U1>   <float64> <float64> <float64> <float64> <float64> <float64>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2D to 1D assumes row-wise application\n",
    "display(sf.Frame(np.arange(8).reshape(2, 4), index=tuple('xy')))\n",
    "display(f1['B'])\n",
    "# The columns of the sf.Frame and the index of the Series have no labels in common \n",
    "sf.Frame(np.arange(8).reshape(2, 4), index=tuple('xy')) * f1['B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "5bf49a32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><th>x</th><td>0</td><td>1</td><td>2</td><td>3</td></tr><tr><th>y</th><td>4</td><td>5</td><td>6</td><td>7</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> 0       1       2       3       <int64>\n",
       "<Index>\n",
       "x       0       1       2       3\n",
       "y       4       5       6       7\n",
       "<<U1>   <int64> <int64> <int64> <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><th>x</th><td>20</td></tr><tr><th>y</th><td>30</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Series: B>\n",
       "<Index>\n",
       "x           20\n",
       "y           30\n",
       "<<U1>       <int64>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><th>x</th><td>0</td><td>20</td><td>40</td><td>60</td></tr><tr><th>y</th><td>120</td><td>150</td><td>180</td><td>210</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<Frame>\n",
       "<Index> 0       1       2       3       <int64>\n",
       "<Index>\n",
       "x       0       20      40      60\n",
       "y       120     150     180     210\n",
       "<<U1>   <int64> <int64> <int64> <int64>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using via_T, can perform column-wise application\n",
    "display(sf.Frame(np.arange(8).reshape(2, 4), index=tuple('xy')))\n",
    "display(f1['B'])\n",
    "sf.Frame(np.arange(8).reshape(2, 4), index=tuple('xy')).via_T * f1['B']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebc191f9",
   "metadata": {},
   "source": [
    "# All the Rest\n",
    "\n",
    "* Complete API documentation: https://static-frame.readthedocs.io/en/latest/api_overview/frame.html\n",
    "* Browse and search the API with examples: https://www.staticframe.dev"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ec5b03",
   "metadata": {},
   "source": [
    "# All the Rest: NumPy-Style Interfaces\n",
    "\n",
    "* StaticFrame supports common NumPy interfaces and methods (Same as Pandas)\n",
    "* Attributes:\n",
    "    * `sf.Frame.shape`\n",
    "    * `sf.Frame.ndim`\n",
    "    * `sf.Frame.size`\n",
    "    * `sf.Frame.nbytes`\n",
    "    * `sf.Frame.T`\n",
    "* Logical operations (by axis):\n",
    "    * `sf.Frame.all()`\n",
    "    * `sf.Frame.any()`\n",
    "* Mathematical operations (by axis):\n",
    "    * `sf.Frame.sum()`\n",
    "    * `sf.Frame.min()`\n",
    "    * `sf.Frame.max()`\n",
    "    * `sf.Frame.mean()`\n",
    "    * `sf.Frame.median()`\n",
    "    * `sf.Frame.std()`\n",
    "    * `sf.Frame.var()`\n",
    "    * `sf.Frame.prod()`\n",
    "    * `sf.Frame.cumsum()`\n",
    "    * `sf.Frame.cumprod()`\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73d0c18",
   "metadata": {},
   "source": [
    "# All the Rest: Common Interfaces with Pandas\n",
    "* `sf.Frame.isin()`\n",
    "* `sf.Frame.head()`, `sf.Frame.tail()`\n",
    "* `sf.Frame.cov()`, `sf.Frame.var()`\n",
    "* `sf.Frame.clip()`\n",
    "* `sf.Frame.count()`\n",
    "* `sf.Frame.equals()`\n",
    "* `sf.Frame.sample()`\n",
    "* `sf.Frame.sort_values()`, `sf.Frame.sort_index()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31487250",
   "metadata": {},
   "source": [
    "# All the Rest: Handling Duplicated Values\n",
    "\n",
    "\n",
    "* Pandas: `pd.duplicated()`. `pd.DataFrame.drop_duplicates()`\n",
    "* StaticFrame: `sf.Frame.duplicated()`, `sf.Frame.drop_duplicated()`, "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f74530",
   "metadata": {},
   "source": [
    "# All the Rest: Joins\n",
    "* Pandas: `pd.DataFrame.join()` with a `how` parameter (‘left’, ‘right’, ‘outer’, ‘inner’)\n",
    "* StaticFrame:\n",
    "    * `sf.Frame.join_left()`\n",
    "    * `sf.Frame.join_right()`\n",
    "    * `sf.Frame.join_outer()`\n",
    "    * `sf.Frame.join_inner()`    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84de52d",
   "metadata": {},
   "source": [
    "# All the Rest: Ranking\n",
    "* Pandas: `pd.DataFrame.rank()` with a `method` parameter of (‘average’, ‘min’, ‘max’, ‘first’, ‘dense’)\n",
    "* StaticFrame:\n",
    "    * `sf.Frame.rank_mean()`\n",
    "    * `sf.Frame.rank_min()`\n",
    "    * `sf.Frame.rank_max()`\n",
    "    * `sf.Frame.rank_ordinal()`\n",
    "    * `sf.Frame.rank_dense()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4459185",
   "metadata": {},
   "source": [
    "# All the Rest: Pivoting\n",
    "* Pivoting\n",
    "    * Pandas: `pd.DataFrame.pivot()`, `pd.DataFrame.pivot_table()`\n",
    "    * StaticFrame: `sf.Frame.pivot()`\n",
    "* Stacking & unstacking are types of pivots\n",
    "    * Pandas: `pd.DataFrame.stack()`, `pd.DataFrame.unstack()`\n",
    "    * StaticFrame: `sf.Frame.pivot_stack()`, `sf.Frame.pivot_unstack()`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2eb6efd",
   "metadata": {},
   "source": [
    "# Performance\n",
    "* In many situations StaticFrame can lead to more efficient systems\n",
    "* Code can be more efficient with memory\n",
    "    * Many operations naturally reuse immutable views\n",
    "    * No need for implementing defensive copies\n",
    "    * See also: https://static-frame.readthedocs.io/en/latest/articles/no_copy.html\n",
    "* Pandas still outperforms StaticFrame in some areas\n",
    "    * Some types of pivots\n",
    "    * Joining\n",
    "    * Windowing\n",
    "* Focus of current development is performance\n",
    "    * Profiling with `cprofile`, `pyinstrument`, `line-profiler` and `gprof2dot` (for call graph analysis)\n",
    "    * C-extensions in ArrayKit, AutoMap\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894432f7",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "* Some applications will benefit from more-specialized DataFrame libraries\n",
    "* More strict interfaces lead to more maintable code \n",
    "* An immutable data model reduces opportunities for error and permits more efficient memory usage\n",
    "* Higher-order Frame containers offer an efficient alternative to representing N-dimensional data with hierarchical indices"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}