biclustering-with-paul-revere.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Biclustering (with Paul Revere)\n",
    "\n",
    "\n",
    "ref: http://scikit-learn.org/stable/auto_examples/bicluster/bicluster_newsgroups.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Biclustering is a helpful clustering procedure that derives cluster membership for both rows and columns. Clustering rows and columns individually is fairly common. I view biclustering as an extension that allows me to ask: \"If these columns are clustered together, which rows (if any) point towards that clustering? Accordingly, if any rows are clustered together, which column values might tell me why they're clustered?\"\n",
    "\n",
    "I have found it helpful with:\n",
    "- Incidence Matrix / Cooccurence data (typically found before converting something into a graph/network structure)\n",
    "- Text data (the [original paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.140.3011&rep=rep1&type=pdf) on coclustering does this)\n",
    "\n",
    "The example below performs coclustering on the awesome \"Paul Revere dataset\" originally analyzed in an [amazing blog post](https://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere/) by [Kieran Healy](https://kieranhealy.org/).\n",
    "\n",
    "## References:\n",
    "> Biclustering algorithms simultaneously cluster rows and columns of a data matrix. These clusters of rows and columns are known as biclusters. Each determines a submatrix of the original data matrix with some desired properties.\n",
    "\n",
    "[sklearn - Biclustering](http://scikit-learn.org/stable/modules/biclustering.html#biclustering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "from sklearn.datasets.twenty_newsgroups import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cluster.bicluster import SpectralCoclustering\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\n",
    "    \"https://raw.githubusercontent.com/kjhealy/revere/master/data/PaulRevereAppD.csv\",\n",
    "    index_col=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>StAndrewsLodge</th>\n",
       "      <th>LoyalNine</th>\n",
       "      <th>NorthCaucus</th>\n",
       "      <th>LongRoomClub</th>\n",
       "      <th>TeaParty</th>\n",
       "      <th>BostonCommittee</th>\n",
       "      <th>LondonEnemies</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Adams.John</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Adams.Samuel</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Allen.Dr</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Appleton.Nathaniel</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ash.Gilbert</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    StAndrewsLodge  LoyalNine  NorthCaucus  LongRoomClub  \\\n",
       "Adams.John                       0          0            1             1   \n",
       "Adams.Samuel                     0          0            1             1   \n",
       "Allen.Dr                         0          0            1             0   \n",
       "Appleton.Nathaniel               0          0            1             0   \n",
       "Ash.Gilbert                      1          0            0             0   \n",
       "\n",
       "                    TeaParty  BostonCommittee  LondonEnemies  \n",
       "Adams.John                 0                0              0  \n",
       "Adams.Samuel               0                1              1  \n",
       "Allen.Dr                   0                0              0  \n",
       "Appleton.Nathaniel         0                1              0  \n",
       "Ash.Gilbert                0                0              0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.index = [i.replace(\".\", \", \") for i in data.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rows: 254 columns: 7\n"
     ]
    }
   ],
   "source": [
    "nrows, ncolumns = data.shape\n",
    "print(\"rows:\", nrows, \"columns:\", ncolumns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SpectralCoclustering(init='k-means++', mini_batch=False, n_clusters=6,\n",
       "                     n_init=10, n_jobs=None, n_svd_vecs=None, random_state=666,\n",
       "                     svd_method='arpack')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cocluster = SpectralCoclustering(n_clusters=6, random_state=666, svd_method=\"arpack\")\n",
    "cocluster.fit(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster:  0 \tMembers: 68 \tOrgs: 3\n",
      "Orgs:    LongRoomClub; BostonCommittee; LondonEnemies\n",
      "Members: Adams, John; Adams, Samuel; Appleton, Nathaniel; Austin, Benjamin; Austin, Samuel; Baldwin, Cyrus; Boyer, Peter; Boynton, Richard; Brackett, Jos; Bradford, John; Brimmer, Herman; Brimmer, Martin; Broomfield, Henry; Brown, Enoch; Brown, John; Cheever, Ezekiel; Church, Benjamin; Cooper, William; Davis, Caleb; Davis, Edward; Davis, William; Dawes, Thomas; Dennie, William; Dexter, Samuel; Fleet, Thomas; Foster, Bos; Gill, Moses; Greenleaf, Joseph; Greenleaf, William; Greenough, Newn; Hancock, Eben; Hancock, John; Hill, Alexander; Hopkins, Caleb; Isaac, Pierce; Ivers, James; Jarvis, Charles; Johnston, Eben; Lambert, John; Mackay, William; Marshall, Thomas; Marson, John; Mason, Jonathan; Noyces, Nat; Otis, James; Parkman, Elias; Partridge, Sam; Phillips, Samuel; Phillips, William; Pierpont, Robert; Pitts, John; Pitts, Samuel; Powell, William; Prince, Job; Pulling, John; Quincy, Josiah; Roylson, Thomas; Ruddock, Abiel; Sweetser, John; Tyler, Royall; Vernon, Fortesque; Waldo, Benjamin; Wendell, Oliver; Whitwell, Samuel; Whitwell, William; Williams, Jonathan; Winslow, John; Winthrop, John\n",
      "---\n",
      "Cluster:  1 \tMembers: 81 \tOrgs: 1\n",
      "Orgs:    TeaParty\n",
      "Members: Barnard, Samuel; Bolter, Thomas; Bradlee, David; Bradlee, Josiah; Bradlee, Nathaniel; Bradlee, Thomas; Bewer, James; Bruce, Stephen; Burton, Benjamin; Campbell, Nicholas; Clarke, Benjamin; Cochran, John; Colesworthy, Gilbert; Collier, Gershom; Crane, John; Davis, Robert; Dolbear, Edward; Eaton, Joseph; Eckley, Unknown; Etheridge, William; Fenno, Samuel; Foster, Samuel; Frothingham, Nathaniel; Gammell, John; Gore, Samuel; Greene, Nathaniel; Hammond, Samuel; Hendley, William; Hewes, George; Hicks, John; Hobbs, Samuel; Hooton, John; Howard, Samuel; Howe, Edward; Hunnewell, Jonathan; Hunnewell, Richard; Hunstable, Thomas; Hunt, Abraham; Ingersoll, Daniel; Kinnison, David; Lee, Joseph; Lincoln, Amos; Loring, Matthew; Machin, Thomas; MacKintosh, Capt; MacNeil, Archibald; May, John; Melville, Thomas; Moore, Thomas; Morse, Anthony; Mountford, Joseph; Newell, Eliphelet; Palmer, Joseph; Parker, Jonathan; Payson, Joseph; Peters, John; Pierce, William; Pitts, Lendall; Porter, Thomas; Prentiss, Henry; Prince, John; Purkitt, Henry; Randall, John; Roby, Joseph; Russell, John; Russell, William; Sessions, Robert; Shed, Joseph; Simpson, Benjamin; Slater, Peter; Spear, Thomas; Sprague, Samuel; Spurr, John; Starr, James; Stearns, Phineas; Stevens, Ebenezer; Wheeler, Josiah; Williams, Jeremiah; Williams, Thomas; Willis, Nathaniel; Wyeth, Joshua\n",
      "---\n",
      "Cluster:  2 \tMembers: 46 \tOrgs: 1\n",
      "Orgs:    StAndrewsLodge\n",
      "Members: Ash, Gilbert; Bell, William; Blake, Increase; Bray, George; Brown, Hugh; Burbeck, Edward; Burbeck, William; Cailleteau, Edward; Callendar, Elisha; Chipman, Seth; Collins, Ezra; Deshon, Moses; Doyle, Peter; Ferrell, Ambrose; Flagg, Josiah; Gould, William; Graham, James; Gray, Wait; Ham, William; Hitchborn, Nathaniel; Hoffins, John; Inglish, Alexander; Jarvis, Edward; Jefferds, Unknown; Jenkins, John; Kerr, Walter; Lewis, Phillip; Marett, Phillip; Marlton, John; McAlpine, William; Milliken, Thomas; Moody, Samuel; Nicholls, Unknown; Obear, Israel; Palfrey, William; Phillips, John; Potter, Edward; Pulling, Richard; Seward, James; Sloper, Ambrose; Stanbridge, Henry; Tabor, Philip; Webb, Joseph; Webster, Thomas; Whitten, John; Wingfield, William\n",
      "---\n",
      "Cluster:  3 \tMembers: 12 \tOrgs: 0\n",
      "Orgs:    \n",
      "Members: Bass, Henry; Chase, Thomas; Collson, Adam; Condy, JamesFoster; Cooper, Samuel; Eayres, Joseph; Grant, Moses; Molineux, William; Proctor, Edward; Story, Elisha; Swan, James; Young, Thomas\n",
      "---\n",
      "Cluster:  4 \tMembers: 37 \tOrgs: 1\n",
      "Orgs:    NorthCaucus\n",
      "Members: Allen, Dr; Avery, John; Ballard, John; Barber, Nathaniel; Boit, John; Breck, William; Burt, Benjamin; Cazneau, Capt; Chadwell, Mr; Champney, Caleb; Chrysty, Thomas; Edes, Benjamin; Emmes, Samuel; Hickling, William; Hitchborn, Thomas; Holmes, Nathaniel; Hoskins, William; Johonnott, Gabriel; Kent, Benjamin; Kimball, Thomas; Lowell, John; Matchett, John; Merrit, John; Morton, Perez; Palms, Richard; Pearce, IsaacJun; Pearce, Isaac; Peck, Thomas; Sharp, Gibbens; Sigourney, John; Stoddard, Asa; Stoddard, Jonathan; Symmes, Eben; Symmes, John; Tileston, Thomas; Warren, Joseph; White, Samuel\n",
      "---\n",
      "Cluster:  5 \tMembers: 10 \tOrgs: 1\n",
      "Orgs:    LoyalNine\n",
      "Members: Barrett, Samuel; Cleverly, Stephen; Crafts, Thomas; Field, Joseph; Peck, Samuel; Revere, Paul; Smith, John; Trott, George; Urann, Thomas; Welles, Henry\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "cluster_data = []\n",
    "for cluster in range(cocluster.n_clusters):\n",
    "    row_ix, col_ix = cocluster.get_indices(cluster)\n",
    "    data_subset = data.iloc[row_ix, col_ix]\n",
    "    c_rows, c_cols = data_subset.shape\n",
    "    orgs = data_subset.columns\n",
    "    names = data_subset.index\n",
    "    print(\"Cluster: \", cluster, \"\\tMembers:\", c_rows, \"\\tOrgs:\", c_cols)\n",
    "    print(\"Orgs:   \", \"; \".join([\"{}\".format(org) for org in orgs]))\n",
    "    print(\"Members:\", \"; \".join([\"{}\".format(name) for name in names]))\n",
    "    print(\"---\")\n",
    "    cluster = {\n",
    "        \"Cluster\": cluster,\n",
    "        \"Members\": \"; \".join([\"{}\".format(name) for name in names]),\n",
    "        \"Orgs\": \"; \".join([\"{}\".format(org) for org in orgs]),\n",
    "    }\n",
    "    cluster_data.append(cluster)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_info = pd.DataFrame(cluster_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cluster</th>\n",
       "      <th>Members</th>\n",
       "      <th>Orgs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Adams, John; Adams, Samuel; Appleton, Nathanie...</td>\n",
       "      <td>LongRoomClub; BostonCommittee; LondonEnemies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Barnard, Samuel; Bolter, Thomas; Bradlee, Davi...</td>\n",
       "      <td>TeaParty</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Ash, Gilbert; Bell, William; Blake, Increase; ...</td>\n",
       "      <td>StAndrewsLodge</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Bass, Henry; Chase, Thomas; Collson, Adam; Con...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Allen, Dr; Avery, John; Ballard, John; Barber,...</td>\n",
       "      <td>NorthCaucus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>Barrett, Samuel; Cleverly, Stephen; Crafts, Th...</td>\n",
       "      <td>LoyalNine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Cluster                                            Members  \\\n",
       "0        0  Adams, John; Adams, Samuel; Appleton, Nathanie...   \n",
       "1        1  Barnard, Samuel; Bolter, Thomas; Bradlee, Davi...   \n",
       "2        2  Ash, Gilbert; Bell, William; Blake, Increase; ...   \n",
       "3        3  Bass, Henry; Chase, Thomas; Collson, Adam; Con...   \n",
       "4        4  Allen, Dr; Avery, John; Ballard, John; Barber,...   \n",
       "5        5  Barrett, Samuel; Cleverly, Stephen; Crafts, Th...   \n",
       "\n",
       "                                           Orgs  \n",
       "0  LongRoomClub; BostonCommittee; LondonEnemies  \n",
       "1                                      TeaParty  \n",
       "2                                StAndrewsLodge  \n",
       "3                                                \n",
       "4                                   NorthCaucus  \n",
       "5                                     LoyalNine  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format and Sort DataFrame with MultiIndex\n",
    "Since rows and columns now have hierarchical information with cluster membership, we can reindex the dataframe to take this into account and index easily on cluster. This should allow us to see the block cluster structure along the diagonal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_mi = data.copy()\n",
    "\n",
    "cocluster_rows = list(zip(cocluster.row_labels_, data.index))\n",
    "row_index = pd.MultiIndex.from_tuples(cocluster_rows, names=[\"cluster\", \"person\"])\n",
    "data_mi.index = row_index\n",
    "data_mi = data_mi.sort_index(axis=0)\n",
    "\n",
    "cocluster_columns = list(zip(cocluster.column_labels_, data.columns))\n",
    "col_index = pd.MultiIndex.from_tuples(cocluster_columns, names=[\"cluster\", \"org\"])\n",
    "data_mi.columns = col_index\n",
    "data_mi = data_mi.sort_index(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>cluster</th>\n",
       "      <th colspan=\"3\" halign=\"left\">0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>org</th>\n",
       "      <th>BostonCommittee</th>\n",
       "      <th>LondonEnemies</th>\n",
       "      <th>LongRoomClub</th>\n",
       "      <th>TeaParty</th>\n",
       "      <th>StAndrewsLodge</th>\n",
       "      <th>NorthCaucus</th>\n",
       "      <th>LoyalNine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster</th>\n",
       "      <th>person</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">0</th>\n",
       "      <th>Adams, John</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Adams, Samuel</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Appleton, Nathaniel</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Austin, Benjamin</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Austin, Samuel</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Baldwin, Cyrus</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boyer, Peter</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boynton, Richard</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brackett, Jos</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bradford, John</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "cluster                                   0                             \\\n",
       "org                         BostonCommittee LondonEnemies LongRoomClub   \n",
       "cluster person                                                           \n",
       "0       Adams, John                       0             0            1   \n",
       "        Adams, Samuel                     1             1            1   \n",
       "        Appleton, Nathaniel               1             0            0   \n",
       "        Austin, Benjamin                  0             1            0   \n",
       "        Austin, Samuel                    0             1            0   \n",
       "        Baldwin, Cyrus                    0             1            0   \n",
       "        Boyer, Peter                      0             1            0   \n",
       "        Boynton, Richard                  1             1            0   \n",
       "        Brackett, Jos                     0             1            0   \n",
       "        Bradford, John                    1             1            0   \n",
       "\n",
       "cluster                            1              2           4         5  \n",
       "org                         TeaParty StAndrewsLodge NorthCaucus LoyalNine  \n",
       "cluster person                                                             \n",
       "0       Adams, John                0              0           1         0  \n",
       "        Adams, Samuel              0              0           1         0  \n",
       "        Appleton, Nathaniel        0              0           1         0  \n",
       "        Austin, Benjamin           0              0           0         0  \n",
       "        Austin, Samuel             0              0           0         0  \n",
       "        Baldwin, Cyrus             0              0           0         0  \n",
       "        Boyer, Peter               0              0           0         0  \n",
       "        Boynton, Richard           0              0           0         0  \n",
       "        Brackett, Jos              0              0           0         0  \n",
       "        Bradford, John             0              0           0         0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mi.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify Coclustering structure\n",
    "\"The resulting bicluster structure is block-diagonal, since each row and each column belongs to exactly one bicluster.\" - [sklearn SpectralCoclustering Docs](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.bicluster.SpectralCoclustering.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6YAAACdCAYAAACn12UYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAUAUlEQVR4nO3df7DldX3f8edrF1ABrWt2ZVJ+iCbUSjqGZe9C7BoqpjFinGBbJdKYpKQzWztsxUinxfSHMTOZxsloTEKk3Si/Jv4YmkhDIyKMhYI0rntX1vBLKoMgMJDdxY1BcaAL7/5xvrternvu/d4f3/s9557nY+bMOd9fn8/73Pu+33ve8/18PydVhSRJkiRJfVnTdwCSJEmSpMlmYSpJkiRJ6pWFqSRJkiSpVxamkiRJkqReWZhKkiRJknplYSpJkiRJ6tVIFaZJ3pzkviT3J7mk73ik+SR5MMmdSXYnmW7WvSzJTUm+0Tyv6ztOCSDJ5Un2JLlrxrrD5msG/qA5H/9VktP7i1yTbkju/maSR5vz7+4kb5mx7f1N7t6X5Of6iVoaSHJikpuT3JPk7iQXNes9/2qkzZG7nZx/R6YwTbIW+CPgHOBU4Pwkp/YbldTK2VV1WlVNNcuXAF+sqlOALzbL0ii4EnjzrHXD8vUc4JTmsRW4bIVilA7nSn44dwF+rzn/nlZV1wM0nx3eCfxEc8zHms8YUl8OABdX1anATwEXNnnq+VejbljuQgfn35EpTIEzgPur6oGqegb4DHBuzzFJi3EucFXz+irgbT3GIh1SVbcC3561eli+ngtcXQNfBl6a5EdXJlLp+Ybk7jDnAp+pqqer6pvA/Qw+Y0i9qKrHquqrzesngXuB4/H8qxE3R+4Os6Tz7ygVpscDD89YfoS537g0Cgq4McmuJFubdcdV1WPN68eB4/oJTWplWL56TtY42NYMdbx8xm0T5q5GVpKTgY3ADjz/aozMyl3o4Pw7SoWpNI5eX1WnMxh2c2GSs2ZurKpiULxKI8981Zi5DPgx4DTgMeDD/YYjzS3JscCfAe+tqr+duc3zr0bZYXK3k/PvKBWmjwInzlg+oVknjayqerR53gNcy2C4wl8fHHLTPO/pL0JpXsPy1XOyRlpV/XVVPVtVzwF/zA+Gi5m7GjlJjmTwwf6TVfXZZrXnX428w+VuV+ffUSpMdwKnJHllkqMY3Dh7Xc8xSUMlOSbJiw++Bt4E3MUgb3+12e1XgT/vJ0KplWH5eh3wK83skD8FfGfGkDOpd7PuufsnDM6/MMjddyZ5QZJXMphA5isrHZ90UJIAnwDuraqPzNjk+VcjbVjudnX+PWLpIS+PqjqQZBvwBWAtcHlV3d1zWNJcjgOuHfzNcgTwqaq6IclO4Jok/xJ4CDivxxilQ5J8GngDsD7JI8AHgN/h8Pl6PfAWBhMXPAVcsOIBS40hufuGJKcxGP74IPCvAKrq7iTXAPcwmFHywqp6to+4pcYW4JeBO5Psbtb9Bp5/NfqG5e75XZx/MxjSLkmSJElSP0ZpKK8kSZIkaQJZmEqSJEmSemVhKkmSJEnqlYWpJEmSJKlXFqaSJEmSpF6NXGGaZGvfMUiLZf5qnJm/Gmfmr8aVuatxtpz5O3KFKeAfp8aZ+atxZv5qnJm/GlfmrsbZqi5MJUmSJEkTJFXVdwyHrF+/vo4+5hg2rN/QdyjSouzdt9f81dgyfzXOzF+ttDvu/dbQbRtfc1LrdlYyd2fHvJA4l9soxTJp5srd2eb6vdxx77eoA98nR7xozuNm9lfPPEkd+H4O195IFaabNk3V7Tum+w5DkiRJmtO6zduGbtu/89IVjKS92TH3GecoxTJp5srd2eb6vbT9G5i539P3XcNzT+05bGHqUF5JkiRJUq8sTCVJkiRJveq0ME3y0iR/muTrSe5N8rou+5MkSZIkjZ8jOm7/94EbqurtSY4Cju64P0mSJEnSmOmsME3yd4CzgH8BUFXPAM901Z8kSZIkaTx1OZT3lcBe4IokdyT5eJJjOuxPkiRJkjSGuixMjwBOBy6rqo3A94BLZu+UZGuS6STTe/ft7TAcSZIkSdIo6rIwfQR4pKp2NMt/yqBQfZ6q2l5VU1U15RdjS5IkSdLk6awwrarHgYeTvLpZ9TPAPV31J0mSJEkaT13PyvtvgE82M/I+AFzQcX+SJEmSpDHTaWFaVbuBqS77kCRJkiSNty7vMZUkSZIkaV4WppIkSZKkXlmYSpIkSZJ6larqO4ZD1hz98nrBq89rte/+nZd2HM3yWLd529Bty/EeZre/2DZntjMuP9vlslw/w+Xof66++45ztknOGUmS5vqMN5eF/K9fbrP77ro/jY6VrjuGfU7ccuYUu3ZN53DHeMVUkiRJktQrC1NJkiRJUq8sTCVJkiRJver0e0yTPAg8CTwLHKgqv9NUkiRJkvQ8nRamjbOrat8K9CNJkiRJGkMO5ZUkSZIk9WpRhWmSo1ruWsCNSXYl2bqYviRJkiRJq9u8hWmSW5KcPGP5DGBny/ZfX1WnA+cAFyY56zDtb00ynWS6Dny/ZbOSJEmSpNWizT2m/wW4IckfAMczKDIvaNN4VT3aPO9Jci1wBnDrrH22A9sB1hz98mofuiRJkiRpNZi3MK2qLyR5N3ATsA/YWFWPz3dckmOANVX1ZPP6TcBvLTVgSZIkSdLqMm9hmuQ/AecBZwGvBW5JcnFVfW6eQ48Drk1ysJ9PVdUNS4xXkiRJkrTKtBnK+yPAGVX1feAvk9wAfByYszCtqgeAn1x6iJIkSZKk1azNUN73AiQ5uqqeqqqHgJ/tPDJJkiRJ0kRoMyvv65LcA3y9Wf7JJB/rPDJJkiRJ0kRo8z2mHwV+DngCoKq+xuB+U0mSJEmSlixVc39DS5IdVXVmkjuqamOz7mtVtez3j27aNFW375he7mYlSdIqsm7ztr5DkNi/89K+Q5CeZ+a5cXZ+Lse22eb6Gxh23NP3XcNzT+3J4ba1mfzo4ST/EKgkRwIXAfe2OE6SJEmSpHm1Gcr7buBC4HjgUeC0ZlmSJEmSpCVrMyvvPuCXViAWSZIkSdIEGlqYJvlDYOgNqFX1njYdJFkLTAOPVtVbFxyhJEmSJGlVm2so7zSwC3ghcDrwjeZxGnDUAvrwnlRJkiRJ0lBDr5hW1VUASf418PqqOtAs/1fgtjaNJzkB+Hngt4H3LTlaSZIkSdKq02byo3XAS2YsH9usa+OjwL8DnltgXJIkSZKkCdHm62J+B7gjyc1AgLOA35zvoCRvBfZU1a4kb5hjv63AVoATTzqpRTiSJEmSpNWkzay8VyT5PHBms+rfV9XjLdreAvxCkrcwuE/1JUn+pKreNav97cB2gE2bpoZOtiRJkiRJWp3aDOUFWAvsBfYDfy/JWfMdUFXvr6oTqupk4J3A/5pdlEqSJEmSNO8V0yQfAn4RuJsf3CtawK0dxiVJkiRJmhBt7jF9G/Dqqnp6sZ1U1S3ALYs9XpIkSZK0erUZyvsAcGTXgUiSJEmSJlObK6ZPAbuTfBE4dNW0qt7TWVSSJEmSpInRpjC9rnlIkiRJkrTs2nxdzFUrEYik8bVu87a+Q5A0QfbvvLTvEKSx+d/X9u9l9vvx72z8zPU7W+y25TCz/S1nfnnofkML0yTXVNV5Se5kMAvv81TVa5cYoyRJkiRJc14xvah5futKBCJJkiRJmkxDC9Oqeqx5fmjlwpEkSZIkTZo2XxezKElemOQrSb6W5O4kH+yqL0mSJEnS+GozK+9iPQ28saq+m+RI4EtJPl9Vw+94lSRJkiRNnM4K06oq4LvN4pHN44cmUZIkSZIkTbZ5h/Im2ZLkpiT/N8kDSb6Z5IE2jSdZm2Q3sAe4qap2LDVgSZIkSdLq0uaK6SeAXwd2Ac8upPGqehY4LclLgWuT/IOqumvmPkm2AlsBTjzppIU0L0mSJElaBdpMfvSdqvp8Ve2pqicOPhbSSVX9DXAz8ObDbNteVVNVNbVh/YaFNCtJkiRJWgXaFKY3J/ndJK9LcvrBx3wHJdnQXCklyYuAnwW+vsR4JUmSJEmrTJuhvGc2z1Mz1hXwxnmO+1HgqiRrGRTA11TVXyw8REmSJEnSajZvYVpVZy+m4ar6K2DjYo6VJEmSJE2OeQvTZjjurwAnz9y/qt7TXViSJEmSpEnRZijv9cCXgTuB57oNR5IkSZI0adoUpi+sqvd1HokkSZIkaSKlqubeIfl14LvAXwBPH1xfVd9e7mDWHP3yesGrzzvstv07L13u7lbcus3bhm5byPub2c7s45arj+WIZZT7aNvm7J9n13m42P5WIs6V/llIkrQarcRnJ02mrnNrOT4Lbjlzil27pnO4bW2umD4D/C7wHxjMxkvz/KoFRyJJkiRJ0ixtCtOLgR+vqn1dByNJkiRJmjxrWuxzP/BU14FIkiRJkiZTmyum3wN2J7mZ599jOufXxSQ5EbgaOI7B0N/tVfX7S4hVkiRJkrQKtSlM/0fzWKgDwMVV9dUkLwZ2Jbmpqu5ZRFuSJEmSpFVq3sK0qq5aTMNV9RjwWPP6yST3AscDFqaSJEmSpEPmLUyTfJMfzMZ7SFW1npU3ycnARmDHAmKTJEmSJE2ANkN5p2a8fiHwDuBlbTtIcizwZ8B7q+pvD7N9K7AVgCOPbdusJEmSJGmVmHdW3qp6Ysbj0ar6KPDzbRpPciSDovSTVfXZIe1vr6qpqprKES9aUPCSJEmSpPHXZijv6TMW1zC4gtrmuACfAO6tqo8sOkJJkiRJ0qrWZijvh2e8PgA8CJzX4rgtwC8DdybZ3az7jaq6fkERSpIkSZJWtTaz8p69mIar6ktAFnOsJEmSJGlyDC1Mk7xvrgMdnitJkiRJWg5zXTF98YpFIUmSJEmaWEML06r64EoGIkmSJEmaTG1m1z0B+EMGkxkB3AZcVFWPLHcwG19zErfvuPTQ8rrN24buO9e2cbF/Z7v3OtdxC9m20j+z5eqvi/fU9mc/u++Z+861bbEW0ubMfbuIZa7+Zvcx1+9IkqTVqO3/6Nn7dfEZpK2FxOL/9tVlHD63zfs9psAVwHXA320e/7NZJ0mSJEnSkrUpTDdU1RVVdaB5XAls6DguSZIkSdKEaFOYPpHkXUnWNo93AU90HZgkSZIkaTK0KUx/DTgPeBx4DHg7cMF8ByW5PMmeJHctLURJkiRJ0mo27+RHVfUQ8AuLaPtK4FLg6kUcK0mSJEmaEPNeMU1yVZKXzlhel+Ty+Y6rqluBby8xPkmSJEnSKtdmKO9rq+pvDi5U1X5gY3chSZIkSZImSZvCdE2SdQcXkryMFkOA20qyNcl0kum9+/YuV7OSJEmSpDHRpsD8MPCXSf57s/wO4LeXK4Cq2g5sB9i0aaqWq11JkiRJ0nhoM/nR1UmmgTc2q/5pVd3TbViSJEmSpEnRakhuU4guqBhN8mngDcD6JI8AH6iqTyw4QkmSJEnSqrZs94rOVlXnd9W2JEmSJGn1aDP5kSRJkiRJnbEwlSRJkiT1ysJUkiRJktSrVI3ON7Rs2jRVt++YPrS8bvO2ofvu33npovqY2ebsNubathxmv5+Zfcy1bbFtLiW2PmNZiT6W43e9kFja9tdFm8t1nCRJWjkr/dlwJT7HqVvj8Blvy5lT7No1ncNt84qpJEmSJKlXFqaSJEmSpF5ZmEqSJEmSetVpYZrkzUnuS3J/kku67EuSJEmSNJ46K0yTrAX+CDgHOBU4P8mpXfUnSZIkSRpPXV4xPQO4v6oeqKpngM8A53bYnyRJkiRpDHVZmB4PPDxj+ZFmnSRJkiRJh/Q++VGSrUmmk0zv3be373AkSZIkSSusy8L0UeDEGcsnNOuep6q2V9VUVU1tWL+hw3AkSZIkSaOoy8J0J3BKklcmOQp4J3Bdh/1JkiRJksbQEV01XFUHkmwDvgCsBS6vqru76k+SJEmSNJ46K0wBqup64Pou+5AkSZIkjbfeJz+SJEmSJE02C1NJkiRJUq8sTCVJkiRJvUpV9R3DIUn2At8D9vUdi7RI6zF/Nb7MX40z81fjytzVOFto/r6iqg77HaEjVZgCJJmuqqm+45AWw/zVODN/Nc7MX40rc1fjbDnz16G8kiRJkqReWZhKkiRJkno1ioXp9r4DkJbA/NU4M39XQJLfSvKPF3jMg0nWdxXTKmH+alyZuxpny5a/I3ePqSRJoyZJGPzPfK6n/h8EpqrKCVIkSavSKF4xlSSpd0lOTnJfkquBu4ATk5yf5M4kdyX5ULPfO5J8pHl9UZIHmtevSnL7Ydq9Msnbm9cPJvlgkq827f79Zv2PJLkxyd1JPg5kxvHvSvKVJLuT/Lcka5O8Isk3kqxPsibJbUne1PkPSZKkZWJhKknScKcAH6uqnwD+H/Ah4I3AacDmJG8DbgN+utn/p4EnkhzfvL61RR/7qup04DLg3zbrPgB8qen3WuAkgCSvAX4R2FJVpwHPAr9UVQ81sV0GXAzcU1U3LumdS5K0gixMJUka7qGq+nLzejNwS1XtraoDwCeBs6rqceDYJC8GTgQ+BZzFoDC9rUUfn22edwEnN6/PAv4EoKo+B+xv1v8MsAnYmWR3s/yqZr+PAy8B3s0PClxJksbCEX0HIEnSCPtey/3+D3ABcB+DYvTXgNcxuHo5n6eb52eZ//9ygKuq6v0/tCE5GjihWTwWeLJF35IkjQSvmEqS1M5XgH/U3Me5Fjgf+N/NttsYXKW8FbgDOBt4uqq+s8i+bgX+OUCSc4B1zfovAm9P8vJm28uSvKLZ9iEGV3H/M/DHi+xXkqReeMVUkqQWquqxJJcANzO4cvm5qvrzZvNtDIbx3lpVzyZ5GPj6Err7IPDpJHczuBr7rSaGe5L8R+DGJGsY3Pd6YZKTGQw13tL0/8+SXFBVVywhBkmSVoxfFyNJkiRJ6pVDeSVJkiRJvbIwlSRJkiT1ysJUkiRJktQrC1NJkiRJUq8sTCVJkiRJvbIwlSRJkiT1ysJUkiRJktQrC1NJkiRJUq/+P7C4g0NYCGw1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1152x144 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(data_mi.values.T, cmap=plt.cm.Blues, aspect=\"auto\", origin=\"lower\")\n",
    "plt.xlabel(\"row index\")\n",
    "plt.ylabel(\"column index\")\n",
    "plt.show()\n",
    "# I plotted the transpose so it doesn't give us a long image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief Analysis\n",
    "Let's take a look and see where the actual Loyal Nine members ended up. The organization ended up in our `Cluster 5`, but how many members of that bicluster match?\n",
    "\n",
    "Loyal Nine [members](https://books.google.com/books?id=iNgNCgAAQBAJ&pg=PA26#v=onepage&q&f=false\n",
    "):\n",
    "\n",
    "- John Avery, a distiller and club secretary;\n",
    "- John Smith and Stephen Cleverly, both braziers;\n",
    "- Thomas Crafts, a printer;\n",
    "- Benjamin Edes, who along with John Gill produced the important Boston Gazette;\n",
    "- Thomas Chase, a distiller;\n",
    "- Joseph Field, a ship's captain;\n",
    "- George Trott, a jeweler;\n",
    "- Henry Bass, a merchant related to Samuel Adams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "loyal_nine = [\n",
    "    \"Avery, John\",\n",
    "    \"Smith, John\",\n",
    "    \"Cleverly, Stephen\",\n",
    "    \"Crafts, Thomas\",\n",
    "    \"Edes, Benjamin\",\n",
    "    \"Chase, Thomas\",\n",
    "    \"Field, Joseph\",\n",
    "    \"Trott, George\",\n",
    "    \"Bass, Henry\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>cluster</th>\n",
       "      <th colspan=\"3\" halign=\"left\">0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>org</th>\n",
       "      <th>BostonCommittee</th>\n",
       "      <th>LondonEnemies</th>\n",
       "      <th>LongRoomClub</th>\n",
       "      <th>TeaParty</th>\n",
       "      <th>StAndrewsLodge</th>\n",
       "      <th>NorthCaucus</th>\n",
       "      <th>LoyalNine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cluster</th>\n",
       "      <th>person</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>Bass, Henry</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chase, Thomas</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
       "      <th>Avery, John</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Edes, Benjamin</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">5</th>\n",
       "      <th>Cleverly, Stephen</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Crafts, Thomas</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Field, Joseph</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Smith, John</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Trott, George</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "cluster                                 0                                   1  \\\n",
       "org                       BostonCommittee LondonEnemies LongRoomClub TeaParty   \n",
       "cluster person                                                                  \n",
       "3       Bass, Henry                     0             1            0        1   \n",
       "        Chase, Thomas                   0             1            0        1   \n",
       "4       Avery, John                     0             1            0        0   \n",
       "        Edes, Benjamin                  0             0            0        0   \n",
       "5       Cleverly, Stephen               0             0            0        0   \n",
       "        Crafts, Thomas                  0             0            0        0   \n",
       "        Field, Joseph                   0             0            0        0   \n",
       "        Smith, John                     0             0            0        0   \n",
       "        Trott, George                   0             0            0        0   \n",
       "\n",
       "cluster                                2           4         5  \n",
       "org                       StAndrewsLodge NorthCaucus LoyalNine  \n",
       "cluster person                                                  \n",
       "3       Bass, Henry                    0           1         1  \n",
       "        Chase, Thomas                  0           1         1  \n",
       "4       Avery, John                    0           0         1  \n",
       "        Edes, Benjamin                 0           1         1  \n",
       "5       Cleverly, Stephen              0           0         1  \n",
       "        Crafts, Thomas                 1           0         1  \n",
       "        Field, Joseph                  0           0         1  \n",
       "        Smith, John                    0           0         1  \n",
       "        Trott, George                  0           0         1  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_mi[data_mi.index.get_level_values(\"person\").isin(loyal_nine)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `Cluster 5` we recovered all individuals who were *exclusively* members of the Loyal Nine, plus Thomas Crafts. `Cluster 4` pulled two Loyal Nine members also part of the North Caucus and London Enemies, and `Cluster 3` holds members that were additionally part of the North Cacus, Tea Party, and London Enemies each. Also note Cluster 3 contains no organizations -- it's members are part of several organizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export to Excel\n",
    "It's helpful to export this to Excel because the MultiIndex formats nicely into merged cells and retains the hierarchical indexing structure.\n",
    "\n",
    "The data is on the `data` sheet, and cluster details are on the `clusters` sheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sheets = [(data_mi, \"data\"), (cluster_info, \"clusters\")]\n",
    "\n",
    "filename = \"bicluster.xlsx\"\n",
    "writer = pd.ExcelWriter(filename, engine=\"xlsxwriter\")\n",
    "for data in data_sheets:\n",
    "    data[0].to_excel(writer, sheet_name=data[1])\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}