brainome_302_Generating_Probabilities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cfebebd9",
   "metadata": {},
   "source": [
    "![brainome logo](./images/brainome_logo.png)\n",
    "# 302 Generating Probabilities\n",
    "Brainome's *Random Forest* and *Neural Network* model predictors can also generate probabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "900275fe",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "This notebook assumes brainome is installed as per notebook [brainome_101_Quick_Start](./brainome_101_Quick_Start.ipynb)\n",
    "\n",
    "The data sets are:\n",
    "* [titanic_train.csv](https://download.brainome.ai/data/public/titanic_train.csv) for training data\n",
    "* [titanic_predict.csv](https://download.brainome.ai/data/public/titanic_predict.csv) for predictions\n",
    "\n",
    "\n",
    "Predictors require numpy to run and optionally scipy to generate a confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "655a3e12",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[33mDEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621\u001B[0m\n",
      "brainome v1.006-18-beta\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "-rw-r--r--  1 andys  staff   858B Sep 20 16:55 titanic_predict.csv\n"
     ]
    }
   ],
   "source": [
    "!python3 -m pip install brainome --quiet\n",
    "!brainome --version\n",
    "\n",
    "import urllib.request as request\n",
    "response2 = request.urlretrieve('https://download.brainome.ai/data/public/titanic_predict.csv', 'titanic_predict.csv')\n",
    "%ls -lh titanic_predict.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2062bec9",
   "metadata": {},
   "source": [
    "## Generate a predictor\n",
    "The predictor filename is `predictor_302.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a287f5b4",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Could not detect a GPU. Neural Network generation will be slow.\n",
      "\n",
      "\n",
      "Created predictor_302.py\n",
      "-rw-r--r--  1 andys  staff    38K Sep 20 16:56 predictor_302.py\n"
     ]
    }
   ],
   "source": [
    "!brainome https://download.brainome.ai/data/public/titanic_train.csv -y -o predictor_302.py -modelonly -q\n",
    "print('\\nCreated predictor_302.py')\n",
    "!ls -lh predictor_302.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33177319",
   "metadata": {},
   "source": [
    "## Generating classification probabilities for a data set\n",
    "Rather than picking a single class, this feature outputs the probabilities for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f4dc7f42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[33mDEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621\u001B[0m\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.9/site-packages (1.3.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.9/site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.9/site-packages (from pandas) (2021.1)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.9/site-packages (from pandas) (1.20.0)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas) (1.16.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "--------------------------- Prediction Probabilities ---------------------------\n",
      "[['died' 'survived']\n",
      " ['0.10347254522374348' '0.8965274547762565']\n",
      " ['0.8357493590851747' '0.16425064091482533']\n",
      " ['0.5723091324568758' '0.42769086754312424']\n",
      " ['0.8357493590851747' '0.16425064091482533']\n",
      " ['0.8833638515749265' '0.11663614842507353']\n",
      " ['0.8204901964380503' '0.17950980356194968']\n",
      " ['0.8215620538923634' '0.17843794610763664']\n",
      " ['0.10347254522374348' '0.8965274547762565']\n",
      " ['0.42322660063987716' '0.5767733993601228']\n",
      " ['0.7495210975955352' '0.2504789024044648']\n",
      " ['0.8833638515749265' '0.11663614842507353']]\n"
     ]
    }
   ],
   "source": [
    "# using pandas to read csv data\n",
    "%pip install pandas\n",
    "import pandas as pd\n",
    "import predictor_302 as predictor\n",
    "# reading csv file\n",
    "predict_data = pd.read_csv('titanic_predict.csv', na_values=[], na_filter=False)\n",
    "# REQUIRED: strip the headers from dataset\n",
    "predict_values = predict_data.values\n",
    "probabilities_output = predictor.predict(predict_values, return_probabilities=True)\n",
    "print(' Prediction Probabilities '.center(80, '-'))\n",
    "print(probabilities_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11de2d52",
   "metadata": {},
   "source": [
    "## Combining probabilities into the source data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b2253144",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PassengerId</td>\n",
       "      <td>Cabin_Class</td>\n",
       "      <td>Name</td>\n",
       "      <td>Sex</td>\n",
       "      <td>Age</td>\n",
       "      <td>Sibling_Spouse</td>\n",
       "      <td>Parent_Children</td>\n",
       "      <td>Ticket_Number</td>\n",
       "      <td>Fare</td>\n",
       "      <td>Cabin_Number</td>\n",
       "      <td>Port_of_Embarkation</td>\n",
       "      <td>died</td>\n",
       "      <td>survived</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>881</td>\n",
       "      <td>2</td>\n",
       "      <td>Shelley, Mrs. William (Imanita Parrish Hall)</td>\n",
       "      <td>female</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>230433</td>\n",
       "      <td>26.0</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.10347254522374348</td>\n",
       "      <td>0.8965274547762565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>882</td>\n",
       "      <td>3</td>\n",
       "      <td>Markun, Mr. Johann</td>\n",
       "      <td>male</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349257</td>\n",
       "      <td>7.8958</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.8357493590851747</td>\n",
       "      <td>0.16425064091482533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>883</td>\n",
       "      <td>3</td>\n",
       "      <td>Dahlberg, Miss. Gerda Ulrika</td>\n",
       "      <td>female</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7552</td>\n",
       "      <td>10.5167</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.5723091324568758</td>\n",
       "      <td>0.42769086754312424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>884</td>\n",
       "      <td>2</td>\n",
       "      <td>Banfield, Mr. Frederick James</td>\n",
       "      <td>male</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>C.A./SOTON 34068</td>\n",
       "      <td>10.5</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.8357493590851747</td>\n",
       "      <td>0.16425064091482533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>885</td>\n",
       "      <td>3</td>\n",
       "      <td>Sutehall, Mr. Henry Jr</td>\n",
       "      <td>male</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>SOTON/OQ 392076</td>\n",
       "      <td>7.05</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.8833638515749265</td>\n",
       "      <td>0.11663614842507353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>886</td>\n",
       "      <td>3</td>\n",
       "      <td>Rice, Mrs. William (Margaret Norton)</td>\n",
       "      <td>female</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>382652</td>\n",
       "      <td>29.125</td>\n",
       "      <td></td>\n",
       "      <td>Q</td>\n",
       "      <td>0.8204901964380503</td>\n",
       "      <td>0.17950980356194968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>887</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.0</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.8215620538923634</td>\n",
       "      <td>0.17843794610763664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.0</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>0.10347254522374348</td>\n",
       "      <td>0.8965274547762565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>889</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen Carrie\"\"</td>\n",
       "      <td>female</td>\n",
       "      <td></td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.45</td>\n",
       "      <td></td>\n",
       "      <td>S</td>\n",
       "      <td>0.42322660063987716</td>\n",
       "      <td>0.5767733993601228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.0</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>0.7495210975955352</td>\n",
       "      <td>0.2504789024044648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>891</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.75</td>\n",
       "      <td></td>\n",
       "      <td>Q</td>\n",
       "      <td>0.8833638515749265</td>\n",
       "      <td>0.11663614842507353</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0            1                                             2   \\\n",
       "0   PassengerId  Cabin_Class                                          Name   \n",
       "1           881            2  Shelley, Mrs. William (Imanita Parrish Hall)   \n",
       "2           882            3                            Markun, Mr. Johann   \n",
       "3           883            3                  Dahlberg, Miss. Gerda Ulrika   \n",
       "4           884            2                 Banfield, Mr. Frederick James   \n",
       "5           885            3                        Sutehall, Mr. Henry Jr   \n",
       "6           886            3          Rice, Mrs. William (Margaret Norton)   \n",
       "7           887            2                         Montvila, Rev. Juozas   \n",
       "8           888            1                  Graham, Miss. Margaret Edith   \n",
       "9           889            3      Johnston, Miss. Catherine Helen Carrie\"\"   \n",
       "10          890            1                         Behr, Mr. Karl Howell   \n",
       "11          891            3                           Dooley, Mr. Patrick   \n",
       "\n",
       "        3    4               5                6                 7        8   \\\n",
       "0      Sex  Age  Sibling_Spouse  Parent_Children     Ticket_Number     Fare   \n",
       "1   female   25               0                1            230433     26.0   \n",
       "2     male   33               0                0            349257   7.8958   \n",
       "3   female   22               0                0              7552  10.5167   \n",
       "4     male   28               0                0  C.A./SOTON 34068     10.5   \n",
       "5     male   25               0                0   SOTON/OQ 392076     7.05   \n",
       "6   female   39               0                5            382652   29.125   \n",
       "7     male   27               0                0            211536     13.0   \n",
       "8   female   19               0                0            112053     30.0   \n",
       "9   female                    1                2        W./C. 6607    23.45   \n",
       "10    male   26               0                0            111369     30.0   \n",
       "11    male   32               0                0            370376     7.75   \n",
       "\n",
       "              9                    10                   11  \\\n",
       "0   Cabin_Number  Port_of_Embarkation                 died   \n",
       "1                                   S  0.10347254522374348   \n",
       "2                                   S   0.8357493590851747   \n",
       "3                                   S   0.5723091324568758   \n",
       "4                                   S   0.8357493590851747   \n",
       "5                                   S   0.8833638515749265   \n",
       "6                                   Q   0.8204901964380503   \n",
       "7                                   S   0.8215620538923634   \n",
       "8            B42                    S  0.10347254522374348   \n",
       "9                                   S  0.42322660063987716   \n",
       "10          C148                    C   0.7495210975955352   \n",
       "11                                  Q   0.8833638515749265   \n",
       "\n",
       "                     12  \n",
       "0              survived  \n",
       "1    0.8965274547762565  \n",
       "2   0.16425064091482533  \n",
       "3   0.42769086754312424  \n",
       "4   0.16425064091482533  \n",
       "5   0.11663614842507353  \n",
       "6   0.17950980356194968  \n",
       "7   0.17843794610763664  \n",
       "8    0.8965274547762565  \n",
       "9    0.5767733993601228  \n",
       "10   0.2504789024044648  \n",
       "11  0.11663614842507353  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "predict_header = predict_data.columns.values\n",
    "full_output = np.concatenate((\n",
    "    np.concatenate((predict_header.reshape(1, -1), predict_data)), probabilities_output), axis=1)\n",
    "pd.DataFrame(full_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e36fb88",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "- Check out [303 Predictor Json Measurements](./brainome_303_Predictor_Json_Measurements.ipynb)\n",
    "- Check out [300 Put your model to work](./brainome_300_Integrating_Predictors.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}