Initial_Data_Exploration_ and_Visualizations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 21,
        "hidden": false,
        "row": 0,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "# Higgs Boson detection in experiments from the LHC\n",
    "\n",
    "\n",
    "\n",
    "#### ==================================================\n",
    "\n",
    "\n",
    "\n",
    "##### The following are a collection of  measurments of particles that were the result experiments derived from the collision of particles at CERN's Large Hadron Collider.\n",
    "\n",
    "#### The Large Hadron Collider\n",
    "\n",
    "##### Considered to be the most advanced and one of the most precise peices of machinery to ever be asssembled, this almost 17 mile length tunnel lined with super conducting magnets accelerates particles near the speed of light in order to study the energy and  particles produced by the collisions of such particles. The particles are contained in an unltra high vaccum in order not to just reduce resistance, but to completely isolate the particles. This results in the most accurate measurement s possible for each collison, and eschews the chance of interfernce from outside pressures, radiations, energies or particles.\n",
    "\n",
    "\n",
    "#### The Standard Model and BSM \n",
    "\n",
    "##### While the Standard Model seeks to describe the interactions of fundamental particles at the quantum level, LHC experiments are designed to test the properties of matter beyond the Standar Model of physics(BSM). This includes particles such as  multiple flavors of positively charged leptons (and their neutral counterparts, neutrinos. The most widely known lepton is the electron), and various types of bosons. The most important particle specifically for this experiment is the  detection of the Higgs Boson and the measured results are used to test and produce new theories in BSM physics. \n",
    "\n",
    "#### The Higgs Boson\n",
    "##### The importance of this particle in modern physics cannot be understated. Considered a \"missing link\" in the standard model, the presence of this particle after being solely theoretical for almost half a century, gives validity to the entirety of the standard model of physics.\n",
    "\n",
    "#### Detection of the Higgs Boson\n",
    "##### Higgs Bosons are usually detected ( from doing this dataset and the associated research) generally occur the most in collisions where the observed lepton_eta , lepton_phi, and missing_energy_phi having median values of 0.001894, 0.000971 and 0.002060 respectively. On the other side of this the median at which the didnt occur( and should not be tested at in later sets since it's not helping the predictions) have median values of 0.000920, -0.003570 and 0.000100 respectively\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 4,
        "hidden": false,
        "row": 21,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "## Glossary\n",
    "##### Torodial - Donut shaped, the shape itself being important for collisions, measuments and modeling of events at the sub-atomic level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 4,
        "hidden": false,
        "row": 29,
        "width": 4
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "## Initial Data Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "hidden": true
       },
       "report_default": {
        "hidden": true
       }
      }
     }
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats, integrate\n",
    "import matplotlib.pyplot as plt\n",
    "# thers 28 columns to this trunicated\n",
    "pd.set_option('display.max_columns', 500)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import psycopg2\n",
    "from sqlalchemy import create_engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Take a look at the data\n",
    "# print(pd.read_csv(file, nrows=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "hidden": true
       },
       "report_default": {
        "hidden": true
       }
      }
     }
    }
   },
   "outputs": [],
   "source": [
    "column_names = ['sig_or_back','lepton_pT', 'lepton_eta', 'lepton_phi', 'missing_energy_magnitude', 'missing_energy_phi', 'jet_1_pt', 'jet_1_eta', 'jet_1_phi', 'jet_1_b-tag', 'jet_2_pt', 'jet_2_eta', 'jet_2_phi', 'jet_2_b-tag', 'jet_3_pt', 'jet_3_eta', 'jet_3_phi', 'jet_3_b-tag', 'jet_4_pt', 'jet_4_eta', 'jet_4_phi', 'jet_4_b-tag', 'm_jj', 'm_jjj', 'm_lv', 'm_jlv', 'm_bb', 'm_wbb','m_wwbb']\n",
    "#column names for later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pandas csv text\n",
    "# set the table name\n",
    "table_name='higgs'\n",
    "# set the database uri to connect to\n",
    "con='postgresql://donovanadams:Dad8e3cc@localhost:5432/datascience'\n",
    "# the sql query well need\n",
    "# did a normal distribution off of the notebook and found that the first 1000000 sample matches the given data in terms of stats\n",
    "sql = 'SELECT * FROM higgs LIMIT 1000000;'\n",
    "# chunking, for some reazont it gives a key error\n",
    "df_all_rows_and_columns = pd.read_sql_query(sql,con, index_col=None, coerce_float=True, params=None,\n",
    "             parse_dates=None, chunksize=None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 3,
        "hidden": false,
        "row": 33,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "## Descriptive Statistics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 4,
        "hidden": false,
        "row": 25,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "source": [
    "### Data size\n",
    "#### The data contains 29 columns and 10,999,999 rows for a total of 318,999,971 measured data points\n",
    "#### Doing a  Gaussian distributin of the entire data set let to the first 1000000 being discovered ad pretty much representative of the entire sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 8,
        "hidden": false,
        "row": 36,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "outputs": [],
   "source": [
    "#testing data length\n",
    "sql_count = 'SELECT COUNT(*)FROM higgs'\n",
    "df_all_rows_and_columns_row_amount =pd.read_sql_query(sql_count,con)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all_rows_and_columns_row_amount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get total counts\n",
    "sql_head = 'SELECT * FROM higgs LIMIT 10;'\n",
    "df_all_rows_and_columns_head = pd.read_sql_query(sql_head,con)\n",
    "df_all_rows_and_columns_head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "col": 0,
        "height": 10,
        "hidden": false,
        "row": 44,
        "width": 12
       },
       "report_default": {
        "hidden": false
       }
      }
     }
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sig_or_back</th>\n",
       "      <th>lepton_pt</th>\n",
       "      <th>lepton_eta</th>\n",
       "      <th>lepton_phi</th>\n",
       "      <th>missing_energy_magnitude</th>\n",
       "      <th>missing_energy_phi</th>\n",
       "      <th>jet_1_pt</th>\n",
       "      <th>jet_1_eta</th>\n",
       "      <th>jet_1_phi</th>\n",
       "      <th>jet_1_b_tag</th>\n",
       "      <th>jet_2_pt</th>\n",
       "      <th>jet_2_eta</th>\n",
       "      <th>jet_2_phi</th>\n",
       "      <th>jet_2_b_tag</th>\n",
       "      <th>jet_3_pt</th>\n",
       "      <th>jet_3_eta</th>\n",
       "      <th>jet_3_phi</th>\n",
       "      <th>jet_3_b_tag</th>\n",
       "      <th>jet_4_pt</th>\n",
       "      <th>jet_4_eta</th>\n",
       "      <th>jet_4_phi</th>\n",
       "      <th>jet_4_b_tag</th>\n",
       "      <th>m_jj</th>\n",
       "      <th>m_jjj</th>\n",
       "      <th>m_lv</th>\n",
       "      <th>m_jlv</th>\n",
       "      <th>m_bb</th>\n",
       "      <th>m_wbb</th>\n",
       "      <th>m_wwbb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.530247</td>\n",
       "      <td>0.991743</td>\n",
       "      <td>0.000459</td>\n",
       "      <td>-0.000874</td>\n",
       "      <td>0.999347</td>\n",
       "      <td>0.000189</td>\n",
       "      <td>0.990050</td>\n",
       "      <td>0.000978</td>\n",
       "      <td>0.000793</td>\n",
       "      <td>0.999928</td>\n",
       "      <td>0.993009</td>\n",
       "      <td>0.000822</td>\n",
       "      <td>0.000736</td>\n",
       "      <td>1.001355</td>\n",
       "      <td>0.992009</td>\n",
       "      <td>-0.001198</td>\n",
       "      <td>0.001231</td>\n",
       "      <td>0.998396</td>\n",
       "      <td>0.986613</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>-0.001177</td>\n",
       "      <td>1.001427</td>\n",
       "      <td>1.034352</td>\n",
       "      <td>1.024893</td>\n",
       "      <td>1.050781</td>\n",
       "      <td>1.010207</td>\n",
       "      <td>0.972978</td>\n",
       "      <td>1.033122</td>\n",
       "      <td>0.959923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.499085</td>\n",
       "      <td>0.565644</td>\n",
       "      <td>1.008984</td>\n",
       "      <td>1.006978</td>\n",
       "      <td>0.599987</td>\n",
       "      <td>1.006277</td>\n",
       "      <td>0.474265</td>\n",
       "      <td>1.010897</td>\n",
       "      <td>1.005582</td>\n",
       "      <td>1.028016</td>\n",
       "      <td>0.500251</td>\n",
       "      <td>1.009674</td>\n",
       "      <td>1.006583</td>\n",
       "      <td>1.049267</td>\n",
       "      <td>0.487096</td>\n",
       "      <td>1.008979</td>\n",
       "      <td>1.005991</td>\n",
       "      <td>1.193145</td>\n",
       "      <td>0.506325</td>\n",
       "      <td>1.007618</td>\n",
       "      <td>1.006706</td>\n",
       "      <td>1.400702</td>\n",
       "      <td>0.674698</td>\n",
       "      <td>0.380407</td>\n",
       "      <td>0.165093</td>\n",
       "      <td>0.397981</td>\n",
       "      <td>0.523875</td>\n",
       "      <td>0.364490</td>\n",
       "      <td>0.312483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.274697</td>\n",
       "      <td>-2.434980</td>\n",
       "      <td>-1.742510</td>\n",
       "      <td>0.001475</td>\n",
       "      <td>-1.743940</td>\n",
       "      <td>0.138602</td>\n",
       "      <td>-2.969730</td>\n",
       "      <td>-1.741240</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.188981</td>\n",
       "      <td>-2.913090</td>\n",
       "      <td>-1.742370</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.263608</td>\n",
       "      <td>-2.729660</td>\n",
       "      <td>-1.742070</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.365354</td>\n",
       "      <td>-2.497260</td>\n",
       "      <td>-1.742690</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.079009</td>\n",
       "      <td>0.260719</td>\n",
       "      <td>0.152692</td>\n",
       "      <td>0.185844</td>\n",
       "      <td>0.051633</td>\n",
       "      <td>0.319164</td>\n",
       "      <td>0.351971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.590570</td>\n",
       "      <td>-0.738322</td>\n",
       "      <td>-0.873041</td>\n",
       "      <td>0.577362</td>\n",
       "      <td>-0.870931</td>\n",
       "      <td>0.678535</td>\n",
       "      <td>-0.689225</td>\n",
       "      <td>-0.866433</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.656839</td>\n",
       "      <td>-0.693501</td>\n",
       "      <td>-0.870734</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.650501</td>\n",
       "      <td>-0.700719</td>\n",
       "      <td>-0.869470</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.618254</td>\n",
       "      <td>-0.715023</td>\n",
       "      <td>-0.873144</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.790643</td>\n",
       "      <td>0.846026</td>\n",
       "      <td>0.985759</td>\n",
       "      <td>0.767527</td>\n",
       "      <td>0.674419</td>\n",
       "      <td>0.819697</td>\n",
       "      <td>0.770572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.853737</td>\n",
       "      <td>-0.000054</td>\n",
       "      <td>-0.002461</td>\n",
       "      <td>0.892327</td>\n",
       "      <td>0.000893</td>\n",
       "      <td>0.894086</td>\n",
       "      <td>0.003935</td>\n",
       "      <td>0.001824</td>\n",
       "      <td>1.086540</td>\n",
       "      <td>0.890264</td>\n",
       "      <td>0.001032</td>\n",
       "      <td>0.001009</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.896371</td>\n",
       "      <td>0.000173</td>\n",
       "      <td>-0.000197</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.868477</td>\n",
       "      <td>0.000372</td>\n",
       "      <td>-0.001374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.894886</td>\n",
       "      <td>0.950997</td>\n",
       "      <td>0.989782</td>\n",
       "      <td>0.916946</td>\n",
       "      <td>0.873933</td>\n",
       "      <td>0.947708</td>\n",
       "      <td>0.872042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.236590</td>\n",
       "      <td>0.739188</td>\n",
       "      <td>0.871549</td>\n",
       "      <td>1.294640</td>\n",
       "      <td>0.871274</td>\n",
       "      <td>1.169000</td>\n",
       "      <td>0.690165</td>\n",
       "      <td>0.868867</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>1.201870</td>\n",
       "      <td>0.695564</td>\n",
       "      <td>0.872647</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>1.222150</td>\n",
       "      <td>0.698334</td>\n",
       "      <td>0.873059</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>1.221170</td>\n",
       "      <td>0.713269</td>\n",
       "      <td>0.869941</td>\n",
       "      <td>3.101960</td>\n",
       "      <td>1.025030</td>\n",
       "      <td>1.083820</td>\n",
       "      <td>1.020950</td>\n",
       "      <td>1.142860</td>\n",
       "      <td>1.138930</td>\n",
       "      <td>1.140520</td>\n",
       "      <td>1.059720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>10.381200</td>\n",
       "      <td>2.434870</td>\n",
       "      <td>1.743240</td>\n",
       "      <td>11.613600</td>\n",
       "      <td>1.743250</td>\n",
       "      <td>6.841410</td>\n",
       "      <td>2.969670</td>\n",
       "      <td>1.741450</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>8.836790</td>\n",
       "      <td>2.913210</td>\n",
       "      <td>1.743170</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>10.764700</td>\n",
       "      <td>2.730010</td>\n",
       "      <td>1.742880</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>10.320900</td>\n",
       "      <td>2.498010</td>\n",
       "      <td>1.743370</td>\n",
       "      <td>3.101960</td>\n",
       "      <td>40.192400</td>\n",
       "      <td>20.372800</td>\n",
       "      <td>6.287520</td>\n",
       "      <td>9.960750</td>\n",
       "      <td>11.685300</td>\n",
       "      <td>11.496500</td>\n",
       "      <td>8.374500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          sig_or_back       lepton_pt      lepton_eta      lepton_phi  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.530247        0.991743        0.000459       -0.000874   \n",
       "std          0.499085        0.565644        1.008984        1.006978   \n",
       "min          0.000000        0.274697       -2.434980       -1.742510   \n",
       "25%          0.000000        0.590570       -0.738322       -0.873041   \n",
       "50%          1.000000        0.853737       -0.000054       -0.002461   \n",
       "75%          1.000000        1.236590        0.739188        0.871549   \n",
       "max          1.000000       10.381200        2.434870        1.743240   \n",
       "\n",
       "       missing_energy_magnitude  missing_energy_phi        jet_1_pt  \\\n",
       "count            1000000.000000      1000000.000000  1000000.000000   \n",
       "mean                   0.999347            0.000189        0.990050   \n",
       "std                    0.599987            1.006277        0.474265   \n",
       "min                    0.001475           -1.743940        0.138602   \n",
       "25%                    0.577362           -0.870931        0.678535   \n",
       "50%                    0.892327            0.000893        0.894086   \n",
       "75%                    1.294640            0.871274        1.169000   \n",
       "max                   11.613600            1.743250        6.841410   \n",
       "\n",
       "            jet_1_eta       jet_1_phi     jet_1_b_tag        jet_2_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000978        0.000793        0.999928        0.993009   \n",
       "std          1.010897        1.005582        1.028016        0.500251   \n",
       "min         -2.969730       -1.741240        0.000000        0.188981   \n",
       "25%         -0.689225       -0.866433        0.000000        0.656839   \n",
       "50%          0.003935        0.001824        1.086540        0.890264   \n",
       "75%          0.690165        0.868867        2.173080        1.201870   \n",
       "max          2.969670        1.741450        2.173080        8.836790   \n",
       "\n",
       "            jet_2_eta       jet_2_phi     jet_2_b_tag        jet_3_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000822        0.000736        1.001355        0.992009   \n",
       "std          1.009674        1.006583        1.049267        0.487096   \n",
       "min         -2.913090       -1.742370        0.000000        0.263608   \n",
       "25%         -0.693501       -0.870734        0.000000        0.650501   \n",
       "50%          0.001032        0.001009        0.000000        0.896371   \n",
       "75%          0.695564        0.872647        2.214870        1.222150   \n",
       "max          2.913210        1.743170        2.214870       10.764700   \n",
       "\n",
       "            jet_3_eta       jet_3_phi     jet_3_b_tag        jet_4_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean        -0.001198        0.001231        0.998396        0.986613   \n",
       "std          1.008979        1.005991        1.193145        0.506325   \n",
       "min         -2.729660       -1.742070        0.000000        0.365354   \n",
       "25%         -0.700719       -0.869470        0.000000        0.618254   \n",
       "50%          0.000173       -0.000197        0.000000        0.868477   \n",
       "75%          0.698334        0.873059        2.548220        1.221170   \n",
       "max          2.730010        1.742880        2.548220       10.320900   \n",
       "\n",
       "            jet_4_eta       jet_4_phi     jet_4_b_tag            m_jj  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000066       -0.001177        1.001427        1.034352   \n",
       "std          1.007618        1.006706        1.400702        0.674698   \n",
       "min         -2.497260       -1.742690        0.000000        0.079009   \n",
       "25%         -0.715023       -0.873144        0.000000        0.790643   \n",
       "50%          0.000372       -0.001374        0.000000        0.894886   \n",
       "75%          0.713269        0.869941        3.101960        1.025030   \n",
       "max          2.498010        1.743370        3.101960       40.192400   \n",
       "\n",
       "                m_jjj            m_lv           m_jlv            m_bb  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         1.024893        1.050781        1.010207        0.972978   \n",
       "std          0.380407        0.165093        0.397981        0.523875   \n",
       "min          0.260719        0.152692        0.185844        0.051633   \n",
       "25%          0.846026        0.985759        0.767527        0.674419   \n",
       "50%          0.950997        0.989782        0.916946        0.873933   \n",
       "75%          1.083820        1.020950        1.142860        1.138930   \n",
       "max         20.372800        6.287520        9.960750       11.685300   \n",
       "\n",
       "                m_wbb          m_wwbb  \n",
       "count  1000000.000000  1000000.000000  \n",
       "mean         1.033122        0.959923  \n",
       "std          0.364490        0.312483  \n",
       "min          0.319164        0.351971  \n",
       "25%          0.819697        0.770572  \n",
       "50%          0.947708        0.872042  \n",
       "75%          1.140520        1.059720  \n",
       "max         11.496500        8.374500  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will at first print in scientific notation due to some of the measurements involved\n",
    "\n",
    "df_all_rows_and_columns.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "extensions": {
     "jupyter_dashboards": {
      "version": 1,
      "views": {
       "grid_default": {
        "hidden": true
       },
       "report_default": {
        "hidden": true
       }
      }
     }
    }
   },
   "source": [
    "## Grouping by Signal or Background ( for signal or background)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sig_or_back</th>\n",
       "      <th>lepton_pt</th>\n",
       "      <th>lepton_eta</th>\n",
       "      <th>lepton_phi</th>\n",
       "      <th>missing_energy_magnitude</th>\n",
       "      <th>missing_energy_phi</th>\n",
       "      <th>jet_1_pt</th>\n",
       "      <th>jet_1_eta</th>\n",
       "      <th>jet_1_phi</th>\n",
       "      <th>jet_1_b_tag</th>\n",
       "      <th>jet_2_pt</th>\n",
       "      <th>jet_2_eta</th>\n",
       "      <th>jet_2_phi</th>\n",
       "      <th>jet_2_b_tag</th>\n",
       "      <th>jet_3_pt</th>\n",
       "      <th>jet_3_eta</th>\n",
       "      <th>jet_3_phi</th>\n",
       "      <th>jet_3_b_tag</th>\n",
       "      <th>jet_4_pt</th>\n",
       "      <th>jet_4_eta</th>\n",
       "      <th>jet_4_phi</th>\n",
       "      <th>jet_4_b_tag</th>\n",
       "      <th>m_jj</th>\n",
       "      <th>m_jjj</th>\n",
       "      <th>m_lv</th>\n",
       "      <th>m_jlv</th>\n",
       "      <th>m_bb</th>\n",
       "      <th>m_wbb</th>\n",
       "      <th>m_wwbb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000000.0</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.966601</td>\n",
       "      <td>0.001011</td>\n",
       "      <td>0.000554</td>\n",
       "      <td>0.941716</td>\n",
       "      <td>0.001149</td>\n",
       "      <td>1.016472</td>\n",
       "      <td>-0.000018</td>\n",
       "      <td>0.001463</td>\n",
       "      <td>0.991091</td>\n",
       "      <td>1.002574</td>\n",
       "      <td>0.000978</td>\n",
       "      <td>-0.000727</td>\n",
       "      <td>0.951313</td>\n",
       "      <td>0.998794</td>\n",
       "      <td>0.000953</td>\n",
       "      <td>-0.002351</td>\n",
       "      <td>0.972978</td>\n",
       "      <td>1.003462</td>\n",
       "      <td>0.000137</td>\n",
       "      <td>0.002159</td>\n",
       "      <td>1.016954</td>\n",
       "      <td>1.042500</td>\n",
       "      <td>1.033615</td>\n",
       "      <td>1.052396</td>\n",
       "      <td>0.998205</td>\n",
       "      <td>0.897825</td>\n",
       "      <td>1.010167</td>\n",
       "      <td>0.923321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.506000</td>\n",
       "      <td>0.997564</td>\n",
       "      <td>1.005950</td>\n",
       "      <td>0.527316</td>\n",
       "      <td>1.005969</td>\n",
       "      <td>0.482208</td>\n",
       "      <td>1.008500</td>\n",
       "      <td>1.005504</td>\n",
       "      <td>1.028577</td>\n",
       "      <td>0.505407</td>\n",
       "      <td>0.997826</td>\n",
       "      <td>1.006053</td>\n",
       "      <td>1.044942</td>\n",
       "      <td>0.494597</td>\n",
       "      <td>0.995048</td>\n",
       "      <td>1.006425</td>\n",
       "      <td>1.187550</td>\n",
       "      <td>0.517460</td>\n",
       "      <td>0.987695</td>\n",
       "      <td>1.006414</td>\n",
       "      <td>1.405600</td>\n",
       "      <td>0.691928</td>\n",
       "      <td>0.399344</td>\n",
       "      <td>0.164967</td>\n",
       "      <td>0.359982</td>\n",
       "      <td>0.384973</td>\n",
       "      <td>0.332907</td>\n",
       "      <td>0.283584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.274697</td>\n",
       "      <td>-2.434980</td>\n",
       "      <td>-1.742510</td>\n",
       "      <td>0.000576</td>\n",
       "      <td>-1.743940</td>\n",
       "      <td>0.138235</td>\n",
       "      <td>-2.969730</td>\n",
       "      <td>-1.741240</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.189107</td>\n",
       "      <td>-2.913090</td>\n",
       "      <td>-1.742370</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.263608</td>\n",
       "      <td>-2.729660</td>\n",
       "      <td>-1.742070</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.365354</td>\n",
       "      <td>-2.497260</td>\n",
       "      <td>-1.742690</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.083492</td>\n",
       "      <td>0.198676</td>\n",
       "      <td>0.150271</td>\n",
       "      <td>0.132006</td>\n",
       "      <td>0.058877</td>\n",
       "      <td>0.295112</td>\n",
       "      <td>0.352331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.601917</td>\n",
       "      <td>-0.728583</td>\n",
       "      <td>-0.870266</td>\n",
       "      <td>0.561644</td>\n",
       "      <td>-0.869389</td>\n",
       "      <td>0.704826</td>\n",
       "      <td>-0.674372</td>\n",
       "      <td>-0.864216</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.658349</td>\n",
       "      <td>-0.676016</td>\n",
       "      <td>-0.870734</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.651028</td>\n",
       "      <td>-0.682514</td>\n",
       "      <td>-0.875017</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.625556</td>\n",
       "      <td>-0.692537</td>\n",
       "      <td>-0.869259</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.793720</td>\n",
       "      <td>0.834300</td>\n",
       "      <td>0.985775</td>\n",
       "      <td>0.751364</td>\n",
       "      <td>0.702525</td>\n",
       "      <td>0.835940</td>\n",
       "      <td>0.767889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.848430</td>\n",
       "      <td>0.001894</td>\n",
       "      <td>0.000971</td>\n",
       "      <td>0.858574</td>\n",
       "      <td>0.002060</td>\n",
       "      <td>0.921111</td>\n",
       "      <td>-0.001016</td>\n",
       "      <td>0.001270</td>\n",
       "      <td>1.086540</td>\n",
       "      <td>0.900965</td>\n",
       "      <td>0.001032</td>\n",
       "      <td>-0.001313</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.901903</td>\n",
       "      <td>0.001993</td>\n",
       "      <td>-0.004080</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.882594</td>\n",
       "      <td>0.001205</td>\n",
       "      <td>0.002613</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.898383</td>\n",
       "      <td>0.964665</td>\n",
       "      <td>0.989796</td>\n",
       "      <td>0.927161</td>\n",
       "      <td>0.854631</td>\n",
       "      <td>0.938892</td>\n",
       "      <td>0.845691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.197250</td>\n",
       "      <td>0.730422</td>\n",
       "      <td>0.872104</td>\n",
       "      <td>1.225602</td>\n",
       "      <td>0.872360</td>\n",
       "      <td>1.188420</td>\n",
       "      <td>0.675311</td>\n",
       "      <td>0.868867</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>1.217610</td>\n",
       "      <td>0.677108</td>\n",
       "      <td>0.869318</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>1.233040</td>\n",
       "      <td>0.684680</td>\n",
       "      <td>0.869731</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>1.246240</td>\n",
       "      <td>0.692448</td>\n",
       "      <td>0.873825</td>\n",
       "      <td>3.101960</td>\n",
       "      <td>1.030370</td>\n",
       "      <td>1.110920</td>\n",
       "      <td>1.023630</td>\n",
       "      <td>1.160860</td>\n",
       "      <td>0.998158</td>\n",
       "      <td>1.084030</td>\n",
       "      <td>0.990793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.154050</td>\n",
       "      <td>2.434870</td>\n",
       "      <td>1.743240</td>\n",
       "      <td>7.568670</td>\n",
       "      <td>1.743250</td>\n",
       "      <td>5.818070</td>\n",
       "      <td>2.969670</td>\n",
       "      <td>1.741450</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>7.509390</td>\n",
       "      <td>2.913210</td>\n",
       "      <td>1.743170</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>7.760710</td>\n",
       "      <td>2.730010</td>\n",
       "      <td>1.742880</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>8.447930</td>\n",
       "      <td>2.498010</td>\n",
       "      <td>1.743370</td>\n",
       "      <td>3.101960</td>\n",
       "      <td>27.563000</td>\n",
       "      <td>13.908400</td>\n",
       "      <td>4.638880</td>\n",
       "      <td>10.691400</td>\n",
       "      <td>9.471660</td>\n",
       "      <td>8.267250</td>\n",
       "      <td>6.482470</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sig_or_back       lepton_pt      lepton_eta      lepton_phi  \\\n",
       "count    1000000.0  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean           1.0        0.966601        0.001011        0.000554   \n",
       "std            0.0        0.506000        0.997564        1.005950   \n",
       "min            1.0        0.274697       -2.434980       -1.742510   \n",
       "25%            1.0        0.601917       -0.728583       -0.870266   \n",
       "50%            1.0        0.848430        0.001894        0.000971   \n",
       "75%            1.0        1.197250        0.730422        0.872104   \n",
       "max            1.0        6.154050        2.434870        1.743240   \n",
       "\n",
       "       missing_energy_magnitude  missing_energy_phi        jet_1_pt  \\\n",
       "count            1000000.000000      1000000.000000  1000000.000000   \n",
       "mean                   0.941716            0.001149        1.016472   \n",
       "std                    0.527316            1.005969        0.482208   \n",
       "min                    0.000576           -1.743940        0.138235   \n",
       "25%                    0.561644           -0.869389        0.704826   \n",
       "50%                    0.858574            0.002060        0.921111   \n",
       "75%                    1.225602            0.872360        1.188420   \n",
       "max                    7.568670            1.743250        5.818070   \n",
       "\n",
       "            jet_1_eta       jet_1_phi     jet_1_b_tag        jet_2_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean        -0.000018        0.001463        0.991091        1.002574   \n",
       "std          1.008500        1.005504        1.028577        0.505407   \n",
       "min         -2.969730       -1.741240        0.000000        0.189107   \n",
       "25%         -0.674372       -0.864216        0.000000        0.658349   \n",
       "50%         -0.001016        0.001270        1.086540        0.900965   \n",
       "75%          0.675311        0.868867        2.173080        1.217610   \n",
       "max          2.969670        1.741450        2.173080        7.509390   \n",
       "\n",
       "            jet_2_eta       jet_2_phi     jet_2_b_tag        jet_3_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000978       -0.000727        0.951313        0.998794   \n",
       "std          0.997826        1.006053        1.044942        0.494597   \n",
       "min         -2.913090       -1.742370        0.000000        0.263608   \n",
       "25%         -0.676016       -0.870734        0.000000        0.651028   \n",
       "50%          0.001032       -0.001313        0.000000        0.901903   \n",
       "75%          0.677108        0.869318        2.214870        1.233040   \n",
       "max          2.913210        1.743170        2.214870        7.760710   \n",
       "\n",
       "            jet_3_eta       jet_3_phi     jet_3_b_tag        jet_4_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000953       -0.002351        0.972978        1.003462   \n",
       "std          0.995048        1.006425        1.187550        0.517460   \n",
       "min         -2.729660       -1.742070        0.000000        0.365354   \n",
       "25%         -0.682514       -0.875017        0.000000        0.625556   \n",
       "50%          0.001993       -0.004080        0.000000        0.882594   \n",
       "75%          0.684680        0.869731        2.548220        1.246240   \n",
       "max          2.730010        1.742880        2.548220        8.447930   \n",
       "\n",
       "            jet_4_eta       jet_4_phi     jet_4_b_tag            m_jj  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000137        0.002159        1.016954        1.042500   \n",
       "std          0.987695        1.006414        1.405600        0.691928   \n",
       "min         -2.497260       -1.742690        0.000000        0.083492   \n",
       "25%         -0.692537       -0.869259        0.000000        0.793720   \n",
       "50%          0.001205        0.002613        0.000000        0.898383   \n",
       "75%          0.692448        0.873825        3.101960        1.030370   \n",
       "max          2.498010        1.743370        3.101960       27.563000   \n",
       "\n",
       "                m_jjj            m_lv           m_jlv            m_bb  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         1.033615        1.052396        0.998205        0.897825   \n",
       "std          0.399344        0.164967        0.359982        0.384973   \n",
       "min          0.198676        0.150271        0.132006        0.058877   \n",
       "25%          0.834300        0.985775        0.751364        0.702525   \n",
       "50%          0.964665        0.989796        0.927161        0.854631   \n",
       "75%          1.110920        1.023630        1.160860        0.998158   \n",
       "max         13.908400        4.638880       10.691400        9.471660   \n",
       "\n",
       "                m_wbb          m_wwbb  \n",
       "count  1000000.000000  1000000.000000  \n",
       "mean         1.010167        0.923321  \n",
       "std          0.332907        0.283584  \n",
       "min          0.295112        0.352331  \n",
       "25%          0.835940        0.767889  \n",
       "50%          0.938892        0.845691  \n",
       "75%          1.084030        0.990793  \n",
       "max          8.267250        6.482470  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "signal_present_group_sql = \"SELECT * FROM higgs WHERE sig_or_back =1.0 LIMIT 1000000\"\n",
    "signal_present_df= pd.read_sql_query(signal_present_group_sql,con)\n",
    "signal_present_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sig_or_back</th>\n",
       "      <th>lepton_pt</th>\n",
       "      <th>lepton_eta</th>\n",
       "      <th>lepton_phi</th>\n",
       "      <th>missing_energy_magnitude</th>\n",
       "      <th>missing_energy_phi</th>\n",
       "      <th>jet_1_pt</th>\n",
       "      <th>jet_1_eta</th>\n",
       "      <th>jet_1_phi</th>\n",
       "      <th>jet_1_b_tag</th>\n",
       "      <th>jet_2_pt</th>\n",
       "      <th>jet_2_eta</th>\n",
       "      <th>jet_2_phi</th>\n",
       "      <th>jet_2_b_tag</th>\n",
       "      <th>jet_3_pt</th>\n",
       "      <th>jet_3_eta</th>\n",
       "      <th>jet_3_phi</th>\n",
       "      <th>jet_3_b_tag</th>\n",
       "      <th>jet_4_pt</th>\n",
       "      <th>jet_4_eta</th>\n",
       "      <th>jet_4_phi</th>\n",
       "      <th>jet_4_b_tag</th>\n",
       "      <th>m_jj</th>\n",
       "      <th>m_jjj</th>\n",
       "      <th>m_lv</th>\n",
       "      <th>m_jlv</th>\n",
       "      <th>m_bb</th>\n",
       "      <th>m_wbb</th>\n",
       "      <th>m_wwbb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000000.0</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.00000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.020871</td>\n",
       "      <td>0.000267</td>\n",
       "      <td>-0.002114</td>\n",
       "      <td>1.063465</td>\n",
       "      <td>0.000285</td>\n",
       "      <td>0.962276</td>\n",
       "      <td>0.000648</td>\n",
       "      <td>-0.000682</td>\n",
       "      <td>1.010123</td>\n",
       "      <td>0.981327</td>\n",
       "      <td>0.000838</td>\n",
       "      <td>0.000243</td>\n",
       "      <td>1.055956</td>\n",
       "      <td>0.984483</td>\n",
       "      <td>-0.001167</td>\n",
       "      <td>0.001219</td>\n",
       "      <td>1.029647</td>\n",
       "      <td>0.966676</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.97679</td>\n",
       "      <td>1.026102</td>\n",
       "      <td>1.015107</td>\n",
       "      <td>1.049206</td>\n",
       "      <td>1.022764</td>\n",
       "      <td>1.057827</td>\n",
       "      <td>1.059064</td>\n",
       "      <td>1.001548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.625350</td>\n",
       "      <td>1.021208</td>\n",
       "      <td>1.006557</td>\n",
       "      <td>0.666746</td>\n",
       "      <td>1.005754</td>\n",
       "      <td>0.463539</td>\n",
       "      <td>1.012382</td>\n",
       "      <td>1.006222</td>\n",
       "      <td>1.027131</td>\n",
       "      <td>0.493268</td>\n",
       "      <td>1.023288</td>\n",
       "      <td>1.006497</td>\n",
       "      <td>1.051552</td>\n",
       "      <td>0.479197</td>\n",
       "      <td>1.024574</td>\n",
       "      <td>1.006179</td>\n",
       "      <td>1.199524</td>\n",
       "      <td>0.492099</td>\n",
       "      <td>1.030492</td>\n",
       "      <td>1.006256</td>\n",
       "      <td>1.39242</td>\n",
       "      <td>0.661810</td>\n",
       "      <td>0.362121</td>\n",
       "      <td>0.166330</td>\n",
       "      <td>0.433894</td>\n",
       "      <td>0.637256</td>\n",
       "      <td>0.398141</td>\n",
       "      <td>0.340407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.274697</td>\n",
       "      <td>-2.434980</td>\n",
       "      <td>-1.742510</td>\n",
       "      <td>0.000992</td>\n",
       "      <td>-1.743940</td>\n",
       "      <td>0.138602</td>\n",
       "      <td>-2.969730</td>\n",
       "      <td>-1.741240</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.188981</td>\n",
       "      <td>-2.913090</td>\n",
       "      <td>-1.742370</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.263608</td>\n",
       "      <td>-2.729660</td>\n",
       "      <td>-1.742070</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.365354</td>\n",
       "      <td>-2.497260</td>\n",
       "      <td>-1.742690</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.075070</td>\n",
       "      <td>0.264307</td>\n",
       "      <td>0.118455</td>\n",
       "      <td>0.185844</td>\n",
       "      <td>0.051633</td>\n",
       "      <td>0.334524</td>\n",
       "      <td>0.356169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.577760</td>\n",
       "      <td>-0.750010</td>\n",
       "      <td>-0.873595</td>\n",
       "      <td>0.596081</td>\n",
       "      <td>-0.869345</td>\n",
       "      <td>0.654534</td>\n",
       "      <td>-0.706059</td>\n",
       "      <td>-0.869759</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.653943</td>\n",
       "      <td>-0.714871</td>\n",
       "      <td>-0.871844</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.650501</td>\n",
       "      <td>-0.719834</td>\n",
       "      <td>-0.869470</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.609005</td>\n",
       "      <td>-0.740008</td>\n",
       "      <td>-0.870924</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.787319</td>\n",
       "      <td>0.855960</td>\n",
       "      <td>0.985724</td>\n",
       "      <td>0.783731</td>\n",
       "      <td>0.630428</td>\n",
       "      <td>0.797807</td>\n",
       "      <td>0.775405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.861790</td>\n",
       "      <td>0.000920</td>\n",
       "      <td>-0.003570</td>\n",
       "      <td>0.935576</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>0.863856</td>\n",
       "      <td>0.002945</td>\n",
       "      <td>0.000715</td>\n",
       "      <td>1.086540</td>\n",
       "      <td>0.878303</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.001009</td>\n",
       "      <td>1.107440</td>\n",
       "      <td>0.891454</td>\n",
       "      <td>-0.000737</td>\n",
       "      <td>0.000357</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.851925</td>\n",
       "      <td>-0.000461</td>\n",
       "      <td>-0.000264</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.891037</td>\n",
       "      <td>0.940312</td>\n",
       "      <td>0.989758</td>\n",
       "      <td>0.908924</td>\n",
       "      <td>0.928330</td>\n",
       "      <td>0.965961</td>\n",
       "      <td>0.918975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.286000</td>\n",
       "      <td>0.750876</td>\n",
       "      <td>0.869884</td>\n",
       "      <td>1.379230</td>\n",
       "      <td>0.870611</td>\n",
       "      <td>1.149490</td>\n",
       "      <td>0.706999</td>\n",
       "      <td>0.867204</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>1.184370</td>\n",
       "      <td>0.716934</td>\n",
       "      <td>0.870982</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>1.208450</td>\n",
       "      <td>0.718359</td>\n",
       "      <td>0.873614</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>1.192940</td>\n",
       "      <td>0.738254</td>\n",
       "      <td>0.871605</td>\n",
       "      <td>3.10196</td>\n",
       "      <td>1.019060</td>\n",
       "      <td>1.047753</td>\n",
       "      <td>1.017810</td>\n",
       "      <td>1.118440</td>\n",
       "      <td>1.326340</td>\n",
       "      <td>1.208490</td>\n",
       "      <td>1.131690</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.0</td>\n",
       "      <td>12.098900</td>\n",
       "      <td>2.434870</td>\n",
       "      <td>1.743240</td>\n",
       "      <td>12.843900</td>\n",
       "      <td>1.743260</td>\n",
       "      <td>9.940390</td>\n",
       "      <td>2.969670</td>\n",
       "      <td>1.741450</td>\n",
       "      <td>2.173080</td>\n",
       "      <td>8.928450</td>\n",
       "      <td>2.913210</td>\n",
       "      <td>1.743170</td>\n",
       "      <td>2.214870</td>\n",
       "      <td>10.764700</td>\n",
       "      <td>2.730010</td>\n",
       "      <td>1.742880</td>\n",
       "      <td>2.548220</td>\n",
       "      <td>10.320900</td>\n",
       "      <td>2.498010</td>\n",
       "      <td>1.743370</td>\n",
       "      <td>3.10196</td>\n",
       "      <td>40.192400</td>\n",
       "      <td>20.372800</td>\n",
       "      <td>6.287520</td>\n",
       "      <td>10.202400</td>\n",
       "      <td>12.840000</td>\n",
       "      <td>11.496500</td>\n",
       "      <td>8.374500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sig_or_back       lepton_pt      lepton_eta      lepton_phi  \\\n",
       "count    1000000.0  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean           0.0        1.020871        0.000267       -0.002114   \n",
       "std            0.0        0.625350        1.021208        1.006557   \n",
       "min            0.0        0.274697       -2.434980       -1.742510   \n",
       "25%            0.0        0.577760       -0.750010       -0.873595   \n",
       "50%            0.0        0.861790        0.000920       -0.003570   \n",
       "75%            0.0        1.286000        0.750876        0.869884   \n",
       "max            0.0       12.098900        2.434870        1.743240   \n",
       "\n",
       "       missing_energy_magnitude  missing_energy_phi        jet_1_pt  \\\n",
       "count            1000000.000000      1000000.000000  1000000.000000   \n",
       "mean                   1.063465            0.000285        0.962276   \n",
       "std                    0.666746            1.005754        0.463539   \n",
       "min                    0.000992           -1.743940        0.138602   \n",
       "25%                    0.596081           -0.869345        0.654534   \n",
       "50%                    0.935576            0.000100        0.863856   \n",
       "75%                    1.379230            0.870611        1.149490   \n",
       "max                   12.843900            1.743260        9.940390   \n",
       "\n",
       "            jet_1_eta       jet_1_phi     jet_1_b_tag        jet_2_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000648       -0.000682        1.010123        0.981327   \n",
       "std          1.012382        1.006222        1.027131        0.493268   \n",
       "min         -2.969730       -1.741240        0.000000        0.188981   \n",
       "25%         -0.706059       -0.869759        0.000000        0.653943   \n",
       "50%          0.002945        0.000715        1.086540        0.878303   \n",
       "75%          0.706999        0.867204        2.173080        1.184370   \n",
       "max          2.969670        1.741450        2.173080        8.928450   \n",
       "\n",
       "            jet_2_eta       jet_2_phi     jet_2_b_tag        jet_3_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         0.000838        0.000243        1.055956        0.984483   \n",
       "std          1.023288        1.006497        1.051552        0.479197   \n",
       "min         -2.913090       -1.742370        0.000000        0.263608   \n",
       "25%         -0.714871       -0.871844        0.000000        0.650501   \n",
       "50%          0.000060        0.001009        1.107440        0.891454   \n",
       "75%          0.716934        0.870982        2.214870        1.208450   \n",
       "max          2.913210        1.743170        2.214870       10.764700   \n",
       "\n",
       "            jet_3_eta       jet_3_phi     jet_3_b_tag        jet_4_pt  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean        -0.001167        0.001219        1.029647        0.966676   \n",
       "std          1.024574        1.006179        1.199524        0.492099   \n",
       "min         -2.729660       -1.742070        0.000000        0.365354   \n",
       "25%         -0.719834       -0.869470        0.000000        0.609005   \n",
       "50%         -0.000737        0.000357        0.000000        0.851925   \n",
       "75%          0.718359        0.873614        2.548220        1.192940   \n",
       "max          2.730010        1.742880        2.548220       10.320900   \n",
       "\n",
       "            jet_4_eta       jet_4_phi    jet_4_b_tag            m_jj  \\\n",
       "count  1000000.000000  1000000.000000  1000000.00000  1000000.000000   \n",
       "mean         0.000023        0.000020        0.97679        1.026102   \n",
       "std          1.030492        1.006256        1.39242        0.661810   \n",
       "min         -2.497260       -1.742690        0.00000        0.075070   \n",
       "25%         -0.740008       -0.870924        0.00000        0.787319   \n",
       "50%         -0.000461       -0.000264        0.00000        0.891037   \n",
       "75%          0.738254        0.871605        3.10196        1.019060   \n",
       "max          2.498010        1.743370        3.10196       40.192400   \n",
       "\n",
       "                m_jjj            m_lv           m_jlv            m_bb  \\\n",
       "count  1000000.000000  1000000.000000  1000000.000000  1000000.000000   \n",
       "mean         1.015107        1.049206        1.022764        1.057827   \n",
       "std          0.362121        0.166330        0.433894        0.637256   \n",
       "min          0.264307        0.118455        0.185844        0.051633   \n",
       "25%          0.855960        0.985724        0.783731        0.630428   \n",
       "50%          0.940312        0.989758        0.908924        0.928330   \n",
       "75%          1.047753        1.017810        1.118440        1.326340   \n",
       "max         20.372800        6.287520       10.202400       12.840000   \n",
       "\n",
       "                m_wbb          m_wwbb  \n",
       "count  1000000.000000  1000000.000000  \n",
       "mean         1.059064        1.001548  \n",
       "std          0.398141        0.340407  \n",
       "min          0.334524        0.356169  \n",
       "25%          0.797807        0.775405  \n",
       "50%          0.965961        0.918975  \n",
       "75%          1.208490        1.131690  \n",
       "max         11.496500        8.374500  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# By those that reported that the readings were from the background\n",
    "background_present_group_sql = \"SELECT * FROM higgs WHERE sig_or_back =0.0 LIMIT 1000000\"\n",
    "background_present_df= pd.read_sql_query(background_present_group_sql,con)\n",
    "background_present_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Missing Energy Magnitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>missing_energy_magnitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.100000e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>9.985364e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>6.000185e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.370090e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5.768160e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>8.916280e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.293060e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.539680e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       missing_energy_magnitude\n",
       "count              1.100000e+07\n",
       "mean               9.985364e-01\n",
       "std                6.000185e-01\n",
       "min                2.370090e-04\n",
       "25%                5.768160e-01\n",
       "50%                8.916280e-01\n",
       "75%                1.293060e+00\n",
       "max                1.539680e+01"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "miss_energy_mag_sql = 'SELECT missing_energy_magnitude FROM higgs;'\n",
    "miss_energy_mag_df= pd.read_sql_query(miss_energy_mag_sql,con)\n",
    "miss_energy_mag_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>missing_energy_magnitude</th>\n",
       "      <th>lepton_eta</th>\n",
       "      <th>lepton_pt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "      <td>1000000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.941878</td>\n",
       "      <td>-0.000125</td>\n",
       "      <td>0.965848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.527368</td>\n",
       "      <td>0.997880</td>\n",
       "      <td>0.505751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000730</td>\n",
       "      <td>-2.434980</td>\n",
       "      <td>0.274697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.561709</td>\n",
       "      <td>-0.730531</td>\n",
       "      <td>0.601002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.858896</td>\n",
       "      <td>0.000920</td>\n",
       "      <td>0.847698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.225572</td>\n",
       "      <td>0.729448</td>\n",
       "      <td>1.196330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.568670</td>\n",
       "      <td>2.434870</td>\n",
       "      <td>6.602600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       missing_energy_magnitude      lepton_eta       lepton_pt\n",
       "count            1000000.000000  1000000.000000  1000000.000000\n",
       "mean                   0.941878       -0.000125        0.965848\n",
       "std                    0.527368        0.997880        0.505751\n",
       "min                    0.000730       -2.434980        0.274697\n",
       "25%                    0.561709       -0.730531        0.601002\n",
       "50%                    0.858896        0.000920        0.847698\n",
       "75%                    1.225572        0.729448        1.196330\n",
       "max                    7.568670        2.434870        6.602600"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lepton_eta- the length of time that the lepton is observed have wildy different \n",
    "# median values, with the occurrences with signal detected having much higher etas at 0.001894\n",
    "# Occurences with  backround detected coming in at around 0.000920\n",
    "occurence_with_signal_sql = 'SELECT missing_energy_magnitude,lepton_eta,lepton_pt FROM higgs WHERE sig_or_back =1.0 LIMIT 1000000;'\n",
    "occurence_with_signal_df = pd.read_sql_query(occurence_with_signal_sql,con)\n",
    "occurence_with_signal_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "extensions": {
   "jupyter_dashboards": {
    "activeView": "grid_default",
    "version": 1,
    "views": {
     "grid_default": {
      "cellMargin": 10,
      "defaultCellHeight": 20,
      "maxColumns": 12,
      "name": "grid",
      "type": "grid"
     },
     "report_default": {
      "name": "report",
      "type": "report"
     }
    }
   }
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}