Logistic_Regression_Full_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<p>\n",
    "DonorsChoose.org receives hundreds of thousands of project proposals each year for classroom projects in need of funding. Right now, a large number of volunteers is needed to manually screen each submission before it's approved to be posted on the DonorsChoose.org website.\n",
    "</p>\n",
    "<p>\n",
    "    Next year, DonorsChoose.org expects to receive close to 500,000 project proposals. As a result, there are three main problems they need to solve:\n",
    "<ul>\n",
    "<li>\n",
    "    How to scale current manual processes and resources to screen 500,000 projects so that they can be posted as quickly and as efficiently as possible</li>F\n",
    "    <li>How to increase the consistency of project vetting across different volunteers to improve the experience for teachers</li>\n",
    "    <li>How to focus volunteer time on the applications that need the most assistance</li>\n",
    "    </ul>\n",
    "</p>    \n",
    "<p>\n",
    "The goal of the competition is to predict whether or not a DonorsChoose.org project proposal submitted by a teacher will be approved, using the text of project descriptions as well as additional metadata about the project, teacher, and school. DonorsChoose.org can then use this information to identify projects most likely to need further review before approval.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes on the Essay Data\n",
    "\n",
    "<ul>\n",
    "Prior to May 17, 2016, the prompts for the essays were as follows:\n",
    "<li>__project_essay_1:__ \"Introduce us to your classroom\"</li>\n",
    "<li>__project_essay_2:__ \"Tell us more about your students\"</li>\n",
    "<li>__project_essay_3:__ \"Describe how your students will use the materials you're requesting\"</li>\n",
    "<li>__project_essay_3:__ \"Close by sharing why your project will make a difference\"</li>\n",
    "</ul>\n",
    "\n",
    "\n",
    "<ul>\n",
    "Starting on May 17, 2016, the number of essays was reduced from 4 to 2, and the prompts for the first 2 essays were changed to the following:<br>\n",
    "<li>__project_essay_1:__ \"Describe your students: What makes your students special? Specific details about their background, your neighborhood, and your school are all helpful.\"</li>\n",
    "<li>__project_essay_2:__ \"About your project: How will these materials make a difference in your students' learning and improve their school lives?\"</li>\n",
    "<br>For all projects with project_submitted_datetime of 2016-05-17 and later, the values of project_essay_3 and project_essay_4 will be NaN.\n",
    "</ul>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Import the necessary Libraries\n",
    "we will need to import libraries that allow for data analysis and data visualization to get acclimated to the dataset. We will be using pandas, numpy, matplotlib and seaborn to conduct this. Data Exploration libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "warnings.filterwarnings(action='ignore', category=UserWarning, module='gensim')\n",
    "warnings.filterwarnings(\"ignore\",'detected Windows; aliasing chunkize to chunkize_serial')\n",
    "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n",
    "\n",
    "import sqlite3\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "import string\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "import re\n",
    "# Tutorial about Python regular expressions: https://pymotw.com/2/re/\n",
    "import string\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models import KeyedVectors\n",
    "import pickle\n",
    "from tqdm import tqdm\n",
    "import os\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Read in the dataset.\n",
    "We will use the pandas .read_csv() method to read in the dataset. Then we will use the. head() method to observe the first few rows of the data, to understand the information better. In our case, the feature(column) headers tell us pretty little. This is fine because we are merely trying to gain insight via classifying new data points by referencing it’s neighboring elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Reading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>description</th>\n",
       "      <th>quantity</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>p233245</td>\n",
       "      <td>LC652 - Lakeshore Double-Space Mobile Drying Rack</td>\n",
       "      <td>1</td>\n",
       "      <td>149.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                        description  quantity  price\n",
       "0  p233245  LC652 - Lakeshore Double-Space Mobile Drying Rack         1  149.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project_data = pd.read_csv(\"C:\\\\VipinML\\\\Assignment 2\\\\Assignments_DonorsChoose_2018\\\\train_data.csv\")\n",
    "resource_data = pd.read_csv(\"C:\\\\VipinML\\Assignment 2\\\\Assignments_DonorsChoose_2018\\\\resources.csv\")\n",
    "#Limit the data for testing purpose since processing takes few hours for full set..\n",
    "\n",
    "#project_data = project_data.head(2000)\n",
    "#resource_data = resource_data.head (2000)\n",
    "\n",
    "resource_data.head(1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of data points in train data (109248, 17)\n",
      "--------------------------------------------------\n",
      "The attributes of data : ['Unnamed: 0' 'id' 'teacher_id' 'teacher_prefix' 'school_state'\n",
      " 'project_submitted_datetime' 'project_grade_category'\n",
      " 'project_subject_categories' 'project_subject_subcategories'\n",
      " 'project_title' 'project_essay_1' 'project_essay_2' 'project_essay_3'\n",
      " 'project_essay_4' 'project_resource_summary'\n",
      " 'teacher_number_of_previously_posted_projects' 'project_is_approved']\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of data points in train data\", project_data.shape)\n",
    "print('-'*50)\n",
    "print(\"The attributes of data :\", project_data.columns.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>teacher_id</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>school_state</th>\n",
       "      <th>Date</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>project_subject_categories</th>\n",
       "      <th>project_subject_subcategories</th>\n",
       "      <th>project_title</th>\n",
       "      <th>project_essay_1</th>\n",
       "      <th>project_essay_2</th>\n",
       "      <th>project_essay_3</th>\n",
       "      <th>project_essay_4</th>\n",
       "      <th>project_resource_summary</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55660</th>\n",
       "      <td>8393</td>\n",
       "      <td>p205479</td>\n",
       "      <td>2bf07ba08945e5d8b2a3f269b2b3cfe5</td>\n",
       "      <td>Mrs.</td>\n",
       "      <td>CA</td>\n",
       "      <td>2016-04-27 00:27:36</td>\n",
       "      <td>Grades PreK-2</td>\n",
       "      <td>Math &amp; Science</td>\n",
       "      <td>Applied Sciences, Health &amp; Life Science</td>\n",
       "      <td>Engineering STEAM into the Primary Classroom</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>My students come from a variety of backgrounds...</td>\n",
       "      <td>Each month I try to do several science or STEM...</td>\n",
       "      <td>It is challenging to develop high quality scie...</td>\n",
       "      <td>My students need STEM kits to learn critical s...</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0       id                        teacher_id teacher_prefix  \\\n",
       "55660        8393  p205479  2bf07ba08945e5d8b2a3f269b2b3cfe5           Mrs.   \n",
       "\n",
       "      school_state                Date project_grade_category  \\\n",
       "55660           CA 2016-04-27 00:27:36          Grades PreK-2   \n",
       "\n",
       "      project_subject_categories            project_subject_subcategories  \\\n",
       "55660             Math & Science  Applied Sciences, Health & Life Science   \n",
       "\n",
       "                                      project_title  \\\n",
       "55660  Engineering STEAM into the Primary Classroom   \n",
       "\n",
       "                                         project_essay_1  \\\n",
       "55660  I have been fortunate enough to use the Fairy ...   \n",
       "\n",
       "                                         project_essay_2  \\\n",
       "55660  My students come from a variety of backgrounds...   \n",
       "\n",
       "                                         project_essay_3  \\\n",
       "55660  Each month I try to do several science or STEM...   \n",
       "\n",
       "                                         project_essay_4  \\\n",
       "55660  It is challenging to develop high quality scie...   \n",
       "\n",
       "                                project_resource_summary  \\\n",
       "55660  My students need STEM kits to learn critical s...   \n",
       "\n",
       "       teacher_number_of_previously_posted_projects  project_is_approved  \n",
       "55660                                            53                    1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how to replace elements in list python: https://stackoverflow.com/a/2582163/4084039\n",
    "cols = ['Date' if x=='project_submitted_datetime' else x for x in list(project_data.columns)]\n",
    "#sort dataframe based on time pandas python: https://stackoverflow.com/a/49702492/4084039\n",
    "project_data['Date'] = pd.to_datetime(project_data['project_submitted_datetime'])\n",
    "project_data.drop('project_submitted_datetime', axis=1, inplace=True)\n",
    "project_data.sort_values(by=['Date'], inplace=True)\n",
    "\n",
    "# how to reorder columns pandas python: https://stackoverflow.com/a/13148611/4084039\n",
    "project_data = project_data[cols]\n",
    "project_data.head(1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 preprocessing of `project_subject_categories`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "catogories = list(project_data['project_subject_categories'].values)\n",
    "# remove special characters from list of strings python: https://stackoverflow.com/a/47301924/4084039\n",
    "\n",
    "# https://www.geeksforgeeks.org/removing-stop-words-nltk-python/\n",
    "# https://stackoverflow.com/questions/23669024/how-to-strip-a-specific-word-from-a-string\n",
    "# https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string-in-python\n",
    "cat_list = []\n",
    "for i in catogories:\n",
    "    temp = \"\"\n",
    "    # consider we have text like this \"Math & Science, Warmth, Care & Hunger\"\n",
    "    for j in i.split(','): # it will split it in three parts [\"Math & Science\", \"Warmth\", \"Care & Hunger\"]\n",
    "        if 'The' in j.split(): # this will split each of the catogory based on space \"Math & Science\"=> \"Math\",\"&\", \"Science\"\n",
    "            j=j.replace('The','') # if we have the words \"The\" we are going to replace it with ''(i.e removing 'The')\n",
    "        j = j.replace(' ','') # we are placeing all the ' '(space) with ''(empty) ex:\"Math & Science\"=>\"Math&Science\"\n",
    "        temp+=j.strip()+\" \" #\" abc \".strip() will return \"abc\", remove the trailing spaces\n",
    "        temp = temp.replace('&','_') # we are replacing the & value into \n",
    "    cat_list.append(temp.strip())\n",
    "    \n",
    "project_data['clean_categories'] = cat_list\n",
    "project_data.drop(['project_subject_categories'], axis=1, inplace=True)\n",
    "\n",
    "from collections import Counter\n",
    "my_counter = Counter()\n",
    "for word in project_data['clean_categories'].values:\n",
    "    my_counter.update(word.split())\n",
    "\n",
    "cat_dict = dict(my_counter)\n",
    "sorted_cat_dict = dict(sorted(cat_dict.items(), key=lambda kv: kv[1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 preprocessing of `project_subject_subcategories`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_catogories = list(project_data['project_subject_subcategories'].values)\n",
    "# remove special characters from list of strings python: https://stackoverflow.com/a/47301924/4084039\n",
    "\n",
    "# https://www.geeksforgeeks.org/removing-stop-words-nltk-python/\n",
    "# https://stackoverflow.com/questions/23669024/how-to-strip-a-specific-word-from-a-string\n",
    "# https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string-in-python\n",
    "\n",
    "sub_cat_list = []\n",
    "for i in sub_catogories:\n",
    "    temp = \"\"\n",
    "    # consider we have text like this \"Math & Science, Warmth, Care & Hunger\"\n",
    "    for j in i.split(','): # it will split it in three parts [\"Math & Science\", \"Warmth\", \"Care & Hunger\"]\n",
    "        if 'The' in j.split(): # this will split each of the catogory based on space \"Math & Science\"=> \"Math\",\"&\", \"Science\"\n",
    "            j=j.replace('The','') # if we have the words \"The\" we are going to replace it with ''(i.e removing 'The')\n",
    "        j = j.replace(' ','') # we are placeing all the ' '(space) with ''(empty) ex:\"Math & Science\"=>\"Math&Science\"\n",
    "        temp +=j.strip()+\" \"#\" abc \".strip() will return \"abc\", remove the trailing spaces\n",
    "        temp = temp.replace('&','_')\n",
    "    sub_cat_list.append(temp.strip())\n",
    "\n",
    "project_data['clean_subcategories'] = sub_cat_list\n",
    "project_data.drop(['project_subject_subcategories'], axis=1, inplace=True)\n",
    "\n",
    "# count of all the words in corpus python: https://stackoverflow.com/a/22898595/4084039\n",
    "my_counter = Counter()\n",
    "for word in project_data['clean_subcategories'].values:\n",
    "    my_counter.update(word.split())\n",
    "    \n",
    "sub_cat_dict = dict(my_counter)\n",
    "sorted_sub_cat_dict = dict(sorted(sub_cat_dict.items(), key=lambda kv: kv[1]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "teacher_cat = list(project_data['teacher_prefix'].values)\n",
    "# remove special characters from list of strings python: https://stackoverflow.com/a/47301924/4084039\n",
    "\n",
    "# https://www.geeksforgeeks.org/removing-stop-words-nltk-python/\n",
    "# https://stackoverflow.com/questions/23669024/how-to-strip-a-specific-word-from-a-string\n",
    "# https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string-in-python\n",
    "cat_list = []\n",
    "for i in teacher_cat:\n",
    "    temp = \"\"\n",
    "    # consider we have text like this \"Math & Science, Warmth, Care & Hunger\"\n",
    "    j = j.replace(' ','') # we are placeing all the ' '(space) with ''(empty) ex:\"Math & Science\"=>\"Math&Science\"\n",
    "    temp+=j.strip()+\" \" #\" abc \".strip() will return \"abc\", remove the trailing spaces\n",
    "    temp = temp.replace('&','_') # we are replacing the & value into \n",
    "    cat_list.append(temp.strip())\n",
    "\n",
    "project_data.drop(['teacher_prefix'], axis=1, inplace=True)\n",
    "project_data['teacher_prefix'] = sub_cat_list\n",
    "    \n",
    "from collections import Counter\n",
    "my_counter = Counter()\n",
    "for word in project_data['teacher_prefix'].values:\n",
    "    my_counter.update(word.split())\n",
    "\n",
    "cat_dict = dict(my_counter)\n",
    "sorted_teacher_dict = dict(sorted(cat_dict.items(), key=lambda kv: kv[1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Text preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge two column text dataframe: \n",
    "project_data[\"essay\"] = project_data[\"project_essay_1\"].map(str) +\\\n",
    "                        project_data[\"project_essay_2\"].map(str) + \\\n",
    "                        project_data[\"project_essay_3\"].map(str) + \\\n",
    "                        project_data[\"project_essay_4\"].map(str)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean project_grade_category\n",
    "project_data[\"project_grade_category\"] = \\\n",
    "project_data.apply(lambda x: (x['project_grade_category'].replace(' ', '_')), axis=1)\n",
    "project_data[\"project_grade_category\"] = \\\n",
    "project_data.apply(lambda x: (x['project_grade_category'].replace('-', '_')), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>teacher_id</th>\n",
       "      <th>school_state</th>\n",
       "      <th>Date</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>project_title</th>\n",
       "      <th>project_essay_1</th>\n",
       "      <th>project_essay_2</th>\n",
       "      <th>project_essay_3</th>\n",
       "      <th>project_essay_4</th>\n",
       "      <th>project_resource_summary</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "      <th>clean_categories</th>\n",
       "      <th>clean_subcategories</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>essay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55660</th>\n",
       "      <td>8393</td>\n",
       "      <td>p205479</td>\n",
       "      <td>2bf07ba08945e5d8b2a3f269b2b3cfe5</td>\n",
       "      <td>CA</td>\n",
       "      <td>2016-04-27 00:27:36</td>\n",
       "      <td>Grades_PreK_2</td>\n",
       "      <td>Engineering STEAM into the Primary Classroom</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>My students come from a variety of backgrounds...</td>\n",
       "      <td>Each month I try to do several science or STEM...</td>\n",
       "      <td>It is challenging to develop high quality scie...</td>\n",
       "      <td>My students need STEM kits to learn critical s...</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>Math_Science</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0       id                        teacher_id school_state  \\\n",
       "55660        8393  p205479  2bf07ba08945e5d8b2a3f269b2b3cfe5           CA   \n",
       "\n",
       "                     Date project_grade_category  \\\n",
       "55660 2016-04-27 00:27:36          Grades_PreK_2   \n",
       "\n",
       "                                      project_title  \\\n",
       "55660  Engineering STEAM into the Primary Classroom   \n",
       "\n",
       "                                         project_essay_1  \\\n",
       "55660  I have been fortunate enough to use the Fairy ...   \n",
       "\n",
       "                                         project_essay_2  \\\n",
       "55660  My students come from a variety of backgrounds...   \n",
       "\n",
       "                                         project_essay_3  \\\n",
       "55660  Each month I try to do several science or STEM...   \n",
       "\n",
       "                                         project_essay_4  \\\n",
       "55660  It is challenging to develop high quality scie...   \n",
       "\n",
       "                                project_resource_summary  \\\n",
       "55660  My students need STEM kits to learn critical s...   \n",
       "\n",
       "       teacher_number_of_previously_posted_projects  project_is_approved  \\\n",
       "55660                                            53                    1   \n",
       "\n",
       "      clean_categories                 clean_subcategories  \\\n",
       "55660     Math_Science  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                           teacher_prefix  \\\n",
       "55660  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                                                   essay  \n",
       "55660  I have been fortunate enough to use the Fairy ...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project_data.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### 1.4.2.3 Using Pretrained Models: TFIDF weighted W2V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/a/47091490/4084039\n",
    "import re\n",
    "\n",
    "def decontracted(phrase):\n",
    "    # specific\n",
    "    phrase = re.sub(r\"won't\", \"will not\", phrase)\n",
    "    phrase = re.sub(r\"can\\'t\", \"can not\", phrase)\n",
    "\n",
    "    # general\n",
    "    phrase = re.sub(r\"n\\'t\", \" not\", phrase)\n",
    "    phrase = re.sub(r\"\\'re\", \" are\", phrase)\n",
    "    phrase = re.sub(r\"\\'s\", \" is\", phrase)\n",
    "    phrase = re.sub(r\"\\'d\", \" would\", phrase)\n",
    "    phrase = re.sub(r\"\\'ll\", \" will\", phrase)\n",
    "    phrase = re.sub(r\"\\'t\", \" not\", phrase)\n",
    "    phrase = re.sub(r\"\\'ve\", \" have\", phrase)\n",
    "    phrase = re.sub(r\"\\'m\", \" am\", phrase)\n",
    "    return phrase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s an RSP (resource specialist), my \\\"typical day\\\" varies depending on the day of the week.  However, typically,  I push in to classrooms and team teach with the general education teacher and/or teac\n",
      "====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "sent = decontracted(project_data['essay'].values[500])\n",
    "print(sent[1:200])\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s an RSP (resource specialist), my  typical day  varies depending on the day of the week.  However, typically,  I push in to classrooms and team teach with the general education teacher and/or teach \n",
      "s an RSP (resource specialist), my  typical day  varies depending on the day of the week.  However, typically,  I push in to classrooms and team teach with the general education teacher and/or teach \n"
     ]
    }
   ],
   "source": [
    "# \\r \\n \\t remove from string python: http://texthandler.com/info/remove-line-breaks-python/\n",
    "sent = sent.replace('\\\\r', ' ')\n",
    "sent = sent.replace('\\\\\"', ' ')\n",
    "sent = sent.replace('\\\\n', ' ')\n",
    "print(sent[1:200])\n",
    "print(sent[1:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s an RSP resource specialist my typical day varies depending on the day of the week However typically I push in to classrooms and team teach with the general education teacher and or teach a small gr\n"
     ]
    }
   ],
   "source": [
    "#remove spacial character: https://stackoverflow.com/a/5843547/4084039\n",
    "sent = re.sub('[^A-Za-z0-9]+', ' ', sent)\n",
    "print(sent[1:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://gist.github.com/sebleier/554280\n",
    "# we are removing the words from the stop words list: 'no', 'nor', 'not'\n",
    "stopwords= ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\",\\\n",
    "            \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', \\\n",
    "            'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their',\\\n",
    "            'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', \\\n",
    "            'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', \\\n",
    "            'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', \\\n",
    "            'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',\\\n",
    "            'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',\\\n",
    "            'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\\\n",
    "            'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', \\\n",
    "            's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', \\\n",
    "            've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn',\\\n",
    "            \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn',\\\n",
    "            \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", \\\n",
    "            'won', \"won't\", 'wouldn', \"wouldn't\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.3 Merging price with project_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        id   price  quantity\n",
      "1  p000002  515.89        21\n",
      "2  p000003  298.97         4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>teacher_id</th>\n",
       "      <th>school_state</th>\n",
       "      <th>Date</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>project_title</th>\n",
       "      <th>project_essay_1</th>\n",
       "      <th>project_essay_2</th>\n",
       "      <th>project_essay_3</th>\n",
       "      <th>project_essay_4</th>\n",
       "      <th>project_resource_summary</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "      <th>clean_categories</th>\n",
       "      <th>clean_subcategories</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>essay</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8393</td>\n",
       "      <td>p205479</td>\n",
       "      <td>2bf07ba08945e5d8b2a3f269b2b3cfe5</td>\n",
       "      <td>CA</td>\n",
       "      <td>2016-04-27 00:27:36</td>\n",
       "      <td>Grades_PreK_2</td>\n",
       "      <td>Engineering STEAM into the Primary Classroom</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>My students come from a variety of backgrounds...</td>\n",
       "      <td>Each month I try to do several science or STEM...</td>\n",
       "      <td>It is challenging to develop high quality scie...</td>\n",
       "      <td>My students need STEM kits to learn critical s...</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>Math_Science</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>725.05</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0       id                        teacher_id school_state  \\\n",
       "0        8393  p205479  2bf07ba08945e5d8b2a3f269b2b3cfe5           CA   \n",
       "\n",
       "                 Date project_grade_category  \\\n",
       "0 2016-04-27 00:27:36          Grades_PreK_2   \n",
       "\n",
       "                                  project_title  \\\n",
       "0  Engineering STEAM into the Primary Classroom   \n",
       "\n",
       "                                     project_essay_1  \\\n",
       "0  I have been fortunate enough to use the Fairy ...   \n",
       "\n",
       "                                     project_essay_2  \\\n",
       "0  My students come from a variety of backgrounds...   \n",
       "\n",
       "                                     project_essay_3  \\\n",
       "0  Each month I try to do several science or STEM...   \n",
       "\n",
       "                                     project_essay_4  \\\n",
       "0  It is challenging to develop high quality scie...   \n",
       "\n",
       "                            project_resource_summary  \\\n",
       "0  My students need STEM kits to learn critical s...   \n",
       "\n",
       "   teacher_number_of_previously_posted_projects  project_is_approved  \\\n",
       "0                                            53                    1   \n",
       "\n",
       "  clean_categories                 clean_subcategories  \\\n",
       "0     Math_Science  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                       teacher_prefix  \\\n",
       "0  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                                               essay   price  quantity  \n",
       "0  I have been fortunate enough to use the Fairy ...  725.05         4  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "price_data = resource_data.groupby('id').agg({'price':'sum', 'quantity':'sum'}).reset_index()\n",
    "project_data = pd.merge(project_data, price_data, on='id', how='left')\n",
    "print (price_data[1:3])\n",
    "project_data.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.3.1 Merge Project Title Count with project_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>teacher_id</th>\n",
       "      <th>school_state</th>\n",
       "      <th>Date</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>project_title</th>\n",
       "      <th>project_essay_1</th>\n",
       "      <th>project_essay_2</th>\n",
       "      <th>project_essay_3</th>\n",
       "      <th>...</th>\n",
       "      <th>project_resource_summary</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "      <th>clean_categories</th>\n",
       "      <th>clean_subcategories</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>essay</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "      <th>project_title_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8393</td>\n",
       "      <td>p205479</td>\n",
       "      <td>2bf07ba08945e5d8b2a3f269b2b3cfe5</td>\n",
       "      <td>CA</td>\n",
       "      <td>2016-04-27 00:27:36</td>\n",
       "      <td>Grades_PreK_2</td>\n",
       "      <td>Engineering STEAM into the Primary Classroom</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>My students come from a variety of backgrounds...</td>\n",
       "      <td>Each month I try to do several science or STEM...</td>\n",
       "      <td>...</td>\n",
       "      <td>My students need STEM kits to learn critical s...</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>Math_Science</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>725.05</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0       id                        teacher_id school_state  \\\n",
       "0        8393  p205479  2bf07ba08945e5d8b2a3f269b2b3cfe5           CA   \n",
       "\n",
       "                 Date project_grade_category  \\\n",
       "0 2016-04-27 00:27:36          Grades_PreK_2   \n",
       "\n",
       "                                  project_title  \\\n",
       "0  Engineering STEAM into the Primary Classroom   \n",
       "\n",
       "                                     project_essay_1  \\\n",
       "0  I have been fortunate enough to use the Fairy ...   \n",
       "\n",
       "                                     project_essay_2  \\\n",
       "0  My students come from a variety of backgrounds...   \n",
       "\n",
       "                                     project_essay_3  ...  \\\n",
       "0  Each month I try to do several science or STEM...  ...   \n",
       "\n",
       "                            project_resource_summary  \\\n",
       "0  My students need STEM kits to learn critical s...   \n",
       "\n",
       "  teacher_number_of_previously_posted_projects  project_is_approved  \\\n",
       "0                                           53                    1   \n",
       "\n",
       "   clean_categories                 clean_subcategories  \\\n",
       "0      Math_Science  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                       teacher_prefix  \\\n",
       "0  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                                               essay   price  quantity  \\\n",
       "0  I have been fortunate enough to use the Fairy ...  725.05         4   \n",
       "\n",
       "   project_title_count  \n",
       "0                    6  \n",
       "\n",
       "[1 rows x 21 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add count (total number of words) in Project Title in each row.\n",
    "\n",
    "project_title_count = project_data['project_title'].str.split().str.len()\n",
    "project_data['project_title_count'] = project_title_count\n",
    "project_data.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.3.2 Essay count of words for each row and  merge with project_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>id</th>\n",
       "      <th>teacher_id</th>\n",
       "      <th>school_state</th>\n",
       "      <th>Date</th>\n",
       "      <th>project_grade_category</th>\n",
       "      <th>project_title</th>\n",
       "      <th>project_essay_1</th>\n",
       "      <th>project_essay_2</th>\n",
       "      <th>project_essay_3</th>\n",
       "      <th>...</th>\n",
       "      <th>teacher_number_of_previously_posted_projects</th>\n",
       "      <th>project_is_approved</th>\n",
       "      <th>clean_categories</th>\n",
       "      <th>clean_subcategories</th>\n",
       "      <th>teacher_prefix</th>\n",
       "      <th>essay</th>\n",
       "      <th>price</th>\n",
       "      <th>quantity</th>\n",
       "      <th>project_title_count</th>\n",
       "      <th>essay_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8393</td>\n",
       "      <td>p205479</td>\n",
       "      <td>2bf07ba08945e5d8b2a3f269b2b3cfe5</td>\n",
       "      <td>CA</td>\n",
       "      <td>2016-04-27 00:27:36</td>\n",
       "      <td>Grades_PreK_2</td>\n",
       "      <td>Engineering STEAM into the Primary Classroom</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>My students come from a variety of backgrounds...</td>\n",
       "      <td>Each month I try to do several science or STEM...</td>\n",
       "      <td>...</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>Math_Science</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>AppliedSciences Health_LifeScience</td>\n",
       "      <td>I have been fortunate enough to use the Fairy ...</td>\n",
       "      <td>725.05</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>285</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0       id                        teacher_id school_state  \\\n",
       "0        8393  p205479  2bf07ba08945e5d8b2a3f269b2b3cfe5           CA   \n",
       "\n",
       "                 Date project_grade_category  \\\n",
       "0 2016-04-27 00:27:36          Grades_PreK_2   \n",
       "\n",
       "                                  project_title  \\\n",
       "0  Engineering STEAM into the Primary Classroom   \n",
       "\n",
       "                                     project_essay_1  \\\n",
       "0  I have been fortunate enough to use the Fairy ...   \n",
       "\n",
       "                                     project_essay_2  \\\n",
       "0  My students come from a variety of backgrounds...   \n",
       "\n",
       "                                     project_essay_3  ...  \\\n",
       "0  Each month I try to do several science or STEM...  ...   \n",
       "\n",
       "  teacher_number_of_previously_posted_projects project_is_approved  \\\n",
       "0                                           53                   1   \n",
       "\n",
       "   clean_categories                 clean_subcategories  \\\n",
       "0      Math_Science  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                       teacher_prefix  \\\n",
       "0  AppliedSciences Health_LifeScience   \n",
       "\n",
       "                                               essay   price quantity  \\\n",
       "0  I have been fortunate enough to use the Fairy ...  725.05        4   \n",
       "\n",
       "   project_title_count  essay_count  \n",
       "0                    6          285  \n",
       "\n",
       "[1 rows x 22 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add count (total number of words) in essay in each row.\n",
    "\n",
    "essay_count = project_data['essay'].str.split().str.len()\n",
    "project_data['essay_count'] = essay_count\n",
    "project_data.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Convert NaN value to mean of the column\n",
    "project_data.fillna(project_data.mean(), inplace=True)\n",
    "project_data.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting data into Train and cross validation(or test): Stratified Sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = project_data['project_is_approved'].values\n",
    "X = project_data.drop(['project_is_approved'], axis=1)\n",
    "\n",
    "# train test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, stratify=y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "catogories_essay = list(project_data['essay'].values)\n",
    "# remove special characters from list of strings python: https://stackoverflow.com/a/47301924/4084039\n",
    "\n",
    "# https://www.geeksforgeeks.org/removing-stop-words-nltk-python/\n",
    "# https://stackoverflow.com/questions/23669024/how-to-strip-a-specific-word-from-a-string\n",
    "# https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string-in-python\n",
    "cat_essay_list = []\n",
    "for i in catogories_essay:\n",
    "    temp = \"\"\n",
    "    # consider we have text like this \"Math & Science, Warmth, Care & Hunger\"\n",
    "    for j in i.split(','): # it will split it in three parts [\"Math & Science\", \"Warmth\", \"Care & Hunger\"]\n",
    "        if 'The' in j.split(): # this will split each of the catogory based on space \"Math & Science\"=> \"Math\",\"&\", \"Science\"\n",
    "            j=j.replace('The','') # if we have the words \"The\" we are going to replace it with ''(i.e removing 'The')\n",
    "        j = j.replace(' ','') # we are placeing all the ' '(space) with ''(empty) ex:\"Math & Science\"=>\"Math&Science\"\n",
    "        temp+=j.strip()+\" \" #\" abc \".strip() will return \"abc\", remove the trailing spaces\n",
    "        temp = temp.replace('&','_') # we are replacing the & value into \n",
    "    cat_essay_list.append(temp.strip())\n",
    "    \n",
    "project_data['clean_essay'] = cat_essay_list\n",
    "\n",
    "from collections import Counter\n",
    "my_counter = Counter()\n",
    "for word in project_data['clean_essay'].values:\n",
    "    my_counter.update(word.split())\n",
    "\n",
    "cat_essay_dict = dict(my_counter)\n",
    "sorted_cat_essay_dict = dict(sorted(cat_essay_dict.items(), key=lambda kv: kv[1]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "catogories_title = list(project_data['project_title'].values)\n",
    "# remove special characters from list of strings python: https://stackoverflow.com/a/47301924/4084039\n",
    "# https://www.geeksforgeeks.org/removing-stop-words-nltk-python/\n",
    "# https://stackoverflow.com/questions/23669024/how-to-strip-a-specific-word-from-a-string\n",
    "# https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string-in-python\n",
    "project_title_list = []\n",
    "for i in catogories_title:\n",
    "    temp = \"\"\n",
    "    # consider we have text like this \"Math & Science, Warmth, Care & Hunger\"\n",
    "    for j in i.split(','): # it will split it in three parts [\"Math & Science\", \"Warmth\", \"Care & Hunger\"]\n",
    "        if 'The' in j.split(): # this will split each of the catogory based on space \"Math & Science\"=> \"Math\",\"&\", \"Science\"\n",
    "            j=j.replace('The','') # if we have the words \"The\" we are going to replace it with ''(i.e removing 'The')\n",
    "        j = j.replace(' ','') # we are placeing all the ' '(space) with ''(empty) ex:\"Math & Science\"=>\"Math&Science\"\n",
    "        temp+=j.strip()+\" \" #\" abc \".strip() will return \"abc\", remove the trailing spaces\n",
    "        temp = temp.replace('&','_') # we are replacing the & value into \n",
    "    project_title_list.append(temp.strip())\n",
    "\n",
    "#project_data.drop('project_title', axis=1, inplace=True)\n",
    "#project_data['project_title'] = project_title_list\n",
    "\n",
    "from collections import Counter\n",
    "my_counter = Counter()\n",
    "for word in project_data['project_title'].values:\n",
    "    my_counter.update(word.split())\n",
    "\n",
    "project_title_dict = dict(my_counter)\n",
    "sorted_project_title_dict = dict(sorted(project_title_dict.items(), key=lambda kv: kv[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combining all the above stundents \n",
    "from tqdm import tqdm\n",
    "X_train_preprocessed_essays = []\n",
    "# tqdm is for printing the status bar\n",
    "for sentance in tqdm(X_train['essay'].values):\n",
    "    sent = decontracted(sentance)\n",
    "    sent = sent.replace('\\\\r', ' ')\n",
    "    sent = sent.replace('\\\\\"', ' ')\n",
    "    sent = sent.replace('\\\\n', ' ')\n",
    "    sent = re.sub('[^A-Za-z0-9]+', ' ', sent)\n",
    "    # https://gist.github.com/sebleier/554280\n",
    "    sent = ' '.join(e for e in sent.split() if e not in stopwords)\n",
    "    X_train_preprocessed_essays.append(sent.lower().strip())\n",
    "   # print (X_train_preprocessed_essays)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combining all the above stundents \n",
    "from tqdm import tqdm\n",
    "X_test_preprocessed_essays = []\n",
    "# tqdm is for printing the status bar\n",
    "for sentance in tqdm(X_test['essay'].values):\n",
    "    sent = decontracted(sentance)\n",
    "    sent = sent.replace('\\\\r', ' ')\n",
    "    sent = sent.replace('\\\\\"', ' ')\n",
    "    sent = sent.replace('\\\\n', ' ')\n",
    "    sent = re.sub('[^A-Za-z0-9]+', ' ', sent)\n",
    "    # https://gist.github.com/sebleier/554280\n",
    "    sent = ' '.join(e for e in sent.split() if e not in stopwords)\n",
    "    X_test_preprocessed_essays.append(sent.lower().strip())\n",
    "   # print (X_test_preprocessed_essays)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Standardize (normalize) the data scale to prep for Logistic regression.\n",
    "Because the distance between pairs of points plays a critical part on the classification, it is necessary to normalize the data This will generate an array of values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.1 Vectorizing Categorical data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- https://www.appliedaicourse.com/course/applied-ai-course-online/lessons/handling-categorical-and-numerical-features/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization of clean_categories for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one \n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000,lowercase=False, binary=True)\n",
    "X_train_categories_one_hot = vectorizer.fit_transform(X_train['clean_categories'].values)\n",
    "X_test_categories_one_hot = vectorizer.transform(X_test['clean_categories'].values)\n",
    "print(vectorizer.get_feature_names())\n",
    "print(\"Shape of matrix X_train_categories_one_hot  after one hot encodig \",X_train_categories_one_hot.shape)\n",
    "print(\"Shape of matrix X_test_categories_one_hot after one hot encodig \",X_test_categories_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Vectorization of project_grade_category for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one \n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000, lowercase=False, binary=True)\n",
    "X_train_project_grade_category_one_hot = vectorizer.fit_transform(X_train['project_grade_category'].values)\n",
    "X_test_project_grade_category_one_hot = vectorizer.transform(X_test['project_grade_category'].values)\n",
    "print(vectorizer.get_feature_names())\n",
    "print(\"Shape of matrix X_train_project_grade_category_one_hot  after one hot encodig \",X_train_project_grade_category_one_hot.shape)\n",
    "print(\"Shape of matrix X_test_project_grade_category_one_hot after one hot encodig \",X_test_project_grade_category_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one \n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000, lowercase=False, binary=True)\n",
    "X_train_school_state_one_hot = vectorizer.fit_transform(X_train['school_state'].values)\n",
    "X_test_school_state_one_hot = vectorizer.transform(X_test['school_state'].values)\n",
    "print(vectorizer.get_feature_names())\n",
    "print(\"Shape of matrix X_train_school_state_one_hot  after one hot encodig \",X_train_school_state_one_hot.shape)\n",
    "print(\"Shape of matrix X_test_school_state_one_hot after one hot encodig \",X_test_school_state_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization of clean_subcategories for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one \n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000,vocabulary=list(sorted_sub_cat_dict.keys()), lowercase=False, binary=True)\n",
    "X_train_sub_categories_one_hot = vectorizer.fit_transform(X_train['clean_subcategories'].values)\n",
    "X_test_sub_categories_one_hot = vectorizer.transform(X_test['clean_subcategories'].values)\n",
    "\n",
    "print(vectorizer.get_feature_names())\n",
    "print(\"Shape of matrix X_train_sub_categories_one_hot  after one hot encodig \",X_train_sub_categories_one_hot.shape)\n",
    "print(\"Shape of matrix X_test_sub_categories_one_hot after oneX_test_sub_categories_one_hot  hot encodig \",X_test_sub_categories_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can do the similar thing with state, teacher_prefix and project_grade_category also"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF of preprocessed_essays for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_model = TfidfVectorizer()\n",
    "tfidf_model.fit(X_train_preprocessed_essays)\n",
    "# we are converting a dictionary with word as a key, and the idf as a value\n",
    "X_train_dictionary = dict(zip(tfidf_model.get_feature_names(), list(tfidf_model.idf_)))\n",
    "X_train_tfidf_words = set(tfidf_model.get_feature_names())\n",
    "print (len(X_train_tfidf_words))\n",
    "X_train_tfidf = tfidf_model.transform(X_train_preprocessed_essays)\n",
    "X_test_tfidf = tfidf_model.transform(X_test_preprocessed_essays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## X_train TFIDF of preprocessed_essays for X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_model = TfidfVectorizer()\n",
    "tfidf_model.fit(X_test_preprocessed_essays)\n",
    "# we are converting a dictionary with word as a key, and the idf as a value\n",
    "X_test_dictionary = dict(zip(tfidf_model.get_feature_names(), list(tfidf_model.idf_)))\n",
    "X_test_tfidf_words = set(tfidf_model.get_feature_names())\n",
    "print (len(X_test_tfidf_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4.2 Vectorizing Text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stronging variables into pickle files python: http://www.jessicayung.com/how-to-use-pickle-to-save-and-load-variables-in-python/\n",
    "# make sure you have the glove_vectors file\n",
    "with open('C:\\\\VipinML\\\\InputData\\\\glove_vectors', 'rb') as f:\n",
    "    model = pickle.load(f)\n",
    "    glove_words =  set(model.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization of preprocessed_essays for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_train_avg_w2v_vectors = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_train_preprocessed_essays): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    cnt_words =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if word in glove_words:\n",
    "            vector += model[word]\n",
    "            cnt_words += 1\n",
    "    if cnt_words != 0:\n",
    "        vector /= cnt_words\n",
    "    X_train_avg_w2v_vectors.append(vector)\n",
    "\n",
    "print(len(X_train_avg_w2v_vectors))\n",
    "print(len(X_train_avg_w2v_vectors[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_test_avg_w2v_vectors = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_test_preprocessed_essays): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    cnt_words =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if word in glove_words:\n",
    "            vector += model[word]\n",
    "            cnt_words += 1\n",
    "    if cnt_words != 0:\n",
    "        vector /= cnt_words\n",
    "    X_test_avg_w2v_vectors.append(vector)\n",
    "\n",
    "print(len(X_test_avg_w2v_vectors))\n",
    "print(len(X_test_avg_w2v_vectors[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TFIDF-W2W Vecorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_train_tfidf_w2v_vectors_pessays = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_train_preprocessed_essays): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    tf_idf_weight =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if (word in glove_words) and (word in X_train_tfidf_words):\n",
    "            vec = model[word] # getting the vector for each word\n",
    "            # here we are multiplying idf value(dictionary[word]) and the tf value((sentence.count(word)/len(sentence.split())))\n",
    "            tf_idf = X_train_dictionary[word]*(sentence.count(word)/len(sentence.split())) # getting the tfidf value for each word\n",
    "            vector += (vec * tf_idf) # calculating tfidf weighted w2v\n",
    "            tf_idf_weight += tf_idf\n",
    "    if tf_idf_weight != 0:\n",
    "        vector /= tf_idf_weight\n",
    "    X_train_tfidf_w2v_vectors_pessays.append(vector)\n",
    "\n",
    "print(len(X_train_tfidf_w2v_vectors_pessays))\n",
    "print(len(X_train_tfidf_w2v_vectors_pessays[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_test_tfidf_w2v_vectors_pessays = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_test_preprocessed_essays): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    tf_idf_weight =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if (word in glove_words) and (word in X_test_tfidf_words):\n",
    "            vec = model[word] # getting the vector for each word\n",
    "            # here we are multiplying idf value(dictionary[word]) and the tf value((sentence.count(word)/len(sentence.split())))\n",
    "            tf_idf = X_test_dictionary[word]*(sentence.count(word)/len(sentence.split())) # getting the tfidf value for each word\n",
    "            vector += (vec * tf_idf) # calculating tfidf weighted w2v\n",
    "            tf_idf_weight += tf_idf\n",
    "    if tf_idf_weight != 0:\n",
    "        vector /= tf_idf_weight\n",
    "    X_test_tfidf_w2v_vectors_pessays.append(vector)\n",
    "\n",
    "print(len(X_test_tfidf_w2v_vectors_pessays))\n",
    "print(len(X_test_tfidf_w2v_vectors_pessays[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_test_tfidf_w2v_vectors_ptitle = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_test['project_title']): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    tf_idf_weight =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if (word in glove_words) and (word in X_test_tfidf_words):\n",
    "            vec = model[word] # getting the vector for each word\n",
    "            # here we are multiplying idf value(dictionary[word]) and the tf value((sentence.count(word)/len(sentence.split())))\n",
    "            tf_idf = X_test_dictionary[word]*(sentence.count(word)/len(sentence.split())) # getting the tfidf value for each word\n",
    "            vector += (vec * tf_idf) # calculating tfidf weighted w2v\n",
    "            tf_idf_weight += tf_idf\n",
    "    if tf_idf_weight != 0:\n",
    "        vector /= tf_idf_weight\n",
    "    X_test_tfidf_w2v_vectors_ptitle.append(vector)\n",
    "\n",
    "print(len(X_test_tfidf_w2v_vectors_ptitle))\n",
    "print(len(X_test_tfidf_w2v_vectors_ptitle[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "\n",
    "X_train_tfidf_w2v_vectors_ptitle = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_train['project_title']): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    tf_idf_weight =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if (word in glove_words) and (word in X_train_tfidf_words):\n",
    "            vec = model[word] # getting the vector for each word\n",
    "            # here we are multiplying idf value(dictionary[word]) and the tf value((sentence.count(word)/len(sentence.split())))\n",
    "            tf_idf = X_train_dictionary[word]*(sentence.count(word)/len(sentence.split())) # getting the tfidf value for each word\n",
    "            vector += (vec * tf_idf) # calculating tfidf weighted w2v\n",
    "            tf_idf_weight += tf_idf\n",
    "    if tf_idf_weight != 0:\n",
    "        vector /= tf_idf_weight\n",
    "    X_train_tfidf_w2v_vectors_ptitle.append(vector)\n",
    "\n",
    "print(len(X_train_tfidf_w2v_vectors_ptitle))\n",
    "print(len(X_train_tfidf_w2v_vectors_ptitle[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization of teacher_prefix  for X_train,X_test, X_cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one hot encoded features\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), vocabulary=list(sorted_teacher_dict.keys()),max_features=5000, lowercase=False, binary=True)\n",
    "X_train_teacher_prefix_data = X_train['teacher_prefix']\n",
    "\n",
    "X_train_teacher_prefix_data.fillna(\"Mrs.\", inplace = True) \n",
    "\n",
    "teacher_prefix_notnull = X_train_teacher_prefix_data[pd.notnull(X_train_teacher_prefix_data)]\n",
    "\n",
    "vectorizer.fit(teacher_prefix_notnull.values)\n",
    "\n",
    "#print(vectorizer.get_feature_names())\n",
    "\n",
    "#print(teacher_prefix_notnull.values)\n",
    "\n",
    "X_train_teacher_prefix_one_hot = vectorizer.fit_transform(teacher_prefix_notnull.values)\n",
    "print(\"Shape of matrix after one hot encodig \",X_train_teacher_prefix_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use count vectorizer to convert the values into one hot encoded features\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer1 = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000,vocabulary=list(sorted_teacher_dict.keys()), lowercase=False, binary=True)\n",
    "X_test_teacher_prefix_data = X_test['teacher_prefix']\n",
    "X_test_teacher_prefix_data.fillna(\"Mrs.\", inplace = True) \n",
    "teacher_prefix_notnull = X_test_teacher_prefix_data[pd.notnull(X_test_teacher_prefix_data)]\n",
    "vectorizer.fit(teacher_prefix_notnull.values)\n",
    "X_test_teacher_prefix_one_hot = vectorizer1.transform(teacher_prefix_notnull.values)\n",
    "print(\"Shape of matrix after one hot encodig \",X_test_teacher_prefix_one_hot.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization of price for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.head(1)\n",
    "X_test.head(1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Normalizer\n",
    "normalizer = Normalizer()\n",
    "# normalizer.fit(X_train['price'].values)\n",
    "# this will rise an error Expected 2D array, got 1D array instead: \n",
    "# array=[105.22 215.96  96.01 ... 368.98  80.53 709.67].\n",
    "# Reshape your data either using \n",
    "# array.reshape(-1, 1) if your data has a single feature \n",
    "# array.reshape(1, -1)  if it contains a single sample.\n",
    "\n",
    "#normalizer.fit(X_train['price'].values.reshape(-1,1))\n",
    "\n",
    "X_train_price_norm = normalizer.transform(X_train['price'].values.reshape(1,-1))\n",
    "X_test_price_norm = normalizer.transform(X_test['price'].values.reshape(1,-1))\n",
    "X_train_price_norm= X_train_price_norm.reshape(-1,1)\n",
    "X_test_price_norm=X_test_price_norm.reshape(-1,1)\n",
    "\n",
    "print(\"After vectorizations\")\n",
    "print (type(X_train_price_norm))\n",
    "print(X_train_price_norm.shape, y_train.shape)\n",
    "print(X_test_price_norm.shape, y_test.shape)\n",
    "\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Normalization of Project Title Count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "#print (X_train['project_title_count'])\n",
    "\n",
    "X_train_project_title_count_norm = normalizer.fit_transform(X_train['project_title_count'].values.reshape(1,-1))\n",
    "X_test_project_title_count_norm = normalizer.transform(X_test['project_title_count'].values.reshape(1,-1))\n",
    "\n",
    "X_train_project_title_count_norm= X_train_project_title_count_norm.reshape(-1,1)\n",
    "X_test_project_title_count_norm=X_test_project_title_count_norm.reshape(-1,1)\n",
    "\n",
    "print(\"After vectorizations\")\n",
    "print(X_train_project_title_count_norm.shape, y_train.shape)\n",
    "print(X_test_project_title_count_norm.shape, y_test.shape)\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Normalization of essay count words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "#print (X_train['project_title_count'])\n",
    "\n",
    "X_train_essay_count_norm = normalizer.fit_transform(X_train['essay_count'].values.reshape(1,-1))\n",
    "X_test_essay_count_norm = normalizer.transform(X_test['essay_count'].values.reshape(1,-1))\n",
    "\n",
    "X_train_essay_count_norm= X_train_essay_count_norm.reshape(-1,1)\n",
    "X_test_essay_count_norm=X_test_essay_count_norm.reshape(-1,1)\n",
    "\n",
    "print(\"After vectorizations\")\n",
    "print(X_train_essay_count_norm.shape, y_train.shape)\n",
    "print(X_test_essay_count_norm.shape, y_test.shape)\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "#print (X_train['project_title_count'])\n",
    "\n",
    "X_train_quantity_norm = normalizer.fit_transform(X_train['quantity'].values.reshape(1,-1))\n",
    "X_test_quantity_norm = normalizer.transform(X_test['quantity'].values.reshape(1,-1))\n",
    "\n",
    "X_train_quantity_norm= X_train_quantity_norm.reshape(-1,1)\n",
    "X_test_quantity_norm=X_test_quantity_norm.reshape(-1,1)\n",
    "\n",
    "print(\"After vectorizations\")\n",
    "print(X_train_quantity_norm.shape, y_train.shape)\n",
    "print(X_test_quantity_norm.shape, y_test.shape)\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Normalizer\n",
    "normalizer = Normalizer()\n",
    "\n",
    "X_train_teacher_number_of_previously_posted_projects_norm = normalizer.fit_transform(X_train['teacher_number_of_previously_posted_projects'].values.reshape(1,-1))\n",
    "X_test_teacher_number_of_previously_posted_projects_norm = normalizer.transform(X_test['teacher_number_of_previously_posted_projects'].values.reshape(1,-1))\n",
    "\n",
    "X_train_teacher_number_of_previously_posted_projects_norm= X_train_teacher_number_of_previously_posted_projects_norm.reshape(-1,1)\n",
    "X_test_teacher_number_of_previously_posted_projects_norm=X_test_teacher_number_of_previously_posted_projects_norm.reshape(-1,1)\n",
    "\n",
    "print(\"After vectorizations\")\n",
    "print(X_train_teacher_number_of_previously_posted_projects_norm.shape, y_train.shape)\n",
    "print(X_test_teacher_number_of_previously_posted_projects_norm.shape, y_test.shape)\n",
    "print(\"=\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of words of preprocessed_essays for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are considering only the words which appeared in at least 10 documents(rows or projects).\n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4),  max_features=5000)\n",
    "X_train_text_bow = vectorizer.fit_transform(X_train_preprocessed_essays)\n",
    "X_test_text_bow = vectorizer.transform(X_test_preprocessed_essays)\n",
    "\n",
    "print(\"Shape of matrix X_train_text_bow after one hot encodig \",X_train_text_bow.shape)\n",
    "print(\"Shape of matrix X_test_text_bow after one hot encodig \",X_test_text_bow.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of words of project_title for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PROJECT_TITLE BOW\n",
    "# We are considering only the words which appeared in at least 10 documents(rows or projects). \n",
    "vectorizer = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000)\n",
    "X_train_project_title_bow = vectorizer.fit_transform(X_train['project_title'])\n",
    "X_test_project_title_bow = vectorizer.transform(X_test['project_title'])\n",
    "\n",
    "print(\"Shape of matrix X_train_project_title_bow after one hot encodig \",X_train_project_title_bow .shape)\n",
    "print(\"Shape of matrix X_test_project_title_bow after one hot encodig \",X_test_project_title_bow .shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF of preprocessed_essays for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train_text_tfidf = vectorizer.fit_transform(X_train_preprocessed_essays)\n",
    "X_test_text_tfidf = vectorizer.transform(X_test_preprocessed_essays)\n",
    "\n",
    "print(\"Shape of matrix X_train_text_tfidf after one hot encodig \",X_train_text_tfidf.shape)\n",
    "print(\"Shape of matrix X_test_text_tfidf after one hot encodig \",X_test_text_tfidf.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF of Project Title for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(min_df=10)\n",
    "X_train_project_title_tfidf = vectorizer.fit_transform((X_train['project_title']))\n",
    "X_test_project_title_tfidf = vectorizer.transform((X_test['project_title']))\n",
    "\n",
    "print(\"Shape of matrix  X_train_project_title_tfidf after one hot encodig \",X_train_project_title_tfidf.shape)\n",
    "print(\"Shape of matrix  X_test_project_title_tfidf after one hot encodig \",X_test_project_title_tfidf.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TFIDF AVG W2V for Project Title for X_train,X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_train_project_title_avg_w2v_vectors = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_train['project_title']): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    cnt_words =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if word in glove_words:\n",
    "            vector += model[word]\n",
    "            cnt_words += 1\n",
    "    if cnt_words != 0:\n",
    "        vector /= cnt_words\n",
    "    X_train_project_title_avg_w2v_vectors.append(vector)\n",
    "\n",
    "print(len(X_train_project_title_avg_w2v_vectors))\n",
    "print(len(X_train_project_title_avg_w2v_vectors[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# average Word2Vec\n",
    "# compute average word2vec for each review.\n",
    "X_test_project_title_avg_w2v_vectors = []; # the avg-w2v for each sentence/review is stored in this list\n",
    "for sentence in tqdm(X_test['project_title']): # for each review/sentence\n",
    "    vector = np.zeros(300) # as word vectors are of zero length\n",
    "    cnt_words =0; # num of words with a valid vector in the sentence/review\n",
    "    for word in sentence.split(): # for each word in a review/sentence\n",
    "        if word in glove_words:\n",
    "            vector += model[word]\n",
    "            cnt_words += 1\n",
    "    if cnt_words != 0:\n",
    "        vector /= cnt_words\n",
    "    X_test_project_title_avg_w2v_vectors.append(vector)\n",
    "\n",
    "print(len(X_test_project_title_avg_w2v_vectors))\n",
    "print(len(X_test_project_title_avg_w2v_vectors[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge two sparse matrices: https://stackoverflow.com/a/19710648/4084039\n",
    "from scipy.sparse import hstack\n",
    "\n",
    "\n",
    "X_tr = hstack((X_train_price_norm,X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot)).tocsr()\n",
    "X_te = hstack((X_test_price_norm,X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot)).tocsr()\n",
    "\n",
    "#print (X_train_price_norm)\n",
    "X_tr_bow = hstack((X_train_price_norm,X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,\\\n",
    "                  X_train_project_title_bow )).tocsr()\n",
    "X_te_bow = hstack((X_test_price_norm,X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,\\\n",
    "                   X_test_project_title_bow)).tocsr()\n",
    "\n",
    "#X_tr_bow = hstack((X_train_price_norm,X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,\\\n",
    "                #   X_train_text_bow,X_train_project_title_bow)).tocsr()\n",
    "#X_te_bow = hstack((X_test_price_norm,X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,\\\n",
    "                  # X_test_text_bow,X_test_project_title_bow)).tocsr()\n",
    "\n",
    "X_tr_tfidf = hstack((X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,X_train_price_norm,\\\n",
    "                     X_train_project_title_tfidf)).tocsr()\n",
    "X_te_tfidf = hstack((X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,X_test_price_norm,\\\n",
    "                     X_test_project_title_tfidf)).tocsr()\n",
    "\n",
    "\n",
    "#X_tr_tfidf = hstack((X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,X_train_price_norm,\\\n",
    "#                     X_train_project_title_tfidf,X_train_tfidf)).tocsr()\n",
    "#X_te_tfidf = hstack((X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,X_test_price_norm,\\\n",
    "#                     X_test_project_title_tfidf,X_test_tfidf)).tocsr()\n",
    "\n",
    "X_tr_tfidf_w2v = hstack((X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,X_train_price_norm,X_train_project_title_avg_w2v_vectors)).tocsr()\n",
    "X_te_tfidf_w2v = hstack((X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,X_test_price_norm,X_test_project_title_avg_w2v_vectors)).tocsr()\n",
    "\n",
    "\n",
    "X_tr_avg_w2v = hstack((X_train_sub_categories_one_hot,X_train_teacher_prefix_one_hot,X_train_price_norm,X_train_project_title_avg_w2v_vectors)).tocsr()\n",
    "X_te_avg_w2v = hstack((X_test_sub_categories_one_hot,X_test_teacher_prefix_one_hot,X_test_price_norm,X_test_project_title_avg_w2v_vectors)).tocsr()\n",
    "\n",
    "#set 5\n",
    "\n",
    "X_te_set5 =hstack((X_test_school_state_one_hot,X_test_categories_one_hot,X_test_sub_categories_one_hot,\\\n",
    "                  X_test_teacher_prefix_one_hot,X_test_quantity_norm,X_test_price_norm,X_test_project_grade_category_one_hot,\\\n",
    "                  X_test_teacher_number_of_previously_posted_projects_norm,X_test_price_norm,\\\n",
    "                  X_test_project_title_count_norm,X_test_essay_count_norm))\n",
    "\n",
    "X_tr_set5 =hstack((X_train_school_state_one_hot,X_train_categories_one_hot, X_train_sub_categories_one_hot,\\\n",
    "                  X_train_teacher_prefix_one_hot,X_train_quantity_norm,X_train_price_norm,X_train_project_grade_category_one_hot,\\\n",
    "                  X_train_teacher_number_of_previously_posted_projects_norm,X_train_price_norm,X_train_project_title_count_norm,\\\n",
    "                  X_train_essay_count_norm))\n",
    "\n",
    "X_te_tfidf_avg_w2v = hstack((X_test_school_state_one_hot,X_test_categories_one_hot,X_test_sub_categories_one_hot,\\\n",
    "                  X_test_tfidf_w2v_vectors_pessays,X_test_tfidf_w2v_vectors_ptitle))\n",
    "\n",
    "X_tr_tfidf_avg_w2v = hstack((X_train_school_state_one_hot,X_train_categories_one_hot, X_train_sub_categories_one_hot,\\\n",
    "                  X_train_tfidf_w2v_vectors_pessays,X_train_tfidf_w2v_vectors_ptitle))\n",
    "\n",
    "#print(\"Final Data matrix\")\n",
    "print(X_tr.shape, y_train.shape)\n",
    "print(X_te.shape, y_test.shape)\n",
    "\n",
    "print(\"=\"*100)\n",
    "\n",
    "print(X_tr_tfidf.shape, y_train.shape)\n",
    "print(X_te_tfidf.shape, y_test.shape)\n",
    "print(\"=\"*100)\n",
    "\n",
    "print(X_tr_set5.shape,y_train.shape )\n",
    "print(X_te_set5.shape,y_test.shape )\n",
    "print(\"=\"*100)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 5: Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ol>\n",
    "    <li><strong>[Task-1] Logistic Regression(either SGDClassifier with log loss, or LogisticRegression) on these feature sets</strong>\n",
    "        <ul>\n",
    "            <li><font color='red'>Set 1</font>: categorical, numerical features + project_title(BOW) + preprocessed_eassay (`BOW with bi-grams` with `min_df=10` and `max_features=5000`)</li>\n",
    "            <li><font color='red'>Set 2</font>: categorical, numerical features + project_title(TFIDF)+  preprocessed_eassay (`TFIDF with bi-grams` with `min_df=10` and `max_features=5000`)</li>\n",
    "            <li><font color='red'>Set 3</font>: categorical, numerical features + project_title(AVG W2V)+  preprocessed_eassay (AVG W2V)</li>\n",
    "            <li><font color='red'>Set 4</font>: categorical, numerical features + project_title(TFIDF W2V)+  preprocessed_essay (TFIDF W2V)</li>        </ul>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li><strong>Hyper paramter tuning (find best hyper parameters corresponding the algorithm that you choose)</strong>\n",
    "        <ul>\n",
    "    <li>Find the best hyper parameter which will give the maximum <a href='https://www.appliedaicourse.com/course/applied-ai-course-online/lessons/receiver-operating-characteristic-curve-roc-curve-and-auc-1/'>AUC</a> value</li>\n",
    "    <li>Find the best hyper paramter using k-fold cross validation or simple cross validation data</li>\n",
    "    <li>Use gridsearch cv or randomsearch cv or you can also write your own for loops to do this task of hyperparameter tuning</li>          \n",
    "        </ul>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li><strong>Representation of results</strong>\n",
    "        <ul>\n",
    "    <li>You need to plot the performance of model both on train data and cross validation data for each hyper parameter, like shown in the figure.\n",
    "    <img src='train_cv_auc.JPG' width=300px></li>\n",
    "    <li>Once after you found the best hyper parameter, you need to train your model with it, and find the AUC on test data and plot the ROC curve on both train and test.\n",
    "    <img src='train_test_auc.JPG' width=300px></li>\n",
    "    <li>Along with plotting ROC curve, you need to print the <a href='https://www.appliedaicourse.com/course/applied-ai-course-online/lessons/confusion-matrix-tpr-fpr-fnr-tnr-1/'>confusion matrix</a> with predicted and original labels of test data points. Please visualize your confusion matrices using <a href='https://seaborn.pydata.org/generated/seaborn.heatmap.html'>seaborn heatmaps.\n",
    "    <img src='confusion_matrix.png' width=300px></li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li><strong>[Task-2] Apply Logistic Regression on the below feature set <font color='red'> Set 5 </font> by finding the best hyper parameter as suggested in step 2 and step 3.</strong>\n",
    "    <li> Consider these set of features <font color='red'> Set 5 :</font>\n",
    "            <ul>\n",
    "                <li><strong>school_state</strong> : categorical data</li>\n",
    "                <li><strong>clean_categories</strong> : categorical data</li>\n",
    "                <li><strong>clean_subcategories</strong> : categorical data</li>\n",
    "                <li><strong>project_grade_category</strong> :categorical data</li>\n",
    "                <li><strong>teacher_prefix</strong> : categorical data</li>\n",
    "                <li><strong>quantity</strong> : numerical data</li>\n",
    "                <li><strong>teacher_number_of_previously_posted_projects</strong> : numerical data</li>\n",
    "                <li><strong>price</strong> : numerical data</li>\n",
    "                <li><strong>sentiment score's of each of the essay</strong> : numerical data</li>\n",
    "                <li><strong>number of words in the title</strong> : numerical data</li>\n",
    "                <li><strong>number of words in the combine essays</strong> : numerical data</li>\n",
    "            </ul>\n",
    "        And apply the Logistic regression on these features by finding the best hyper paramter as suggested in step 2 and step 3 <br>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li><strong>Conclusion</strong>\n",
    "        <ul>\n",
    "    <li>You need to summarize the results at the end of the notebook, summarize it in the table format. To print out a table please refer to this prettytable library<a href='http://zetcode.com/python/prettytable/'>  link</a> \n",
    "        <img src='summary.JPG' width=400px>\n",
    "    </li>\n",
    "        </ul>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>2. Logistic Regression </h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>2.4 Appling Logistic Regression on different kind of featurization as mentioned in the instructions</h2>\n",
    "\n",
    "<br>Apply Logistic Regression on different kind of featurization as mentioned in the instructions\n",
    "<br> For Every model that you work on make sure you do the step 2 and step 3 of instrucations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_predict(clf, data):\n",
    "    # roc_auc_score(y_true, y_score) the 2nd parameter should be probability estimates of the positive class\n",
    "    # not the predicted outputs\n",
    "    y_data_pred = []\n",
    "    tr_loop = data.shape[0] - data.shape[0]%1000\n",
    "    # consider you X_tr shape is 49041, then your tr_loop will be 49041 - 49041%1000 = 49000\n",
    "    # in this for loop we will iterate unti the last 1000 multiplier\n",
    "    for i in range(0, tr_loop, 1000):\n",
    "        y_data_pred.extend(clf.predict_proba(data[i:i+1000])[:,1])\n",
    "    # we will be predicting for the last data points\n",
    "    if data.shape[0]%1000 !=0:\n",
    "        y_data_pred.extend(clf.predict_proba(data[tr_loop:])[:,1])\n",
    "    \n",
    "    return y_data_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we are writing our own function for predict, with defined thresould\n",
    "# we will pick a threshold that will give the least fpr\n",
    "def find_best_threshold(threshould, fpr, tpr):\n",
    "    t = threshould[np.argmax(tpr*(1-fpr))]\n",
    "    # (tpr*(1-fpr)) will be maximum if your fpr is very low and tpr is very high\n",
    "    print(\"the maximum value of tpr*(1-fpr)\", max(tpr*(1-fpr)), \"for threshold\", np.round(t,3))\n",
    "    return t\n",
    "\n",
    "def predict_with_best_t(proba, threshould):\n",
    "    predictions = []\n",
    "    for i in proba:\n",
    "        if i>=threshould:\n",
    "            predictions.append(1)\n",
    "        else:\n",
    "            predictions.append(0)\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic_regression_validation(X_train,y_train,X_test,y_test):\n",
    "    from sklearn import metrics\n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    from sklearn.metrics import roc_auc_score\n",
    "    \"\"\"\n",
    "    y_true : array, shape = [n_samples] or [n_samples, n_classes]\n",
    "    True binary labels or binary label indicators.\n",
    "\n",
    "    y_score : array, shape = [n_samples] or [n_samples, n_classes]\n",
    "    Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of\n",
    "    decisions (as returned by “decision_function” on some classifiers). \n",
    "    For binary y_true, y_score is supposed to be the score of the class with greater label.\n",
    "\n",
    "    \"\"\"\n",
    "    # Create regularization penalty space\n",
    "    penalty = ['l1', 'l2']\n",
    "    penalty = ['l2']\n",
    "\n",
    "    # Create regularization hyperparameter space\n",
    "    C = np.logspace(-3, 3, 10)\n",
    "   # print (C)\n",
    "    \n",
    "     # Create hyperparameter options\n",
    "    Hyper_parameters = dict(C=C, penalty=penalty)\n",
    "    \n",
    "    #Using GridSearchCV\n",
    "    model = GridSearchCV(LogisticRegression(class_weight='balanced',random_state=0,solver='sag'), \\\n",
    "                         Hyper_parameters, scoring = 'f1', cv=3, n_jobs=-1)\n",
    "        \n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    print(\"model.best_estimator_ = %s\" % model.best_estimator_)\n",
    "    print(\"model.score = %s\" % model.score(X_test, y_test))\n",
    "\n",
    "    y_train_pred = model.predict_proba(X_train)    \n",
    "    y_test_pred = model.predict_proba(X_test)\n",
    "\n",
    "   # calculate accuracy of class predictions\n",
    "    from sklearn import metrics\n",
    "   # print (metrics.accuracy_score(y_test, y_test_pred[:, 1]))\n",
    "\n",
    "    #the ROC curve is plotted with TPR against the FPR where TPR is on y-axis and FPR is on the x-axis.\n",
    "    train_fpr, train_tpr, tr_thresholds = roc_curve(y_train, y_train_pred[:, 1])\n",
    "    test_fpr, test_tpr, te_thresholds = roc_curve(y_test, y_test_pred[:, 1])\n",
    "    trainAUC= float(\"{0:.2f}\".format(auc(train_fpr, train_tpr)))\n",
    "    testAUC = float(\"{0:.2f}\".format(auc(test_fpr, test_tpr)))\n",
    "\n",
    "    \n",
    "    plt.plot(train_fpr, train_tpr, label=\"train AUC =\"+str(trainAUC))\n",
    "    plt.plot(test_fpr, test_tpr, label=\"Test AUC =\"+str(testAUC))\n",
    "    plt.legend()\n",
    "    plt.xlabel(\"FPR\")\n",
    "    plt.ylabel(\"TPR\")\n",
    "    plt.title(\"AUC PLOT\")\n",
    "    plt.grid()\n",
    "    plt.show()\n",
    "     \n",
    "    print(\"=\"*100)\n",
    "    from sklearn.metrics import confusion_matrix\n",
    "    best_t = find_best_threshold(tr_thresholds, train_fpr, train_tpr)\n",
    "    print(\"Train confusion matrix\")\n",
    "    print(confusion_matrix(y_train, predict_with_best_t(y_train_pred[:, 1], best_t)))\n",
    "    print(\"Test confusion matrix\")\n",
    "    print(confusion_matrix(y_test, predict_with_best_t(y_test_pred[:, 1], best_t)))\n",
    "    \n",
    "    return [trainAUC,testAUC]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logistic_regression_for_Best_Hyper_Parameter(X_train,y_train,X_test,y_test, HyperParameter):\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    from sklearn.metrics import roc_auc_score\n",
    "\n",
    "    model = LogisticRegression(class_weight='balanced',C=HyperParameter)\n",
    "\n",
    "    model.fit(X_train, y_train)\n",
    "    \n",
    "    y_train_pred = model.predict_proba(X_train)    \n",
    "    y_test_pred = model.predict_proba(X_test)\n",
    "   \n",
    "    #The ROC curve is plotted with TPR against the FPR where TPR is on y-axis and FPR is on the x-axis.\n",
    "    train_fpr, train_tpr, tr_thresholds = roc_curve(y_train, y_train_pred[:, 1])\n",
    "    test_fpr, test_tpr, te_thresholds = roc_curve(y_test, y_test_pred[:, 1])\n",
    "\n",
    "    import seaborn as sns\n",
    "   \n",
    "    print(\"=\"*100)\n",
    "    from sklearn.metrics import confusion_matrix\n",
    "    best_t = find_best_threshold(tr_thresholds, train_fpr, train_tpr)\n",
    "    print(\"Train confusion matrix\")\n",
    "    train_matrix = confusion_matrix(y_train, predict_with_best_t(y_train_pred[:, 1], best_t))\n",
    "    print(train_matrix)\n",
    "                                   \n",
    "    print(\"Test confusion matrix\")\n",
    "    test_matrix = confusion_matrix(y_test, predict_with_best_t(y_test_pred[:, 1], best_t))\n",
    "    print(test_matrix)\n",
    "   \n",
    "    trainAUC= float(\"{0:.2f}\".format(auc(train_fpr, train_tpr)))\n",
    "    testAUC = float(\"{0:.2f}\".format(auc(test_fpr, test_tpr)))\n",
    "\n",
    "    \n",
    "    plt.plot(train_fpr, train_tpr, label=\"train AUC =\"+str(trainAUC))\n",
    "    plt.plot(test_fpr, test_tpr, label=\"Test AUC =\"+str(testAUC))\n",
    "    \n",
    "    # plot AUC curve. AUC curve should show best accuracy rate, since best aplha is used in the logic.\n",
    "    plt.plot(train_fpr, train_tpr, label=\"train AUC =\"+str(trainAUC))\n",
    "    plt.plot(test_fpr, test_tpr, label=\"Test AUC =\"+str(testAUC))\n",
    "    plt.legend()\n",
    "    plt.xlabel(\"FPR\")\n",
    "    plt.ylabel(\"TPR\")\n",
    "    plt.title(\"AUC PLOT\")\n",
    "    plt.grid()\n",
    "    \n",
    "     # Confusiomatrix heatmap.\n",
    "    \n",
    "    fig, (ax1, ax2) = plt.subplots(1,2,sharex=True, sharey=True)\n",
    "  #  plt.figure(figsize=(30, 60))\n",
    "    g1=sns.heatmap(test_matrix, annot=True,fmt='',cbar=True, linewidths =0.3, ax=ax1)\n",
    "    g1.set_xlabel(\"Test confusion matrix\")\n",
    "    g1.axes.get_xaxis().set_visible(True)\n",
    "    g1.axes.get_yaxis().set_visible(False)\n",
    "        \n",
    "    g2=sns.heatmap(train_matrix,annot=True,fmt='',cbar=True,linewidths =0.3,ax=ax2)\n",
    "    g2.set_xlabel(\"Train confusion matrix\")\n",
    "    g2.axes.get_xaxis().set_visible(True)\n",
    "    g2.axes.get_yaxis().set_visible(False)\n",
    "    \n",
    "    plt.show()\n",
    "    return [trainAUC,testAUC]\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the normalized data into training and test sets\n",
    "Logic below is simialr as covred in kanalysis_cross_validation(X,y), but here logic is for to  calculate confusion matrix, acuarcy ration for best K as we already foound best K after trying best accuracy for multiple K values.\n",
    "We can apply  K -fold CV to either the hyperparameter tuning, performance reporting, or both. The advantage of this approach is that the performance is less sensitive to unfortunate splits of data. In addition, it utilize data better since each example can be used for both training and validation/testing.\n",
    "\n",
    "Let's use  K -Fold CV to select the hyperparamter n_neighbors of the KNeighborsClassifier:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to speculate the performance of the model using ROC Curve?\n",
    "#### An excellent model has AUC near to the 1 which means it has good measure of separability. A poor model has AUC near to the 0 which means it has worst measure of separability. In fact it means it is reciprocating the result. It is predicting 0s as 1s and 1s as 0s. And when AUC is 0.5, it means model has no class separation capacity whatsoever"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the normalized data into training and test sets\n",
    "This step is required to prepare us for the fitting (i.e. training) the #model later. The “X” variable is a collection of all the features. The “y” variable is the target label which specifies the #classification of 1 or 0 based. Our goal will be to identify which category the new data point should fall into. Evaluate the predictions. Evaluate the Model by reviewing the classification report or confusion matrix. By reviewing these tables, we are able to evaluate how accurate our model is with new values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function is used for passsing various Hyper parameter in a loop,  and get the best parameter \n",
    "# that would give best accuracy. For  each Hyperparemer, predicted value and accuracy  is calculated. \n",
    "# best Hyperparameter  is reHyper  for best accuracy. this is like gridCVSearch but shown plots for various hymer parametr.\n",
    "# after best Hyperparam is returnbed, that is used  and again best AUC plot is drawn.\n",
    "# Items 4 in your query, first three items are covered here.\n",
    "\n",
    "\n",
    "def LogicRegression_HyperParam_Analysis(X_train,y_train,X_test,y_test):\n",
    "    from sklearn import model_selection\n",
    "    from mlxtend.plotting import plot_decision_regions\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.metrics import accuracy_score\n",
    "    from sklearn.metrics import roc_auc_score\n",
    "    import math\n",
    "    # Import classification report and confusion matrix to evaluate predictions\n",
    "    from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "    train_auc = []\n",
    "    test_auc = []\n",
    "    #Hyperparams = [10**-6, 10**-5,10**-4, 10**-2,10**0, 10,20, 30,10**2, 10**3, 10**4]\n",
    "    Hyperparams = np.logspace(-4,4, 30)\n",
    "    #print (Hyperparams)\n",
    "    #Hyperparams= [10**x for x in range (-4,5)]\n",
    "    \n",
    "       \n",
    "    best_accuracy=0.0001\n",
    "    LogHyperparams =[]\n",
    "    for i in Hyperparams:\n",
    "        if (i >0  and math.log(i) >0):\n",
    "            HyperParameter = math.log(i)\n",
    "            LogHyperparams.append(HyperParameter)\n",
    "            model = LogisticRegression(class_weight='balanced',C=HyperParameter)\n",
    "\n",
    "            # fitting the model on crossvalidation train\n",
    "            model.fit(X_train, y_train)\n",
    "\n",
    "            # predict the response on the crossvalidation train\n",
    "            y_train_pred = model.predict(X_train)  # predicting the value using cross validation data. \n",
    "\n",
    "            # predict the response on the crossvalidation test\n",
    "            y_test_pred = model.predict(X_test)  # predicting the value using cross validation data. \n",
    "\n",
    "\n",
    "            # evaluate CV accuracy\n",
    "            acc = accuracy_score(y_test, y_test_pred, normalize=True) * float(100)  # I get the accuracy score. \n",
    "            print (\"accuracy score = %s, Hyper Parameter  = %s \" , acc, HyperParameter)\n",
    "            if acc > best_accuracy:\n",
    "                best_accuracy =acc\n",
    "                HyperParam = HyperParameter\n",
    "                \n",
    "               \n",
    "           # print('\\n Test Accuracy for Hyper Parameter = %s is %s' % (HyperParameter, acc))\n",
    "           # print(\"=========================================\")\n",
    "\n",
    "            # roc_auc_score(y_true, y_score) the 2nd parameter should be probability estimates of the positive class\n",
    "            # not the predicted outputs        \n",
    "            train_auc.append(roc_auc_score(y_train,y_train_pred))\n",
    "            test_auc.append(roc_auc_score(y_test, y_test_pred))\n",
    "        else:\n",
    "            continue\n",
    "    \n",
    "    print('\\n Test Accuracy for Hyper Parameter = %s is %s' % (HyperParam, best_accuracy))\n",
    "    \n",
    "    plt.plot(LogHyperparams, train_auc, label='Train AUC')\n",
    "    plt.plot(LogHyperparams, test_auc, label='Test AUC')\n",
    "\n",
    "    plt.scatter(LogHyperparams, train_auc, label='Train AUC points')\n",
    "    plt.scatter(LogHyperparams, test_auc, label='Test AUC points')\n",
    "\n",
    "    plt.legend()\n",
    "    plt.xlabel(\"Log (Hyperparams: Hyperparameter)\")\n",
    "    plt.ylabel(\"AUC\")\n",
    "    plt.title(\"AUC vrs Log(Hyperparameter)\")\n",
    "    plt.grid()\n",
    "    plt.show()\n",
    "       \n",
    "    return HyperParam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.0 Applying Logistic Regression  Set 1: categorical, numerical features + project_title(BOW) + preprocessed_essay (BOW),<font color='red'> SET 1</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Logistic Regression feature selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import warnings filter\n",
    "from warnings import simplefilter\n",
    "# ignore all future warnings\n",
    "simplefilter(action='ignore', category=FutureWarning)\n",
    "trainAUC, testAUC = logistic_regression_validation (X_tr,y_train,X_te,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.1 Applying Logistic Regression on BOW - Set 1: categorical, numerical features + project_title(BOW) + preprocessed_essay (BOW),<font color='red'> SET 1</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainAUC_bow, testAUC_bow = logistic_regression_validation (X_tr_bow,y_train,X_te_bow,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.2 Applying Logistic Regression  on TFIDF Set 2: categorical, numerical features + project_title(TFIDF)+ preprocessed_essay (TFIDF),<font color='red'> SET 2</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainAUC_tfidf, testAUC_tfidf = logistic_regression_validation (X_tr_tfidf,y_train,X_te_tfidf,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.2 Applying Logistic Regression on TFIDF Avg W2V <font color='red'> SET 2</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainAUC_avg_w2v, testAUC_avg_w2v = logistic_regression_validation (X_tr_tfidf_avg_w2v,y_train,X_te_tfidf_avg_w2v,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.3 Applying Logistic Regression on AVG W2V,<font color='red'> SET 3</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "trainAUC_avg_w2v, testAUC_avg_w2v = logistic_regression_validation (X_tr_avg_w2v,y_train,X_te_avg_w2v,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.4 Applying Logistic Regression on TFIDF W2V,<font color='red'> SET 4</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainAUC_tfidf_w2v, testAUC_tfidf_w2v = logistic_regression_validation(X_tr_tfidf_w2v,y_train,X_te_tfidf_w2v,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.2 Applying Logistic Regression  on TFIDF Set 5: categorical, numerical features <font color='red'> SET 5</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import warnings filter\n",
    "from warnings import simplefilter\n",
    "# ignore all future warnings\n",
    "simplefilter(action='ignore', category=FutureWarning)\n",
    "trainAUA_set, testAUC_set = logistic_regression_validation (X_tr_set5,y_train,X_te_set5,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5  Feature selection for Best Hyper Parameter /font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 2.5.1 <font color='red'> Hyper Param-Analysis  <font color='blue'> categorical, numerical features + project_title(BOW) + preprocessed_essay (BOW),<font color='red'> SET 1</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_hyperparam= LogicRegression_HyperParam_Analysis(X_tr_bow,y_train,X_te_bow,y_test)\n",
    "print (\"Hyper Param to apply is %s\" % bow_hyperparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5.2 <font color='red'> Hyper Param -Analysis   <font color='blue'>  on TFIDF Set 2: categorical, numerical features + project_title(TFIDF)+ preprocessed_essay (TFIDF),<font color='red'> SET 2</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tfidf_hyperparam= LogicRegression_HyperParam_Analysis(X_tr_tfidf,y_train,X_te_tfidf,y_test)\n",
    "print (\"Hyper Param to apply is %s\" % tfidf_hyperparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5.3 <font color='red'> Hyper Parameter-Analysis     <font color='blue'>  on AVG W2V - categorical, numerical features + project_title(AVG W2V )+ preprocessed_essay (AVG W2V ),<font color='red'> SET 3</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avgw2v_hyperparam= LogicRegression_HyperParam_Analysis(X_tr_avg_w2v,y_train,X_te_avg_w2v,y_test)\n",
    "print (\"Hyper Param to apply is %s\" % avgw2v_hyperparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5.3 <font color='red'> Hyper Parameter-Analysis     <font color='blue'>  on TFIDF W2V - categorical, numerical features + project_title(TFIDF W2V )+ preprocessed_essay (TFIDF W2V ),<font color='red'> SET 3</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidfw2v_hyperparam= LogicRegression_HyperParam_Analysis(X_tr_tfidf_w2v,y_train,X_te_tfidf_w2v,y_test)\n",
    "print (\"Hyper Param to apply is %s\" % tfidfw2v_hyperparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper Param Analysis on TFIDF Set 5: categorical, numerical features + SET 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logic Regression Analysis on Best Hyper Parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set5_hyperparam = LogicRegression_HyperParam_Analysis(X_tr_set5,y_train,X_te_set5,y_test)\n",
    "print (\"Hyper Param to apply is %s\" % set5_hyperparam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidfw2v_hyperparam = LogicRegression_HyperParam_Analysis(X_tr_tfidf_w2v,y_train,X_te_tfidf_w2v,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HyperParameter = set5_hyperparam\n",
    "trainAUC_analysis, testAUC_analysis = logistic_regression_for_Best_Hyper_Parameter(X_tr_set5,y_train,X_te_set5,y_test,HyperParameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HyperParameter =bow_hyperparam\n",
    "trainAUC_bow_analysis, testAUC_bow_analysis =logistic_regression_for_Best_Hyper_Parameter(X_tr_bow,y_train,X_te_bow,y_test, HyperParameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HyperParameter = avgw2v_hyperparam\n",
    "trainAUC_avg_w2v_analysis, testAUC_avg_w2v_analysis = logistic_regression_for_Best_Hyper_Parameter(X_tr_avg_w2v,y_train,X_te_avg_w2v,y_test, HyperParameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HyperParameter = tfidf_hyperparam\n",
    "trainAUC_tfidf_analysis, testAUC_tfidf_analysis = logistic_regression_for_Best_Hyper_Parameter(X_tr_tfidf,y_train,X_te_tfidf,y_test,HyperParameter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HyperParameter = tfidfw2v_hyperparam\n",
    "trainAUC_tfidf_w2v_analysis, testAUC_tfidf_w2v_analysis = logistic_regression_for_Best_Hyper_Parameter(X_tr_tfidf_w2v,y_train,X_te_tfidf_w2v,y_test,HyperParameter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.0 <font color='Blue'> Pretty Table  SET 3</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Draw Pretty Table using GridCVSearch\n",
    "\n",
    "from prettytable import PrettyTable\n",
    "    \n",
    "LRTable = PrettyTable()\n",
    "\n",
    "LRTable.field_names = [\"Model Type\", \"Train AUC\", \"Test AUC\"]\n",
    "LRTable.add_row([\"Regular\", trainAUC, testAUC])\n",
    "LRTable.add_row([\"BoW\", trainAUC_bow, testAUC_bow])\n",
    "LRTable.add_row([\"TFIDF\", trainAUC_tfidf, testAUC_tfidf])\n",
    "LRTable.add_row([\"TFIDF_Avg_W2V\", trainAUC_avg_w2v, testAUC_avg_w2v])\n",
    "LRTable.add_row([\"TFIDF_W2V\", trainAUC_tfidf_w2v, testAUC_tfidf_w2v])\n",
    "print (LRTable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Draw Pretty Table using Best Analysis. Pretty table is drawn based on \n",
    "#best AUC is calcukated by passing varipus Hyperparameter in loop.\n",
    "\n",
    "from prettytable import PrettyTable\n",
    "    \n",
    "LRTable = PrettyTable()\n",
    "\n",
    "LRTable.field_names = [\"Model Type\", \"Train AUC\", \"Test AUC\"]\n",
    "LRTable.add_row([\"Regular_Analyis\", trainAUC_analysis, testAUC_analysis])\n",
    "LRTable.add_row([\"BoW_Analysis\", trainAUC_bow_analysis, testAUC_bow_analysis])\n",
    "LRTable.add_row([\"TFIDF_Analysis\", trainAUC_tfidf, testAUC_tfidf])\n",
    "LRTable.add_row([\"TFIDF_Avg_W2V_Analysis\", trainAUC_avg_w2v_analysis, testAUC_avg_w2v_analysis])\n",
    "LRTable.add_row([\"TFIDF_W2V_Analysis\", trainAUC_tfidf_w2v_analysis, testAUC_tfidf_w2v_analysis])\n",
    "print (LRTable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>3. Conclusions</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic Regression is a stastical method for analyzing a dataset in which there are one or more independent variables that determine the outcome. \n",
    "\n",
    "## Summary of above program as below:\n",
    "\n",
    "Lot of plots are drawn for different data set between train and test data. Test data is very near to train data.\n",
    "Please see pretty table for all comparasions.\n",
    "\n",
    "### Step 1: Import the necessary Libraries\n",
    "we will need to import libraries that allow for data analysis and data visualization to get acclimated to the dataset. We will be using pandas, numpy, matplotlib and seaborn to conduct this. Data Exploration libraries\n",
    "\n",
    "### Step 2: Read in the dataset.\n",
    "We will use the pandas .read_csv() method to read in the dataset. Then we will use the. head() method to observe the first few rows of the data, to understand the information better. In our case, the feature(column) headers tell us pretty little. This is fine because we are merely trying to gain insight via classifying new data points by referencing it’s neighboring elements.\n",
    "\n",
    "### Step 3: Standardize (normalize) the data scale to prep for Logistic regression.\n",
    "Because the distance between pairs of points plays a critical part on the classification, it is necessary to normalize the data This will generate an array of values. \n",
    "\n",
    "### Step 4: Split the normalized data into training and test sets.\n",
    "This step is required to prepare us for the fitting (i.e. training) the model later. The “X” variable is a collection of all the features. The “y” variable is the target label which specifies the classification of 1 or 0 based. Our goal will be to identify which category the new data point should fall into.\n",
    "\n",
    "\n",
    "### Step 5: Create and Train the Model.\n",
    "Here we create a Logistic Regression Object and use the .fit() method to train the model. Upon completion of the model we should receive confirmation that the training has been complete\n",
    "\n",
    "Please see functions as covered below, used in above program: def logistic_regression_validation(X,y): def \n",
    "\n",
    "### Step 6: Make Predictions.\n",
    "Here we review where our model was accurate and where it misclassified elements.\n",
    "\n",
    "Please see functions as covered below, used in above program: def logistic_regression_validation(X,y):\n",
    "\n",
    "### Step 7: Evaluate the predictions.\n",
    "\n",
    "Evaluate the Model by reviewing the classification report or confusion matrix. By reviewing these tables, we are able to evaluate how accurate our model is with new values.\n",
    "\n",
    "def logistic_regression_validation(X,y):\n",
    "\n",
    "### Setp 8:Classification Report :\n",
    "This tells us our model was around 84% accurate… Print out classification report and confusion matrix\n",
    "\n",
    "I have covered various set to show confusion matrix.\n",
    "\n",
    "Please see section 2. covered various data sets and created confusion matrix.\n",
    "\n",
    "### Step 9: Evaluate alternative Hyper Parameter for better predictions.\n",
    "To simplify the process of evaluating multiple cases of Alpha values, we create a function to derive the error using the average where our predictions were not equal to the test values.\n",
    "\n",
    "Please see section 2. covered various data sets and created error accuracy reports.\n",
    "\n",
    "### Step 10: Adjust Hyper Parameter value per error rate evaluations \n",
    "This is just fine tuning our model to increase accuracy. We will need to retrain our model with the new Alpha.\n",
    "Please see section 3 in above program. we have created confusion matrix for optimal Alpha value for various data sets. As we can see for optimal Alpha, Accuracy is much higher - so prediction is much better.\n",
    "\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "+"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}