04_Machine_learning_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4. Machine Learning Classification\n",
    "App to analyze web-site search logs (internal search)<br>\n",
    "**This script:** scikit-learn for site-search classifications<br>\n",
    "Authors: dan.wendling@nih.gov, <br>\n",
    "Last modified: 2018-09-09\n",
    "\n",
    "\n",
    "## THIS SCRIPT (A WORK IN PROGRESS)\n",
    "\n",
    "Some rough, not-quite in order, partly not-functioning machine learning \n",
    "code that will eventually result in a dataframe of classification choices \n",
    "for the ~30% of visitor search queries that the UMLS Metathesaurus is not \n",
    "able to classify into broader categories (see 01_Pre-processing.py). \n",
    "Some entries will be automatically assignable if misspellings can be \n",
    "overcome with confidence.\n",
    "\n",
    "Desired end result in the future (feature name / column name is first row):\n",
    "\n",
    "| adjustedQueryCase  | preferredTerm                     | SemanticType       | SemanticGroup |\n",
    "| gallbladder cancer | Malignant neoplasm of gallbladder | Neoplastic Process | Disorders     |\n",
    "\n",
    "\n",
    "What this script is trying to generate currently:\n",
    "    \n",
    "| adjustedQueryCase  | UmlsApproximate | pred-LinearSVC   | pred-LogisticRegression | pred-NaiveBayesMultinomial |\n",
    "| cbd oil            | nan             | Organic Chemical | Intellectual Product    | Organic Chemical           |\n",
    "\n",
    "(cbd oil is a marajuana-based product that many visitors seem to be \n",
    "interested in. Variations on cbd will be avilable in the training set. \n",
    "(An alternative to this work is clustering, but would it cluster correctly...)\n",
    "\n",
    "\n",
    "Feature/column explanations:\n",
    "    \n",
    "adjustedQueryCase - Lowercased version of query with most punctuation removed.\n",
    "preferredTerm - Can be assigned later if needed, but these are the UMLS system\n",
    "    picks from more than 200 medical vocabularies. 01_Pre-processing.py will \n",
    "    assign when possible; this script does not use, as described below.\n",
    "    The UMLS system has 10s of thousands of these or more.\n",
    "SemanticType - Select one of 130 ontology categories. Eventually I will need\n",
    "    to select two or more of these categories for searches that look like \n",
    "    \"cancer exercise.\" For now I am okay capturing one category.\n",
    "SemanticGroup - One of 15 ontology super-categories.\n",
    "UmlsApproximate - A new run of the UMLS API that will be set to more liberal\n",
    "matching.\n",
    "pred-LinearSVC, etc. - Predictions from the various models. This can be used for eyeballing\n",
    "entries and adding manual assignments, to get the under-represented classes \n",
    "some more content for future matching.\n",
    "\n",
    "\n",
    "## Script contents\n",
    "\n",
    "1. Start-up / What to put into place, where; dataframe mods\n",
    "2. Eyeball level of balance among classes under study, in training set\n",
    "3. Training set: Calculate tf-idf vector for each query\n",
    "4. Training set, Chi square: Find the terms most correlated with each item\n",
    "5. Train, test, predict with a multi-class classifier, Naive Bayes-Multinomial\n",
    "6. Model selection - Which among four models is the BEST model for this dataset?\n",
    "\n",
    "Asperational, not working\n",
    "7. Look deeper, with a confusion matrix, into the most successful model of\n",
    "   our group, LinearSVC (Linear Support Vector Classification)\n",
    "8. Understand the misclassifications. Should we change the model, or not?\n",
    "\n",
    "Somewhat working\n",
    "9. Chi-square to find terms MOST CORRELATED with each category\n",
    "10. TryLinearSVCdf - Unmatched terms with LinearSVC\n",
    "11. TryLogisticRegressiondf - Unmatched terms with LogisticRegression\n",
    "\n",
    "Not working\n",
    "12. Final report by category\n",
    "\n",
    "\n",
    "## FIXMEs\n",
    "\n",
    "Things Dan wrote for Dan; modify as needed. There are more FIXMEs in context.\n",
    "\n",
    "* [ ] \n",
    "\n",
    "- Biggest problem: I have under-represented classes such as for NLM\n",
    "  products, that we are building manually. See file search-seed_the_ML.xlsx - \n",
    "  these are not matchable to the UMLS API as currently configured (it is \n",
    "  configured for high-confidence matches). We're working now to improve \n",
    "  category prediction for things not found in the UMLS datasets, such as \n",
    "  misspellings, NLM products and services, partial NLM Web page titles \n",
    "  (I scrape the site so I have a file of these, but they are verbose), \n",
    "  historical names, commercial product names, etc. These will be added \n",
    "  to the \"GoldStandard\" file. We started with the highest-frequency \n",
    "  unmatched; hopefully ML can take over some or most of this. Clustering \n",
    "  and FuzzyWuzzy will probably help here.\n",
    "  \n",
    "- Dan will add second and third runs for the UMLS API, as described in \n",
    "  01_Pre-processing.py, to resolve non-English queries and provide a feature\n",
    "  (column) of UMLS API guesses, whose prediction scores were too low to \n",
    "  return in the \"normalized string\" procedure I am using in the single \n",
    "  current UMLS API run. Then an editor can perhaps choose among the UMLS, \n",
    "  LinearSVC, LogisticRegression, etc., predictions.\n",
    "  \n",
    "- For the ML code below, I am trying to assign from the ~130 \n",
    "  *SemanticTypeName* categories (see file 01_Pre-processing_files/\n",
    "  SemanticNetworkReference.xlsx). I think using *SemanticTypeName* is \n",
    "  best for the project; we could also try to match to the 15 \n",
    "  super-categories and then create more routines to match to the 130 \n",
    "  sub-categories.\n",
    "  \n",
    "- Add FuzzyWuzzy, perhaps fix misspellings in place (in col adjustedQueryCase).\n",
    "  If confidence is high that it will be the right fix...\n",
    "- Add stemming, lemmatization?\n",
    "- Future: Add the ability to assign one query to multiple categories.\n",
    "- More FIXMEs may appear in context below.\n",
    "\n",
    "\n",
    "## INFLUENCES\n",
    "\n",
    "- Susan Li, https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f\n",
    "  (This code came from her code; I don't know what all of it does. Still has some of her info in comments)\n",
    "- Andreas Mueller, https://github.com/amueller/introduction_to_ml_with_python\n",
    "  (I am looking to add procedures from here that will assist in manual assign-\n",
    "  ments. Not sure what to add; LDA-based charts look useful.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Start-up / What to put into place, where; dataframe mods\n",
    "\n",
    "Training file: ApiAssignedSearches.xlsx (successful matches)\n",
    "Unmatched terms we want to predict for: search-seed_the_ML.xlsx\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import pie, axis, show\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# Set working directory\n",
    "os.chdir('/Users/wendlingd/Projects/webDS/_util')\n",
    "\n",
    "\n",
    "'''\n",
    "Bring in and adjust training file, ApiAssignedSearches.xlsx - previous \n",
    "successful assignments; we will need adjustedQueryCase, preferredTerm, \n",
    "SemanticTypeName, SemanticGroup...\n",
    "'''\n",
    "\n",
    "df = pd.read_excel('02_UMLS_API_files/ApiAssignedSearches.xlsx')\n",
    "\n",
    "\n",
    "# OR...\n",
    "# df = ApiAssignedSearches\n",
    "\n",
    "\n",
    "# Don't use preferredTerm for now - will be too inaccurate\n",
    "df = df.drop(['preferredTerm'], axis=1) # drop col\n",
    "\n",
    "# Don't try to process any non-Roman characters; eyeball and remove\n",
    "# df = df[17:]\n",
    "\n",
    "df.info()\n",
    "# df.head()\n",
    "# df.columns\n",
    "\n",
    "# 6/23: Trouble with fit, so trying this - remove integer data type, perhaps\n",
    "# Remove int values in adjustedQueryCase by removal or coerced data type change\n",
    "df['adjustedQueryCase'] = df['adjustedQueryCase'].astype(str)\n",
    "\n",
    "\n",
    "df = df.sort_values(by='adjustedQueryCase', ascending=True)\n",
    "df = df.reset_index()\n",
    "df.drop(['index'], axis=1, inplace=True)\n",
    "\n",
    "'''\n",
    "df.drop(12038, inplace=True)\n",
    "df.drop(10714, inplace=True)\n",
    "df.drop(6822, inplace=True)\n",
    "'''\n",
    "df.drop(26905, inplace=True)\n",
    "df.drop(26904, inplace=True)\n",
    "df.drop(26903, inplace=True)\n",
    "\n",
    "'''\n",
    "To preserve changes to training file for future sessions\n",
    "\n",
    "# Useful to write out the cleaned up version; if you do re-processing, you can skip a bunch of work.\n",
    "writer = pd.ExcelWriter('01_Pre-processing_files/ApiAssignedSearches.xlsx')\n",
    "df.to_excel(writer,'ApiAssignedSearches')\n",
    "# df2.to_excel(writer,'Sheet2')\n",
    "writer.save()\n",
    "'''\n",
    "\n",
    "# add a column encoding the product as an integer, because categorical \n",
    "# variables are often better represented by integers than strings\n",
    "df['category_id'] = df['SemanticTypeName'].factorize()[0]\n",
    "\n",
    "# create a couple of dictionaries for future use\n",
    "category_id_df = df[['SemanticTypeName', \n",
    "                     'category_id']].drop_duplicates().sort_values('category_id')\n",
    "category_to_id = dict(category_id_df.values)\n",
    "id_to_category = dict(category_id_df[['category_id', 'SemanticTypeName']].values)\n",
    "\n",
    "# what the first rows look like after the mods\n",
    "df.head()\n",
    "\n",
    "# Bring in entries to match\n",
    "unassignedAfterUmls1 = pd.read_excel('02_UMLS_API_files/unassignedAfterUmls1.xlsx')\n",
    "unassignedAfterUmls1 = unassignedAfterUmls1.drop(['timesSearched'], axis=1) # drop col\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "'''\n",
    "*** I'M STUCK ON THIS ONE (FILE BELOW) ***\n",
    "\n",
    "Not sure of your definition of fuzzy matching, but I use it to describe \n",
    "misspellings and also this - verbose product, service, etc. names.\n",
    "\n",
    "Advice on how this should be implemented would be very useful!\n",
    "\n",
    "People often look for words within web pages - librarians call this a \n",
    "\"known item search,\" for a web page/product page/service page they are \n",
    "trying to get to. Many person names are in our biography pages. Product,\n",
    "service names, etc. Multiple searches for \"Kenneth Walker\" - people\n",
    "are probably trying to get to https://www.nlm.nih.gov/news/nlm_mourns_ken_walker.html,\n",
    "or that's where we want them to get to. \n",
    "\n",
    "I use SEO Spider to scrape web page content, including page titles, in this \n",
    "case\n",
    "they probably are trying to get to the page titled \"NLM Mourns the Loss \n",
    "of H. Kenneth Walker, MD, MACP, FAAN, Former Chair of the National \n",
    "Library of Medicine Board of Regents.\"\n",
    "\n",
    "Do I have to vectorize this whole title? How should the matches between\n",
    "visitor search terms and verbose title pages, be implemented? Name \n",
    "extraction first?\n",
    "\n",
    "I also have to do this with the list of 200+ named NLM products (mostly\n",
    "databases such as pubmed.gov).\n",
    "'''\n",
    "\n",
    "# This data needs to be used to fuzzy match against web page names.\n",
    "ShouldBeFuzzyMatched = pd.read_excel('03_ML-classification_files/ShouldBeFuzzyMatched.xlsx')\n",
    "\n",
    "ShouldBeFuzzyMatched.head(n=10)\n",
    "\n",
    "'''\n",
    "ShouldBeFuzzyMatched is not used in this script; please suggest methods. \n",
    "For the above example I would like eventually to end up with:\n",
    "\n",
    "| adjustedQueryCase | preferredTerm | SemanticType | SemanticGroup         |\n",
    "| kenneth walker    | NLM News 2018 | NLM Web Page | Intellectual Products |\n",
    "\n",
    "\n",
    "FYI, preferredTerm is not part of this script currrently, I am dropping \n",
    "that column above. But the value of preferredTerm would be \n",
    "ShouldBeFuzzyMatched['ContentGroup']) when a match is made to the page title.\n",
    "Regarding SemanticType, to the original ontology of ~130 types I have added \n",
    "several NLM-specific type names such as \"NLM Web Page.\" Not many in the\n",
    "training set yet.\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. Eyeball level of balance among classes under study, in training set\n",
    "# Less than 1 minute\n",
    "# =======================================================================\n",
    "'''\n",
    "Perhaps this should be used with item 12 below, Final report, after that\n",
    "has been run.\n",
    "'''\n",
    "\n",
    "fig = plt.figure(figsize=(10,20))\n",
    "df.groupby('SemanticTypeName').adjustedQueryCase.count().sort_index(ascending=False).plot.barh(ylim=0, fontsize=6, color=\"slateblue\")\n",
    "fig.subplots_adjust(left=0.3)\n",
    "plt.title(\"Eyeball level of balance among classes\", fontsize=12)\n",
    "plt.xlabel(\"Number of queries\")\n",
    "plt.show()\n",
    "\n",
    "\n",
    "'''\n",
    "(Wow, lots of variation. Many under-represented classes.)\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. Training set: Calculate tf-idf vector for each query\n",
    "# Less than 1 minute\n",
    "# ========================================================\n",
    "'''\n",
    "Current classifiers and learning algorithms can not directly process \n",
    "text in its original form; most of them expect numerical feature vectors \n",
    "with a fixed size, rather than raw text of variable length. Therefore, \n",
    "in this preprocessing step, comment text will be converted to a more manageable \n",
    "representation.\n",
    "\n",
    "One common approach for extracting features from text is to use the \n",
    "\"bag of words\" model, where for each document, a complaint narrative \n",
    "in our case, the presence (and often the frequency) of words is taken \n",
    "into consideration, but the order in which they occur is ignored.\n",
    "\n",
    "Specifically, for each term in our dataset, we will calculate a measure \n",
    "called Term Frequency, Inverse Document Frequency, abbreviated to tf-idf. \n",
    "\n",
    "We will use sklearn.feature_extraction.text.TfidfVectorizer to calculate \n",
    "a tf-idf vector for each of our consumer complaint narratives.\n",
    "\n",
    "Cf. http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html\n",
    "'''\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf = TfidfVectorizer(sublinear_tf=True, # True = Use a logarithmic form for frequency\n",
    "                        min_df=5,   # minimum number of documents a word must be present in to be kept\n",
    "                        norm='l2',  # to ensure all our feature vectors have a euclidian norm of 1\n",
    "                        encoding='latin-1', \n",
    "                        ngram_range=(1, 2), # both unigrams and bigrams\n",
    "                        stop_words='english') # remove common \"noise\" words, limit resulting features to useful ones\n",
    "\n",
    "features = tfidf.fit_transform(df.adjustedQueryCase).toarray()\n",
    "labels = df.category_id\n",
    "features.shape\n",
    "\n",
    "\n",
    "'''\n",
    "Shape example, (29289, 75036)\n",
    "\n",
    "Now, each of 29289 queries is represented by 75036 features, representing \n",
    "the tf-idf score for different unigrams and bigrams.\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. Training set, Chi square: Find the terms most correlated with each item\n",
    "# Less than 1 minute\n",
    "# ===========================================================================\n",
    "'''\n",
    "We can use sklearn.feature_selection.chi2 to find the terms that are the \n",
    "most correlated with each of the products.\n",
    "\n",
    "Cf. http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html\n",
    "'''\n",
    "\n",
    "from sklearn.feature_selection import chi2\n",
    "\n",
    "N = 2\n",
    "for Product, category_id in sorted(category_to_id.items()):\n",
    "  features_chi2 = chi2(features, labels == category_id)\n",
    "  indices = np.argsort(features_chi2[0])\n",
    "  feature_names = np.array(tfidf.get_feature_names())[indices]\n",
    "  unigrams = [v for v in feature_names if len(v.split(' ')) == 1]\n",
    "  bigrams = [v for v in feature_names if len(v.split(' ')) == 2]\n",
    "  print(\"# '{}':\".format(Product))\n",
    "  print(\"  . Most correlated unigrams:\\n       . {}\".format('\\n       . '.join(unigrams[-N:])))\n",
    "  print(\"  . Most correlated bigrams:\\n       . {}\".format('\\n       . '.join(bigrams[-N:])))\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. Train, test, predict with a multi-class classifier, Naive Bayes-Multinomial\n",
    "# Less than 1 minute\n",
    "# ===============================================================================\n",
    "'''\n",
    "To train supervised classifiers, we first transformed the “Consumer \n",
    "complaint narrative” into a vector of numbers. We explored vector \n",
    "representations such as TF-IDF weighted vectors.\n",
    "\n",
    "After having a vector representation of the text, we can train \n",
    "supervised-learning classifiers to train unseen “Consumer complaint narrative” \n",
    "entries and predict which of our products to assign them to.\n",
    "\n",
    "Here we will vectorize with CountVectorizer and transform with TfidfTransformer.\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['adjustedQueryCase'], df['SemanticTypeName'], random_state = 0)\n",
    "count_vect = CountVectorizer()\n",
    "X_train_counts = count_vect.fit_transform(X_train)\n",
    "tfidf_transformer = TfidfTransformer()\n",
    "X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
    "\n",
    "\n",
    "'''\n",
    "Now that we have all the features and labels, we can start training the \n",
    "classifier. There are a number of algorithms that might be useful for the \n",
    "current dataset. Naive Bayes is a common go-to. The model most suitable \n",
    "for word counts is the multinomial variant.\n",
    "\n",
    "Cf. http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html\n",
    "\n",
    "'''\n",
    "\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB().fit(X_train_tfidf, y_train)\n",
    "\n",
    "\n",
    "# After fitting the training set, let’s try a few predictions.\n",
    "\n",
    "# Tests\n",
    "print(clf.predict(count_vect.transform([\"herpes i\"])))\n",
    "print(clf.predict(count_vect.transform([\"bemer\"])))\n",
    "print(clf.predict(count_vect.transform([\"dental journals\"])))\n",
    "print(clf.predict(count_vect.transform([\"intermittent fasting\"])))\n",
    "print(clf.predict(count_vect.transform([\"cardiac tamponade pericardial lymphoma\"])))\n",
    "print(clf.predict(count_vect.transform([\"fisioterapia\"])))\n",
    "print(clf.predict(count_vect.transform([\"diabete\"])))\n",
    "print(clf.predict(count_vect.transform([\"journal of clinical and diagnostic research\"])))\n",
    "print(clf.predict(count_vect.transform([\"hippocrates\"])))\n",
    "print(clf.predict(count_vect.transform([\"the new england journal of medicine\"])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 6. Model selection - Which among four models is the BEST model for this dataset?\n",
    "# ~1 minute\n",
    "# REQUIRES AT LEAST 5 EXAMPLES PER CLASS\n",
    "# =================================================================================\n",
    "'''\n",
    "Let's benchmark four models used for this type of dataset, evaluate their \n",
    "accuracy, and visualize their classification accuracy for our dataset.\n",
    "\n",
    "1. (Multinomial) Naive Bayes, http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html\n",
    "2. Logistic Regression, http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n",
    "3. Linear Support Vector Classification, http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html\n",
    "4. Random Forest, http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html\n",
    "'''\n",
    "\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "models = [\n",
    "    MultinomialNB(),\n",
    "    LogisticRegression(random_state=0),\n",
    "    LinearSVC(),\n",
    "    RandomForestClassifier(n_estimators=200, max_depth=3, random_state=0),\n",
    "]\n",
    "CV = 5\n",
    "cv_df = pd.DataFrame(index=range(CV * len(models)))\n",
    "entries = []\n",
    "for model in models:\n",
    "  model_name = model.__class__.__name__\n",
    "  accuracies = cross_val_score(model, features, labels, scoring='accuracy', cv=CV)\n",
    "  for fold_idx, accuracy in enumerate(accuracies):\n",
    "    entries.append((model_name, fold_idx, accuracy))\n",
    "cv_df = pd.DataFrame(entries, columns=['model_name', 'fold_idx', 'accuracy'])\n",
    "\n",
    "\n",
    "import seaborn as sns\n",
    "# Cf. https://seaborn.pydata.org/generated/seaborn.boxplot.html\n",
    "\n",
    "sns.boxplot(x='model_name', y='accuracy', data=cv_df).set_title(\"Classifier performance (box plot)\")\n",
    "sns.stripplot(x='model_name', y='accuracy', data=cv_df, \n",
    "              size=8, jitter=True, edgecolor=\"gray\", linewidth=2)\n",
    "plt.show()\n",
    "\n",
    "cv_df.groupby('model_name').accuracy.mean()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 7. Look deeper, with a confusion matrix, into the most successful model of\n",
    "# our group, LinearSVC (Linear Support Vector Classification)\n",
    "# Less than 1 minute\n",
    "# ========================================================================\n",
    "'''\n",
    "Too many categories to display. Future, could use SemanticGroup (only 15 classes)\n",
    "\n",
    "Continuing with LinearSVC, the most-accurate model of the ones we tested, \n",
    "let's create a confusion matrix to show the discrepancies between predicted \n",
    "and actual labels within the categories.\n",
    "\n",
    "Parameters: http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "model = LinearSVC()\n",
    "\n",
    "X_train, X_test, y_train, y_test, indices_train, indices_test = train_test_split(\n",
    "        features, labels, df.index, test_size=0.33, random_state=0)\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "conf_mat = confusion_matrix(y_test, y_pred)\n",
    "fig, ax = plt.subplots(figsize=(10,8))\n",
    "sns.heatmap(conf_mat, annot=True, fmt='d',\n",
    "            xticklabels=category_id_df.Product.values, \n",
    "            yticklabels=category_id_df.Product.values)\n",
    "plt.rcParams.update({'font.size': 8})\n",
    "plt.ylabel('Actual')\n",
    "plt.subplots_adjust(left=0.5, bottom=0.5)\n",
    "plt.xlabel('Predicted')\n",
    "plt.show()\n",
    "\n",
    "# Reduce long tag to see this as it was intended, with the Actual and \n",
    "# Predicted labels, 'Credit reporting, credit repair services, or other \n",
    "# personal consumer reports'.\n",
    "\n",
    "'''\n",
    "The vast majority of the predictions end up on the diagonal (predicted \n",
    "label = actual label), where we want them to be. \n",
    "\n",
    "However, there are a number of misclassifications, and it might be \n",
    "interesting to see what those are caused by.\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 8. Understand the misclassifications. Should we change the model, or not?\n",
    "# Less than 1 minute\n",
    "# ==========================================================================\n",
    "'''\n",
    "From Susan Li's blog post, not working with this dataset.\n",
    "\n",
    "Uses dictionary id_to_category.\n",
    "'''\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "for predicted in category_id_df.category_id:\n",
    "  for actual in category_id_df.category_id:\n",
    "    if predicted != actual and conf_mat[actual, predicted] >= 6:\n",
    "      print(\"'{}' predicted as '{}' : {} examples.\".format(\n",
    "              id_to_category[actual], id_to_category[predicted], \n",
    "              conf_mat[actual, predicted]))\n",
    "      display(df.loc[indices_test[(y_test == actual) & \n",
    "                                  (y_pred == predicted)]]\n",
    "      [['SemanticTypeName', 'adjustedQueryCase']])\n",
    "      print('')\n",
    "\n",
    "'''\n",
    "When things belong in multiple categories, errors will happen; not directly\n",
    "fixable.\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 9. Chi-square to find terms MOST CORRELATED with each category\n",
    "# Less than 1 minute\n",
    "# ========================================================================\n",
    "'''\n",
    "Again, we use the chi-squared test to find the terms that are the most \n",
    "correlated with each of the categories.\n",
    "\n",
    "In IPython console:\n",
    "    Start recording print output: %logstart dan1.txt\n",
    "    Stop recording print output: %logstop\n",
    "\n",
    "Or don't specifiy file name and then look for ipython_log.py\n",
    "https://ipython.org/ipython-doc/3/interactive/reference.html\n",
    "'''\n",
    "\n",
    "model.fit(features, labels)\n",
    "\n",
    "from sklearn.feature_selection import chi2\n",
    "\n",
    "N = 3\n",
    "stringCapture = \"\"\n",
    "for Product, category_id in sorted(category_to_id.items()):\n",
    "  indices = np.argsort(model.coef_[category_id])\n",
    "  feature_names = np.array(tfidf.get_feature_names())[indices]\n",
    "  unigrams = [v for v in reversed(feature_names) if len(v.split(' ')) == 1][:N]\n",
    "  bigrams = [v for v in reversed(feature_names) if len(v.split(' ')) == 2][:N]\n",
    "  print(\"# '{}':\".format(Product))\n",
    "  print(\"  . Top unigrams:\\n       . {}\".format('\\n       . '.join(unigrams)))\n",
    "  print(\"  . Top bigrams:\\n       . {}\".format('\\n       . '.join(bigrams)))\n",
    "  stringCapture += '\\n\\n' + str(Product) + '\\n Top unigrams:\\n   ' + str(unigrams) + '\\n Top bigrams:\\n   ' + str(bigrams)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 10. TryLinearSVCdf - Unmatched terms with LinearSVC\n",
    "# Less than 1 minute\n",
    "# ========================================================================\n",
    "\n",
    "TryLinearSVCdf = pd.DataFrame()\n",
    "TryLinearSVCdf['adjustedQueryCase'] = \"\"\n",
    "TryLinearSVCdf['pred-LinearSVC'] = \"\"\n",
    "\n",
    "TryLinearSVC = unassignedAfterUmls1['adjustedQueryCase'].astype(str)\n",
    "\n",
    "text_features = tfidf.transform(TryLinearSVC)\n",
    "\n",
    "predictions = model.predict(text_features)\n",
    "\n",
    "\n",
    "for queryTerm, predicted in zip(TryLinearSVC, predictions):\n",
    "    TryLinearSVCdf = TryLinearSVCdf.append(pd.DataFrame({'adjustedQueryCase': queryTerm, \n",
    "            'pred-LinearSVC': id_to_category[predicted]}, index=[0]), ignore_index=True, sort=True)\n",
    "\n",
    "TryLinearSVCdf = TryLinearSVCdf[['adjustedQueryCase', 'pred-LinearSVC']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 11. TryLogisticRegressiondf - Unmatched terms with LogisticRegression\n",
    "# Less than 1 minute\n",
    "# ========================================================================\n",
    "'''\n",
    "https://towardsdatascience.com/logistic-regression-using-python-sklearn-numpy-mnist-handwriting-recognition-matplotlib-a6b31e2b166a\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logisticRegr = LogisticRegression(random_state=0)\n",
    "\n",
    "logisticRegr.fit(X_train, y_train)\n",
    "\n",
    "predictions = logisticRegr.predict(X_train)\n",
    "\n",
    "'''\n",
    "# Use score method to get accuracy of model\n",
    "score = logisticRegr.score(x_test, y_test)\n",
    "print(score)\n",
    "'''\n",
    "\n",
    "TryLogisticRegressionDf = pd.DataFrame()\n",
    "TryLogisticRegressionDf['adjustedQueryCase'] = \"\"\n",
    "TryLogisticRegressionDf['pred-LogisticReg'] = \"\"\n",
    "\n",
    "TryLogisticRegression = unassignedAfterUmls1['adjustedQueryCase'].astype(str)\n",
    "\n",
    "text_features = tfidf.transform(TryLogisticRegression)\n",
    "\n",
    "for queryTerm, predicted in zip(TryLogisticRegression, predictions):\n",
    "    TryLogisticRegressionDf = TryLogisticRegressionDf.append(pd.DataFrame({'adjustedQueryCase': queryTerm, \n",
    "            'pred-LogisticReg': id_to_category[predicted]}, index=[0]), ignore_index=True, sort=True)\n",
    "\n",
    "TryLogisticRegressionDf = TryLogisticRegressionDf[['adjustedQueryCase', 'pred-LogisticReg']]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# JOIN NEW DATAFRAMES\n",
    "\n",
    "twoGuesses = pd.merge(TryLinearSVCdf, TryLogisticRegressionDf)\n",
    "\n",
    "writer = pd.ExcelWriter('01_Pre-processing_files/twoGuesses.xlsx')\n",
    "twoGuesses.to_excel(writer,'twoGuesses')\n",
    "# df2.to_excel(writer,'Sheet2')\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 12. Final report by category\n",
    "# Less than 1 minute\n",
    "# ========================================================================\n",
    "'''\n",
    "Classes where more training data is needed, or other changes need to be\n",
    "made.\n",
    "'''\n",
    "  \n",
    "from sklearn import metrics\n",
    "\n",
    "print(metrics.classification_report(y_test, y_pred, \n",
    "                                    target_names=df['SemanticTypeName'].unique()))\n",
    "\n",
    "\n",
    "'''\n",
    "\n",
    "                                         precision    recall  f1-score   support\n",
    "\n",
    "                 NLM Product or Service       0.66      0.48      0.55       299\n",
    "                   Quantitative Concept       0.31      0.22      0.26        78\n",
    "Nucleic Acid, Nucleoside, or Nucleotide       0.40      0.14      0.21        57\n",
    "    Therapeutic or Preventive Procedure       0.66      0.47      0.55       383\n",
    "                                  Plant       0.53      0.12      0.19       178\n",
    "                       Organic Chemical       0.26      0.94      0.41      1212\n",
    "                   Intellectual Product       0.50      0.29      0.36       270\n",
    "        Amino Acid, Peptide, or Protein       0.68      0.24      0.35       409\n",
    "                         Cell Component       0.65      0.36      0.46        36\n",
    "                Pharmacologic Substance       0.29      0.15      0.20       143\n",
    "  Indicator, Reagent, or Diagnostic Aid       0.20      0.08      0.12        12\n",
    "                       Temporal Concept       0.36      0.20      0.26        45\n",
    "                    Nucleotide Sequence       0.00      0.00      0.00         3\n",
    "                   Laboratory Procedure       0.57      0.54      0.55        98\n",
    "   Body Part, Organ, or Organ Component       0.55      0.39      0.45       199\n",
    "                                Finding       0.40      0.26      0.32       334\n",
    "                    Disease or Syndrome       0.68      0.51      0.58      1040\n",
    "                        Spatial Concept       0.20      0.02      0.04        43\n",
    "                    Manufactured Object       0.28      0.11      0.16        91\n",
    "                                   Cell       0.66      0.61      0.64        44\n",
    "                         Gene or Genome       0.95      0.54      0.69       448\n",
    "                                Vitamin       0.00      0.00      0.00         4\n",
    "                     Immunologic Factor       0.78      0.34      0.47       116\n",
    "          Cell or Molecular Dysfunction       0.40      0.14      0.21        14\n",
    "                   Diagnostic Procedure       0.62      0.41      0.50       104\n",
    "                     Molecular Function       0.47      0.24      0.32        29\n",
    "                            semTypeName       0.77      0.66      0.71       176\n",
    "                     Neoplastic Process       0.00      0.00      0.00         5\n",
    "       Self-help or Relief Organization       0.20      0.16      0.18        19\n",
    "                Body Location or Region       0.58      0.37      0.45       119\n",
    "                        Sign or Symptom       0.00      0.00      0.00        31\n",
    "                   Acquired Abnormality       0.59      0.28      0.38       169\n",
    "                         Medical Device       0.21      0.18      0.19        28\n",
    "                 Anatomical Abnormality       0.74      0.67      0.70       116\n",
    "                    Injury or Poisoning       0.41      0.33      0.37        27\n",
    "                     Clinical Attribute       0.53      0.44      0.48       125\n",
    "       Mental or Behavioral Dysfunction       0.26      0.15      0.19       131\n",
    "                    Pathologic Function       0.54      0.24      0.33        59\n",
    "                       Population Group       1.00      0.33      0.50         9\n",
    "                    Embryonic Structure       0.40      0.17      0.24        12\n",
    "                      Regulation or Law       0.18      0.08      0.11        98\n",
    "                    Qualitative Concept       0.62      0.22      0.33        94\n",
    "                 Congenital Abnormality       0.29      0.10      0.15        72\n",
    "                     Functional Concept       0.00      0.00      0.00        35\n",
    "                  Occupational Activity       0.47      0.24      0.32        67\n",
    "                         Mental Process       0.29      0.19      0.23        62\n",
    "     Professional or Occupational Group       0.25      0.09      0.13        11\n",
    "                           Organization       0.50      0.29      0.37        90\n",
    "                                   Food       0.00      0.00      0.00        74\n",
    "                              Eukaryote       0.00      0.00      0.00        20\n",
    "                  Phenomenon or Process       0.54      0.22      0.32        58\n",
    "       Health Care Related Organization       0.50      0.10      0.17        49\n",
    "                      Organism Function       0.12      0.06      0.09        31\n",
    "               Organ or Tissue Function       0.36      0.15      0.21        34\n",
    "                        Social Behavior       0.31      0.08      0.13        59\n",
    "                        Idea or Concept       0.96      0.32      0.48        78\n",
    "                              Bacterium       0.00      0.00      0.00         2\n",
    "                               Chemical       0.56      0.17      0.26        29\n",
    "          Natural Phenomenon or Process       0.31      0.17      0.22        24\n",
    "                               Activity       0.00      0.00      0.00         2\n",
    "                   Professional Society       0.45      0.41      0.43       123\n",
    "                   Health Care Activity       0.33      0.06      0.10        17\n",
    "               Element, Ion, or Isotope       0.17      0.05      0.07        21\n",
    "                   Physiologic Function       0.38      0.25      0.30        32\n",
    "         Daily or Recreational Activity       0.65      0.47      0.54       129\n",
    "    Biomedical Occupation or Discipline       0.00      0.00      0.00        10\n",
    "           Chemical Viewed Structurally       0.73      0.40      0.52        20\n",
    "                               Receptor       0.63      0.32      0.42        38\n",
    "                                  Virus       0.75      0.17      0.27        18\n",
    "                                 Tissue       0.00      0.00      0.00        21\n",
    "                     Organism Attribute       0.00      0.00      0.00         7\n",
    "           Chemical Viewed Functionally       0.50      0.31      0.38        32\n",
    "                    Individual Behavior       0.12      0.12      0.12         8\n",
    "                              Age Group       0.27      0.50      0.35         8\n",
    "                        Group Attribute       0.44      0.41      0.42        17\n",
    "           NLM Organizational Component       0.27      0.25      0.26        12\n",
    "                          Cell Function       0.44      0.21      0.28        39\n",
    "               Occupation or Discipline       0.43      0.07      0.12        90\n",
    "                        Geographic Area       0.56      0.74      0.64        31\n",
    "                          Clinical Drug       0.40      0.10      0.16        20\n",
    "                                 Fungus       0.45      0.49      0.47        49\n",
    "                      Research Activity       0.14      0.07      0.10        14\n",
    "                              Substance       0.00      0.00      0.00         2\n",
    "         Environmental Effect of Humans       1.00      0.29      0.44         7\n",
    "              Patient or Disabled Group       0.00      0.00      0.00         4\n",
    "                                  Human       0.38      0.19      0.25        16\n",
    "              Laboratory or Test Result       0.00      0.00      0.00         1\n",
    "                                Reptile       0.00      0.00      0.00         2\n",
    "          Experimental Model of Disease       0.17      0.05      0.08        20\n",
    "          Biologically Active Substance       0.00      0.00      0.00        16\n",
    "                      Conceptual Entity       0.47      0.21      0.29        39\n",
    "          Biomedical or Dental Material       0.69      0.19      0.30        47\n",
    "                                 Mammal       0.00      0.00      0.00         1\n",
    "                    Amino Acid Sequence       0.11      0.11      0.11        18\n",
    "                         Body Substance       0.00      0.00      0.00         3\n",
    "                              Amphibian       0.00      0.00      0.00         6\n",
    "                      Biologic Function       1.00      0.08      0.15        12\n",
    "                                   Bird       0.00      0.00      0.00         0\n",
    "                   Anatomical Structure       0.50      0.12      0.20         8\n",
    "                                Hormone       0.00      0.00      0.00        16\n",
    "                         Classification       0.00      0.00      0.00         6\n",
    "                                   Fish       0.00      0.00      0.00         3\n",
    "                                 Animal       0.00      0.00      0.00         1\n",
    "                                  Event       0.31      0.31      0.31        13\n",
    "                 Body Space or Junction       0.00      0.00      0.00         3\n",
    "                             Antibiotic       0.00      0.00      0.00        14\n",
    "    Governmental or Regulatory Activity       0.77      0.38      0.51        26\n",
    "                   Educational Activity       0.00      0.00      0.00         2\n",
    "                               Archaeon       0.00      0.00      0.00        13\n",
    "       Hazardous or Poisonous Substance       0.00      0.00      0.00         5\n",
    "                       Machine Activity       0.40      0.32      0.35        19\n",
    "                       Genetic Function       0.00      0.00      0.00        14\n",
    "                     Inorganic Chemical       0.00      0.00      0.00         4\n",
    "                               Behavior       0.75      0.30      0.43        10\n",
    "     Human-caused Phenomenon or Process       0.00      0.00      0.00         7\n",
    "   Molecular Biology Research Technique       0.44      0.31      0.36        13\n",
    "                            Body System       0.40      0.44      0.42         9\n",
    "                               Language       0.50      0.07      0.12        15\n",
    "                               Organism       0.00      0.00      0.00         5\n",
    "                                  Group       0.29      0.17      0.21        12\n",
    "                           Family Group       0.00      0.00      0.00         2\n",
    "                        Research Device       0.00      0.00      0.00         1\n",
    "                        Physical Object       0.00      0.00      0.00         2\n",
    "                            NLM Product       0.00      0.00      0.00         1\n",
    "\n",
    "                            avg / total       0.51      0.42      0.40      8878                                 \n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}