Comprehend Sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract a single column containing free-text feedback and run sentiment analysis on it\n",
    "\n",
    "Import pandas, set filename, load contents from file from CSV into dataframe, print dataframe shape to verify load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "filename = '~/Downloads/AWS Summit London 2018_May 22, 2018_04.25.csv'\n",
    "\n",
    "df = pd.read_csv(filename,index_col=False, header=0,low_memory=False)\n",
    "print(\"Data loaded into pandas dataframe with shape: {}\".format(df.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the desired column from the dataframe by name. Drop empty values at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_name = 'Q17'\n",
    "feedback = df.dropna(subset=[column_name])[column_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print sample of 5 records. Note it's a dataframe, so it comes with keys and metadata. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print feedback.sample(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import modules. \n",
    "* boto3 is our AWS SDK. \n",
    "* operator allows us to sort dictionary objects creating lists. \n",
    "* sys gives us some line printing utility\n",
    "\n",
    "Define a function to allow us to overprint one line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import operator\n",
    "import sys\n",
    "\n",
    "def restart_line():\n",
    "    sys.stdout.write('\\r')\n",
    "    sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise variables\n",
    "\n",
    "Do the work.\n",
    "* Create a comprehend client\n",
    "* itterate over the contents of feedback\n",
    "* send each line to comprehend.detect_sentiment and get results\n",
    "* increment the counter on the highest probability sentiment\n",
    "* if it's a positive, increment positives\n",
    "* increment the record count\n",
    "* print status\n",
    "* once done itterating, print the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = 0\n",
    "positives = []\n",
    "scoredict = {\n",
    "                \"Neutral\" : 0,\n",
    "                \"Positive\" : 0,\n",
    "                \"Negative\" :0,\n",
    "                \"Mixed\" : 0\n",
    "            }\n",
    "\n",
    "comprehendClient = boto3.client('comprehend')\n",
    "\n",
    "for i in feedback.sample(n=5):\n",
    "    comprehendClientResponse = comprehendClient.detect_sentiment(\n",
    "        Text=i,\n",
    "        LanguageCode='en')\n",
    "    scores = comprehendClientResponse['SentimentScore']\n",
    "    sorted_scores = sorted(scores.items(), key=operator.itemgetter(1))\n",
    "    \n",
    "    scoredict[sorted_scores[3][0]] += 1 \n",
    "    count = count +1\n",
    "    if sorted_scores[3][0] == 'Positive':\n",
    "        positives.append(sorted_scores[3][0])\n",
    "\n",
    "    sys.stdout.write(\"Processed {} records. Positives: {}\".format(count,len(positives)))\n",
    "    sys.stdout.flush()\n",
    "    restart_line()\n",
    "        \n",
    "print(\"\\nTotal number of records: {}.\\n\".format(count))\n",
    "\n",
    "for result in sorted(scoredict.items(), \n",
    "                     reverse=True, \n",
    "                     key=operator.itemgetter(1)):\n",
    "    print(\"{} : {}\".format(result[1],result[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}