Day14_作業.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 本日課程-文字預處理，部分內容前面章節可能提過，這裡會將前處理所需技巧串起"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from nltk.corpus import stopwords\n",
    "#tsv是指用tab分開字元的檔案\n",
    "dataset=pd.read_csv('movie_feedback.csv', header=None, encoding='Big5')\n",
    "X = dataset[0].values\n",
    "Y = dataset[1].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                                                       0  1\n",
       "0      the rock is destined to be the 21st century's ...  1\n",
       "1      the gorgeously elaborate continuation of \" the...  1\n",
       "2                         effective but too-tepid biopic  1\n",
       "3      if you sometimes like to go to the movies to h...  1\n",
       "4      emerges as something rare , an issue movie tha...  1\n",
       "...                                                  ... ..\n",
       "10657  a terrible movie that some people will neverth...  0\n",
       "10658  there are many definitions of 'time waster' bu...  0\n",
       "10659  as it stands , crocodile hunter has the hurrie...  0\n",
       "10660  the thing looks like a made-for-home-video qui...  0\n",
       "10661  enigma is well-made , but it's just too dry an...  0\n",
       "\n",
       "[10662 rows x 2 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>the rock is destined to be the 21st century's ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>the gorgeously elaborate continuation of \" the...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>effective but too-tepid biopic</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>if you sometimes like to go to the movies to h...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>emerges as something rare , an issue movie tha...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>10657</th>\n      <td>a terrible movie that some people will neverth...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>10658</th>\n      <td>there are many definitions of 'time waster' bu...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>10659</th>\n      <td>as it stands , crocodile hunter has the hurrie...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>10660</th>\n      <td>the thing looks like a made-for-home-video qui...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>10661</th>\n      <td>enigma is well-made , but it's just too dry an...</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>10662 rows × 2 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review before preprocessing : the rock is destined to be the 21st century's new \" conan \" and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal . \n"
     ]
    }
   ],
   "source": [
    "print('review before preprocessing : {}'.format(X[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 運用re.sub去除部分字元"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re \n",
    "# 去除a-zA-Z以外的字元，並將他們取代為空格' '\n",
    "review=re.sub('[^a-zA-Z]',' ',X[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after re.sub : the rock is destined to be the   st century s new   conan   and that he s going to make a splash even greater than arnold schwarzenegger   jean claud van damme or steven segal   \n"
     ]
    }
   ],
   "source": [
    "print('review after re.sub : {}'.format(review))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 將所有字母轉為小寫:因為大部分情境區分大小寫並不能提供而外訊息，如CV內顏色無法提供額外訊息時我們會將圖像轉為灰階，藉此降低複雜度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after lower : the rock is destined to be the   st century s new   conan   and that he s going to make a splash even greater than arnold schwarzenegger   jean claud van damme or steven segal   \n"
     ]
    }
   ],
   "source": [
    "#把全部變成小寫\n",
    "review=review.lower()\n",
    "print('review after lower : {}'.format(review))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 斷詞"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after split : ['the', 'rock', 'is', 'destined', 'to', 'be', 'the', 'st', 'century', 's', 'new', 'conan', 'and', 'that', 'he', 's', 'going', 'to', 'make', 'a', 'splash', 'even', 'greater', 'than', 'arnold', 'schwarzenegger', 'jean', 'claud', 'van', 'damme', 'or', 'steven', 'segal']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "#把review裡面的單字切開\n",
    "print('review after split : {}'.format(review.split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* tokenize 相較於split會是更好的選擇，如 split 無法分開 word. 這種case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after tokenize : ['the', 'rock', 'is', 'destined', 'to', 'be', 'the', 'st', 'century', 's', 'new', 'conan', 'and', 'that', 'he', 's', 'going', 'to', 'make', 'a', 'splash', 'even', 'greater', 'than', 'arnold', 'schwarzenegger', 'jean', 'claud', 'van', 'damme', 'or', 'steven', 'segal']\n"
     ]
    }
   ],
   "source": [
    "review = nltk.word_tokenize(review)\n",
    "print('review after tokenize : {}'.format(review))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## stopwords: 移除贅字，此步驟為前處理的重要步驟之一，過多的贅字不僅無法提供更多訊息，還會干擾到模型的訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "[nltk_data] Downloading package stopwords to\n[nltk_data]     C:\\Users\\zero0\\AppData\\Roaming\\nltk_data...\n[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {},
     "execution_count": 11
    }
   ],
   "source": [
    "#處理文字，有建立好的文字褲會幫我們移除不想要的文字\n",
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after removeing stopwords : ['rock', 'destined', 'st', 'century', 'new', 'conan', 'going', 'make', 'splash', 'even', 'greater', 'arnold', 'schwarzenegger', 'jean', 'claud', 'van', 'damme', 'steven', 'segal']\n"
     ]
    }
   ],
   "source": [
    "review=[word for word in review if not word in set(stopwords.words('English'))]\n",
    "print('review after removeing stopwords : {}'.format(review))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming: 詞幹提取\n",
    " * ex. loves,loved都變成love\n",
    " * 中文沒有詞幹提取的需求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "ps=PorterStemmer()\n",
    "review=[ps.stem(word) for word in review]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "review after stemming : ['rock', 'destin', 'st', 'centuri', 'new', 'conan', 'go', 'make', 'splash', 'even', 'greater', 'arnold', 'schwarzenegg', 'jean', 'claud', 'van', 'damm', 'steven', 'segal']\n"
     ]
    }
   ],
   "source": [
    "print('review after stemming : {}'.format(review))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 練習清理所有的句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dataset=pd.read_csv('movie_feedback.csv',encoding = 'Big5',names=['feedback', 'label'] )\n",
    "X = dataset[0].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus=[]\n",
    "row=len(X)\n",
    "for i in range(0,row):\n",
    "    review=re.sub('[^a-zA-Z]',' ',X[i])\n",
    "    review=review.lower()\n",
    "    review=review.split()\n",
    "    ps=PorterStemmer()\n",
    "    ## 這裡先不用stopwords 因為 review中很多反定詞會被移掉 如isn't good, 會變成 good\n",
    "    review=[ps.stem(word) for word in review ]\n",
    "    review=' '.join(review)\n",
    "    corpus.append(review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 轉bag-of-words vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "#Creating bag of word model\n",
    "#tokenization(符號化)\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "#max_features是要建造幾個column，會按造字出現的高低去篩選 \n",
    "cv = CountVectorizer(max_features=1500)\n",
    "#toarray是建造matrixs\n",
    "#X現在為sparsity就是很多零的matrix\n",
    "X_ = cv.fit_transform(corpus).toarray()\n",
    "Y_ = dataset[1].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 選擇練習: 將處理好數據放入 naive_bayes模型，並預測評論為正向或負面，詳細原理之後章節會解釋。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "GaussianNB(priors=None, var_smoothing=1e-09)"
      ]
     },
     "metadata": {},
     "execution_count": 37
    }
   ],
   "source": [
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_, Y_, test_size = 0.1)\n",
    "\n",
    "# Feature Scaling\n",
    "\n",
    "#Naive Bayes\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "classifier = GaussianNB()\n",
    "classifier.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "message='I really like this movie!!'\n",
    "## 要使用一樣的前處理\n",
    "review=re.sub('[^a-zA-Z]',' ',message)\n",
    "review=review.lower()\n",
    "review=review.split()\n",
    "ps=PorterStemmer()\n",
    "review=[ps.stem(word) for word in review]\n",
    "review = ' '.join(review)\n",
    "input_ = cv.transform([review]).toarray()\n",
    "prediction = classifier.predict(input_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "array([1], dtype=int64)"
      ]
     },
     "metadata": {},
     "execution_count": 39
    }
   ],
   "source": [
    "prediction ## 1代表正向評價"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "message='A terrible movie  !!'\n",
    "review=re.sub('[^a-zA-Z]',' ',message)\n",
    "review=review.lower()\n",
    "review=review.split()\n",
    "ps=PorterStemmer()\n",
    "review=[ps.stem(word) for word in review]\n",
    "review = ' '.join(review)\n",
    "input_ = cv.transform([review]).toarray()\n",
    "prediction = classifier.predict(input_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction ## 0代表負面評價"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}