Day6-Jieba_作業.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標: 使用Jieba進行各項的斷詞操作\n",
    "\n",
    "這份作業我們會使用中文飯店評論資料集來作為斷詞練習。 [資料集:ChnSentiCorp_htl_all](https://github.com/SophonPlus/ChineseNlpCorpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 讀入文本資料"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   label                                             review\n",
       "0      1  距离川沙公路较近,但是公交指示不对,如果是\"蔡陆线\"的话,会非常麻烦.建议用别的路线.房间较...\n",
       "1      1                       商务大床房，房间很大，床有2M宽，整体感觉经济实惠不错!\n",
       "2      1         早餐太差，无论去多少人，那边也不加食品的。酒店应该重视一下这个问题了。房间本身很好。\n",
       "3      1  宾馆在小街道上，不大好找，但还好北京热心同胞很多~宾馆设施跟介绍的差不多，房间很小，确实挺小...\n",
       "4      1               CBD中心,周围没什么店铺,说5星有点勉强.不知道为什么卫生间没有电吹风"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>label</th>\n      <th>review</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>距离川沙公路较近,但是公交指示不对,如果是\"蔡陆线\"的话,会非常麻烦.建议用别的路线.房间较...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>商务大床房，房间很大，床有2M宽，整体感觉经济实惠不错!</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>早餐太差，无论去多少人，那边也不加食品的。酒店应该重视一下这个问题了。房间本身很好。</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>宾馆在小街道上，不大好找，但还好北京热心同胞很多~宾馆设施跟介绍的差不多，房间很小，确实挺小...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>CBD中心,周围没什么店铺,说5星有点勉强.不知道为什么卫生间没有电吹风</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# hint: 可利用pandas讀取CSV\n",
    "###<your code>###\n",
    "\n",
    "pd_corpus= pd.read_csv('./ChnSentiCorp_htl_all.csv')\n",
    "\n",
    "pd_corpus.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total: 7766, Positive: 5322, Negative: 2444\n"
     ]
    }
   ],
   "source": [
    "#確認所有留言,正評價(label=1)與負評價數量(label=0)\n",
    "###<your code>###\n",
    "pd_positive=pd_corpus[pd_corpus['label'] == 1]\n",
    "pd_negative=pd_corpus[pd_corpus['label'] == 0]\n",
    "\n",
    "print(f'Total: {len(pd_corpus)}, Positive: {len(pd_positive)}, Negative: {len(pd_negative)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#移除缺失值\n",
    "###<your code>###\n",
    "pd_corpus=pd_corpus.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 建構結巴斷詞Function\n",
    "\n",
    "建構可將讀入的Pandas DataFrame的文本資料，外加一欄名為`cut`的review欄位斷詞結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class JiebaCutingClass(object):\n",
    "    '''Class to use jeiba to parse corpus from dataframe and cut the corpus\n",
    "    \n",
    "    Parameters\n",
    "    -----------\n",
    "    key_to_cut: str\n",
    "        the dataframe key to parse the sentence for jieba cutting\n",
    "    dic: str\n",
    "        the dictionary to use for jieba, default is None (use default dictionary)\n",
    "    userdict: str\n",
    "        the user defined dictionary to use for jieba, default is None\n",
    "    '''\n",
    "    \n",
    "    def __init__(self, key_to_cut:str, dic:str=None, userdict:str=None):\n",
    "        \n",
    "        if dic is not None:\n",
    "            jieba.set_dictionary(dic)\n",
    "        \n",
    "        if userdict is not None:\n",
    "            jieba.load_userdict(userdict)\n",
    "        \n",
    "        self.key_to_cut = key_to_cut\n",
    "        \n",
    "        #將paddle 功能開啟\n",
    "        ###<your code>###\n",
    "        jieba.enable_paddle()\n",
    "    @staticmethod\n",
    "    def cut_single_sentence(sentence, use_paddle=False, use_full=False, use_search=False):\n",
    "        \n",
    "        if use_search:\n",
    "            # hint:使用收尋引擎模式進行斷詞\n",
    "            ###<your code>###\n",
    "            out=jieba.cut_for_search(sentence)\n",
    "        else:\n",
    "            # hint:非收尋引擎模式斷詞，請注意有精確模式、全模式與paddle模式\n",
    "            ###<your code>###\n",
    "            out=jieba.cut(sentence, use_paddle=use_paddle, cut_all= use_full)\n",
    "        return out\n",
    "            \n",
    "    \n",
    "    def cut_corpus(self, corpus: pd.DataFrame, mode: str) -> pd.DataFrame:\n",
    "        '''Method to read and cut sentence from dataframe and append another column named cut\n",
    "        \n",
    "        Paremeters\n",
    "        --------------\n",
    "        corpus: pd.DataFrame\n",
    "            Input corpus in dataframe\n",
    "        mode: str\n",
    "            Jieba mode to be used\n",
    "        \n",
    "        Return\n",
    "        ----------------\n",
    "        out: pd.Dataframe\n",
    "            Output corpus in dataframe\n",
    "        '''\n",
    "        \n",
    "        # checking valid mode\n",
    "        if mode not in ['paddle', 'full', 'precise', 'search']:\n",
    "            raise TypeError(f'only support `paddle`, `full`, `precise`, and `search` mode, but get {mode}')\n",
    "            \n",
    "        # cut the corpus based on mode\n",
    "        # hint: 根據mode來呼叫相對應的斷詞method\n",
    "        if mode == 'paddle':\n",
    "            ###<your code>###\n",
    "            out = self._paddle_cut(corpus)\n",
    "        elif mode == 'full':\n",
    "            ###<your code>###\n",
    "            out = self._full_cut(corpus)\n",
    "        elif mode == 'precise':\n",
    "            ###<your code>###\n",
    "            out = self._precise_cut(corpus)\n",
    "        elif mode == 'search':\n",
    "            ###<your code>###\n",
    "            out = self._search_cut(corpus)\n",
    "        return out\n",
    "    \n",
    "    def _paddle_cut(self, corpus):\n",
    "        '''paddle mode\n",
    "        '''\n",
    "        #enable paddle\n",
    "        #hint:先啟用paddle mode\n",
    "        ### <your code> ###\n",
    "        jieba.enable_paddle()\n",
    "        out = []\n",
    "        # hint:將讀進的文本進行斷詞，並將結果append回out的陣列中\n",
    "        for single_review in corpus[self.key_to_cut]:\n",
    "            ###<your code>###\n",
    "            out.append([word for word in JiebaCutingClass.cut_single_sentence(single_review, use_paddle=True)])\n",
    "        corpus['cut'] = out\n",
    "        \n",
    "        return corpus\n",
    "    \n",
    "    def _full_cut(self, corpus):\n",
    "        '''full mode\n",
    "        '''\n",
    "        \n",
    "        out = []\n",
    "        # hint:將讀進的文本進行斷詞，並將結果append回out的陣列中\n",
    "        for single_review in corpus[self.key_to_cut]:\n",
    "            ###<your code>###\n",
    "            out.append([word for word in JiebaCutingClass.cut_single_sentence(single_review, use_full=True)])\n",
    "        corpus['cut'] = out\n",
    "        \n",
    "        return corpus\n",
    "    \n",
    "    def _precise_cut(self, corpus):\n",
    "        '''precise mode\n",
    "        '''\n",
    "        \n",
    "        out = []\n",
    "        # hint:將讀進的文本進行斷詞，並將結果append回out的陣列中\n",
    "        for single_review in corpus[self.key_to_cut]:\n",
    "            ###<your code>###\n",
    "            out.append([word for word in JiebaCutingClass.cut_single_sentence(single_review)])\n",
    "        corpus['cut'] = out\n",
    "        \n",
    "        return corpus\n",
    "    \n",
    "    def _search_cut(self, corpus):\n",
    "        '''search mode\n",
    "        '''\n",
    "        \n",
    "        out = []\n",
    "        # hint:將讀進的文本進行斷詞，並將結果append回out的陣列中\n",
    "        for single_review in corpus[self.key_to_cut]:\n",
    "            ###<your code>###\n",
    "            out.append([word for word in JiebaCutingClass.cut_single_sentence(single_review, use_search=True)])\n",
    "        corpus['cut'] = out\n",
    "        \n",
    "        return corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用建構好的斷詞物件對文本進行斷詞"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "Paddle enabled successfully......\n",
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache C:\\Users\\zero0\\AppData\\Local\\Temp\\jieba.cache\n",
      "Loading model cost 0.963 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   label                                             review  \\\n",
       "0      1  距离川沙公路较近,但是公交指示不对,如果是\"蔡陆线\"的话,会非常麻烦.建议用别的路线.房间较...   \n",
       "1      1                       商务大床房，房间很大，床有2M宽，整体感觉经济实惠不错!   \n",
       "2      1         早餐太差，无论去多少人，那边也不加食品的。酒店应该重视一下这个问题了。房间本身很好。   \n",
       "3      1  宾馆在小街道上，不大好找，但还好北京热心同胞很多~宾馆设施跟介绍的差不多，房间很小，确实挺小...   \n",
       "4      1               CBD中心,周围没什么店铺,说5星有点勉强.不知道为什么卫生间没有电吹风   \n",
       "\n",
       "                                                 cut  \n",
       "0  [距离, 川沙, 公路, 较近, ,, 但是, 公交, 指示, 不, 对, ,, 如果, 是...  \n",
       "1  [商务, 大床, 房, ，, 房间, 很大, ，, 床有, 2M, 宽, ，, 整体, 感觉...  \n",
       "2  [早餐, 太, 差, ，, 无论, 去, 多少, 人, ，, 那边, 也, 不加, 食品, ...  \n",
       "3  [宾馆, 在, 小, 街道, 上, ，, 不大好, 找, ，, 但, 还好, 北京, 热心,...  \n",
       "4  [CBD, 中心, ,, 周围, 没什么, 店铺, ,, 说, 5, 星, 有点, 勉强, ...  "
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>label</th>\n      <th>review</th>\n      <th>cut</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>距离川沙公路较近,但是公交指示不对,如果是\"蔡陆线\"的话,会非常麻烦.建议用别的路线.房间较...</td>\n      <td>[距离, 川沙, 公路, 较近, ,, 但是, 公交, 指示, 不, 对, ,, 如果, 是...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>商务大床房，房间很大，床有2M宽，整体感觉经济实惠不错!</td>\n      <td>[商务, 大床, 房, ，, 房间, 很大, ，, 床有, 2M, 宽, ，, 整体, 感觉...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>早餐太差，无论去多少人，那边也不加食品的。酒店应该重视一下这个问题了。房间本身很好。</td>\n      <td>[早餐, 太, 差, ，, 无论, 去, 多少, 人, ，, 那边, 也, 不加, 食品, ...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>宾馆在小街道上，不大好找，但还好北京热心同胞很多~宾馆设施跟介绍的差不多，房间很小，确实挺小...</td>\n      <td>[宾馆, 在, 小, 街道, 上, ，, 不大好, 找, ，, 但, 还好, 北京, 热心,...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>CBD中心,周围没什么店铺,说5星有点勉强.不知道为什么卫生间没有电吹风</td>\n      <td>[CBD, 中心, ,, 周围, 没什么, 店铺, ,, 说, 5, 星, 有点, 勉强, ...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "source": [
    "### 請使用精確模式與預設字典對文本進行斷詞\n",
    "\n",
    "# hint:請先實例化JiebaCutingClass，再進行斷詞\n",
    "###<your code>###\n",
    "###<your code>### #為了避免處理時間過久, 這裡我們只使用前50個進行斷詞\n",
    "jieba_cut=JiebaCutingClass(key_to_cut='review')\n",
    "pd_cut = jieba_cut.cut_corpus(pd_corpus.loc[:50, :], mode='precise')\n",
    "\n",
    "pd_cut.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 載入繁中字典為指定的字詞進行斷詞"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "Paddle enabled successfully......\n",
      "Building prefix dict from c:\\Users\\zero0\\OneDrive\\cupoy\\NLP\\Day006\\dict.txt.big ...\n",
      "Loading model from cache C:\\Users\\zero0\\AppData\\Local\\Temp\\jieba.u229e21d64adf4c52e9e89689445ae0bf.cache\n",
      "Paddle模式: ['我', '愛', 'cupoy', '自然', '語言', '處理', '馬拉松', '課程']\n",
      "Loading model cost 1.571 seconds.\n",
      "Prefix dict has been built successfully.\n",
      "全模式: ['我', '愛', 'cupoy', '自然', '自然語言', '語言', '處理', '馬拉', '馬拉松', '課程']\n",
      "搜尋模式: ['我', '愛', 'cupoy', '自然', '語言', '自然語言', '處理', '馬拉', '馬拉松', '課程']\n",
      "精確模式: ['我', '愛', 'cupoy', '自然語言', '處理', '馬拉松', '課程']\n"
     ]
    }
   ],
   "source": [
    "test_string = '我愛cupoy自然語言處理馬拉松課程'\n",
    "jieba_cut = JiebaCutingClass(key_to_cut='', dic='./dict.txt.big')\n",
    "\n",
    "\n",
    "out_string = JiebaCutingClass.cut_single_sentence(test_string, use_paddle=True) ###<your code>### #paddle 模式\n",
    "print(f'Paddle模式: {[string for string in out_string]}')\n",
    "\n",
    "out_string = JiebaCutingClass.cut_single_sentence(test_string, use_full=True) ###<your code>### #全模式\n",
    "print(f'全模式: {[string for string in out_string]}')\n",
    "\n",
    "out_string = JiebaCutingClass.cut_single_sentence(test_string, use_search=True) ###<your code>### #搜尋模式\n",
    "print(f'搜尋模式: {[string for string in out_string]}')\n",
    "\n",
    "out_string = JiebaCutingClass.cut_single_sentence(test_string) ###<your code>### #精確模式\n",
    "print(f'精確模式: {[string for string in out_string]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.6 64-bit (conda)",
   "metadata": {
    "interpreter": {
     "hash": "7b4b4feff2f24a0f0a34464dbe537a36fda679851528fb8735cb41fa49dffb2d"
    }
   }
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}