SMP2018_EDA_and_Baseline_Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $$SMP2018中文人机对话技术评测（ECDT）$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **不要直接修改此文件，可把该文件拷贝至自己的文件夹下再进行操作**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 下面是一个完整的针对 [SMP2018中文人机对话技术评测（ECDT）](http://smp2018.cips-smp.org/ecdt_index.html) 的实验，由该实验训练的基线模型能达到评测排行榜的前三的水平。\n",
    "2. 通过本实验，可以掌握处理自然语言文本数据的一般方法。\n",
    "3. 推荐自己修改此文件，达到更好的实验效果，比如改变以下几个超参数 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# 词嵌入的维度\n",
    "embedding_word_dims = 32\n",
    "# 批次大小\n",
    "batch_size = 30\n",
    "# 周期\n",
    "epochs = 20\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 本实验还可以改进的地方举例 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 预处理阶段使用其它的分词工具\n",
    "2. 采用字符向量和词向量结合的方式\n",
    "3. 使用预先训练好的词向量\n",
    "4. 改变模型结构\n",
    "5. 改变模型超参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入依赖库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import collections\n",
    "import jieba\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, LSTM, Dense\n",
    "from keras.utils import to_categorical,plot_model\n",
    "from keras.callbacks import TensorBoard, Callback\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import requests \n",
    "\n",
    "import time\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "\n",
    "# 计算 F1 值的函数\n",
    "def f1(y_true, y_pred):\n",
    "    def recall(y_true, y_pred):\n",
    "        \"\"\"Recall metric.\n",
    "\n",
    "        Only computes a batch-wise average of recall.\n",
    "\n",
    "        Computes the recall, a metric for multi-label classification of\n",
    "        how many relevant items are selected.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
    "        recall = true_positives / (possible_positives + K.epsilon())\n",
    "        return recall\n",
    "\n",
    "    def precision(y_true, y_pred):\n",
    "        \"\"\"Precision metric.\n",
    "\n",
    "        Only computes a batch-wise average of precision.\n",
    "\n",
    "        Computes the precision, a metric for multi-label classification of\n",
    "        how many selected items are relevant.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
    "        precision = true_positives / (predicted_positives + K.epsilon())\n",
    "        return precision\n",
    "    precision = precision(y_true, y_pred)\n",
    "    recall = recall(y_true, y_pred)\n",
    "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取自定义时间格式的字符串\n",
    "def get_customization_time():\n",
    "    # return '2018_10_10_18_11_45' 年月日时分秒\n",
    "    time_tuple = time.localtime(time.time())\n",
    "    customization_time = \"{}_{}_{}_{}_{}_{}\".format(time_tuple[0], time_tuple[1], time_tuple[2], time_tuple[3], time_tuple[4], time_tuple[5])\n",
    "    return customization_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [下载SMP2018官方数据](https://worksheets.codalab.org/worksheets/0x27203f932f8341b79841d50ce0fd684f/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_train_data_url = \"https://worksheets.codalab.org/rest/bundles/0x0161fd2fb40d4dd48541c2643d04b0b8/contents/blob/\"\n",
    "raw_test_data_url = \"https://worksheets.codalab.org/rest/bundles/0x1f96bc12222641209ad057e762910252/contents/blob/\"\n",
    "\n",
    "# 如果不存在 SMP2018 数据，则下载\n",
    "if (not os.path.exists('./data/train.json')) or (not os.path.exists('./data/dev.json')):\n",
    "    raw_train = requests.get(raw_train_data_url) \n",
    "    raw_test = requests.get(raw_test_data_url) \n",
    "    if not os.path.exists('./data'):\n",
    "        os.makedirs('./data')\n",
    "    with open(\"./data/train.json\", \"wb\") as code:\n",
    "         code.write(raw_train.content)\n",
    "    with open(\"./data/dev.json\", \"wb\") as code:\n",
    "         code.write(raw_test.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_json_data(path):\n",
    "    # read data\n",
    "    data_df = pd.read_json(path)\n",
    "    # change row and colunm\n",
    "    data_df = data_df.transpose()\n",
    "    # change colunm order\n",
    "    data_df = data_df[['query', 'label']]\n",
    "    return data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_df = get_json_data(path=\"data/train.json\")\n",
    "\n",
    "test_data_df = get_json_data(path=\"data/dev.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>今天东莞天气如何</td>\n",
       "      <td>weather</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>从观音桥到重庆市图书馆怎么走</td>\n",
       "      <td>map</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>鸭蛋怎么腌？</td>\n",
       "      <td>cookbook</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>怎么治疗牛皮癣</td>\n",
       "      <td>health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>唠什么</td>\n",
       "      <td>chat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            query     label\n",
       "0        今天东莞天气如何   weather\n",
       "1  从观音桥到重庆市图书馆怎么走       map\n",
       "2          鸭蛋怎么腌？  cookbook\n",
       "3         怎么治疗牛皮癣    health\n",
       "4             唠什么      chat"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>毛泽东的诗哦。</td>\n",
       "      <td>poetry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>有房有车吗微笑</td>\n",
       "      <td>chat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2013年亚洲冠军联赛恒广州恒大比赛时间。</td>\n",
       "      <td>match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>若相惜不弃下一句是什么？</td>\n",
       "      <td>poetry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>苹果翻译成英语</td>\n",
       "      <td>translation</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   query        label\n",
       "0                毛泽东的诗哦。       poetry\n",
       "1                有房有车吗微笑         chat\n",
       "2  2013年亚洲冠军联赛恒广州恒大比赛时间。        match\n",
       "3           若相惜不弃下一句是什么？       poetry\n",
       "4                苹果翻译成英语  translation"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2299</td>\n",
       "      <td>2299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>2299</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>中国新闻网网站</td>\n",
       "      <td>chat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          query label\n",
       "count      2299  2299\n",
       "unique     2299    31\n",
       "top     中国新闻网网站  chat\n",
       "freq          1   455"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>770</td>\n",
       "      <td>770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>770</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>查下安徽电视台今天节目单</td>\n",
       "      <td>chat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               query label\n",
       "count            770   770\n",
       "unique           770    31\n",
       "top     查下安徽电视台今天节目单  chat\n",
       "freq               1   154"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取所以标签，也就是分类的类别\n",
    "labels = list(set(train_data_df['label'].tolist()))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# All labels\n",
    "labels = ['website', 'tvchannel', 'lottery', 'chat', 'match',\n",
    "          'datetime', 'weather', 'bus', 'novel', 'video', 'riddle',\n",
    "          'calc', 'telephone', 'health', 'contacts', 'epg', 'app', 'music',\n",
    "          'cookbook', 'stock', 'map', 'message', 'poetry', 'cinemas', 'news',\n",
    "          'flight', 'translation', 'train', 'schedule', 'radio', 'email']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label_numbers:\t 31\n"
     ]
    }
   ],
   "source": [
    "label_numbers = len(labels)\n",
    "print('label_numbers:\\t', label_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 标签和对应ID的映射字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_2_index_dict = dict([(label, index) for index, label in enumerate(labels)])\n",
    "index_2_label_dict = dict([(index, label) for index, label in enumerate(labels)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [结巴分词](https://github.com/fxsjy/jieba)示例，下面将使用结巴分词对原数据进行处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /tmp/jieba.cache\n",
      "Loading model cost 0.903 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['他', '来到', '了', '网易', '杭研', '大厦']\n"
     ]
    }
   ],
   "source": [
    "seg_list = jieba.cut(\"他来到了网易杭研大厦\")  # 默认是精确模式\n",
    "print(list(seg_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 序列化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def use_jieba_cut(a_sentence):\n",
    "    return list(jieba.cut(a_sentence))\n",
    "\n",
    "train_data_df['cut_query'] = train_data_df['query'].apply(use_jieba_cut)\n",
    "test_data_df['cut_query'] = test_data_df['query'].apply(use_jieba_cut)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "      <th>cut_query</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>今天东莞天气如何</td>\n",
       "      <td>weather</td>\n",
       "      <td>[今天, 东莞, 天气, 如何]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>从观音桥到重庆市图书馆怎么走</td>\n",
       "      <td>map</td>\n",
       "      <td>[从, 观音桥, 到, 重庆市, 图书馆, 怎么, 走]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>鸭蛋怎么腌？</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[鸭蛋, 怎么, 腌, ？]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>怎么治疗牛皮癣</td>\n",
       "      <td>health</td>\n",
       "      <td>[怎么, 治疗, 牛皮癣]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>唠什么</td>\n",
       "      <td>chat</td>\n",
       "      <td>[唠, 什么]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>阳澄湖大闸蟹的做法。</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[阳澄湖, 大闸蟹, 的, 做法, 。]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>昆山大润发在哪里</td>\n",
       "      <td>map</td>\n",
       "      <td>[昆山, 大润发, 在, 哪里]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>红烧肉怎么做？嗯？</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[红烧肉, 怎么, 做, ？, 嗯, ？]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>南京到厦门的火车票</td>\n",
       "      <td>train</td>\n",
       "      <td>[南京, 到, 厦门, 的, 火车票]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>6的平方</td>\n",
       "      <td>calc</td>\n",
       "      <td>[6, 的, 平方]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            query     label                     cut_query\n",
       "0        今天东莞天气如何   weather              [今天, 东莞, 天气, 如何]\n",
       "1  从观音桥到重庆市图书馆怎么走       map  [从, 观音桥, 到, 重庆市, 图书馆, 怎么, 走]\n",
       "2          鸭蛋怎么腌？  cookbook                [鸭蛋, 怎么, 腌, ？]\n",
       "3         怎么治疗牛皮癣    health                 [怎么, 治疗, 牛皮癣]\n",
       "4             唠什么      chat                       [唠, 什么]\n",
       "5      阳澄湖大闸蟹的做法。  cookbook          [阳澄湖, 大闸蟹, 的, 做法, 。]\n",
       "6        昆山大润发在哪里       map              [昆山, 大润发, 在, 哪里]\n",
       "7       红烧肉怎么做？嗯？  cookbook         [红烧肉, 怎么, 做, ？, 嗯, ？]\n",
       "8       南京到厦门的火车票     train           [南京, 到, 厦门, 的, 火车票]\n",
       "9            6的平方      calc                    [6, 的, 平方]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取数据的所有词汇\n",
    "def get_all_vocab_from_data(data, colunm_name):\n",
    "    train_vocab_list = []\n",
    "    max_cut_query_lenth = 0\n",
    "    for cut_query in data[colunm_name]:\n",
    "        if len(cut_query) > max_cut_query_lenth:\n",
    "            max_cut_query_lenth = len(cut_query)\n",
    "        train_vocab_list += cut_query\n",
    "    return train_vocab_list, max_cut_query_lenth   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_vocab_list, max_cut_query_lenth = get_all_vocab_from_data(train_data_df, 'cut_query')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words：\t 11498\n"
     ]
    }
   ],
   "source": [
    "print('Number of words：\\t', len(train_vocab_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max_cut_query_lenth:\t 26\n"
     ]
    }
   ],
   "source": [
    "print('max_cut_query_lenth:\\t', max_cut_query_lenth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_vocab_list, test_max_cut_query_lenth = get_all_vocab_from_data(train_data_df, 'cut_query')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_max_cut_query_lenth:\t 26\n"
     ]
    }
   ],
   "source": [
    "print('test_max_cut_query_lenth:\\t', test_max_cut_query_lenth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['今天', '东莞', '天气', '如何', '从', '观音桥', '到', '重庆市', '图书馆', '怎么']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_vocab_list[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_vocab_counter = collections.Counter(train_vocab_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of different words:\t 2887\n"
     ]
    }
   ],
   "source": [
    "print('Number of different words:\\t', len(train_vocab_counter.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 不同种类的词汇个数，预留一个位置给不存在的词汇（不存在的词汇标记为0）  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_features = len(train_vocab_counter.keys()) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2888\n"
     ]
    }
   ],
   "source": [
    "print(max_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('的', 605),\n",
       " ('。', 341),\n",
       " ('我', 320),\n",
       " ('你', 297),\n",
       " ('怎么', 273),\n",
       " ('？', 251),\n",
       " ('什么', 210),\n",
       " ('到', 165),\n",
       " ('给', 154),\n",
       " ('做', 148)]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 10 words with the highest frequency\n",
    "train_vocab_counter.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 统计低频词语"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word_times_zero:\t 1978\n",
      "word_times_zero/all:\t 0.685140284031867\n"
     ]
    }
   ],
   "source": [
    "word_times_zero = 0\n",
    "for word, word_times in train_vocab_counter.items():\n",
    "    if word_times <=1:\n",
    "        word_times_zero+=1\n",
    "print('word_times_zero:\\t', word_times_zero)\n",
    "print('word_times_zero/all:\\t', word_times_zero/len(train_vocab_counter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 制作词汇字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_train_vocab_dict(train_vocab_counter):\n",
    "    word_2_index, index_2_word = {}, {}\n",
    "    # Reserve 0 for masking via pad_sequences\n",
    "    index_number = 1\n",
    "    for word, word_times in train_vocab_counter.most_common():\n",
    "        word_2_index[word] = index_number\n",
    "        index_2_word[index_number] = word\n",
    "        index_number += 1\n",
    "    return word_2_index, index_2_word "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_2_index_dict, index_2_word_dict = create_train_vocab_dict(train_vocab_counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2\n"
     ]
    }
   ],
   "source": [
    "print(word_2_index_dict['的'], word_2_index_dict['。'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "的 。\n"
     ]
    }
   ],
   "source": [
    "print(index_2_word_dict[1], index_2_word_dict[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "今天东莞天气如何 weather ['今天', '东莞', '天气', '如何']\n",
      "从观音桥到重庆市图书馆怎么走 map ['从', '观音桥', '到', '重庆市', '图书馆', '怎么', '走']\n",
      "鸭蛋怎么腌？ cookbook ['鸭蛋', '怎么', '腌', '？']\n",
      "怎么治疗牛皮癣 health ['怎么', '治疗', '牛皮癣']\n",
      "唠什么 chat ['唠', '什么']\n",
      "阳澄湖大闸蟹的做法。 cookbook ['阳澄湖', '大闸蟹', '的', '做法', '。']\n",
      "昆山大润发在哪里 map ['昆山', '大润发', '在', '哪里']\n",
      "红烧肉怎么做？嗯？ cookbook ['红烧肉', '怎么', '做', '？', '嗯', '？']\n",
      "南京到厦门的火车票 train ['南京', '到', '厦门', '的', '火车票']\n",
      "6的平方 calc ['6', '的', '平方']\n"
     ]
    }
   ],
   "source": [
    "pq= 0\n",
    "for index, row in train_data_df.iterrows():\n",
    "    print(row[0], row[1], row[2])\n",
    "    pq+=1\n",
    "    if pq==10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_2_index_dict.get('的2', 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vectorize_data(data, label_2_index_dict, word_2_index_dict, max_cut_query_lenth):\n",
    "    x_train = []\n",
    "    y_train = []\n",
    "    for index, row in data.iterrows():\n",
    "        query_sentence = row[2]\n",
    "        label = row[1]\n",
    "        # 字典找不到的情况下用 0 填充\n",
    "        x = [word_2_index_dict.get(w, 0) for w in query_sentence]\n",
    "        y = [label_2_index_dict[label]]\n",
    "        x_train.append(x)\n",
    "        y_train.append(y)\n",
    "    return (pad_sequences(x_train, maxlen=max_cut_query_lenth),\n",
    "            pad_sequences(y_train, maxlen=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, y_train = vectorize_data(train_data_df, label_2_index_dict, word_2_index_dict, max_cut_query_lenth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test, y_test = vectorize_data(test_data_df, label_2_index_dict, word_2_index_dict, test_max_cut_query_lenth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0  33 318  27  90] [7]\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0], y_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(y_train, label_numbers)\n",
    "y_test = to_categorical(y_test, label_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2299, 26) (2299, 31)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(770, 26) (770, 31)\n"
     ]
    }
   ],
   "source": [
    "print(x_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 存储预处理过的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print(type(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savez(\"preprocessed_data\", x_train, y_train, x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 直接加载预处理的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用已经经过预处理的数据，默认不使用\n",
    "use_preprocessed_data = True\n",
    "\n",
    "if use_preprocessed_data == True:\n",
    "    preprocessed_data = np.load('preprocessed_data.npz')\n",
    "    x_train, y_train, x_test, y_test = preprocessed_data['arr_0'], preprocessed_data['arr_1'], preprocessed_data['arr_2'], preprocessed_data['arr_3'],"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2299, 26) (2299, 31)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 设计模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_SMP2018_lstm_model(max_features, max_cut_query_lenth, label_numbers):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim=max_features, output_dim=32, input_length=max_cut_query_lenth))\n",
    "    model.add(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2))\n",
    "    model.add(Dense(label_numbers, activation='softmax'))\n",
    "    # try using different optimizers and different optimizer configs\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=[f1])\n",
    "\n",
    "    plot_model(model, to_file='SMP2018_lstm_model.png', show_shapes=True)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not find max_features variable, use default max_features values:\t2888\n",
      "not find max_cut_query_lenth, use default max_features values:\t26\n",
      "not find label_numbers, use default max_features values:\t31\n"
     ]
    }
   ],
   "source": [
    "if 'max_features'  not in  dir():\n",
    "    max_features = 2888\n",
    "    print('not find max_features variable, use default max_features values:\\t{}'.format(max_features))\n",
    "if 'max_cut_query_lenth'  not in  dir():\n",
    "    max_cut_query_lenth = 26\n",
    "    print('not find max_cut_query_lenth, use default max_features values:\\t{}'.format(max_cut_query_lenth))\n",
    "if 'label_numbers'  not in  dir():\n",
    "    label_numbers = 31\n",
    "    print('not find label_numbers, use default max_features values:\\t{}'.format(label_numbers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_SMP2018_lstm_model(max_features, max_cut_query_lenth, label_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 20\n",
    "epochs = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train...\n",
      "Train on 1839 samples, validate on 460 samples\n",
      "Epoch 1/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 3.1404 - f1: 0.0000e+00 - val_loss: 2.9658 - val_f1: 0.0000e+00\n",
      "Epoch 2/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.8634 - f1: 0.0404 - val_loss: 2.5949 - val_f1: 0.1618\n",
      "Epoch 3/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.3841 - f1: 0.3354 - val_loss: 2.1469 - val_f1: 0.4080\n",
      "Epoch 4/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.9530 - f1: 0.4240 - val_loss: 1.8311 - val_f1: 0.4429\n",
      "Epoch 5/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.5153 - f1: 0.5092 - val_loss: 1.4660 - val_f1: 0.5133\n",
      "Epoch 6/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.1055 - f1: 0.6257 - val_loss: 1.2311 - val_f1: 0.6446\n",
      "Epoch 7/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.7985 - f1: 0.7558 - val_loss: 1.0519 - val_f1: 0.6857\n",
      "Epoch 8/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.5887 - f1: 0.8245 - val_loss: 0.9113 - val_f1: 0.7443\n",
      "Epoch 9/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.4365 - f1: 0.8729 - val_loss: 0.8589 - val_f1: 0.7589\n",
      "Epoch 10/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.3196 - f1: 0.9178 - val_loss: 0.8198 - val_f1: 0.7948\n",
      "Epoch 11/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.2584 - f1: 0.9379 - val_loss: 0.7777 - val_f1: 0.8046\n",
      "Epoch 12/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.1941 - f1: 0.9593 - val_loss: 0.7518 - val_f1: 0.8343\n",
      "Epoch 13/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.1540 - f1: 0.9726 - val_loss: 0.7506 - val_f1: 0.8322\n",
      "Epoch 14/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.1220 - f1: 0.9808 - val_loss: 0.7529 - val_f1: 0.8195\n",
      "Epoch 15/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.1044 - f1: 0.9837 - val_loss: 0.7723 - val_f1: 0.8226\n",
      "Epoch 16/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0884 - f1: 0.9868 - val_loss: 0.7465 - val_f1: 0.8326\n",
      "Epoch 17/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0722 - f1: 0.9900 - val_loss: 0.7687 - val_f1: 0.8240\n",
      "Epoch 18/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0594 - f1: 0.9941 - val_loss: 0.7584 - val_f1: 0.8252\n",
      "Epoch 19/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0497 - f1: 0.9947 - val_loss: 0.7572 - val_f1: 0.8302\n",
      "Epoch 20/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0437 - f1: 0.9958 - val_loss: 0.7714 - val_f1: 0.8260\n",
      "Epoch 21/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0398 - f1: 0.9972 - val_loss: 0.7631 - val_f1: 0.8246\n",
      "Epoch 22/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0321 - f1: 0.9973 - val_loss: 0.7698 - val_f1: 0.8276\n",
      "Epoch 23/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0313 - f1: 0.9967 - val_loss: 0.7809 - val_f1: 0.8288\n",
      "Epoch 24/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0272 - f1: 0.9989 - val_loss: 0.7797 - val_f1: 0.8218\n",
      "Epoch 25/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0240 - f1: 0.9986 - val_loss: 0.7531 - val_f1: 0.8345\n",
      "Epoch 26/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0236 - f1: 0.9978 - val_loss: 0.7988 - val_f1: 0.8201\n",
      "Epoch 27/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0207 - f1: 0.9986 - val_loss: 0.8156 - val_f1: 0.8259\n",
      "Epoch 28/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0185 - f1: 0.9995 - val_loss: 0.7938 - val_f1: 0.8185\n",
      "Epoch 29/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0189 - f1: 0.9981 - val_loss: 0.7839 - val_f1: 0.8169\n",
      "Epoch 30/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0145 - f1: 1.0000 - val_loss: 0.8001 - val_f1: 0.8296\n",
      "Epoch 31/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0163 - f1: 0.9984 - val_loss: 0.8265 - val_f1: 0.8116\n",
      "Epoch 32/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0167 - f1: 0.9970 - val_loss: 0.8117 - val_f1: 0.8320\n",
      "Epoch 33/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0131 - f1: 0.9997 - val_loss: 0.8121 - val_f1: 0.8224\n",
      "Epoch 34/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0098 - f1: 1.0000 - val_loss: 0.8158 - val_f1: 0.8277\n",
      "Epoch 35/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0133 - f1: 0.9986 - val_loss: 0.8314 - val_f1: 0.8242\n",
      "Epoch 36/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0099 - f1: 1.0000 - val_loss: 0.8447 - val_f1: 0.8231\n",
      "Epoch 37/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0081 - f1: 1.0000 - val_loss: 0.8237 - val_f1: 0.8312\n",
      "Epoch 38/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0117 - f1: 0.9986 - val_loss: 0.8239 - val_f1: 0.8155\n",
      "Epoch 39/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0082 - f1: 1.0000 - val_loss: 0.8470 - val_f1: 0.8204\n",
      "Epoch 40/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0072 - f1: 1.0000 - val_loss: 0.8471 - val_f1: 0.8262\n",
      "Epoch 41/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0098 - f1: 0.9992 - val_loss: 0.8262 - val_f1: 0.8323\n",
      "Epoch 42/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0110 - f1: 0.9978 - val_loss: 0.8577 - val_f1: 0.8205\n",
      "Epoch 43/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0069 - f1: 0.9997 - val_loss: 0.8587 - val_f1: 0.8226\n",
      "Epoch 44/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0059 - f1: 0.9995 - val_loss: 0.8217 - val_f1: 0.8253\n",
      "Epoch 45/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0054 - f1: 0.9995 - val_loss: 0.8342 - val_f1: 0.8269\n",
      "Epoch 46/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0044 - f1: 1.0000 - val_loss: 0.8494 - val_f1: 0.8310\n",
      "Epoch 47/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0040 - f1: 1.0000 - val_loss: 0.8496 - val_f1: 0.8341\n",
      "Epoch 48/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0044 - f1: 1.0000 - val_loss: 0.8640 - val_f1: 0.8269\n",
      "Epoch 49/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0049 - f1: 1.0000 - val_loss: 0.8453 - val_f1: 0.8321\n",
      "Epoch 50/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0033 - f1: 1.0000 - val_loss: 0.9022 - val_f1: 0.8279\n",
      "Epoch 51/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0074 - f1: 0.9989 - val_loss: 0.9145 - val_f1: 0.8171\n",
      "Epoch 52/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0050 - f1: 0.9995 - val_loss: 0.9031 - val_f1: 0.8159\n",
      "Epoch 53/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0083 - f1: 0.9981 - val_loss: 0.9456 - val_f1: 0.8084\n",
      "Epoch 54/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0057 - f1: 0.9995 - val_loss: 0.8972 - val_f1: 0.8265\n",
      "Epoch 55/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0027 - f1: 1.0000 - val_loss: 0.8921 - val_f1: 0.8284\n",
      "Epoch 56/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0027 - f1: 1.0000 - val_loss: 0.8885 - val_f1: 0.8318\n",
      "Epoch 57/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0024 - f1: 1.0000 - val_loss: 0.9059 - val_f1: 0.8283\n",
      "Epoch 58/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0028 - f1: 1.0000 - val_loss: 0.9045 - val_f1: 0.8233\n",
      "Epoch 59/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0022 - f1: 1.0000 - val_loss: 0.9238 - val_f1: 0.8302\n",
      "Epoch 60/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0024 - f1: 0.9997 - val_loss: 0.9383 - val_f1: 0.8209\n",
      "Epoch 61/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0030 - f1: 1.0000 - val_loss: 0.9409 - val_f1: 0.8157\n",
      "Epoch 62/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0023 - f1: 1.0000 - val_loss: 0.9529 - val_f1: 0.8255\n",
      "Epoch 63/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0046 - f1: 0.9997 - val_loss: 0.9899 - val_f1: 0.8158\n",
      "Epoch 64/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0040 - f1: 0.9995 - val_loss: 0.9625 - val_f1: 0.8138\n",
      "Epoch 65/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0039 - f1: 0.9997 - val_loss: 0.9493 - val_f1: 0.8135\n",
      "Epoch 66/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0024 - f1: 1.0000 - val_loss: 0.9872 - val_f1: 0.8151\n",
      "Epoch 67/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0058 - f1: 0.9989 - val_loss: 0.9106 - val_f1: 0.8146\n",
      "Epoch 68/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0045 - f1: 0.9997 - val_loss: 0.9383 - val_f1: 0.8191\n",
      "Epoch 69/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0018 - f1: 1.0000 - val_loss: 0.9366 - val_f1: 0.8184\n",
      "Epoch 70/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0022 - f1: 1.0000 - val_loss: 1.0150 - val_f1: 0.8079\n",
      "Epoch 71/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0018 - f1: 1.0000 - val_loss: 0.9735 - val_f1: 0.8136\n",
      "Epoch 72/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0028 - f1: 1.0000 - val_loss: 0.9194 - val_f1: 0.8333\n",
      "Epoch 73/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0016 - f1: 1.0000 - val_loss: 0.9224 - val_f1: 0.8321\n",
      "Epoch 74/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 0.9445 - val_f1: 0.8249\n",
      "Epoch 75/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 9.3293e-04 - f1: 1.0000 - val_loss: 0.9333 - val_f1: 0.8304\n",
      "Epoch 76/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 9.6054e-04 - f1: 1.0000 - val_loss: 0.9360 - val_f1: 0.8264\n",
      "Epoch 77/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0034 - f1: 0.9992 - val_loss: 0.9103 - val_f1: 0.8299\n",
      "Epoch 78/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0018 - f1: 1.0000 - val_loss: 0.9219 - val_f1: 0.8332\n",
      "Epoch 79/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 0.9166 - val_f1: 0.8372\n",
      "Epoch 80/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.4605e-04 - f1: 1.0000 - val_loss: 0.9176 - val_f1: 0.8362\n",
      "Epoch 81/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0015 - f1: 0.9995 - val_loss: 0.9786 - val_f1: 0.8179\n",
      "Epoch 82/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 0.9525 - val_f1: 0.8336\n",
      "Epoch 83/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 0.9509 - val_f1: 0.8278\n",
      "Epoch 84/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.4968e-04 - f1: 1.0000 - val_loss: 0.9370 - val_f1: 0.8280\n",
      "Epoch 85/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.7447e-04 - f1: 1.0000 - val_loss: 0.9853 - val_f1: 0.8197\n",
      "Epoch 86/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 9.7422e-04 - f1: 1.0000 - val_loss: 0.9614 - val_f1: 0.8287\n",
      "Epoch 87/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.7887e-04 - f1: 1.0000 - val_loss: 0.9664 - val_f1: 0.8255\n",
      "Epoch 88/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.4960e-04 - f1: 1.0000 - val_loss: 0.9559 - val_f1: 0.8309\n",
      "Epoch 89/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.1952e-04 - f1: 1.0000 - val_loss: 0.9667 - val_f1: 0.8296\n",
      "Epoch 90/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.7391e-04 - f1: 1.0000 - val_loss: 0.9579 - val_f1: 0.8348\n",
      "Epoch 91/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.7227e-04 - f1: 1.0000 - val_loss: 0.9605 - val_f1: 0.8326\n",
      "Epoch 92/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.7311e-04 - f1: 1.0000 - val_loss: 0.9678 - val_f1: 0.8298\n",
      "Epoch 93/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.2191e-04 - f1: 1.0000 - val_loss: 0.9664 - val_f1: 0.8352\n",
      "Epoch 94/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.5526e-04 - f1: 1.0000 - val_loss: 0.9601 - val_f1: 0.8335\n",
      "Epoch 95/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.0580e-04 - f1: 1.0000 - val_loss: 0.9861 - val_f1: 0.8351\n",
      "Epoch 96/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 1.0249 - val_f1: 0.8131\n",
      "Epoch 97/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.6099e-04 - f1: 1.0000 - val_loss: 1.0059 - val_f1: 0.8183\n",
      "Epoch 98/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 8.8925e-04 - f1: 1.0000 - val_loss: 1.0204 - val_f1: 0.8181\n",
      "Epoch 99/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 7.3515e-04 - f1: 1.0000 - val_loss: 1.0013 - val_f1: 0.8184\n",
      "Epoch 100/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 8.3419e-04 - f1: 1.0000 - val_loss: 1.0341 - val_f1: 0.8164\n",
      "Epoch 101/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0035 - f1: 0.9989 - val_loss: 0.9958 - val_f1: 0.8191\n",
      "Epoch 102/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0053 - f1: 0.9989 - val_loss: 1.0122 - val_f1: 0.8265\n",
      "Epoch 103/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0013 - f1: 1.0000 - val_loss: 1.0103 - val_f1: 0.8209\n",
      "Epoch 104/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0053 - f1: 0.9989 - val_loss: 1.1036 - val_f1: 0.8036\n",
      "Epoch 105/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0020 - f1: 1.0000 - val_loss: 1.0385 - val_f1: 0.8145\n",
      "Epoch 106/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 1.0522 - val_f1: 0.8209\n",
      "Epoch 107/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 0.0010 - f1: 1.0000 - val_loss: 1.1168 - val_f1: 0.8123\n",
      "Epoch 108/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0040 - f1: 0.9989 - val_loss: 1.1633 - val_f1: 0.8004\n",
      "Epoch 109/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0034 - f1: 0.9995 - val_loss: 1.0392 - val_f1: 0.8165\n",
      "Epoch 110/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0053 - f1: 0.9995 - val_loss: 0.9886 - val_f1: 0.8305\n",
      "Epoch 111/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 1.0026 - val_f1: 0.8277\n",
      "Epoch 112/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 9.9556e-04 - f1: 1.0000 - val_loss: 0.9939 - val_f1: 0.8292\n",
      "Epoch 113/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0021 - f1: 0.9995 - val_loss: 1.0387 - val_f1: 0.8113\n",
      "Epoch 114/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.0578e-04 - f1: 1.0000 - val_loss: 1.0518 - val_f1: 0.8205\n",
      "Epoch 115/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.6815e-04 - f1: 1.0000 - val_loss: 1.0576 - val_f1: 0.8170\n",
      "Epoch 116/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.7931e-04 - f1: 1.0000 - val_loss: 1.0544 - val_f1: 0.8152\n",
      "Epoch 117/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.7737e-04 - f1: 1.0000 - val_loss: 1.0665 - val_f1: 0.8173\n",
      "Epoch 118/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.3346e-04 - f1: 1.0000 - val_loss: 1.0667 - val_f1: 0.8168\n",
      "Epoch 119/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.2447e-04 - f1: 1.0000 - val_loss: 1.0798 - val_f1: 0.8110\n",
      "Epoch 120/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.6263e-04 - f1: 1.0000 - val_loss: 1.0843 - val_f1: 0.8114\n",
      "Epoch 121/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 7.0956e-04 - f1: 1.0000 - val_loss: 1.0713 - val_f1: 0.8091\n",
      "Epoch 122/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.3038e-04 - f1: 1.0000 - val_loss: 1.0521 - val_f1: 0.8218\n",
      "Epoch 123/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 7.8382e-04 - f1: 1.0000 - val_loss: 1.0743 - val_f1: 0.8143\n",
      "Epoch 124/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.2436e-04 - f1: 1.0000 - val_loss: 1.0851 - val_f1: 0.8174\n",
      "Epoch 125/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.3625e-04 - f1: 1.0000 - val_loss: 1.1127 - val_f1: 0.8153\n",
      "Epoch 126/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.4003e-04 - f1: 1.0000 - val_loss: 1.0980 - val_f1: 0.8134\n",
      "Epoch 127/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.1112e-04 - f1: 1.0000 - val_loss: 1.0975 - val_f1: 0.8158\n",
      "Epoch 128/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.7792e-04 - f1: 1.0000 - val_loss: 1.0916 - val_f1: 0.8216\n",
      "Epoch 129/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.5881e-04 - f1: 1.0000 - val_loss: 1.0802 - val_f1: 0.8241\n",
      "Epoch 130/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.0872e-04 - f1: 1.0000 - val_loss: 1.0879 - val_f1: 0.8220\n",
      "Epoch 131/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.8239e-04 - f1: 1.0000 - val_loss: 1.0904 - val_f1: 0.8256\n",
      "Epoch 132/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.1858e-04 - f1: 1.0000 - val_loss: 1.0941 - val_f1: 0.8200\n",
      "Epoch 133/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.3054e-04 - f1: 1.0000 - val_loss: 1.0942 - val_f1: 0.8181\n",
      "Epoch 134/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.2964e-04 - f1: 1.0000 - val_loss: 1.0947 - val_f1: 0.8209\n",
      "Epoch 135/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.8267e-04 - f1: 1.0000 - val_loss: 1.0943 - val_f1: 0.8270\n",
      "Epoch 136/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.2101e-04 - f1: 1.0000 - val_loss: 1.0975 - val_f1: 0.8256\n",
      "Epoch 137/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.3244e-04 - f1: 1.0000 - val_loss: 1.0967 - val_f1: 0.8294\n",
      "Epoch 138/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 9.5624e-05 - f1: 1.0000 - val_loss: 1.1052 - val_f1: 0.8288\n",
      "Epoch 139/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 3.0982e-04 - f1: 1.0000 - val_loss: 1.1630 - val_f1: 0.8185\n",
      "Epoch 140/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.4933e-04 - f1: 1.0000 - val_loss: 1.1253 - val_f1: 0.8189\n",
      "Epoch 141/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.3635e-04 - f1: 1.0000 - val_loss: 1.1227 - val_f1: 0.8225\n",
      "Epoch 142/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.1006e-04 - f1: 1.0000 - val_loss: 1.1184 - val_f1: 0.8251\n",
      "Epoch 143/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.2128e-04 - f1: 1.0000 - val_loss: 1.1187 - val_f1: 0.8309\n",
      "Epoch 144/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.1148e-04 - f1: 1.0000 - val_loss: 1.1111 - val_f1: 0.8275\n",
      "Epoch 145/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.2230e-04 - f1: 1.0000 - val_loss: 1.1228 - val_f1: 0.8313\n",
      "Epoch 146/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.0086e-04 - f1: 1.0000 - val_loss: 1.1273 - val_f1: 0.8293\n",
      "Epoch 147/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.5006e-04 - f1: 1.0000 - val_loss: 1.1418 - val_f1: 0.8300\n",
      "Epoch 148/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0024 - f1: 0.9995 - val_loss: 1.1586 - val_f1: 0.8162\n",
      "Epoch 149/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.8215e-04 - f1: 1.0000 - val_loss: 1.0818 - val_f1: 0.8347\n",
      "Epoch 150/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.4316e-04 - f1: 1.0000 - val_loss: 1.0817 - val_f1: 0.8312\n",
      "Epoch 151/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.3920e-04 - f1: 1.0000 - val_loss: 1.0768 - val_f1: 0.8339\n",
      "Epoch 152/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.1098e-04 - f1: 1.0000 - val_loss: 1.0783 - val_f1: 0.8360\n",
      "Epoch 153/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.2897e-04 - f1: 1.0000 - val_loss: 1.0821 - val_f1: 0.8310\n",
      "Epoch 154/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.6505e-04 - f1: 1.0000 - val_loss: 1.0777 - val_f1: 0.8308\n",
      "Epoch 155/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0015 - f1: 0.9989 - val_loss: 1.1645 - val_f1: 0.8087\n",
      "Epoch 156/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0011 - f1: 1.0000 - val_loss: 1.0931 - val_f1: 0.8377\n",
      "Epoch 157/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.3027e-04 - f1: 1.0000 - val_loss: 1.1130 - val_f1: 0.8289\n",
      "Epoch 158/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.2346e-04 - f1: 1.0000 - val_loss: 1.1253 - val_f1: 0.8290\n",
      "Epoch 159/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0015 - f1: 0.9995 - val_loss: 1.0921 - val_f1: 0.8284\n",
      "Epoch 160/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0042 - f1: 0.9989 - val_loss: 1.1114 - val_f1: 0.8177\n",
      "Epoch 161/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 0.0021 - f1: 0.9995 - val_loss: 1.0611 - val_f1: 0.8253\n",
      "Epoch 162/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.5093e-04 - f1: 1.0000 - val_loss: 1.0799 - val_f1: 0.8289\n",
      "Epoch 163/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.7466e-04 - f1: 1.0000 - val_loss: 1.0400 - val_f1: 0.8241\n",
      "Epoch 164/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.2356e-04 - f1: 1.0000 - val_loss: 1.0486 - val_f1: 0.8242\n",
      "Epoch 165/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.2108e-04 - f1: 1.0000 - val_loss: 1.0388 - val_f1: 0.8244\n",
      "Epoch 166/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.6586e-04 - f1: 1.0000 - val_loss: 1.0512 - val_f1: 0.8222\n",
      "Epoch 167/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.1654e-04 - f1: 1.0000 - val_loss: 1.0569 - val_f1: 0.8237\n",
      "Epoch 168/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.1511e-04 - f1: 1.0000 - val_loss: 1.0831 - val_f1: 0.8220\n",
      "Epoch 169/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.7949e-04 - f1: 1.0000 - val_loss: 1.0916 - val_f1: 0.8185\n",
      "Epoch 170/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.3409e-04 - f1: 1.0000 - val_loss: 1.0991 - val_f1: 0.8198\n",
      "Epoch 171/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.0768e-04 - f1: 1.0000 - val_loss: 1.1065 - val_f1: 0.8176\n",
      "Epoch 172/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.2211e-04 - f1: 1.0000 - val_loss: 1.1115 - val_f1: 0.8155\n",
      "Epoch 173/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 2.1365e-04 - f1: 1.0000 - val_loss: 1.1072 - val_f1: 0.8128\n",
      "Epoch 174/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.8997e-05 - f1: 1.0000 - val_loss: 1.1027 - val_f1: 0.8172\n",
      "Epoch 175/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.0339e-04 - f1: 1.0000 - val_loss: 1.0983 - val_f1: 0.8200\n",
      "Epoch 176/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.2617e-04 - f1: 1.0000 - val_loss: 1.1052 - val_f1: 0.8191\n",
      "Epoch 177/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.2080e-05 - f1: 1.0000 - val_loss: 1.1063 - val_f1: 0.8178\n",
      "Epoch 178/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.9384e-05 - f1: 1.0000 - val_loss: 1.1094 - val_f1: 0.8225\n",
      "Epoch 179/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 7.3541e-05 - f1: 1.0000 - val_loss: 1.1141 - val_f1: 0.8212\n",
      "Epoch 180/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.3658e-04 - f1: 1.0000 - val_loss: 1.1153 - val_f1: 0.8277\n",
      "Epoch 181/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.2540e-05 - f1: 1.0000 - val_loss: 1.1166 - val_f1: 0.8268\n",
      "Epoch 182/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.7952e-04 - f1: 1.0000 - val_loss: 1.1366 - val_f1: 0.8256\n",
      "Epoch 183/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.6589e-05 - f1: 1.0000 - val_loss: 1.1376 - val_f1: 0.8260\n",
      "Epoch 184/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 7.5782e-05 - f1: 1.0000 - val_loss: 1.1378 - val_f1: 0.8295\n",
      "Epoch 185/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.7658e-05 - f1: 1.0000 - val_loss: 1.1392 - val_f1: 0.8247\n",
      "Epoch 186/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 6.0105e-05 - f1: 1.0000 - val_loss: 1.1360 - val_f1: 0.8307\n",
      "Epoch 187/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.7825e-05 - f1: 1.0000 - val_loss: 1.1352 - val_f1: 0.8317\n",
      "Epoch 188/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.0959e-05 - f1: 1.0000 - val_loss: 1.1367 - val_f1: 0.8331\n",
      "Epoch 189/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 7.8447e-05 - f1: 1.0000 - val_loss: 1.1430 - val_f1: 0.8286\n",
      "Epoch 190/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 5.0137e-05 - f1: 1.0000 - val_loss: 1.1394 - val_f1: 0.8309\n",
      "Epoch 191/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.1537e-05 - f1: 1.0000 - val_loss: 1.1423 - val_f1: 0.8339\n",
      "Epoch 192/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 8.0265e-05 - f1: 1.0000 - val_loss: 1.1264 - val_f1: 0.8333\n",
      "Epoch 193/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 4.7630e-05 - f1: 1.0000 - val_loss: 1.1351 - val_f1: 0.8315\n",
      "Epoch 194/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 9.6157e-05 - f1: 1.0000 - val_loss: 1.1569 - val_f1: 0.8323\n",
      "Epoch 195/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 4.0880e-05 - f1: 1.0000 - val_loss: 1.1556 - val_f1: 0.8376\n",
      "Epoch 196/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 1.0660e-04 - f1: 1.0000 - val_loss: 1.1486 - val_f1: 0.8388\n",
      "Epoch 197/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 4.7847e-05 - f1: 1.0000 - val_loss: 1.1383 - val_f1: 0.8360\n",
      "Epoch 198/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 4.5485e-05 - f1: 1.0000 - val_loss: 1.1384 - val_f1: 0.8355\n",
      "Epoch 199/300\n",
      "1839/1839 [==============================] - 7s 4ms/step - loss: 6.6492e-05 - f1: 1.0000 - val_loss: 1.1453 - val_f1: 0.8361\n",
      "Epoch 200/300\n",
      "1839/1839 [==============================] - 7s 4ms/step - loss: 3.0428e-05 - f1: 1.0000 - val_loss: 1.1482 - val_f1: 0.8382\n",
      "Epoch 201/300\n",
      "1839/1839 [==============================] - 7s 4ms/step - loss: 3.2437e-05 - f1: 1.0000 - val_loss: 1.1498 - val_f1: 0.8383\n",
      "Epoch 202/300\n",
      "1839/1839 [==============================] - 7s 4ms/step - loss: 3.4816e-05 - f1: 1.0000 - val_loss: 1.1570 - val_f1: 0.8361\n",
      "Epoch 203/300\n",
      "1839/1839 [==============================] - 6s 4ms/step - loss: 3.9865e-05 - f1: 1.0000 - val_loss: 1.1833 - val_f1: 0.8354\n",
      "Epoch 204/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 6.4216e-05 - f1: 1.0000 - val_loss: 1.1805 - val_f1: 0.8315\n",
      "Epoch 205/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 4.5160e-05 - f1: 1.0000 - val_loss: 1.1617 - val_f1: 0.8412\n",
      "Epoch 206/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 4.1668e-05 - f1: 1.0000 - val_loss: 1.1689 - val_f1: 0.8394\n",
      "Epoch 207/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.4791e-05 - f1: 1.0000 - val_loss: 1.1736 - val_f1: 0.8394\n",
      "Epoch 208/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.3336e-05 - f1: 1.0000 - val_loss: 1.1760 - val_f1: 0.8379\n",
      "Epoch 209/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.2662e-05 - f1: 1.0000 - val_loss: 1.1767 - val_f1: 0.8362\n",
      "Epoch 210/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.6002e-05 - f1: 1.0000 - val_loss: 1.1850 - val_f1: 0.8359\n",
      "Epoch 211/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 1.9812e-05 - f1: 1.0000 - val_loss: 1.1848 - val_f1: 0.8351\n",
      "Epoch 212/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 3.1796e-05 - f1: 1.0000 - val_loss: 1.1862 - val_f1: 0.8332\n",
      "Epoch 213/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.5655e-05 - f1: 1.0000 - val_loss: 1.1919 - val_f1: 0.8299\n",
      "Epoch 214/300\n",
      "1839/1839 [==============================] - 5s 3ms/step - loss: 3.0894e-05 - f1: 1.0000 - val_loss: 1.1916 - val_f1: 0.8326\n",
      "Epoch 215/300\n",
      "1839/1839 [==============================] - 6s 3ms/step - loss: 2.0926e-05 - f1: 1.0000 - val_loss: 1.1944 - val_f1: 0.8318\n",
      "Epoch 216/300\n",
      "  60/1839 [..............................] - ETA: 5s - loss: 9.7486e-06 - f1: 1.0000"
     ]
    }
   ],
   "source": [
    "print('Train...')\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          callbacks=[TensorBoard(log_dir='../logs/{}'.format(\"SMP2018_lstm_{}\".format(get_customization_time())))],\n",
    "          validation_split=0.2\n",
    "          )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "770/770 [==============================] - 0s 240us/step\n",
      "Test score: 0.7415416103291821\n",
      "Test f1: 0.8223602949798882\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test,\n",
    "                       batch_size=batch_size, verbose=1)\n",
    "\n",
    "print('Test score:', score[0])\n",
    "print('Test f1:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat_test = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(770, 31)\n"
     ]
    }
   ],
   "source": [
    "print(y_hat_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 将 one-hot 张量转换成对应的整数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_hat_test, axis=1).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = np.argmax(y_test, axis=1).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查看多分类的 准确率、召回率、F1 值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.90      0.95        21\n",
      "           1       0.86      0.75      0.80         8\n",
      "           2       1.00      0.95      0.98        21\n",
      "           3       0.52      0.57      0.54        23\n",
      "           4       0.91      0.91      0.91        11\n",
      "           5       0.82      0.97      0.89        34\n",
      "           6       0.25      0.17      0.20         6\n",
      "           7       0.86      0.86      0.86        22\n",
      "           8       1.00      0.88      0.93         8\n",
      "           9       0.89      1.00      0.94         8\n",
      "          10       0.95      0.95      0.95        21\n",
      "          11       1.00      0.62      0.77         8\n",
      "          12       0.62      0.70      0.66        60\n",
      "          13       0.86      0.90      0.88        20\n",
      "          14       0.55      0.58      0.56        19\n",
      "          15       0.76      0.78      0.77        36\n",
      "          16       0.87      0.90      0.89       154\n",
      "          17       0.57      0.50      0.53         8\n",
      "          18       0.86      0.75      0.80         8\n",
      "          19       0.74      0.95      0.83        21\n",
      "          20       0.87      0.83      0.85        24\n",
      "          21       1.00      0.75      0.86         8\n",
      "          22       0.67      0.67      0.67         9\n",
      "          23       1.00      1.00      1.00         8\n",
      "          24       0.62      0.56      0.59        18\n",
      "          25       0.92      1.00      0.96        24\n",
      "          26       0.75      0.30      0.43        10\n",
      "          27       0.71      0.55      0.62        22\n",
      "          28       0.71      0.65      0.68        23\n",
      "          29       0.73      0.61      0.67        18\n",
      "          30       0.94      0.94      0.94        89\n",
      "\n",
      "   micro avg       0.82      0.82      0.82       770\n",
      "   macro avg       0.80      0.76      0.77       770\n",
      "weighted avg       0.82      0.82      0.81       770\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_true, y_pred))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}