設計你的神經網路.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![設計你的神經網路](images/create_your_NN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們終於要開始做生命中第一個神經網路..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 讀入套件\n",
    "\n",
    "這裡我們讀入一些套件, 今天暫時不要理會細節。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再來是我們標準數據分析動作!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: KERAS_BACKEND=tensorflow\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%env KERAS_BACKEND=tensorflow\n",
    "\n",
    "# 標準數據分析、畫圖套件\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 神經網路方面\n",
    "from keras.datasets import mnist\n",
    "from keras.utils import np_utils\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "# 互動設計用\n",
    "from ipywidgets import interact_manual"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 讀入 MNIST 數據庫\n",
    "\n",
    "MNIST 是有一堆 0-9 的手寫數字圖庫。有 6 萬筆訓練資料, 1 萬筆測試資料。它是 \"Modified\" 版的 NIST 數據庫, 原來的版本有更多資料。這個 Modified 的版本是由 LeCun, Cortes, 及 Burges 等人做的。可以參考這個數據庫的[原始網頁](http://yann.lecun.com/exdb/mnist/)。\n",
    "\n",
    "MNIST 可以說是 Deep Learning 最有名的範例, 它被 Deep Learning 大師 Hinton 稱為「機器學習的果蠅」。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 由 Keras 讀入 MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras 很貼心的幫我們準備好 MNIST 數據庫, 我們可以這樣讀進來 (第一次要花點時間)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們來看看訓練資料是不是 6 萬筆、測試資料是不是有 1 筆。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "訓練資料總筆數為 60000 筆資料\n",
      "測試資料總筆數為 10000 筆資料\n"
     ]
    }
   ],
   "source": [
    "print(f'訓練資料總筆數為 {len(x_train)} 筆資料')\n",
    "print(f'測試資料總筆數為 {len(x_test)} 筆資料')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 數據庫的內容\n",
    "\n",
    "每筆輸入 (x) 就是一個手寫的 0-9 中一個數字的圖檔, 大小為 28x28。而輸出 (y) 當然就是「正確答案」。我們來看看編訓練資料的 x 輸入、輸出的部份分別長什麼樣子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_xy(n=0):\n",
    "    ax = plt.gca()\n",
    "    X = x_train[n]\n",
    "    plt.xticks([], [])\n",
    "    plt.yticks([], [])\n",
    "    plt.imshow(X.reshape(28,28), cmap = 'Greys')\n",
    "    print(f'本資料 y 給定的答案為: {y_train[n]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "efe8cd7e35224195ad7b5fe986c03357",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='n', max=59999), Button(description='Run Interact', style…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interact_manual(show_xy, n=(0,59999));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 62\n",
    "X = x_train[n]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,  29, 218, 255, 234,  45,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,  31, 216, 253, 253, 253, 218,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         69, 244, 253, 253, 196, 253, 220,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  33,\n",
       "        214, 253, 253, 205,   9, 176, 220,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 196,\n",
       "        253, 253, 230,  57,   0,  17,  22,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 122, 248,\n",
       "        253, 230,  58,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  78, 248, 253,\n",
       "        253, 144,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  92, 253, 253,\n",
       "        194,  11,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,  28, 204, 253, 253,\n",
       "        144,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,  99, 253, 253, 233,\n",
       "         57, 100, 196, 196, 196, 175,  57,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 195, 253, 253, 193,\n",
       "        181, 253, 253, 253, 253, 253, 228,  23,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 228, 253, 253, 253,\n",
       "        253, 253, 184,  98, 210, 253, 253, 191,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 228, 253, 253, 253,\n",
       "        240, 174,  20,   0,  46, 253, 253, 227,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 228, 253, 253, 253,\n",
       "        181,   0,   0,   0,  66, 253, 253, 139,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 228, 253, 198, 198,\n",
       "         91,   0,   0,  34, 226, 253, 249,  85,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 228, 253, 221,  28,\n",
       "          0,   0,   7, 140, 253, 253, 207,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0, 151, 253, 253, 168,\n",
       "         46,  54, 174, 253, 253, 220,  37,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,  43, 229, 253, 253,\n",
       "        253, 253, 253, 253, 251,  80,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  42, 227, 252,\n",
       "        253, 253, 253, 252, 135,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 102,\n",
       "        159, 152, 123,  98,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0]], dtype=uint8)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 輸入格式整理\n",
    "\n",
    "我們現在要用標準神經網路學學手寫辨識。原來的每筆數據是個 28x28 的矩陣 (array), 但標準神經網路只吃「平平的」, 也就是每次要 28x28=784 長的向量。因此我們要用 `reshape` 調校一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(60000, 784)/255\n",
    "x_test = x_test.reshape(10000, 784)/255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 輸出格式整理\n",
    "\n",
    "我們可能會想, 我們想學的函數是這樣的型式:\n",
    "\n",
    "$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}$$\n",
    "\n",
    "其實這樣不太好! 為什麼呢? 比如說我們的輸入 x 是一張 0 的圖, 因為我們訓練的神經網路總會有點誤差, 所以可能會得到:\n",
    "\n",
    "$$\\hat{f}(x) = 0.5$$\n",
    "\n",
    "那這意思是有可能是 0, 也有可能是 1 嗎!!?? 可是 0 和 1 根本不像啊。換句話說分類的問題這樣做其實不合理!\n",
    "\n",
    "於是我們會做 \"1-hot enconding\", 也就是\n",
    "\n",
    "* 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0]\n",
    "* 5 -> [0, 0, 0, 0, 0, 1, 0, 0, 0]\n",
    "\n",
    "等等。因為分類問題基本上都要做這件事, Keras 其實已幫我們準備好套件!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[87]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np_utils.to_categorical(y_train, 10)\n",
    "y_test = np_utils.to_categorical(y_test, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們來看看剛剛某號數據的答案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 1.], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 87\n",
    "y_train[n]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "和我們想的一樣! 至此我們可以打造我們的神經網路了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 打造第一個神經網路\n",
    "\n",
    "我們決定了我們的函數是\n",
    "\n",
    "$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}^{10}$$\n",
    "\n",
    "這個樣子。而我們又說第一次要用標準神網路試試, 所以我們只需要再決定要幾個隱藏層、每層要幾個神經元, 用哪個激發函數就可以了。\n",
    "\n",
    "### 3.1 決定神經網路架構、讀入相關套件\n",
    "\n",
    "假如我們要這麼做:\n",
    "\n",
    "* 使用 <span style=\"color:red;\"> 3 </span> 個 hidden layers\n",
    "* Hidden layer 1 用 <span style=\"color:red;\"> 6 </span> 個神經元\n",
    "* Hidden layer 2 用 <span style=\"color:red;\"> 28 </span> 個神經元\n",
    "* Hidden layer 3 用 <span style=\"color:red;\"> 2 </span> 個神經元\n",
    "\n",
    "* Activation Function 唯一指名 <span style=\"color:red;\">relu</span>\n",
    "\n",
    "於是從 Keras 把相關套件讀進來。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 建構我們的神經網路\n",
    "\n",
    "和以前做迴歸或機器學習一樣, 我們就打開個「函數學習機」。標準一層一層傳遞的神經網路叫 `Sequential`, 於是我們打開一個空的神經網路。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0925 15:42:11.123747 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們每次用 `add` 去加一層, 從第一個隱藏層開始。而第一個隱藏層因為 Keras 當然猜不到輸入有幾個 features, 所以我們要告訴它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0925 15:43:31.974145 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "W0925 15:43:31.978698 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.add(Dense(6, input_dim=784, activation='relu'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二層 hidden layer 因為前面輸出是 4, 現在輸入是 2, 就不用再說了! 這裡的 2 只告訴 Keras, 我們第二層是用 2 個神經元!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(28, activation='relu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(2, activation='relu'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "輸出有 10 個數字, 所以輸出層的神經元是 10 個! 而如果我們的網路輸出是 \n",
    "\n",
    "$$(y_1, y_2, \\ldots, y_{10})$$\n",
    "\n",
    "我們還希望\n",
    "\n",
    "$$\\sum_{i=1}^{10} y_i = 1$$\n",
    "\n",
    "這可能嗎, 結果是很容易, 就用 `softmax` 當激發函數就可以!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我們的第一個神經網路就建好了!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 組裝\n",
    "\n",
    "和之前比較不一樣的是我們還要做 `compile` 才正式把我們的神經網路建好。你可以發現我們還需要做幾件事:\n",
    "\n",
    "* 決定使用的 loss function, 一般是 `mse`\n",
    "* 決定 optimizer, 我們用標準的 SGD\n",
    "* 設 learning rate\n",
    "\n",
    "為了一邊訓練一邊看到結果, 我們加設\n",
    "\n",
    "    metrics=['accuracy']\n",
    "    \n",
    "本行基本上和我們的神經網路功能沒有什麼關係。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0925 15:47:01.226172 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='mse', optimizer=SGD(lr=0.087), metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 檢視我們的神經網路"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以檢視我們神經網路的架構, 可以確認一下是不是和我們想像的一樣。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 看 model 的 summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 6)                 4710      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 28)                196       \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 2)                 58        \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 10)                30        \n",
      "=================================================================\n",
      "Total params: 4,994\n",
      "Trainable params: 4,994\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很快算算參數數目和我們想像是否是一樣的!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "196"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "6*28 + 28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 訓練你的第一個神經網路\n",
    "\n",
    "恭喜! 我們完成了第一個神經網路。現在要訓練的時候, 你會發現不是像以前沒頭沒腦把訓練資料送進去就好。這裡我們還有兩件事要決定:\n",
    "\n",
    "* 一次要訓練幾筆資料 (`batch_size`), 我們就 100 筆調一次參數好了\n",
    "* 這 6 萬筆資料一共要訓練幾次 (`epochs`), 我們訓練個 20 次試試\n",
    "\n",
    "於是最精彩的就來了。你要有等待的心理準備..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0925 15:52:50.015725 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
      "\n",
      "W0925 15:52:50.030764 140666028386048 deprecation_wrapper.py:119] From /home/yenlung/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 2s 37us/step - loss: 0.0898 - acc: 0.1730\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0894 - acc: 0.1889\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0888 - acc: 0.1629\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 1s 20us/step - loss: 0.0880 - acc: 0.1486\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0869 - acc: 0.1602\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0857 - acc: 0.1954\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0835 - acc: 0.2005\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0817 - acc: 0.2059\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0811 - acc: 0.2080\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0807 - acc: 0.2101\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0805 - acc: 0.2132\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0803 - acc: 0.2174\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0801 - acc: 0.2233\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 1s 18us/step - loss: 0.0798 - acc: 0.2279\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0796 - acc: 0.2333\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0794 - acc: 0.2373\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 1s 20us/step - loss: 0.0791 - acc: 0.2433\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0789 - acc: 0.2509\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0786 - acc: 0.2572\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 1s 19us/step - loss: 0.0783 - acc: 0.2649\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7feee475ef60>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, batch_size=100, epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 試用我們的結果\n",
    "\n",
    "我們來用比較炫的方式來看看可愛的神經網路學習成果。對指令有問題可以參考我們之前的 MOOC 影片教學。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interact_manual"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們 \"predict\" 放的是我們神經網路的學習結果。這裡用 `predict_classes` 會讓我們 Keras 選 10 個輸出機率最大的那類。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict = model.predict_classes(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7, 7, 1, ..., 7, 8, 7])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不要忘了我們的 `x_test` 每筆資料已經換成 784 維的向量, 我們要整型回 28x28 的矩陣才能當成圖形顯示出來!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(測試編號):\n",
    "    plt.imshow(x_test[測試編號].reshape(28,28), cmap='Greys')\n",
    "    print('神經網路判斷為:', predict[測試編號])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "神經網路判斷為: 3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAOBElEQVR4nO3de6hd9ZnG8efxFpJY0EyOGqyaThGcMFJbNlFxrEqY4C3GipeKFIsyCZiI1QrjDSqImIit9I+xcBylp0O1KbReghIVKWj/sGQrmRiNM16SaaOJOV6gXiKdmHf+ONvhGM/+7eNea1/M+/3AYe+93r32et3mOWuf9Vtr/xwRArDv22/QDQDoD8IOJEHYgSQIO5AEYQeSOKCfG5s7d27Mnz+/n5sEUtm6daveeecdT1WrFHbbZ0r6uaT9Jf17RKwqPX/+/PlqNptVNgmgoNFotK11/THe9v6S/k3SWZIWSLrU9oJuXw9Ab1X5m32hpNci4o2I+Juk30haWk9bAOpWJexHSvrLpMfbWss+x/Yy203bzfHx8QqbA1BFlbBPdRDgC+feRsRoRDQiojEyMlJhcwCqqBL2bZKOmvT465LeqtYOgF6pEvb1ko61/Q3bB0n6vqRH62kLQN26HnqLiN22V0p6QhNDb/dHxEu1dQagVpXG2SPicUmP19QLgB7idFkgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSqDSLayYvv/xy29rY2Fhx3bvuuqvStvfs2VOs77df735nr1ixolhfvXp1sT5z5sw620EFlcJue6ukDyR9Kml3RDTqaApA/erYs58REe/U8DoAeoi/2YEkqoY9JD1p+3nby6Z6gu1ltpu2m+Pj4xU3B6BbVcN+SkR8R9JZklbY/u7eT4iI0YhoRERjZGSk4uYAdKtS2CPirdbtTkkPSVpYR1MA6td12G3Ptv21z+5LWixpU12NAahXlaPxh0t6yPZnr/NARKyrpasBWLNmTbG+cuXKtrX333+/uG7rPepap3H0qq9fcs899xTrixYtKtaXLl1aZzuooOuwR8Qbkr5VYy8AeoihNyAJwg4kQdiBJAg7kARhB5LgEteWN998s1ifMWNG29oRRxxRXPeSSy4p1m+88cZifdasWcV6yUcffVSsz5s3r+vXxlcLe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9pbrrruuUn1Ydbr8FnmwZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhn3we89957bWtnnHFGpdc+8cQTi/XFixdXen30D3t2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfavgA8//LBYX7VqVdva66+/Xlx39uzZxfratWuL9ZkzZxbrGB4d9+y277e90/amScvm2H7K9qut20N72yaAqqbzMf6Xks7ca9kNkp6OiGMlPd16DGCIdQx7RDwjae/zMZdKGmvdH5N0fs19AahZtwfoDo+I7ZLUuj2s3RNtL7PdtN0cHx/vcnMAqur50fiIGI2IRkQ0RkZGer05AG10G/a3bc+TpNbtzvpaAtAL3Yb9UUmXt+5fLumRetoB0Csdx9ltPyjpdElzbW+T9BNJqyT91vaVkv4s6aJeNrmvK12PLklXX311sb5mzZq2NdvFdUvzzkvSc889V6yfdNJJxfqcOXOKdfRPx7BHxKVtSotq7gVAD3G6LJAEYQeSIOxAEoQdSIKwA0lwiWsNPv7442J9bGysWL/77ruL9U6XqXYaXivpNKXzeeedV6wffPDBxfqKFSva1m655Zbiulw+Wy/27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPsNbj99tuL9dWrV1d6/eOPP75YP+ecc9rWlixZUlz3kUfKX0Wwbt26Yn3jxo3Feum/vdPls52+xnrWrFnFei/t2LGjWO90ae9BBx1UZzvTwp4dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRPRtY41GI5rNZt+21y+dxqLPPffcYr10zbdUnpJZ6u1137t27SrWH3jggWJ9+fLlXW/7mGOOKdbXr19frFf5GuuHH364WH/llVeK9euvv75YP+CA3pzi0mg01Gw2p/yCA/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+zoqdJ01AsXLiyuu2XLlmJ9dHS0WB8ZGWlbu+aaa4rrHnjggcV6p2vxBzVVdaVxdtv3295pe9OkZbfaftP2htbP2XU2DKB+0/kY/0tJZ06x/O6IOKH183i9bQGoW8ewR8Qzktp/FgPwlVDlAN1K2xtbH/MPbfck28tsN203x8fHK2wOQBXdhv0Xkr4p6QRJ2yX9tN0TI2I0IhoR0SgdMAHQW12FPSLejohPI2KPpHsllQ+rAhi4rsJue96kh9+TtKndcwEMh44X1dp+UNLpkuba3ibpJ5JOt32CpJC0VVL3Fy1jn1Yab77vvvuK6y5atKhY73StfOkckgsuuKC47r333lusH3LIIcX6MOoY9oi4dIrF5f9LAIYOp8sCSRB2IAnCDiRB2IEkCDuQBFM2o6d2797dttbraYtLl7HecccdxXVnzJhRdzsDx54dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnB2VbN26tVi/6qqr2taefPLJmrv5vNNOO61tbV8cR++EPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4O4qeffbZYv3CCy8s1t9999062/lSFi9ePLBtDyP27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPs+7g9e/YU65s3by7WO42jH3300cX6unXr2tZOPfXU4rq7du0q1s8///xifebMmcV6Nh337LaPsv0H25ttv2T7mtbyObafsv1q6/bQ3rcLoFvT+Ri/W9KPI+IfJJ0kaYXtBZJukPR0RBwr6enWYwBDqmPYI2J7RLzQuv+BpM2SjpS0VNJY62ljksqfqQAM1Jc6QGd7vqRvS/qTpMMjYrs08QtB0mFt1llmu2m7OT4+Xq1bAF2bdthtHyzpd5J+FBF/ne56ETEaEY2IaIyMjHTTI4AaTCvstg/URNB/HRG/by1+2/a8Vn2epJ29aRFAHToOvdm2pPskbY6In00qPSrpckmrWreP9KRDVPLEE08U60uWLCnWFyxYUKw/9thjxXpp2uRPPvmkuO7EP732brvttmIdnzedcfZTJP1A0ou2N7SW3aSJkP/W9pWS/izpot60CKAOHcMeEX+U1O5X7KJ62wHQK5wuCyRB2IEkCDuQBGEHkiDsQBJc4roP2LFjR9vaFVdcUem1b7755mJ9+fLlxfratWvb1jqNo995553F+nHHHVes4/PYswNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyz7wO2bNnStlb1q8Auu+yyYj0iivXSWHqncfRrr722WMeXw54dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnB2VdJo2+eKLL25bu+givn28n9izA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAS05mf/ShJv5J0hKQ9kkYj4ue2b5X0L5I+u2D6poh4vFeNor2TTz65bW337t197ATDbDon1eyW9OOIeMH21yQ9b/upVu3uiLird+0BqMt05mffLml76/4HtjdLOrLXjQGo15f6m932fEnflvSn1qKVtjfavt/2oW3WWWa7abtZ9SuSAHRv2mG3fbCk30n6UUT8VdIvJH1T0gma2PP/dKr1ImI0IhoR0RgZGamhZQDdmFbYbR+oiaD/OiJ+L0kR8XZEfBoReyTdK2lh79oEUFXHsHvi60Hvk7Q5In42afm8SU/7nqRN9bcHoC7TORp/iqQfSHrR9obWspskXWr7BEkhaauk8ty9AAZqOkfj/yhpqi//Zkwd+ArhDDogCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjoj+bcwel/Q/kxbNlfRO3xr4coa1t2HtS6K3btXZ2zERMeX3v/U17F/YuN2MiMbAGigY1t6GtS+J3rrVr974GA8kQdiBJAYd9tEBb79kWHsb1r4keutWX3ob6N/sAPpn0Ht2AH1C2IEkBhJ222fa/i/br9m+YRA9tGN7q+0XbW+w3RxwL/fb3ml706Rlc2w/ZfvV1u2Uc+wNqLdbbb/Zeu822D57QL0dZfsPtjfbfsn2Na3lA33vCn315X3r+9/stveX9N+S/lnSNknrJV0aES/3tZE2bG+V1IiIgZ+AYfu7kj6U9KuI+MfWsjslvRcRq1q/KA+NiH8dkt5ulfThoKfxbs1WNG/yNOOSzpf0Qw3wvSv0dbH68L4NYs++UNJrEfFGRPxN0m8kLR1AH0MvIp6R9N5ei5dKGmvdH9PEP5a+a9PbUIiI7RHxQuv+B5I+m2Z8oO9doa++GETYj5T0l0mPt2m45nsPSU/aft72skE3M4XDI2K7NPGPR9JhA+5nbx2n8e6nvaYZH5r3rpvpz6saRNinmkpqmMb/TomI70g6S9KK1sdVTM+0pvHulymmGR8K3U5/XtUgwr5N0lGTHn9d0lsD6GNKEfFW63anpIc0fFNRv/3ZDLqt250D7uf/DdM03lNNM64heO8GOf35IMK+XtKxtr9h+yBJ35f06AD6+ALbs1sHTmR7tqTFGr6pqB+VdHnr/uWSHhlgL58zLNN4t5tmXAN+7wY+/XlE9P1H0tmaOCL/uqSbB9FDm77+XtJ/tn5eGnRvkh7UxMe6/9XEJ6IrJf2dpKclvdq6nTNEvf2HpBclbdREsOYNqLd/0sSfhhslbWj9nD3o967QV1/eN06XBZLgDDogCcIOJEHYgSQIO5AEYQeSIOxAEoQdSOL/AERtM2F2tw7mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test(87)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "656c021ed3cf46b0bbcf36177dc789c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=4999, description='測試編號', max=9999), Button(description='Run Interact', …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.test(測試編號)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact_manual(test, 測試編號=(0, 9999))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到底測試資料總的狀況如何呢? 我們可以給我們神經網路「考一下試」。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 27us/step\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.07798466575145721\n",
      "正確率 0.27\n"
     ]
    }
   ],
   "source": [
    "print('loss:', score[0])\n",
    "print('正確率', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 訓練好的神經網路存起來!\n",
    "\n",
    "如果對訓練成果滿意, 我們當然不想每次都再訓練一次! 我們可以把神經網路的架構和訓練好的參數都存起來, 以供日後使用!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之前還沒裝 pyh5 要在終端機 (Anaconda Prompt) 下安裝:\n",
    "    \n",
    "    conda install h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_json = model.to_json()\n",
    "open('stupid_model.json', 'w').write(model_json)\n",
    "model.save_weights('stupid_model_weights.h5')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}