ch02/02_karaage_ai_book_image_classification_on_camera_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "02_karaage_ai_book_image_classification_on_camera_2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOhKCvDCXQlPVmO5UTv3VT+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/karaage0703/karaage-ai-book/blob/master/ch02/02_karaage_ai_book_image_classification_on_camera_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dBJYhtDxkQom"
      },
      "source": [
        "# カメラでの画像分類（性能改善版）\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fgUg8-4VgjM_"
      },
      "source": [
        "学習させたモデルを使ってカメラ画像に対して画像分類をします。\n",
        "\n",
        "注：このノートブックはハードウェアアクセラレータをGPUよりNone(CPU)で実行した方がレスポンスが良い場合もあります。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MR8zDuZpcrsa"
      },
      "source": [
        "## 事前準備"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OtYbzcsPkg6e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9b6bbedf-cbf5-443f-ceca-759206e5214e"
      },
      "source": [
        "%tensorflow_version 2.x\n",
        "import tensorflow as tf\n",
        "print(tf.__version__)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.3.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NAzaEMEZluLy"
      },
      "source": [
        "from tensorflow.keras.preprocessing.image import img_to_array, load_img\n",
        "import numpy as np\n",
        "import time"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3UvGkLmkjHN"
      },
      "source": [
        "モデルとラベルをGoogle Driveからコピーします。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aNBqDFwukme5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "97a0aff0-015b-4dbf-be31-4a0d83ff8057"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sRmnz5FjkvyS"
      },
      "source": [
        "!cp '/content/drive/My Drive/my_model_aug.h5' './my_model_aug.h5'\n",
        "!cp '/content/drive/My Drive/efficientnet_model.h5' './efficientnet_model.h5'\n",
        "!cp '/content/drive/My Drive/labels.txt' './labels.txt'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HAMrF1KhzSwl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "883e4326-58ef-47a6-f9ea-aa95548b9256"
      },
      "source": [
        "!ls -lh"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 124M\n",
            "drwx------ 5 root root 4.0K Nov 27 17:12 drive\n",
            "-rw------- 1 root root  34M Nov 27 17:12 efficientnet_model.h5\n",
            "-rw------- 1 root root   12 Nov 27 17:12 labels.txt\n",
            "-rw------- 1 root root  91M Nov 27 17:12 my_model_aug.h5\n",
            "drwxr-xr-x 1 root root 4.0K Nov 13 17:33 sample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVY8Z3TprQ27"
      },
      "source": [
        "## AIモデルとラベルの読み込み"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Dz5nlv_eMEW"
      },
      "source": [
        "EfficientNetモデルを読み込む場合は以下実行ください"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hn73mVWZ1pSK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "909c1a35-6dc9-4a92-da80-4329c95fb90a"
      },
      "source": [
        "# keras_model = tf.keras.models.load_model(\"efficientnet_model.h5\")\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "keras_model = tf.keras.models.load_model('efficientnet_model.h5', custom_objects={'KerasLayer':hub.KerasLayer})\n",
        "keras_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "keras_layer (KerasLayer)     (None, 1280)              4049564   \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 3)                 3843      \n",
            "=================================================================\n",
            "Total params: 4,053,407\n",
            "Trainable params: 3,843\n",
            "Non-trainable params: 4,049,564\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEFCHu-w1p39"
      },
      "source": [
        "参考： https://github.com/tensorflow/tensorflow/issues/26835"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S06WAfD8n-6m"
      },
      "source": [
        "データ水増しして学習させたMNISTベースの自作モデルを読み込む場合は、以下実行ください。\n",
        "（EfficientNetを試したい場合は、こちらはスキップしてください）。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n59NFDa3n8ta",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88922bf6-0911-4275-e8b4-056663468e08"
      },
      "source": [
        "keras_model = tf.keras.models.load_model(\"my_model_aug.h5\")\n",
        "keras_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d (Conv2D)              (None, 64, 64, 32)        896       \n",
            "_________________________________________________________________\n",
            "activation (Activation)      (None, 64, 64, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_1 (Conv2D)            (None, 62, 62, 64)        18496     \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 62, 62, 64)        0         \n",
            "_________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D) (None, 31, 31, 64)        0         \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 31, 31, 64)        0         \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 61504)             0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 128)               7872640   \n",
            "_________________________________________________________________\n",
            "activation_2 (Activation)    (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 3)                 387       \n",
            "_________________________________________________________________\n",
            "activation_3 (Activation)    (None, 3)                 0         \n",
            "=================================================================\n",
            "Total params: 7,892,419\n",
            "Trainable params: 7,892,419\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2I4uwnHmsRl7"
      },
      "source": [
        "ラベルを読み込みます"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DL3IxYtmsTAe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31964216-7a62-43f2-e541-419a9ff0220a"
      },
      "source": [
        "labels = []\n",
        "with open('./labels.txt','r') as f:\n",
        "  for line in f:\n",
        "    labels.append(line.rstrip())\n",
        "print(labels)\n",
        "\n",
        "NUM_CLASSES = len(labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['choki', 'gu', 'pa']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NQRoDZQslQLf"
      },
      "source": [
        "## 画像分類デモの実行"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PsGeC0PshPcx"
      },
      "source": [
        "Webカメラを使うJavascriptのコードをダウンロードします。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-CmL-aZRAz5U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dfeb9055-6397-4f41-d376-b0f3c33bffb2"
      },
      "source": [
        "!wget https://raw.githubusercontent.com/karaage0703/karaage-ai-book/master/util/colab_camera.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-11-27 17:13:47--  https://raw.githubusercontent.com/karaage0703/karaage-ai-book/master/util/colab_camera.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 3036 (3.0K) [text/plain]\n",
            "Saving to: ‘colab_camera.py’\n",
            "\n",
            "\rcolab_camera.py       0%[                    ]       0  --.-KB/s               \rcolab_camera.py     100%[===================>]   2.96K  --.-KB/s    in 0s      \n",
            "\n",
            "2020-11-27 17:13:47 (31.4 MB/s) - ‘colab_camera.py’ saved [3036/3036]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U23HbQgKBYR3"
      },
      "source": [
        "コードを読み込みます。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tO53JtEdA4Tt"
      },
      "source": [
        "import colab_camera"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ngH5oj5chaYB"
      },
      "source": [
        "処理を行うコールバック関数を定義します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-6NQFFARhV86"
      },
      "source": [
        "import IPython\n",
        "from google.colab import output\n",
        "import cv2\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "from io import BytesIO\n",
        "import base64\n",
        "\n",
        "def run(img_str):\n",
        "  #decode to image\n",
        "  decimg = base64.b64decode(img_str.split(',')[1], validate=True)\n",
        "  decimg = Image.open(BytesIO(decimg))\n",
        "  decimg = np.array(decimg, dtype=np.uint8); \n",
        "  decimg = cv2.cvtColor(decimg, cv2.COLOR_BGR2RGB)\n",
        "\n",
        "  # image classify\n",
        "  img = cv2.resize(decimg, (64, 64))\n",
        "  img = img_to_array(img)\n",
        "  img = img / 255.0\n",
        "  img = img[np.newaxis]\n",
        "  img = np.asarray(img)\n",
        "  preds = keras_model.predict(img)\n",
        "  result = np.argmax(preds[0])\n",
        "\n",
        "  # draw output\n",
        "  out_img = decimg\n",
        "  cv2.putText(out_img, labels[result], (10,50), \\\n",
        "      cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 1, cv2.LINE_AA)\n",
        "\n",
        "  #encode to string\n",
        "  _, encimg = cv2.imencode(\".jpg\", out_img,\n",
        "      [int(cv2.IMWRITE_JPEG_QUALITY), 80])\n",
        "  img_str = encimg.tobytes()\n",
        "  img_str = (\"data:image/jpeg;base64,\" +\n",
        "      base64.b64encode(img_str).decode('utf-8'))\n",
        "  return IPython.display.JSON({'img_str': img_str})\n",
        "\n",
        "output.register_callback('notebook.run', run)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_W8n9Q9AsyL1"
      },
      "source": [
        "デモを実施します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sB8reukWimWG"
      },
      "source": [
        "colab_camera.use_cam()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O257maL9kW7k"
      },
      "source": [
        "# 参考リンク\n",
        "\n",
        "- https://qiita.com/a2kiti/items/f32de4f51a31d609e5a5"
      ]
    }
  ]
}