chapter12_part01_text-generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J-WQ_tQxCrEF"
      },
      "source": [
        "이 노트북은 [케라스 창시자에게 배우는 딥러닝 2판](https://tensorflow.blog/kerasdl2/)의 예제 코드를 담고 있습니다.\n",
        "\n",
        "<table align=\"left\">\n",
        "    <tr>\n",
        "        <td>\n",
        "            <a href=\"https://colab.research.google.com/github/rickiepark/deep-learning-with-python-2nd/blob/main/chapter12_part01_text-generation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "        </td>\n",
        "    </tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "THz7hp89CrEI"
      },
      "source": [
        "# 생성 모델을 위한 딥러닝"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBnsAPKECrEI"
      },
      "source": [
        "## 텍스트 생성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D89JwqPBCrEJ"
      },
      "source": [
        "### 시퀀스 생성을 위한 딥러닝 모델의 간단한 역사"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z4P5rIr6CrEJ"
      },
      "source": [
        "### 시퀀스 데이터를 어떻게 생성할까?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBs8uChTCrEJ"
      },
      "source": [
        "### 샘플링 전략의 중요성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMcqO4_VCrEK"
      },
      "source": [
        "**다른 온도 값을 사용하여 확률 분포의 가중치 바꾸기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "baD0AHOUCrEK"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "def reweight_distribution(original_distribution, temperature=0.5):\n",
        "    distribution = np.log(original_distribution) / temperature\n",
        "    distribution = np.exp(distribution)\n",
        "    return distribution / np.sum(distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9hCKGeqJCrEL"
      },
      "source": [
        "### 케라스를 사용한 텍스트 생성 모델 구현"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NmTMyS1ZCrEM"
      },
      "source": [
        "#### 데이터 준비"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IM6Umf4uCrEM"
      },
      "source": [
        "**IMDB 영화 리뷰 데이터셋 다운로드하고 압축 풀기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fS_z2t7PCrEN",
        "outputId": "cc1ec0c3-a0dd-4d47-e52a-1905678f2f59"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-11-06 02:19:00--  https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
            "Resolving ai.stanford.edu (ai.stanford.edu)... 171.64.68.10\n",
            "Connecting to ai.stanford.edu (ai.stanford.edu)|171.64.68.10|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 84125825 (80M) [application/x-gzip]\n",
            "Saving to: ‘aclImdb_v1.tar.gz’\n",
            "\n",
            "aclImdb_v1.tar.gz   100%[===================>]  80.23M  4.30MB/s    in 28s     \n",
            "\n",
            "2023-11-06 02:19:28 (2.92 MB/s) - ‘aclImdb_v1.tar.gz’ saved [84125825/84125825]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
        "!tar -xf aclImdb_v1.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcHe7T_QCrEN"
      },
      "source": [
        "**텍스트 파일(한 파일 = 한 샘플)에서 데이터셋 만들기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hb2YjZ86CrEO",
        "outputId": "4544c2ba-0845-4aef-adb7-bd4ba20a5fa4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 100006 files belonging to 1 classes.\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "dataset = keras.utils.text_dataset_from_directory(\n",
        "    directory=\"aclImdb\", label_mode=None, batch_size=256)\n",
        "dataset = dataset.map(lambda x: tf.strings.regex_replace(x, \"<br />\", \" \"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-6pyZNQCrEO"
      },
      "source": [
        "**`TextVectorization` 층 준비하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "aHTeHqNfCrEO"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.layers import TextVectorization\n",
        "\n",
        "sequence_length = 100\n",
        "vocab_size = 15000\n",
        "text_vectorization = TextVectorization(\n",
        "    max_tokens=vocab_size,\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=sequence_length,\n",
        ")\n",
        "text_vectorization.adapt(dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLhj695ZCrEO"
      },
      "source": [
        "**언어 모델링 데이터셋 만들기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "pCPsdys6CrEP"
      },
      "outputs": [],
      "source": [
        "def prepare_lm_dataset(text_batch):\n",
        "    vectorized_sequences = text_vectorization(text_batch)\n",
        "    x = vectorized_sequences[:, :-1]\n",
        "    y = vectorized_sequences[:, 1:]\n",
        "    return x, y\n",
        "\n",
        "lm_dataset = dataset.map(prepare_lm_dataset, num_parallel_calls=4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZDJxIp9CrEP"
      },
      "source": [
        "#### 트랜스포머 기반의 시퀀스-투-시퀀스 모델"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "O40uNWAWCrEP"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "class PositionalEmbedding(layers.Layer):\n",
        "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.token_embeddings = layers.Embedding(\n",
        "            input_dim=input_dim, output_dim=output_dim)\n",
        "        self.position_embeddings = layers.Embedding(\n",
        "            input_dim=sequence_length, output_dim=output_dim)\n",
        "        self.sequence_length = sequence_length\n",
        "        self.input_dim = input_dim\n",
        "        self.output_dim = output_dim\n",
        "\n",
        "    def call(self, inputs):\n",
        "        length = tf.shape(inputs)[-1]\n",
        "        positions = tf.range(start=0, limit=length, delta=1)\n",
        "        embedded_tokens = self.token_embeddings(inputs)\n",
        "        embedded_positions = self.position_embeddings(positions)\n",
        "        return embedded_tokens + embedded_positions\n",
        "\n",
        "    def compute_mask(self, inputs, mask=None):\n",
        "        return tf.math.not_equal(inputs, 0)\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super(PositionalEmbedding, self).get_config()\n",
        "        config.update({\n",
        "            \"output_dim\": self.output_dim,\n",
        "            \"sequence_length\": self.sequence_length,\n",
        "            \"input_dim\": self.input_dim,\n",
        "        })\n",
        "        return config\n",
        "\n",
        "\n",
        "class TransformerDecoder(layers.Layer):\n",
        "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.embed_dim = embed_dim\n",
        "        self.dense_dim = dense_dim\n",
        "        self.num_heads = num_heads\n",
        "        self.attention_1 = layers.MultiHeadAttention(\n",
        "          num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.attention_2 = layers.MultiHeadAttention(\n",
        "          num_heads=num_heads, key_dim=embed_dim)\n",
        "        self.dense_proj = keras.Sequential(\n",
        "            [layers.Dense(dense_dim, activation=\"relu\"),\n",
        "             layers.Dense(embed_dim),]\n",
        "        )\n",
        "        self.layernorm_1 = layers.LayerNormalization()\n",
        "        self.layernorm_2 = layers.LayerNormalization()\n",
        "        self.layernorm_3 = layers.LayerNormalization()\n",
        "        self.supports_masking = True\n",
        "\n",
        "    def get_config(self):\n",
        "        config = super(TransformerDecoder, self).get_config()\n",
        "        config.update({\n",
        "            \"embed_dim\": self.embed_dim,\n",
        "            \"num_heads\": self.num_heads,\n",
        "            \"dense_dim\": self.dense_dim,\n",
        "        })\n",
        "        return config\n",
        "\n",
        "    # def get_causal_attention_mask(self, inputs):\n",
        "    #     input_shape = tf.shape(inputs)\n",
        "    #     batch_size, sequence_length = input_shape[0], input_shape[1]\n",
        "    #     i = tf.range(sequence_length)[:, tf.newaxis]\n",
        "    #     j = tf.range(sequence_length)\n",
        "    #     mask = tf.cast(i >= j, dtype=\"int32\")\n",
        "    #     mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))\n",
        "    #     mult = tf.concat(\n",
        "    #         [tf.expand_dims(batch_size, -1),\n",
        "    #          tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
        "    #     return tf.tile(mask, mult)\n",
        "\n",
        "    def call(self, inputs, encoder_outputs, mask=None):\n",
        "        # causal_mask = self.get_causal_attention_mask(inputs)\n",
        "        # if mask is not None:\n",
        "        #     padding_mask = tf.cast(\n",
        "        #         mask[:, tf.newaxis, :], dtype=\"int32\")\n",
        "        #     padding_mask = tf.minimum(padding_mask, causal_mask)\n",
        "        attention_output_1 = self.attention_1(\n",
        "            query=inputs,\n",
        "            value=inputs,\n",
        "            key=inputs,\n",
        "            use_causal_mask=True)\n",
        "            # attention_mask=causal_mask)\n",
        "        attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
        "        attention_output_2 = self.attention_2(\n",
        "            query=attention_output_1,\n",
        "            value=encoder_outputs,\n",
        "            key=encoder_outputs\n",
        "            # attention_mask=padding_mask,\n",
        "        )\n",
        "        attention_output_2 = self.layernorm_2(\n",
        "            attention_output_1 + attention_output_2)\n",
        "        proj_output = self.dense_proj(attention_output_2)\n",
        "        return self.layernorm_3(attention_output_2 + proj_output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_qO94YIGCrEQ"
      },
      "source": [
        "**간단한 트랜스포머 기반 언어 모델**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "LGCHdjUqCrEQ"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "embed_dim = 256\n",
        "latent_dim = 2048\n",
        "num_heads = 2\n",
        "\n",
        "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
        "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
        "x = TransformerDecoder(embed_dim, latent_dim, num_heads)(x, x)\n",
        "outputs = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
        "model = keras.Model(inputs, outputs)\n",
        "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"rmsprop\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXrXEm3ICrEQ"
      },
      "source": [
        "### 가변 온도 샘플링을 사용한 텍스트 생성 콜백"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zeARJmFICrEQ"
      },
      "source": [
        "**텍스트 생성 콜백**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "RtzXqzI1CrER"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "tokens_index = dict(enumerate(text_vectorization.get_vocabulary()))\n",
        "\n",
        "def sample_next(predictions, temperature=1.0):\n",
        "    predictions = np.asarray(predictions).astype(\"float64\")\n",
        "    predictions = np.log(predictions) / temperature\n",
        "    exp_preds = np.exp(predictions)\n",
        "    predictions = exp_preds / np.sum(exp_preds)\n",
        "    probas = np.random.multinomial(1, predictions, 1)\n",
        "    return np.argmax(probas)\n",
        "\n",
        "class TextGenerator(keras.callbacks.Callback):\n",
        "    def __init__(self,\n",
        "                 prompt,\n",
        "                 generate_length,\n",
        "                 model_input_length,\n",
        "                 temperatures=(1.,),\n",
        "                 print_freq=1):\n",
        "        self.prompt = prompt\n",
        "        self.generate_length = generate_length\n",
        "        self.model_input_length = model_input_length\n",
        "        self.temperatures = temperatures\n",
        "        self.print_freq = print_freq\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        if (epoch + 1) % self.print_freq != 0:\n",
        "            return\n",
        "        for temperature in self.temperatures:\n",
        "            print(\"== Generating with temperature\", temperature)\n",
        "            sentence = self.prompt\n",
        "            for i in range(self.generate_length):\n",
        "                tokenized_sentence = text_vectorization([sentence])\n",
        "                predictions = self.model(tokenized_sentence)\n",
        "                next_token = sample_next(predictions[0, i, :], temperature)\n",
        "                sampled_token = tokens_index[next_token]\n",
        "                sentence += \" \" + sampled_token\n",
        "            print(sentence)\n",
        "\n",
        "prompt = \"This movie\"\n",
        "text_gen_callback = TextGenerator(\n",
        "    prompt,\n",
        "    generate_length=50,\n",
        "    model_input_length=sequence_length,\n",
        "    temperatures=(0.2, 0.5, 0.7, 1., 1.5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IZIt8p-oCrER"
      },
      "source": [
        "**언어 모델 훈련하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "PeBx-DacCrER",
        "outputId": "3124d2da-82f0-4112-c5d4-51ed34fc60f9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.9244== Generating with temperature 0.2\n",
            "This movie movie is is a a lot movie of i the think worst that movie i i was was a a lot lot of of the the movie movie i was think a that lot i of was the not movie be i a am lot not of only the reason\n",
            "== Generating with temperature 0.5\n",
            "This movie movie was has very been long better time than i the dont whole understand time how i stupid watched i it am was sure [UNK] how and much watch more it than really that really it bad was but a i movie am was sure going i to have see\n",
            "== Generating with temperature 0.7\n",
            "This movie movie is was even an dragged interesting released documentary its because interesting i script thought is it funny has not no as longer if and it i whats love to it see was it too was bad a i hard have to been see going what through can the see\n",
            "== Generating with temperature 1.0\n",
            "This movie is was none filled id with always creative pick dancers i lyrics just different understood movies ever say seen that informative happened another in career everyone and in fulfill the i sneaking am and except cant they have are expected reading about that time the and schizophrenic the stages 1980s\n",
            "== Generating with temperature 1.5\n",
            "This movie looks has glad sincerely chen refer prophecy movies might struggling want the to dentist luckily thats yossi outcast honestly no cocktail wonder regarding idol chop will nonsense filming ludicrously gallery likable threw names sick inside american before willem wu harrelson the economy usage available visual criticizing decay shoot scares ensemble\n",
            "391/391 [==============================] - 171s 419ms/step - loss: 5.9244\n",
            "Epoch 2/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.4540== Generating with temperature 0.2\n",
            "This movie is is a a [UNK] great movie movie that that is is a the good movie movie is is a the good movie but is it the is acting the is movie not is a a good good and and the the movie movie is is a a good good\n",
            "== Generating with temperature 0.5\n",
            "This movie is is a a wonderful very movie good is movie a the good plot movie is but the it movie is is so the bad acting that is is not a a good very movie very is good bad but and this the movie movie is is that a the\n",
            "== Generating with temperature 0.7\n",
            "This movie is is about so to there be is a a lot good of and the the ending movie you that just more not than to that enjoy it the is film what is is the a movie sequel is to the be film told is me pretty it good is\n",
            "== Generating with temperature 1.0\n",
            "This movie movie stars hurts is died because again i it remember out it skillfully a star forgettable became as a images [UNK] in of an course insane of [UNK] rocks accent popcorn is and probably russell wouldnt stock very characters interesting that from time the ill story have and been the\n",
            "== Generating with temperature 1.5\n",
            "This movie and [UNK] talking springs mud together bumps very 10 scary addition wed skip party deceased oc for it the captures praising him temporarily all dad is murderer also vs pretty ugly tedious its movies not automobiles show colonel fangs at credit disappointingly instrument about for her briefly oxygen trying goes\n",
            "391/391 [==============================] - 168s 430ms/step - loss: 5.4540\n",
            "Epoch 3/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.3120== Generating with temperature 0.2\n",
            "This movie movie is is a a very great well movie done i and can i see have it ever is seen a it movie is i a have great to but see i it have is to a be great a i very think good it movie is i a think\n",
            "== Generating with temperature 0.5\n",
            "This movie movie is is not a a movie very but much it more is than a a [UNK] really the like movie a that [UNK] it movie is is a a movie movie with is a a lot good of guy the who movie plays was a fantastic lot but of\n",
            "== Generating with temperature 0.7\n",
            "This movie is is an the understatement worst that movie i is cant very get familiar into with the the acting plot was is great so but much i the dont story know just what a i 10 just or because so of i the know songs why the i story guess\n",
            "== Generating with temperature 1.0\n",
            "This movie movie is is never so anyone fresh in and a it good funny as it to was see not only very call funny it without just his thats own a the good joke thing but that i someone would shocked be to a feel movie like and and conventional it\n",
            "== Generating with temperature 1.5\n",
            "This movie day flick posted comedy war lacks consciousness heroic  and implausible dog how many filthy preferably humour and deliciously rick [UNK] realizing note nina mouse id wouldnt get [UNK] carries receiving happened really care marty cage begin khan distributed and as clive warn a confined wes cat does such angeles\n",
            "391/391 [==============================] - 168s 429ms/step - loss: 5.3120\n",
            "Epoch 4/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.2168== Generating with temperature 0.2\n",
            "This movie movie was i i was was i in was the not only to because watch i it was i so was i a was good not i a was movie very i disappointed was i so was i not was to very watch disappointed it i i was was so\n",
            "== Generating with temperature 0.5\n",
            "This movie movie was was probably i i first was but really i too really bad bad it i i was was so bad absolutely i no was i expecting was to so watch bad it it i and was i very was disappointed very i it was was because expecting i\n",
            "== Generating with temperature 0.7\n",
            "This movie is was so the funny worst it i ever can seen say and it i its was funny i i didnt dont really get would to be think awful i i have was ever looking seen for it it i the was fx as i bad can i see saw\n",
            "== Generating with temperature 1.0\n",
            "This movie movie is i made was i so was on lethal that but it i and met taped i it cant was understand good why it those isnt words funny what about it a i good have to missed be the told plot if whats you looking are for scarier 5\n",
            "== Generating with temperature 1.5\n",
            "This movie was favorite hilarious mst3k in sunday mr pass activities that pictures harrys fast locations daughters nothing march offer heck populated boredom distracting if coincidentally they wish alone yuen hewitt wagon holding brain talks your ignores choices crash born marlowe way would it be but disgusting no porky matter detectives how\n",
            "391/391 [==============================] - 168s 429ms/step - loss: 5.2168\n",
            "Epoch 5/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.1444== Generating with temperature 0.2\n",
            "This movie movie was i so really bad bad it i was didnt really really bad bad i i thought thought it it was was bad bad i i was was really bad bad i i dont dont think really i bad really i bad was i bad would i like really\n",
            "== Generating with temperature 0.5\n",
            "This movie movie was was terrible horrible i i can couldnt even believe watch that it i was saw so it bad i it would i say was that bad i i watched would it i i was thought just it dont i like thought it it was i i dont think\n",
            "== Generating with temperature 0.7\n",
            "This movie movie was not bad all i i didnt thought think it they was dont going really to predictable just but didnt i think guess i i liked ended it i was seemed bad great i i thought watched it it i i was felt really like bad i but can\n",
            "== Generating with temperature 1.0\n",
            "This movie movie wasnt i there went was to good i about remember i them saw how this bad it it hes but just i to would buy be though entertaining i i got would to seem have more to and read just any cant and be yet why little she [UNK]\n",
            "== Generating with temperature 1.5\n",
            "This movie if reads composition at than the a procedure dubbed studio continuous problems theme swearing raised in poker 1963 domination spoof oh involving numbing fighters man fetish lethargic joint running luckily up god wearing discovering drowning artists taking voice the roar tension cohorts horribly turns popular to entertainment the vet fine\n",
            "391/391 [==============================] - 168s 429ms/step - loss: 5.1444\n",
            "Epoch 6/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.0852== Generating with temperature 0.2\n",
            "This movie movie was i so really bad bad i it was and so i bad was i so was bad so it bad i it was i so was bad so it bad i it was and so i bad was i so was bad so it bad i it was\n",
            "== Generating with temperature 0.5\n",
            "This movie movie is so so bad bad it it was is horrible bad i i dont really even bad if it you i can have see to it watch i it dont is even so dont bad waste it of is your bad time it ive i ever dont and even\n",
            "== Generating with temperature 0.7\n",
            "This movie movie was i i just really so bad bad it it i really cant makes it it and and so i i cant have believe to they the should jokes this no movie i i would just really dont cant like really it watch i it dont and waste i\n",
            "== Generating with temperature 1.0\n",
            "This movie film is i horrible really but bad it that i i had as both much of i the was worst superb movies the i i love would the have whole ever title seen i the was characters truly are bad so with i people love if i it did as\n",
            "== Generating with temperature 1.5\n",
            "This movie isnt everything big just a another cute digital dog sheet it selection deaths by whats wig wrong conditions although help always much leads wonder being if pretty succeeding offended out alone otherwise but lightweight sam story im targets surprised dumb use heather sex flower dignity mansion fares this tshirt cockpit\n",
            "391/391 [==============================] - 167s 427ms/step - loss: 5.0852\n",
            "Epoch 7/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 5.0332== Generating with temperature 0.2\n",
            "This movie movie is is bad bad it acting is bad bad acting acting bad bad acting acting bad bad acting acting bad bad acting acting bad bad bad bad bad acting acting bad bad bad bad bad bad acting bad bad script bad bad bad bad bad bad bad acting bad\n",
            "== Generating with temperature 0.5\n",
            "This movie movie is was so so bad bad it its is good bad its it bad was its bad bad there it are really terrible bad the it acting was bad bad the this worst movie movie is is bad terrible the the acting worst bad movie the is acting bad\n",
            "== Generating with temperature 0.7\n",
            "This movie movie makes was you even enjoy for if all you its will just find want im to not make really a really movie want i a will movie give you you can that see if this you to if be you a can movie you if can you see do\n",
            "== Generating with temperature 1.0\n",
            "This movie movie wasnt was so art bad for it good dragged there around my  super horror movie was makes this one reason i dont waste a 2 and some time flash and whites the worst acting is nothing to do with nothing out of pornography steals my sister acting this\n",
            "== Generating with temperature 1.5\n",
            "This movie needed just me horrible analyze performed japanese resurrected horrors foulmouthed topless ive girls seen lowly viva 2 satans awkward sport french legion satirical institute movement eighty style highlights ruined film the theyre cheapest loaded group at from times sixty retribution bosses angles cancel biography scenes patriot from curtis pink bin\n",
            "391/391 [==============================] - 167s 427ms/step - loss: 5.0332\n",
            "Epoch 8/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 4.9830== Generating with temperature 0.2\n",
            "This movie movie was was bad a it good was the a acting good was the a acting good was the the movie movie the the movie movie the the movie movie the the movie movie the the movie movie the the movie movie the the movie movie the the movie movie\n",
            "== Generating with temperature 0.5\n",
            "This movie movie was the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n",
            "== Generating with temperature 0.7\n",
            "This movie movie is is a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 1.0\n",
            "This movie movie movie an a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 1.5\n",
            "This movie movie scared an a intriguing while teenagers summarize try a about suspicious method nobudget seeing dream texas upstairs there [UNK] lurking underground rufus men horses to missing presenting visitor uncompromising him impact perform has is whenever hard jolson angela sylvester cent stallone studio gloria turd race he conflicts shelter and\n",
            "391/391 [==============================] - 168s 429ms/step - loss: 4.9830\n",
            "Epoch 9/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 4.9330== Generating with temperature 0.2\n",
            "This movie movie is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is\n",
            "== Generating with temperature 0.5\n",
            "This movie movie for of about of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of of\n",
            "== Generating with temperature 0.7\n",
            "This movie movie is is is is not  it is not it is is not a and is its is is a is is is is is a is a is not a this is the it is an it is a is a is a very romantic it is a\n",
            "== Generating with temperature 1.0\n",
            "This movie movie bad a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 1.5\n",
            "This movie was being generally on horrible 2006 the one costumes long make bad verdict looked month embarrassing old accent admirer absurd of need 2 for hours 50 decent raptors some i maybe instead its source pretty cleared dull warriors psychedelic enhance creators aliens can could kind sum  graduation street exorcist\n",
            "391/391 [==============================] - 168s 428ms/step - loss: 4.9330\n",
            "Epoch 10/10\n",
            "391/391 [==============================] - ETA: 0s - loss: 4.8826== Generating with temperature 0.2\n",
            "This movie movie is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is\n",
            "== Generating with temperature 0.5\n",
            "This movie movie movie bad  a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 0.7\n",
            "This movie movie coming at out a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 1.0\n",
            "This movie movie made made a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a\n",
            "== Generating with temperature 1.5\n",
            "This movie movie 310 my title 3 the one night bad once dawn twist predict saying once writerdirector seem harbour sloppy straight radical artificial ok walken david isnt insects rather following anything several good slapped review meanspirited contains current havoc positive catwoman excuses backing seem buster clever is moments exciting hesitant jackie\n",
            "391/391 [==============================] - 168s 428ms/step - loss: 4.8826\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.callbacks.History at 0x7c15a0390370>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "# 코랩에서 정상 실행만 확인하기 위해 에포크 횟수를 200에서 10으로 줄입니다\n",
        "model.fit(lm_dataset, epochs=10,  # 200\n",
        "          callbacks=[text_gen_callback])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_V8gQCWCrER"
      },
      "source": [
        "### 정리"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "chapter12_part01_text-generation.i",
      "provenance": [],
      "gpuClass": "premium"
    },
    "kernelspec": {
      "display_name": "default:Python",
      "language": "python",
      "name": "conda-env-default-py"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.10"
    },
    "gpuClass": "premium"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}