TextGeneration.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jansoe/AICA/blob/main/TextGeneration.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dB45UYZTUqe"
      },
      "source": [
        "# Transformers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Px2ChByN_XOI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8922ae6-1fae-4def-b036-d36d6ea2351b"
      },
      "source": [
        "!pip install transformers -q"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m46.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m57.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.5/224.5 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import transformers\n",
        "import torch\n",
        "import textwrap \n",
        "\n",
        "import plotly.express as px"
      ],
      "metadata": {
        "id": "y-epkjNq3FJM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mHY90e0aQ6ag"
      },
      "source": [
        "## Input : Tokens"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "83mPXhYNRlTY"
      },
      "source": [
        "model_version = 'EleutherAI/gpt-neo-125m'\n",
        "#model_version = 'dbmdz/german-gpt2'\n",
        "tokenizer = transformers.AutoTokenizer.from_pretrained(model_version)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.tokenize('This is a test of the tokenizer')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OYo84qc9F4Ks",
        "outputId": "3bca795b-ccf9-4c68-c120-084ba754e59e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['This', 'Ġis', 'Ġa', 'Ġtest', 'Ġof', 'Ġthe', 'Ġtoken', 'izer']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9rzPyvlEKScs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b01a9a5b-857a-4acf-e918-7501926e8ef3"
      },
      "source": [
        "tokens = tokenizer('This is a test of the tokenizer')['input_ids']\n",
        "tokens"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1212, 318, 257, 1332, 286, 262, 11241, 7509]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "[tokenizer.convert_ids_to_tokens(t) for t in tokens]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gg_KaTYzFvgq",
        "outputId": "c795b17a-611c-46d6-9a8d-38881d5e3846"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['This', 'Ġis', 'Ġa', 'Ġtest', 'Ġof', 'Ġthe', 'Ġtoken', 'izer']"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(tokens)"
      ],
      "metadata": {
        "id": "r68IvDN5v7ck",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b41776e5-9db9-47c0-e327-89fd35bf2c7e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'This is a test of the tokenizer'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9YhAxm_Axks",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c35132d6-0500-49dc-f02c-997ec5019be5"
      },
      "source": [
        "tokenizer.tokenize('n0thing is corect')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['n', '0', 'thing', 'Ġis', 'Ġcore', 'ct']"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.vocab"
      ],
      "metadata": {
        "id": "FBLbqQnCvVsC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jMBijK00WtWn"
      },
      "source": [
        "### Test of different tokenizer\n",
        "\n",
        "Mögliche Modelle: \n",
        "* https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads\n",
        "* https://huggingface.co/models?pipeline_tag=text-classification&sort=downloads"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "94Nw_Sw6WxSl"
      },
      "source": [
        "model_name = 'bert-base-uncased' #\n",
        "\n",
        "tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.tokenize('!!! Test sentence !!!') "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Hfj8-BT8rFF",
        "outputId": "db7d1bc5-5206-4294-d128-6b28d1d300fd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['!', '!', '!', 'test', 'sentence', '!', '!', '!']"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Output"
      ],
      "metadata": {
        "id": "AasR8bXHT2ml"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_version = 'EleutherAI/gpt-neo-125m'\n",
        "\n",
        "tokenizer = transformers.AutoTokenizer.from_pretrained(model_version)\n",
        "model = transformers.AutoModelForCausalLM.from_pretrained(model_version)\n",
        "#_ = model.to('cuda') # Modell auf GPU schieben"
      ],
      "metadata": {
        "id": "WcdnAL-e1UGY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = \"The sun is\"\n",
        "#prompt = \"Is Peter a male or female name. Answer: male.\"\n",
        "\n",
        "\n",
        "ids = tokenizer(prompt, return_tensors='pt')['input_ids']\n",
        "ids"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nb134yWX2jzU",
        "outputId": "7502ae47-afde-4b02-fbe2-0754e78605e5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 464, 4252,  318]])"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with torch.no_grad():\n",
        "    out = model.forward(ids)"
      ],
      "metadata": {
        "id": "xv15u52I2ZUA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "probabilities = torch.softmax(out.logits, 2).numpy().squeeze()"
      ],
      "metadata": {
        "id": "yQ6egVa83J7k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "top10 = probabilities[-1].argsort()[:-11:-1]\n",
        "top10"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IIO-_26_3d6T",
        "outputId": "e15a8373-7bff-464a-b317-5af920ca1300"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([22751,  7396,   262,   257,   503,   287,  4634,  1464,   319,\n",
              "         991])"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "probabilities[-1][top10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SXYd5d8M7yxx",
        "outputId": "befd405f-93ea-40a0-e2a2-da7ad476ea91"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.13434376, 0.07112848, 0.02940449, 0.02640192, 0.02544004,\n",
              "       0.02446919, 0.02341055, 0.02017728, 0.02001574, 0.01899583],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "decoded_most_probable = [tokenizer.decode(i) for i in top10]\n",
        "\n",
        "px.bar(\n",
        "    x = decoded_most_probable, \n",
        "    y = probabilities[-1][top10], \n",
        "    title = 'Next token probability'\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "tbbmggss8YKx",
        "outputId": "9e3e7b67-6395-4bfe-bd83-210d7368edfb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.18.2.min.js\"></script>                <div id=\"769c3aa0-2ff4-4342-82b8-1171ef79c7ab\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"769c3aa0-2ff4-4342-82b8-1171ef79c7ab\")) {                    Plotly.newPlot(                        \"769c3aa0-2ff4-4342-82b8-1171ef79c7ab\",                        [{\"alignmentgroup\":\"True\",\"hovertemplate\":\"x=%{x}<br>y=%{y}<extra></extra>\",\"legendgroup\":\"\",\"marker\":{\"color\":\"#636efa\",\"pattern\":{\"shape\":\"\"}},\"name\":\"\",\"offsetgroup\":\"\",\"orientation\":\"v\",\"showlegend\":false,\"textposition\":\"auto\",\"x\":[\" shining\",\" rising\",\" the\",\" a\",\" out\",\" in\",\" setting\",\" always\",\" on\",\" still\"],\"xaxis\":\"x\",\"y\":[0.13434375822544098,0.0711284801363945,0.02940448559820652,0.02640192210674286,0.02544003538787365,0.02446918748319149,0.02341054566204548,0.020177284255623817,0.020015738904476166,0.018995828926563263],\"yaxis\":\"y\",\"type\":\"bar\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"x\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"y\"}},\"legend\":{\"tracegroupgap\":0},\"title\":{\"text\":\"Next token probability\"},\"barmode\":\"relative\"},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('769c3aa0-2ff4-4342-82b8-1171ef79c7ab');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cbBk0kwTR6pv"
      },
      "source": [
        "Der Textgenerierung liegt zugrunde, dass das Modell in jedem Schritt für jedes Token im Vokabular eine Wahrscheinlichkeit angibt, mit der das Modell das jeweilige Token für die nächste Position vorhersagt. Nun können wir 'greedy' einfach immer das wahrscheinlichste Token auswählen und zur nächsten Position schreiten. Aber dies führt oft zu durchwachsenen Resultaten (siehe https://arxiv.org/abs/1904.09751).\n",
        "\n",
        "Für eine verbesserte Qualität der erzeugten Texte bieten sich verschiedene Einstellungsmöglichkeiten an:\n",
        "- `repetition_penalty`: Tokens, die bereits generiert wurden, werden um diesen Faktor unwahrscheinlicher\n",
        "- `do_sample`: Es wird nicht das wahrscheinlichste Token genommen sondern anhand der vorhergesagten Wahrscheinlichkeitsverteilung ein Token zufällig ausgewählt\n",
        "  - `temperature`: Je höher die Temperatur, desto höher die Chance, dass auch unwahrscheinliche Tokens ausgewählt werden\n",
        "  - `top_`: Einschränkungen für die zur Verfügung stehenden Tokens:\n",
        "    - `top_k`: es werden nur die wahrscheinlichsten k Tokens zugelassen\n",
        "    - `top_p`: von den wahrscheinlichsten Tokens werden so viele zugelassen, dass deren Gesamtwahrscheinlichkeit `p` nicht überschreitet.\n",
        "\n",
        "Ausführlichere Informationen finden Sie auch [hier](https://towardsdatascience.com/how-to-sample-from-language-models-682bceb97277)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5VRN_y7zFGBT"
      },
      "source": [
        "prompt = 'The sun is'\n",
        "torch.manual_seed(10) # the seed controls the randomness\n",
        "\n",
        "output_sequences = model.generate(\n",
        "    **tokenizer(prompt, return_tensors='pt'),\n",
        "    max_length = 128, # Wieviele Token sollen maximal generiert werden\n",
        "    repetition_penalty = None,\n",
        "    do_sample = True,\n",
        "    penalty_alpha = 0.9,\n",
        "    temperature = 0.7,\n",
        "    top_k = None, # set to None to ignore or 0 < k <= vocab_size\n",
        "    top_p = None, # set to None to ignore or 0 < p <= 1\n",
        "    pad_token_id = tokenizer.eos_token_id\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "output_sequences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "slUNXXRiAqga",
        "outputId": "77cab1bb-265c-4a9a-9ec9-03a798d37d1c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[  464,  4252,   318,   287,   262,  6766,     0,   198,   198,    40,\n",
              "           550,   655,  5201,  3555,   257,  1492,   546,   262,   995,   447,\n",
              "           247,    82,   717,  6591, 25872,    13,   632,   373,   281,  1593,\n",
              "          2589,   329,   502,    11,   475,   314,   550,   284,  2245,   329,\n",
              "           257,  2589,   290,  2074,   340,   257,  2589,   286,   616,  1204,\n",
              "            13,  3244,   314,  7452,   262,  4252,    13,   198,   198,    40,\n",
              "           550,   655,  5201,  3555,   262,   717,  6843,   286,   616,  1492,\n",
              "           290,   314,   550,   281,  4998,   640,    13,   314,   373,   523,\n",
              "          7867,   416,   340,    13,   314,  1807,   340,   373,  1223,   284,\n",
              "           466,   351,   262,  4252,  7396,    11,  1223,   284,  2107,   351,\n",
              "            11,  1223,   284,   466,   351,   262,  4252,   447,   247,    82,\n",
              "          3386,    13,  1114,   502,    11,   428,   373,   262,   717,   640,\n",
              "           314,   550,  1775,   262,  4252,  4485,    13,   314]])"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "decoded = tokenizer.decode(output_sequences[0])\n",
        "print(textwrap.fill(decoded, width = 80, replace_whitespace=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VDuLZvUl9si9",
        "outputId": "b305461f-b5c6-4bfe-b7e3-9f4a63008ad9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The sun is in the sky!\n",
            "\n",
            "I had just finished reading a book about the world’s\n",
            "first solar eclipse. It was an important moment for me, but I had to stop for a\n",
            "moment and consider it a moment of my life. Then I faced the sun.\n",
            "\n",
            "I had just\n",
            "finished reading the first chapter of my book and I had an amazing time. I was\n",
            "so inspired by it. I thought it was something to do with the sun rising,\n",
            "something to live with, something to do with the sun’s forces. For me, this was\n",
            "the first time I had seen the sun rise. I\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whgyeP4Lfi2I"
      },
      "source": [
        "### Aufgabe: Exploration der Parameter\n",
        "\n",
        "Experimentieren Sie mit GPT-2 und den Modellfähigkeiten zur Textgenerierung. Wie können Sie die überzeugensten Texte generieren? Halten Sie Ihre Beobachtungen in geeigneter Weise hier im Notebook fest.\n",
        "\n",
        "- Was beobachten Sie, wenn Sie `repetition_penalty = None` und `do_sample = False` setzten?\n",
        "- Was passiert mit `do_sample = True` und verschiedenen Werten von `temperature` and `top_p`?\n",
        "- Was passiert, wenn Sie mit `prompt_text = '<|endoftext|>'` startet?"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "BlqAZpP7AztE"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}