t81_558_class_11_02_tokenizers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3CF2edFAI4Uj"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_tokenizers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RvbM-RwHI4Ul"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing with Hugging Face**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5dE7A-0aI4Ul"
   },
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Introduction to Hugging Face [[Video]](https://www.youtube.com/watch?v=1IHXSbz02XM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_01_huggingface.ipynb)\n",
    "* **Part 11.2: Hugging Face Tokenizers** [[Video]](https://www.youtube.com/watch?v=U-EGU1RyChg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_tokenizers.ipynb)\n",
    "* Part 11.3: Hugging Face Datasets [[Video]](https://www.youtube.com/watch?v=Mq5ODegT17M&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_03_hf_datasets.ipynb)\n",
    "* Part 11.4: Training Hugging Face Models [[Video]](https://www.youtube.com/watch?v=https://www.youtube.com/watch?v=l69ov6b7DOM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_04_hf_train.ipynb)\n",
    "* Part 11.5: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_05_embedding.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Z4A091yI4Um"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RJmzbge9I4Um",
    "outputId": "4685a76c-cc37-490a-d589-db16eb6e9c4c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vdNN6e45I4Un"
   },
   "source": [
    "# Part 11.2: Hugging Face Tokenizers\n",
    "\n",
    "Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. Consider how the program might break up the following sentences into words.\n",
    "\n",
    "* This is a test.\n",
    "* Ok, but what about this?\n",
    "* Is U.S.A. the same as USA.?\n",
    "* What is the best data-set to use?\n",
    "* I think I will do this-no wait; I will do that.\n",
    "\n",
    "The hugging face includes tokenizers that can break these sentences into words and subwords. Because English, and some other languages, are made up of common word parts, we tokenize subwords. For example, a gerund word, such as \"sleeping,\" will be tokenized into \"sleep\" and \"##ing\".\n",
    "\n",
    "We begin by installing Hugging Face if needed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3jGET2abMjcl",
    "outputId": "35b8be0f-bc7e-42d0-f177-ee097d4a076c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in /usr/local/lib/python3.7/dist-packages (4.17.0)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.6.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (6.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n",
      "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.5)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.63.0)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.11.6)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.4.0)\n",
      "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.49)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.11.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.7.0)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.10.8)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.1.0)\n",
      "Requirement already satisfied: transformers[sentencepiece] in /usr/local/lib/python3.7/dist-packages (4.17.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (4.63.0)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (2.23.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (0.4.0)\n",
      "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (0.0.49)\n",
      "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (1.21.5)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (21.3)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (4.11.3)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (6.0)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (3.6.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (2019.12.20)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (0.11.6)\n",
      "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (0.1.96)\n",
      "Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from transformers[sentencepiece]) (3.17.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers[sentencepiece]) (3.10.0.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers[sentencepiece]) (3.0.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers[sentencepiece]) (3.7.0)\n",
      "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.7/dist-packages (from protobuf->transformers[sentencepiece]) (1.15.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers[sentencepiece]) (2.10)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers[sentencepiece]) (2021.10.8)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers[sentencepiece]) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers[sentencepiece]) (3.0.4)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers[sentencepiece]) (1.1.0)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers[sentencepiece]) (7.1.2)\n"
     ]
    }
   ],
   "source": [
    "# HIDE OUTPUT\n",
    "!pip install transformers\n",
    "!pip install transformers[sentencepiece]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wa1ncodn8y0r"
   },
   "source": [
    "First, we create a Hugging Face tokenizer. There are several different tokenizers available from the Hugging Face hub. For this example, we will make use of the following tokenizer:\n",
    "\n",
    "* distilbert-base-uncased\n",
    "\n",
    "This tokenizer is based on BERT and assumes case-insensitive English text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kSGtW0E7xcK9"
   },
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "model = \"distilbert-base-uncased\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YofjyJw59U2x"
   },
   "source": [
    "We can now tokenize a sample sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NVCmyao2zLQ3",
    "outputId": "6b33031a-8667-4431-aae2-26595195d69d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [101, 19204, 6026, 3793, 2003, 3733, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "encoded = tokenizer('Tokenizing text is easy.')\n",
    "print(encoded)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4QgmlpyezhMy"
   },
   "source": [
    "The result of this tokenization contains two elements:\n",
    "* input_ids - The individual subword indexes, each index uniquely identifies a subword.\n",
    "* attention_mask - Which values in *input_ids* are meaningful and not  padding.\n",
    "This sentence had no padding, so all elements have an attention mask of \"1\". Later, we will request the output to be of a fixed length, introducing padding, which always has an attention mask of \"0\". Though each tokenizer can be implemented differently, the attention mask of a tokenizer is generally either \"0\" or \"1\". \n",
    "\n",
    "Due to subwords and special tokens, the number of tokens may not match the number of words in the source string. We can see the meanings of the individual tokens by converting these IDs back to strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ww3XPc-i2Y6c",
    "outputId": "6179b416-61cd-4142-fa15-ae2296e9eab7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]', 'token', '##izing', 'text', 'is', 'easy', '.', '[SEP]']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens(encoded.input_ids)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7KM7VRJECoGU"
   },
   "source": [
    "As you can see, there are two special tokens placed at the beginning and end of each sequence. We will soon see how we can include or exclude these special tokens. These special tokens can vary per tokenizer; however, [CLS] begins a sequence for this tokenizer, and [SEP] ends a sequence. You will also see that the gerund \"tokening\" is broken into \"token\" and \"*ing\".\n",
    "\n",
    "For this tokenizer, the special tokens occur between 100 and 103. Most Hugging Face tokenizers use this approximate range for special tokens. The value zero (0) typically represents padding. We can display all special tokens with this command.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EtQiOmSl2rXt",
    "outputId": "6f5f2aef-d718-4675-d0a5-9c6c9d960c87"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[PAD]', '[UNK]', '[CLS]', '[SEP]', '[MASK]']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens([0, 100, 101, 102, 103])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1nQ-r6bz3ESN"
   },
   "source": [
    "This tokenizer supports these common tokens:\n",
    "\n",
    "* \\[CLS\\] - Sequence beginning.\n",
    "* \\[SEP\\] - Sequence end.\n",
    "* \\[PAD\\] - Padding.\n",
    "* \\[UNK\\] - Unknown token.\n",
    "* \\[MASK\\] - Mask out tokens for a neural network to predict. Not used in this book, see [MLM paper](https://arxiv.org/abs/2109.01819). \n",
    "\n",
    "It is also possible to tokenize lists of sequences. We can pad and truncate sequences to achieve a standard length by tokenizing many sequences at once.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TI4RZXhc4v9k",
    "outputId": "2ae7f2f9-3d8a-421e-c9b9-4dd933817cd3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Input IDs**\n",
      "[101, 2023, 3185, 2001, 2307, 999, 102, 0, 0, 0, 0]\n",
      "[101, 1045, 6283, 2023, 2693, 1010, 5949, 1997, 2051, 999, 102]\n",
      "[101, 8680, 1029, 102, 0, 0, 0, 0, 0, 0, 0]\n",
      "**Attention Mask**\n",
      "[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]\n",
      "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "text = [\n",
    "    \"This movie was great!\",\n",
    "    \"I hated this move, waste of time!\",\n",
    "    \"Epic?\"\n",
    "]\n",
    "\n",
    "encoded = tokenizer(text, padding=True, add_special_tokens=True)\n",
    "\n",
    "print(\"**Input IDs**\")\n",
    "for a in encoded.input_ids:\n",
    "    print(a)\n",
    "\n",
    "print(\"**Attention Mask**\")\n",
    "for a in encoded.attention_mask:\n",
    "    print(a)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1rIF8TEEF9C-"
   },
   "source": [
    "Notice the **input_id**'s for the three movie review text sequences. Each of these sequences begins with 101 and we pad with zeros. Just before the padding, each group of IDs ends with 102. The attention masks also have zeros for each of the padding entries. \n",
    "\n",
    "We used two parameters to the tokenizer to control the tokenization process. Some other useful [parameters](https://huggingface.co/docs/transformers/main_classes/tokenizer) include:\n",
    "\n",
    "* add_special_tokens (defaults to True) Whether or not to encode the sequences with the special tokens relative to their model.\n",
    "* padding (defaults to False) Activates and controls truncation.\n",
    "* max_length (optional) Controls the maximum length to use by one of the truncation/padding parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "t81_558_class_11_02_tokenizers.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}