allennlp_fundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a whirlwind tour of most of the fundamental concepts underlying AllenNLP. There is nothing for you to do here, if you like you could just Ctrl-Enter all the way to the bottom, but feel free to poke at the results as you go if you're curious about them.\n",
    "\n",
    "Many of these concepts you won't have to worry about that much, but it's good to sort of understand what's going on under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "The default tokenizer in AllenNLP is the spacy tokenizer. You can specify others if you need them. (For instance, if you're using BERT, you want to use the same tokenizer that the BERT model expects.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.tokenizers import WordTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"I don't hate notebooks, I just don't like notebooks!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = WordTokenizer()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.tokenize(text)\n",
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Token Indexers\n",
    "\n",
    "A `TokenIndexer` turns tokens into indices or lists of indices. We won't be able to see how they operate until slightly later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.token_indexers import SingleIdTokenIndexer, TokenCharactersIndexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_indexer = SingleIdTokenIndexer()  # maps tokens to word_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fields\n",
    "\n",
    "Your training examples will be represented as `Instances`, each consisting of typed `Field`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.fields import TextField, LabelField"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `TextField` is for storing text, and also needs one or more `TokenIndexer`s that will be used to convert the text into indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field = TextField(tokens, {\"tokens\": token_indexer})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field._indexed_tokens  # not yet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `LabelField` is for storing a discrete label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_field = LabelField(\"technology\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instances\n",
    "\n",
    "Each `Instance` is just a collection of named `Field`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.instance import Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance = Instance({\"text\": text_field, \"category\": label_field})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vocabulary\n",
    "\n",
    "Based on our instances we construct a `Vocabulary` which contains the various mappings token <-> index, label <-> index, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.vocabulary import Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Vocabulary.from_instances([instance])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that our vocabulary has two mappings, a `tokens` mapping (for the tokens) and a `labels` mapping (for the labels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab._token_to_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field._indexed_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_field._label_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we have constructed the mappings, we haven't yet used them to index the fields in our instance. We have to do that manually (although when you use the allennlp trainer all of this will be taken care of.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.index_fields(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field._indexed_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_field._label_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the `Instance` has been indexed, it then knows how to convert itself to a tensor dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.as_tensor_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it knows how long other instances would need to be padded to if we do batching. (More on this below!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.get_padding_lengths()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batching and Padding\n",
    "\n",
    "When you're doing NLP, you have sequences with different lengths, which means that padding and masking are very important. They're tricky to get right! Luckily, AllenNLP handles most of the details for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"I just don't like notebooks.\"\n",
    "tokens1 = tokenizer.tokenize(text)\n",
    "text_field1 = TextField(tokens1, {\"tokens\": token_indexer})\n",
    "label_field1 = LabelField(\"Joel\")\n",
    "instance1 = Instance({\"text\": text_field1, \"speaker\": label_field1})\n",
    "text2 = \"I do like notebooks.\"\n",
    "tokens2 = tokenizer.tokenize(text2)\n",
    "text_field2 = TextField(tokens2, {\"tokens\": token_indexer})\n",
    "label_field2 = LabelField(\"Tim\")\n",
    "instance2 = Instance({\"text\": text_field2, \"speaker\": label_field2})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.dataset import Batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Vocabulary.from_instances([instance1, instance2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = Batch([instance1, instance2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.index_instances(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that\n",
    "\n",
    "1. the batching is already taken care of for you, and\n",
    "2. the shorter text field is appropriately padded with 0's (the `@@PADDING@@` id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch.as_tensor_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Multiple Indexers\n",
    "\n",
    "In some circumstances you might want to use multiple token indexers. For instance, you might want to index a token using its token_id, but also as a sequence of character_ids. This is as simple as adding extra token indexers to our text fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.data.token_indexers import TokenCharactersIndexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_characters_indexer = TokenCharactersIndexer(min_padding_length=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field = TextField(tokens, {\"tokens\": token_indexer, \"token_characters\": token_characters_indexer})\n",
    "label_field = LabelField(\"technology\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance = Instance({\"text\": text_field, \"label\": label_field})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Vocabulary.from_instances([instance])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that we now have an additional vocabulary namespace for the character ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab._token_to_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.index_fields(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now when we call `instance.as_tensor_dict` we'll get an additional (padded) tensor with the character_ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.as_tensor_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TokenEmbedders\n",
    "\n",
    "Once we have our text represented as ids, we use a `TokenEmbedder` to create tensor embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"I just don't like notebooks.\"\n",
    "tokens1 = tokenizer.tokenize(text)\n",
    "text_field1 = TextField(tokens1, {\"tokens\": token_indexer})\n",
    "label_field1 = LabelField(\"Joel\")\n",
    "instance1 = Instance({\"text\": text_field1, \"speaker\": label_field1})\n",
    "text2 = \"I do like notebooks.\"\n",
    "tokens2 = tokenizer.tokenize(text2)\n",
    "text_field2 = TextField(tokens2, {\"tokens\": token_indexer})\n",
    "label_field2 = LabelField(\"Tim\")\n",
    "instance2 = Instance({\"text\": text_field2, \"speaker\": label_field2})\n",
    "vocab = Vocabulary.from_instances([instance1, instance2])\n",
    "batch = Batch([instance1, instance2])\n",
    "batch.index_instances(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor_dict = batch.as_tensor_dict()\n",
    "tensor_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.modules.token_embedders import Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we define an embedding layer that has a number of embeddings equal to the corresponding vocabulary size, and that consists of 5-dimensional vectors. In this case the embeddings will just be randomly initialized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding = Embedding(num_embeddings=vocab.get_vocab_size(\"tokens\"), embedding_dim=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accordingly, we can apply those embeddings to the indexed tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding(tensor_dict['text']['tokens'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TextFieldEmbedders\n",
    "\n",
    "A text field may have multiple indexed representations of its tokens, in which case it needs multiple corresponding `TokenEmbedder`s. Because of this we typically wrap the token embedders in a `TextFieldEmbedder`, which runs the appropriate token embedder for each representation and then concatenates the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field_embedder = BasicTextFieldEmbedder({\"tokens\": embedding})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice now we apply it to the full tensor dict for the text field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_field_embedder(tensor_dict['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seq2VecEncoders\n",
    "\n",
    "At this point we've ended up with a sequence of tensors. Frequently we'll want to collapse that sequence into a single contextualized tensor representation, which we do with a `Seq2VecEncoder`. (If we wanted to produce a full sequence of contextualized representations we'd instead use a `Seq2SeqEncoder`.\n",
    "\n",
    "In particular, here we'll use a `BagOfEmbeddingsEncoder`, which just sums up the vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.modules.seq2vec_encoders import BagOfEmbeddingsEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = BagOfEmbeddingsEncoder(embedding_dim=text_field_embedder.get_output_dim())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can apply this to the output of our text field embedder to collapse each sequence down to a single element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder(text_field_embedder(tensor_dict['text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using PyTorch directly\n",
    "\n",
    "AllenNLP modules are just PyTorch modules, and we can mix and match them with native PyTorch features. Here we create a `torch.nn.Linear` module and apply it to the output of the `Seq2VecEncoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear = torch.nn.Linear(in_features=text_field_embedder.get_output_dim(), out_features=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear(encoder(text_field_embedder(tensor_dict['text'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We typically encapsulate most of these steps into an allennlp Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from allennlp.models import Model\n",
    "from allennlp.modules.text_field_embedders import TextFieldEmbedder\n",
    "from allennlp.modules.seq2vec_encoders import Seq2VecEncoder\n",
    "from typing import Dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a model that accepts the output of `batch.as_tensor_dict`, and applies the text field embedder, the seq2vec encoder, and linear layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(Model):\n",
    "    def __init__(self, \n",
    "                 vocab: Vocabulary, \n",
    "                 embedder: TextFieldEmbedder, \n",
    "                 encoder: Seq2VecEncoder, \n",
    "                 output_dim: int) -> None:\n",
    "        super().__init__(vocab)\n",
    "        self.embedder = embedder\n",
    "        self.encoder = encoder\n",
    "        self.linear = torch.nn.Linear(in_features=embedder.get_output_dim(), out_features=output_dim)\n",
    "        \n",
    "    def forward(self, text: Dict[str, torch.Tensor], speaker: torch.Tensor) -> Dict[str, torch.Tensor]:\n",
    "        \"\"\"\n",
    "        Notice how the argument names correspond the field names in our instance.\n",
    "        \"\"\"\n",
    "        embedded = self.embedder(text)\n",
    "        encoded = self.encoder(embedded)\n",
    "        output = self.linear(encoded)\n",
    "        \n",
    "        return {\"output\": output}\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MyModel(vocab, text_field_embedder, encoder, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model(**tensor_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}