AlBERTo_End_to_End_(Fine_tuning_+_Predicting)_with_Cloud_TPU_Sentence_Classification_Tasks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AlBERTo End to End (Fine-tuning + Predicting) with Cloud TPU: Sentence Classification Tasks",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "wHQH4OCHZ9bq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Original code licensed by:\n",
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkTLZ3I4_7c_",
        "colab_type": "text"
      },
      "source": [
        "# AlBERTo End to End (Fine-tuning + Predicting) with Cloud TPU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wtjs1QDb3DX",
        "colab_type": "text"
      },
      "source": [
        "## Overview\n",
        "\n",
        "**BERT**, or **B**idirectional **E**mbedding **R**epresentations from **T**ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.\n",
        "\n",
        "In particular we use this Notebook for fine-tuning **AlBERTo**, the first italian undertanding language model for Twitter Language.\n",
        "\n",
        "**Note:**  You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud \n",
        "Storage) bucket for this Colab to run.\n",
        "\n",
        "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) for how to create GCP account and GCS bucket. \n",
        "\n",
        "You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ld-JXlueIuPH",
        "colab_type": "text"
      },
      "source": [
        "## Instructions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POkof5uHaQ_c",
        "colab_type": "text"
      },
      "source": [
        "<h3><a href=\"https://cloud.google.com/tpu/\"><img valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"></a>  &nbsp;&nbsp;Train on TPU</h3>\n",
        "\n",
        "   1. Create a Cloud Storage bucket for your TensorBoard logs at http://console.cloud.google.com/storage and fill in the BUCKET parameter in the \"Parameters\" section below.\n",
        " \n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdMmwCJFaT8F",
        "colab_type": "text"
      },
      "source": [
        "### Set up your TPU environment\n",
        "\n",
        "In this section, you perform the following tasks:\n",
        "\n",
        "*   Set up a Colab TPU running environment\n",
        "*   Verify that you are connected to a TPU device\n",
        "*   Upload your credentials to TPU to access your GCS bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "191zq3ZErihP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUBP35oCDmbF",
        "colab_type": "text"
      },
      "source": [
        "### Prepare and import BERT modules\n",
        "​\n",
        "With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wzwke0sxS6W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "\n",
        "!test -d bert_repo || git clone https://github.com/google-research/bert bert_repo\n",
        "if not 'bert_repo' in sys.path:\n",
        "  sys.path += ['bert_repo']\n",
        "\n",
        "# import python modules defined by BERT\n",
        "from run_classifier import *\n",
        "import modeling\n",
        "import optimization\n",
        "import tokenization"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRu1aKO1D7-Z",
        "colab_type": "text"
      },
      "source": [
        "### Prepare for training\n",
        "\n",
        "This next section of code performs the following tasks:\n",
        "\n",
        "*  Specify task and download training data.\n",
        "*  Specify ALBERTO pretrained model\n",
        "*  Specify GS bucket, create output directory for model checkpoints and eval results.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tYkaAlJNfhul",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "TASK = 'SENTIPOLC_TASK3' #@param {type:\"string\"}\n",
        "\n",
        "TASK_DATA_DIR = './'\n",
        "\n",
        "BUCKET = 'alberto_files' #@param {type:\"string\"}\n",
        "assert BUCKET, 'Must specify an existing GCS bucket name'\n",
        "\n",
        "\n",
        "OUTPUT_DIR = 'gs://{}/{}/models/'.format(BUCKET, TASK)\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n",
        "\n",
        "#CONFIGURE AlBERTo MODEL\n",
        "BERT_CONFIG_FILE = 'gs://alberto_files/alberto_model/bert_config.json' #@param {type:\"string\"}\n",
        "VOCAB_FILE = 'gs://alberto_files/alberto_model/vocabulary_lower_case_128.txt' #@param {type:\"string\"}\n",
        "INIT_CHECKPOINT = 'gs://alberto_files/alberto_model/model.ckpt-1000000'#@param {type:\"string\"}\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oqQ_LHyzcoYW",
        "colab_type": "text"
      },
      "source": [
        "Also we initilize our hyperprams, prepare the training data and initialize TPU config."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pYVYULZiKvUi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#SET THE PARAMETERS\n",
        "TRAIN_BATCH_SIZE = 512\n",
        "PREDICT_BATCH_SIZE = 512\n",
        "EVAL_BATCH_SIZE = 512\n",
        "LEARNING_RATE = 2e-5\n",
        "NUM_TRAIN_EPOCHS = 10.0\n",
        "MAX_SEQ_LENGTH = 128\n",
        "# Warmup is a period of time where hte learning rate \n",
        "# is small and gradually increases--usually helps training.\n",
        "WARMUP_PROPORTION = 0.1\n",
        "# Model configs\n",
        "SAVE_CHECKPOINTS_STEPS = 1000\n",
        "SAVE_SUMMARY_STEPS = 500\n",
        "\n",
        "# Setup TPU related config\n",
        "tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "NUM_TPU_CORES = 8\n",
        "ITERATIONS_PER_LOOP = 1000\n",
        "\n",
        "def get_run_config(output_dir):\n",
        "  return tf.contrib.tpu.RunConfig(\n",
        "    cluster=tpu_cluster_resolver,\n",
        "    model_dir=output_dir,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,\n",
        "    tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "        iterations_per_loop=ITERATIONS_PER_LOOP,\n",
        "        num_shards=NUM_TPU_CORES,\n",
        "        per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gzfl2-tcQUc4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#PREPARE TRAINING SENTENCES\n",
        "!pip install ekphrasis\n",
        "!pip install pandas\n",
        "!pip install numpy\n",
        "\n",
        "from ekphrasis.classes.preprocessor import TextPreProcessor\n",
        "from ekphrasis.classes.tokenizer import SocialTokenizer\n",
        "from ekphrasis.dicts.emoticons import emoticons\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "text_processor = TextPreProcessor (\n",
        "    # terms that will be normalized\n",
        "    normalize=[ 'url' , 'email', 'user', 'percent', 'money', 'phone', 'time', 'date', 'number'] ,\n",
        "    # terms that will be annotated\n",
        "    annotate={\"hashtag\"} ,\n",
        "    fix_html=True ,  # fix HTML tokens\n",
        "\n",
        "    unpack_hashtags=True ,  # perform word segmentation on hashtags\n",
        "\n",
        "    # select a tokenizer. You can use SocialTokenizer, or pass your own\n",
        "    # the tokenizer, should take as input a string and return a list of tokens\n",
        "    tokenizer=SocialTokenizer(lowercase=True).tokenize,\n",
        "    dicts = [ emoticons ]\n",
        ")\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xBoMdhmX3NwH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#SETUP TRAINING AND TEST DATA\n",
        "training_data = pd.read_csv(TASK_DATA_DIR+'alberto_sentipolc16_task1_trainig.tsv',delimiter=\"\\t\", header=None, names = [\"id\",\"class\",\"star\",\"message\"], quoting=csv.QUOTE_NONE, error_bad_lines=False)\n",
        "sentences = training_data.iloc[:,3]\n",
        "labels = training_data.iloc[:,1]\n",
        "\n",
        "#final examples training\n",
        "examples = []\n",
        "\n",
        "import re\n",
        "i = 0\n",
        "for s in sentences:\n",
        "    s = s.lower()\n",
        "    s = str(\" \".join(text_processor.pre_process_doc(s)))\n",
        "    s = re.sub(r\"[^a-zA-ZÀ-ú</>!?♥♡\\s\\U00010000-\\U0010ffff]\", ' ', s)\n",
        "    s = re.sub(r\"\\s+\", ' ', s)\n",
        "    s = re.sub(r'(\\w)\\1{2,}',r'\\1\\1', s)\n",
        "    s = re.sub ( r'^\\s' , '' , s )\n",
        "    s = re.sub ( r'\\s$' , '' , s )\n",
        "    #print(s)\n",
        "    examples.append([labels[i],s])\n",
        "    i = i+1\n",
        "\n",
        "\n",
        "examples = np.array(examples)\n",
        "\n",
        "#final examples test\n",
        "test_data = pd.read_csv(TASK_DATA_DIR+'alberto_sentipolc16_task3_test.tsv',delimiter=\"\\t\", names = [\"id\",\"message\"], header=None, quoting=csv.QUOTE_NONE, error_bad_lines=False)\n",
        "print(test_data.shape)\n",
        "sentences_test = test_data.iloc[:,1]\n",
        "test_ids = test_data.iloc[:,0]\n",
        "sentences = test_data.iloc[:,1]\n",
        "\n",
        "examples_test = []\n",
        "\n",
        "\n",
        "i = 0\n",
        "for s in sentences:\n",
        "    s = s.lower()\n",
        "    s = str(\" \".join(text_processor.pre_process_doc(s)))\n",
        "    s = re.sub(r\"[^a-zA-ZÀ-ú</>!?♥♡\\s\\U00010000-\\U0010ffff]\", ' ', s)\n",
        "    s = re.sub(r\"\\s+\", ' ', s)\n",
        "    s = re.sub(r'(\\w)\\1{2,}',r'\\1\\1', s)\n",
        "    s = re.sub ( r'^\\s' , '' , s )\n",
        "    s = re.sub ( r'\\s$' , '' , s )\n",
        "    #print(s)\n",
        "    examples_test.append([labels[i],s])\n",
        "    i = i+1\n",
        "\n",
        "examples_test = np.array(examples_test)\n",
        "\n",
        "'''\n",
        "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create InputExample's using the constructor provided in the BERT library.\n",
        "\n",
        "    text_a is the text we want to classify, which in this case, is the Request field in our Dataframe.\n",
        "    text_b is used if we're training a model to understand the relationship between sentences (i.e. is text_b a translation of text_a? Is text_b an answer to the question asked by text_a?). This doesn't apply to our task, so we can leave text_b blank.\n",
        "    label is the label for our example, i.e. True, False\n",
        "\n",
        "'''\n",
        "\n",
        "f = lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
        "                                                                   text_a = x[1], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = int(x[0]))\n",
        "\n",
        "f2 = lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
        "                                                                   text_a = x[1], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = 0)\n",
        "\n",
        "train_examples = map(f,examples)\n",
        "train_examples = list(train_examples)\n",
        "\n",
        "test_examples = map(f2,examples_test)\n",
        "test_examples = list(test_examples)\n",
        "\n",
        "train_examples = np.array(train_examples)\n",
        "test_examples = np.array(test_examples)\n",
        "\n",
        "print(test_examples.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u5S1llsEd0b0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Labels used for annotating sentences\n",
        "label_list = [0, 1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8snKD00ZOVNK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Test data just created\n",
        "for r in test_examples:\n",
        "  print(r.text_a)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9RtAVJHs5vXv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#inizialize parameters\n",
        "num_train_steps = int(len(train_examples) / TRAIN_BATCH_SIZE * NUM_TRAIN_EPOCHS)+1\n",
        "num_warmup_steps = int(NUM_TRAIN_EPOCHS * WARMUP_PROPORTION)\n",
        "print(num_train_steps)\n",
        "print(num_warmup_steps)\n",
        "\n",
        "#Inizialize the tokenizer\n",
        "tokenizer = tokenization.FullTokenizer(VOCAB_FILE, do_lower_case=True)\n",
        "tokenizer.tokenize(\"Che bella giornata oggi! Stiamo procedendo bene o no? <url>\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3iFMeqLaSll",
        "colab_type": "text"
      },
      "source": [
        "# Fine-tune and Run Predictions on a pretrained BERT Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXyJRc0OIHEU",
        "colab_type": "text"
      },
      "source": [
        "This section demonstrates fine-tuning from a pre-trained BERT TF Hub module and running predictions.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nwcsdbLuIX2I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BERT_CONFIG= modeling.BertConfig.from_json_file(BERT_CONFIG_FILE)\n",
        "\n",
        "\n",
        "model_fn = model_fn_builder(\n",
        "  bert_config=BERT_CONFIG,\n",
        "  num_labels=len(label_list),\n",
        "  init_checkpoint=INIT_CHECKPOINT,\n",
        "  learning_rate=LEARNING_RATE,\n",
        "  num_train_steps=num_train_steps,\n",
        "  num_warmup_steps=num_warmup_steps,\n",
        "  use_tpu=True,\n",
        "  use_one_hot_embeddings=True\n",
        ")\n",
        "\n",
        "estimator = tf.contrib.tpu.TPUEstimator(\n",
        "  use_tpu=True,\n",
        "  model_fn=model_fn,\n",
        "  config=get_run_config(OUTPUT_DIR),\n",
        "  train_batch_size=TRAIN_BATCH_SIZE,\n",
        "  eval_batch_size=EVAL_BATCH_SIZE,\n",
        "  predict_batch_size=PREDICT_BATCH_SIZE,\n",
        ")\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WaR8i758-Ijt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_features = convert_examples_to_features(\n",
        "      train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "\n",
        "print(\"***** Running training *****\")\n",
        "print(\"  Num examples = %d\", len(train_examples))\n",
        "print(\"  Num labels = %d\", len(label_list))\n",
        "print(\"  Batch size = %d\", TRAIN_BATCH_SIZE)\n",
        "print(\"  Num steps = %d\", num_train_steps)\n",
        "\n",
        "train_input_fn = input_fn_builder(\n",
        "    features=train_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=True,\n",
        "    drop_remainder=True)\n",
        "\n",
        "print('***** Started training at {} *****'.format(datetime.datetime.now()))\n",
        "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "print('***** Finished training at {} *****'.format(datetime.datetime.now()))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eoXRtSPZvdiS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# MODEL PREDICTIONS\n",
        "input_features = convert_examples_to_features(\n",
        "      test_examples,label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  \n",
        "print(len(list(input_features)))  \n",
        "print(len(test_ids))\n",
        "print(len(sentences_test))\n",
        "print('***** Started predictions at {} *****'.format(datetime.datetime.now()))\n",
        "\n",
        "# Eval will be slightly WRONG on the TPU because it will truncate\n",
        "# the last batch.\n",
        "\n",
        "predict_input_fn = input_fn_builder(\n",
        "    features=input_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=False,\n",
        "    drop_remainder=False)\n",
        "\n",
        "\n",
        "predictions = estimator.predict(predict_input_fn)\n",
        "\n",
        "print('***** Finished evaluation at {} *****'.format(datetime.datetime.now()))\n",
        "\n",
        "\n",
        "  \n",
        "#SAVE IN BUCKLET RESULTS AND PRINT THEM\n",
        "output_eval_file = os.path.join(OUTPUT_DIR, \"alberto_sentipolc16_task3_results.tsv\")\n",
        "with tf.gfile.GFile(output_eval_file, \"w\") as writer:\n",
        "  print(\"***** Results *****\")\n",
        "  for example, prediction, id in zip(sentences_test, predictions, test_ids):\n",
        "    print('\\t prediction:%s \\t id:%s \\t text_a: %s' % ( np.argmax(prediction['probabilities']),str(id),str(example) ) )\n",
        "    writer.write(\"%s \\t %s\\n\" % (str(id), np.argmax(prediction['probabilities'])) )\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}