Various_NLP.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "EHn8xRsUtmhT"
      },
      "source": [
        "# Assignment 2\n",
        "#CSE5NLP - Term 3 - 2023\n",
        "\n",
        "This assignment contains a total of three problems/tasks. The mark distribution for these tasks are as follows:\n",
        "\n",
        "* Problem 1: 20 marks\n",
        "* Problem 2: 30 marks\n",
        "* Problem 3: 50 marks\n",
        "\n",
        "Write down your name and student ID below: \n",
        "\n",
        "Name: **Benjamin Kereopa-Yorke**\n",
        "\n",
        "Student ID: **21340711**\n",
        "\n",
        "Please read the following statement before getting started: \n",
        "\n",
        "*A key purpose of this assessment task is to test your own ability to complete the assigned tasks. Therefore, the use of ChatGPT, AI tools or chatbots with similar functionality is prohibited for this assessment task. Students who are found to be in breach of this rule will be subject to normal academic misconduct measures. Additionally, students may be engaged to provide an oral validation of their understanding of their submitted work (e.g., coding).*"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "kqvOFR7HvBLm"
      },
      "source": [
        "## Problem 1 TF-IDF\n",
        "\n",
        "Implement TF-IDF using using Python, Numpy, Pandas and whatever text cleaning library required.\n",
        "\n",
        "The tf–idf is the product of two statistics, term frequency and inverse document frequency. There are various ways for determining the exact values of both statistics, you can use the following formulas.\n",
        "\n",
        "### Term Frequency\n",
        "$$tf_{t,d} = \\log_{10}(count(t,d) +1)$$ \n",
        "\n",
        "* $tf_{t,d}$ is the frequency of the word t in the\n",
        "document d\n",
        "\n",
        "### Inverse Document Frequency\n",
        "$$idf_t = \\log_{10}(\\frac{N}{df_t})$$\n",
        "\n",
        "* $N$ is the total number of documents\n",
        "* $df_t $ is the number of documents in which term t occurs\n",
        "\n",
        "### TF-IDF\n",
        "$$tf\\text{-}idf_{t,d} = tf_{t,d} \\times idf_t $$\n",
        "\n",
        "### What is expected? \n",
        "Your implementation should include the following two functions:\n",
        " * `compute_tfidf_weights(train_docs)`\n",
        " * `word_tfidf_vector(word, tf_df, idf_df)`\n",
        "\n",
        "To revise what TF-IDf is, you can revise the lecture notes and the further reading under Week 7. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "trgGYEeoxnfm"
      },
      "outputs": [],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "# Function to compute TF-IDF weights for a given set of documents\n",
        "def compute_tfidf_weights(train_docs):\n",
        "    # Initialize TfidfVectorizer with specific parameters\n",
        "    vectorizer = TfidfVectorizer(norm=None, smooth_idf=False)\n",
        "    \n",
        "    # Fit the vectorizer to the training documents and transform the documents into their vector representation\n",
        "    tfidf_matrix = vectorizer.fit_transform(train_docs)\n",
        "    \n",
        "    # Convert the TF-IDF matrix into a DataFrame for easier manipulation\n",
        "    docs_tf_idf = pd.DataFrame(tfidf_matrix.toarray(), columns=vectorizer.get_feature_names_out())\n",
        "    \n",
        "    # Compute the Term Frequency (TF) for each word in the documents\n",
        "    docs_tf = docs_tf_idf.applymap(lambda x: x if x==0 else np.log10(x)+1)\n",
        "    \n",
        "    # Compute the Inverse Document Frequency (IDF) for each word in the documents\n",
        "    docs_idf = pd.DataFrame([vectorizer.idf_]*len(train_docs), columns=vectorizer.get_feature_names_out())\n",
        "    \n",
        "    # Return the TF and IDF DataFrames\n",
        "    return docs_tf, docs_idf\n",
        "\n",
        "# Function to compute the TF-IDF vector for a given word\n",
        "def word_tfidf_vector(word, tf_df, idf_df):\n",
        "    # Check if the word is present in the documents\n",
        "    if word in tf_df.columns:\n",
        "        # Compute the TF-IDF value for the word by multiplying its TF and IDF values\n",
        "        tf_idf_value = tf_df[word] * idf_df[word]\n",
        "        \n",
        "        # Return the TF-IDF value as a Numpy array\n",
        "        return tf_idf_value.to_numpy()\n",
        "    else:\n",
        "        # If the word is not found in the documents, print a message and return None\n",
        "        print(\"Word not found in training documents.\")\n",
        "        return None\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "cs5XDT4P-3py"
      },
      "source": [
        "## Problem 2 POS for classification"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "eyQIaUWB6tfR"
      },
      "source": [
        "Robots and chat bots receive different commands to do certain tasks. \n",
        "\n",
        "Write a simple pragram that receive interactions in the form of a sentence and return:\n",
        "* A tuple of (command, object) if the sentence is a command\n",
        "* None if the sentence is not a command\n",
        "\n",
        "To write this function, you can utilize a Part-of-speech tagger or named-entity recognizer from libraries like NLTK and Spacy.\n",
        "\n",
        "Consider the following EXAMPLE sentences:\n",
        "\n",
        "* Commands:\n",
        "  * Grab the book\n",
        "  * Fetch the ball\n",
        "  * Open the jar\n",
        "  * Can hand this spoon to John?\n",
        "\n",
        "* Not commands:\n",
        "  * Hey, how is it going?\n",
        "  * How is your day today?\n",
        "  * Do you like the weather?\n",
        "This list is not exhaustive, your function should be able to handle more cases. \n",
        "\n",
        "Expected outcome:\n",
        "\n",
        "1. A function that performs the task\n",
        "2. If your function has limitations, highlight those limitations with examples. You are not required to submit a different file. Write your answer in a 'Text' block in this notebook. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VPnzEzUZyNVE",
        "outputId": "8c326c82-d0ae-459a-cdf2-f73aab6077a5"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Please enter a sentence: Grab me a book\n",
            "('Grab', 'book')\n"
          ]
        }
      ],
      "source": [
        "# Import necessary library\n",
        "import nltk\n",
        "\n",
        "# Download necessary NLTK data\n",
        "nltk.download('punkt') # This line downloads the 'punkt' package, which includes a pre-trained Punkt tokenizer for several languages. This is used for word tokenization.\n",
        "nltk.download('averaged_perceptron_tagger') # This line downloads the averaged perceptron tagger that is pre-trained on English news text. It is used for part-of-speech tagging.\n",
        "\n",
        "# Define a function to extract command from a sentence\n",
        "def extract_command():\n",
        "    '''\n",
        "    This function requests user input in form of a sentence.\n",
        "    It tokenises the sentence into words, assigns Part of Speech (POS) tags to each word,\n",
        "    and then returns the first verb-noun pair it encounters.\n",
        "    \n",
        "    Output arguments:\n",
        "        tuple (verb, noun) if the sentence is a command\n",
        "        None if the sentence is not a command or no verb-noun pair is found\n",
        "    '''\n",
        "\n",
        "    # Request user input\n",
        "    sentence = input(\"Please enter a sentence: \") # The input is expected to be a sentence in English.\n",
        "\n",
        "    # Tokenise the sentence into words\n",
        "    tokens = nltk.word_tokenize(sentence) # nltk.word_tokenize() function splits the input sentence into individual words (tokens).\n",
        "    \n",
        "    # Perform POS tagging\n",
        "    pos_tags = nltk.pos_tag(tokens) # nltk.pos_tag() function assigns POS tags to each token. \n",
        "\n",
        "    # Initialise variables to hold the command and object\n",
        "    command = None\n",
        "    obj = None\n",
        "\n",
        "    # Loop through each word and its corresponding POS tag\n",
        "    for word, tag in pos_tags:\n",
        "        # Check if the tag indicates a verb. If yes, assign the word to 'command'.\n",
        "        if tag.startswith('VB'): # 'VB' is the POS tag for verbs in base form.\n",
        "            command = word\n",
        "        # Check if the tag indicates a noun. If yes, assign the word to 'obj'.\n",
        "        elif tag.startswith('NN'): # 'NN' is the POS tag for nouns in singular form.\n",
        "            obj = word\n",
        "            # Check if we have found both a command (verb) and an object (noun).\n",
        "            if command is not None: # If yes, return the pair as a tuple.\n",
        "                return (command, obj)\n",
        "            \n",
        "    # If no command-object pair was found in the sentence, return None.\n",
        "    return None\n",
        "\n",
        "# Run the function and print its output\n",
        "print(extract_command()) # This line calls the function and prints the verb-noun pair if found, else prints 'None'.\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "rMLe8iuDKG8A"
      },
      "source": []
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "P14Abk7FKZL9"
      },
      "source": [
        "While the `extract_command()` function provides a simplistic way to parse commands from user inputs, it has several limitations rooted in its assumptions and implementation details.\n",
        "\n",
        "1. **Order of Command-Object:** The function expects the verb (command) to appear before the noun (object) in a sentence. In languages or sentence constructions where this order is reversed, or in cases where additional sentence elements intervene between the verb and noun, the function will not correctly identify the command.\n",
        "\n",
        "2. **Limited Understanding of Sentence Structure:** The function only uses a rudimentary form of sentence parsing, looking for the first verb and noun without considering the overall sentence structure. This can lead to errors in complex sentences with multiple verbs and nouns.\n",
        "\n",
        "3. **Ambiguity Resolution:** In the presence of multiple verbs or nouns, the function simply takes the first one that fits its command-object pattern. It does not have the ability to resolve ambiguities or to consider user intent.\n",
        "\n",
        "4. **Lack of Contextual Understanding:** The function does not take into account the contextual meaning of words. For instance, it would not distinguish between a command and a question or a statement, as long as there's a verb and a noun in sequence. For example, the question \"Can you open the door?\" would be interpreted as the command \"open door\", which may not be the desired interpretation.\n",
        "\n",
        "5. **Tagging Accuracy:** The function's accuracy heavily depends on the accuracy of NLTK's POS tagging, which is not perfect and may vary based on the language and specific domain of the text.\n",
        "\n",
        "6. **Lack of Error Handling:** If the user inputs something that is not a sentence (e.g., random symbols or characters), the function may return unexpected results or even raise an error.\n",
        "\n",
        "7. **Lack of Multi-word Object Handling:** The function only captures single-word objects. If the object of a command consists of multiple words (e.g., \"my green bag\"), the function will only capture the first noun (\"green\").\n",
        "\n",
        "Given these limitations, while this function can serve as a starting point for command extraction, it would need to be significantly enhanced and adapted for more advanced or specific use cases in Natural Language Understanding."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JN4SjrEYuFOx"
      },
      "source": [
        "## Problem 3 Word embedding as features for classification\n",
        "\n",
        "### Task\n",
        "Implement a sentiment classifier based on Twitter data to analyse the sentiments of COVID-19 tweets.  \n",
        "\n",
        "Train and test multiple classification model using necessary libraries with the features being sentence embeddings of tweets. \n",
        "\n",
        "Report the accuracy and F1 score (micro- and macro-averaged) for multiple classifier and discuss the differences. \n",
        "\n",
        "### Dataset\n",
        "The dataset have been provided in Dataset.zip file with the assignment. You are required to use the original tweet text for this classification task. \n",
        "\n",
        "### Tweet representation\n",
        "After necessary pre-processing of the tweets, convert the words into their embeddings, then take the mean of all the word vectors in a tweet to end up with a single vector representing each tweet. The tweet vector is then used for sentiment classification.\n",
        "\n",
        "In the process of finding the embeddings for each word, you can ignore out-of-vocabulary words.\n",
        "\n",
        "### Embedding choice\n",
        "For embedding, you can use GloVe embeddings using Gensim. A sample code is give below, and more information can be found from lab resources on embeddings. \n",
        "\n",
        "However, this is a suggested option. You can use any word embedding of your choice, for example, word2vec, TF-IDF, etc., from any library of your choice.   \n",
        "\n",
        "### Classifier choice\n",
        "You are required to implement the following classifiers: \n",
        "* One tradition classification model (not a neural network based model)\n",
        "* One classifier based on any neural network based model. \n",
        "\n",
        "You can use PyTorch/TensorFlow/scikit-learn to implement your classifier. However, you are free to develop a classifier from scratch. \n",
        "\n",
        "### Your answer must include the following: \n",
        "1. Code for data loading, data pre-processing, training, and testing of the models.  \n",
        "2. A discussion on the comparison between the classifiers based on classifier accuracy and F1 score.\n",
        "\n",
        "### Suggestion (Optional)\n",
        "Consider saving a cleaned up version of the dataset after creating the embeddings to a file which can be loaded and used for further experimentation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F8tWOkp7zmFk",
        "outputId": "3129183e-1a01-4a04-911b-dd976906b3eb"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "<ipython-input-14-b44ef1b245c2>:38: DeprecationWarning: Call to deprecated `glove2word2vec` (KeyedVectors.load_word2vec_format(.., binary=False, no_header=True) loads GLoVE text vectors.).\n",
            "  glove2word2vec(glove_input_file, word2vec_output_file)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Classifier: LogisticRegression\n",
            "Accuracy: 0.4426013691416535\n",
            "F1 Score (Micro): 0.4426013691416535\n",
            "F1 Score (Macro): 0.4476202279106887\n",
            "\n",
            "Classifier: MLPClassifier\n",
            "Accuracy: 0.4439178515007899\n",
            "F1 Score (Micro): 0.4439178515007899\n",
            "F1 Score (Macro): 0.45809377681851765\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "# Imports all of the libraries I want to use to solve the problem space\n",
        "import nltk\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from gensim.models import KeyedVectors\n",
        "from sklearn import preprocessing\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "from gensim.scripts.glove2word2vec import glove2word2vec\n",
        "\n",
        "# Download the necessary NLTK components as this notebook and assignment work were done in Google Colab environments which are often deprecated or barebones\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "\n",
        "stop_words = set(stopwords.words('english'))\n",
        "\n",
        "# Load the training data and the test data, ensuring that the encoding for special characters like those present in these datasets doesn't throw errors\n",
        "train_df = pd.read_csv(\"Corona_NLP_train.csv\", encoding='iso-8859-1')\n",
        "test_df = pd.read_csv(\"Corona_NLP_test.csv\", encoding='iso-8859-1')\n",
        "\n",
        "# Preprocess text\n",
        "def preprocess_text(text):\n",
        "    # Lowercase and tokenise\n",
        "    tokens = word_tokenize(text.lower())\n",
        "    # Remove stopwords and return\n",
        "    return [word for word in tokens if word not in stop_words and word.isalpha()]\n",
        "\n",
        "train_df['OriginalTweet'] = train_df['OriginalTweet'].apply(preprocess_text)\n",
        "test_df['OriginalTweet'] = test_df['OriginalTweet'].apply(preprocess_text)\n",
        "\n",
        "# Load the GloVe model and convert it to word2vec format. I manually downloaded the 6B GloVe model for use in other research and learning besides this, instead of other methods I could have used\n",
        "glove_input_file = 'glove.6B.100d.txt'\n",
        "word2vec_output_file = 'glove.6B.100d.word2vec.txt'\n",
        "\n",
        "glove2word2vec(glove_input_file, word2vec_output_file)\n",
        "\n",
        "# Then I can load the converted file:\n",
        "glove_model = KeyedVectors.load_word2vec_format(word2vec_output_file, binary=False)\n",
        "\n",
        "# Creates a vector for each tweet in the dataset \n",
        "def get_vector(words):\n",
        "    word_embeddings = [glove_model[word] for word in words if word in glove_model]\n",
        "    if len(word_embeddings) == 0:\n",
        "        return np.zeros(glove_model.vector_size)\n",
        "    else:\n",
        "        return np.mean(word_embeddings, axis=0)\n",
        "\n",
        "train_df['vector'] = train_df['OriginalTweet'].apply(get_vector)\n",
        "test_df['vector'] = test_df['OriginalTweet'].apply(get_vector)\n",
        "\n",
        "# Encode the labels\n",
        "label_encoder = preprocessing.LabelEncoder()\n",
        "train_df['Sentiment'] = label_encoder.fit_transform(train_df['Sentiment'])\n",
        "test_df['Sentiment'] = label_encoder.transform(test_df['Sentiment'])\n",
        "\n",
        "# Here below I will train the classifiers. There is an assignment requirement to use two classifiers, one that is not neural network based.\n",
        "\n",
        "# First Classifier = Traditional classifier: Logistic Regression\n",
        "lr = LogisticRegression(max_iter=1000)\n",
        "lr.fit(list(train_df['vector']), train_df['Sentiment'])\n",
        "\n",
        "# Second Classifier =  Neural network-based classifier: Multi-layer Perceptron\n",
        "mlp = MLPClassifier()\n",
        "mlp.fit(list(train_df['vector']), train_df['Sentiment'])\n",
        "\n",
        "# Test the classifiers and report the accuracy and F1 scores\n",
        "for classifier in [lr, mlp]:\n",
        "    y_pred = classifier.predict(list(test_df['vector']))\n",
        "    accuracy = accuracy_score(test_df['Sentiment'], y_pred)\n",
        "    f1_micro = f1_score(test_df['Sentiment'], y_pred, average='micro')\n",
        "    f1_macro = f1_score(test_df['Sentiment'], y_pred, average='macro')\n",
        "\n",
        "    # This prints the classifiers accuracy and F1 scores for easy evaluation by the user\n",
        "    print(f'Classifier: {classifier.__class__.__name__}\\nAccuracy: {accuracy}\\nF1 Score (Micro): {f1_micro}\\nF1 Score (Macro): {f1_macro}\\n')\n",
        "\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Bb51BEaI5lxF"
      },
      "source": [
        "Analysing the results from the two classifiers, **Logistic Regression and MLPClassifier**, it's apparent that both models **exhibit somewhat similar performance levels**, albeit with some nuanced differences.\n",
        "\n",
        "**Accuracy Score**: The accuracy score indicates the overall correct predictions made by the model over all kinds predictions made. The accuracy of the Logistic Regression model is approximately 0.4426 while that of the MLPClassifier is slightly higher at approximately 0.4439. Although the difference is quite small, it signifies that the MLPClassifier is slightly better at making correct predictions for this specific dataset.\n",
        "\n",
        "**F1 Score (Micro)**: The Micro F1 Score computes the F1 Score by considering total true positives, false negatives, and false positives (irrespective of the class). Here, both models exhibit the same F1 Score (Micro) as their accuracy. This is often the case when dealing with binary classification problems or multi-class classification problems treated as binary classification ones (where averaging is performed over samples instead of classes). Similar to the accuracy, the MLPClassifier shows a marginally better performance with an F1 Score (Micro) of about 0.4439 versus the Logistic Regression model's score of 0.4426.\n",
        "\n",
        "**F1 Score (Macro)**: The Macro F1 Score calculates the F1 Score for each class independently and then takes the average (which treats all classes equally). It is a better metric when dealing with imbalanced datasets. For the Macro F1 Score, the MLPClassifier outperforms the Logistic Regression model by a wider margin - 0.4581 compared to 0.4476. This suggests that, on average, the MLPClassifier might be more capable of handling all classes in the dataset, particularly if there is a class imbalance.\n",
        "\n",
        "**Overall, based on the provided metrics, it would appear that the MLPClassifier has a slight edge over the Logistic Regression model for this specific dataset**. However, these are raw performance metrics and may not fully reflect the practical suitability of the model. Factors such as training time, interpretability, and computational resources could play a crucial role in deciding the most appropriate model for a given task. Furthermore, it's important to remember that the choice of evaluation metrics should be aligned with the specific objectives and context of the task. For instance, in some scenarios, precision or recall could be more important than overall accuracy."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}