2-NLP-exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c1a60d8-5d17-47cb-835d-476609db1be1",
   "metadata": {},
   "source": [
    "# Knowledge Graphs Workshop: NLP exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369ee828",
   "metadata": {},
   "source": [
    "## Step 1: First import and install all python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "911ac587-179d-4ceb-89d6-6f254db8fb9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import pandas as pd\n",
    "import helperFunctions as hf\n",
    "import sys\n",
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "import spacy\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import Tree\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de064d16",
   "metadata": {},
   "source": [
    "# Assignment 1\n",
    "In fields like maintenance, important information, such as when maintenance is performed or what is repaired, is not often registered properly but is frequently very crudely written down somewhere. Natural Language Processing, or NLP for short, deals with making computers understand language. Using NLP, useful information and patterns can be extracted from text, which can even be used to predict when maintenance needs to happen. In this exercise, we will go over an aircraft maintenance dataset where a maintenance engineer briefly described the problem the aircraft had and the action that was taken to fix the problem.\n",
    "\n",
    "In this exercise, we will take a look at some basic NLP concepts and talk about the following natural language processing concepts:\n",
    "- Tokenizing\n",
    "- Removing stopwords\n",
    "- Lemmatization\n",
    "- Part-of-speech (POS) tagging\n",
    "- Inserting data into a knowledge graph\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0005976-9f48-43f2-9bd0-6fe0cd1479a1",
   "metadata": {},
   "source": [
    "## Basic tokenization\n",
    "A sentence means nothing to a computer. The first step for a computer to start understanding text is to break it down into a list of each word. This sounds easy enough, but a lot of different decisions can be made when deciding to split a string. For example, do you take into account punctuation or quotation marks? And how do you deal with words such as 'it's,' 'haven't,' 'hasn't,' which are comprised of multiple words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02909683-53bc-47e4-8047-0b603ee17279",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The most basic example is to just split a sample but as you can see this can already give some problems\n",
    "example_sent = \"\"\"This is a sample sentence, showing off basic tokenization. Words like ''it's', 'haven't', 'hasn't' are harder to correctly tokenize.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d283c076",
   "metadata": {},
   "source": [
    " ### Task\n",
    "Generate a list of tokens from the given example sentence. For instance, 'This is a sample sentence' would transform into ['this', 'is', 'sample', 'sentence']. You can either create your version or utilize the word_tokenize function from the NLTK library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad38aaf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Solution\n",
    "print(example_sent.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cd3124",
   "metadata": {},
   "source": [
    "## Stopwords\n",
    "Stopwords are frequently used words that are typically excluded or ignored in NLP because they are deemed to convey little or no meaningful information for analysis. This simplifies text analysis, as a list of the most frequently appearing words is not dominated by common words such as 'and,' 'the,' and 'than."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aeb7a52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of example sentences\n",
    "stopword_list = stopwords.words('english')\n",
    "print(stopword_list)\n",
    "example_sent = \"\"\"After every flight, the aircraft undergoes thorough maintenance checks to ensure optimal performance and safety for the next journey.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74d4026",
   "metadata": {},
   "source": [
    "### Task\n",
    "Remove stopwords from the example sentence. Remember to first use your tokenize function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65d8858d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Solution\n",
    "word_tokens = word_tokenize(example_sent)\n",
    "# converts the words in word_tokens to lower case and then checks whether \n",
    "#they are present in stop_words or not\n",
    "filtered_sentence = [w for w in word_tokens if not w.lower() in stopword_list]\n",
    "#with no lower case conversion\n",
    "filtered_sentence = []\n",
    "for w in word_tokens:\n",
    "    if w not in stopword_list:\n",
    "        filtered_sentence.append(w)\n",
    "print(word_tokens)\n",
    "print(filtered_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad5dff09",
   "metadata": {},
   "source": [
    "# Lemmatization \n",
    "When processing sentences you also need to look at lemmatization. \n",
    "Lemmatization is a NLP technique that involves reducing words to their base or root form, known as the \"lemma.\" The goal of lemmatization is to group together different inflected forms of a word so that they can be analyzed as a single item. When analysing for example the most occuring words it is nice to take the lemma of tokens as we are more interessted in the lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77fde507",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_sentence_1 = \"the engine is leaking because of loose screws\"\n",
    "example_sentence_2 = \"leak in engine because of a loose screw\"\n",
    "example_sentence_3 = \"leaks in engine because of loose screws\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a9245e2",
   "metadata": {},
   "source": [
    "### Task\n",
    "use spacy to create lemmas of words. Play around with different sentences and discuss with your teammates how this can be used for analysing nlp datasets. Remember to first tokenize the sentences!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e00855fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Solution\n",
    "# Process the text using spaCy\n",
    "doc = nlp(example_sentence_1)\n",
    "# Extract lemmatized tokens\n",
    "lemmatized_tokens = [token.lemma_ for token in doc]\n",
    "print(lemmatized_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d951a3e0",
   "metadata": {},
   "source": [
    "# Part of Speech tagging.\n",
    "Part of speech tagging or POS in short involves assigning a grammatical category or part of speech to each word in a sentence. The objective is to analyze and comprehend the syntactic structure of a sentence by determining the role of each word within the context.\n",
    "\n",
    "Accurately tagging a sentence is of significant importance as tagged words can be used for analysing and machine learning. \n",
    "\n",
    "The main parts of speech which can be tagged are nouns, verbs, adjectives, adverbs, pronouns, prepositions, conjunctions, and interjections. POS can be used alongside DEP which stands for Syntactic dependency which is a linguistic concept that represents the grammatical and hierarchical relationships between words in a sentence.\n",
    "\n",
    "\n",
    "Additionally, some sentences can be ambiguous, having multiple meanings depending on the context. A computer, however, cannot decipher this meaning and heavily relies on how the sentences are tagged.\n",
    "\n",
    "While there are no specific tasks in this section, it is essential to discuss within your group what is happening and how this understanding can be utilized to analyze maintenance tasks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0c7ed5d-f59b-4fad-b4e9-b3d8eeabbacf",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(example_sentence_1)\n",
    "# Analyze syntax\n",
    "print(\"Noun phrases:\", [chunk.text for chunk in doc.noun_chunks])\n",
    "print(\"DEP:\", [f'{token.lemma_} {token.dep_}' for token in doc])\n",
    "print(\"POS:\", [f'{token.lemma_} {token.pos_}'for token in doc])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0b1c658",
   "metadata": {},
   "source": [
    "### Lets print a parse tree to show the syntatic structure of a sentence\n",
    "Lets use the same example sentences from before to see how a different sentences can have tress that are similar.\n",
    "A parse tree connsists of the following:\n",
    "- Root Node: Represents the entire sentence.\n",
    "- Intermediate Nodes: Represent phrases or constituents, such as noun phrases (NP), verb phrases (VP), etc.\n",
    "- Leaf Nodes: Represent individual words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c28a575",
   "metadata": {},
   "outputs": [],
   "source": [
    "hf.print_nltk_tree(nlp(example_sentence_1))\n",
    "hf.print_nltk_tree(nlp(example_sentence_2))\n",
    "hf.print_nltk_tree(nlp(example_sentence_3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9c1f04",
   "metadata": {},
   "source": [
    "### Task\n",
    "Discuss with your group why the different sentences have the same tree structure and how this can be useful for analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "388fef5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The trees are the same after removing the stopwords!\n",
    "hf.print_nltk_tree(nlp(hf.remove_stopwords(example_sentence_2)))\n",
    "hf.print_nltk_tree(nlp(hf.remove_stopwords(example_sentence_3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf3f5388",
   "metadata": {},
   "source": [
    "# Assignment 2: Aircraft dataset\n",
    "Now, let's apply the knowledge gained from Exercise 1 and the information provided in the knowledge base slides to analyze an aircraft maintenance dataset. Due to time constraints, we have already coded it for you. If you find additional time after completing all exercises, feel free to experiment, enhance the code, and create your own knowledge graph."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e9a6d5",
   "metadata": {},
   "source": [
    "### Lets get some insights about the dataset\n",
    "As you can see a problem occured in the aircraft such as \"ENGINE IDLE OVERRIDE KILLED ENGINE\" and a maintenance engineer fixed the problem by the action \"REMOVED & REPLACE FUEL SERVO\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66fde627-0d13-4733-b237-0d89f76a5c6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from itables import show\n",
    "df = pd.read_csv('Aircraft_Annotation_DataFile.csv')\n",
    "df.columns = [c.lower() for c in df.columns]\n",
    "df['problem'] = df['problem'].str.strip('.').str.strip()\n",
    "df['action'] = df['action'].str.strip('.').str.strip()\n",
    "show(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a1ae160",
   "metadata": {},
   "source": [
    "Lets analyze the aircraft dataset by looking at the most used verbs in each sentence. This may give valuable insights.\n",
    "\n",
    "PS you should also look at some potential mistakes that are happening and why these mistake might happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f6653e5-8d81-4708-b220-faf287a75408",
   "metadata": {},
   "outputs": [],
   "source": [
    "hf.display_most_used(df['action'].iloc[0:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b6320c6-754e-40f0-87ff-f854d6f100c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "hf.display_most_used(df['problem'].iloc[0:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24fb28df",
   "metadata": {},
   "source": [
    "Task: Discuss the results of this graph with your group. What conclusions can you make and how can this be used in relation to knowledge graphs?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52cecd2d",
   "metadata": {},
   "source": [
    "## Putting it all inside a knowledge base\n",
    "Adapt the code from the previous exercise to transform your parse trees into graphs and load them into the Knowledge Base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6bcfdfc-a47e-4d3b-a3d1-1d15d20efaaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "def create_problem_obj():\n",
    "    pass\n",
    "\n",
    "# g = Graph()\n",
    "# g.namespace_manager.bind('', zorro)\n",
    "# for obj in df.apply(create_problem_obj, axis=1):\n",
    "#     for t in obj_to_triples(obj):\n",
    "#         g.add(t)\n",
    "# g.serialize('nlp_graph.ttl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8c732cb-a47f-49ae-a4b3-e6c4ac3f16b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext ipython_sparql_pandas\n",
    "from helperFunctions import GraphDB\n",
    "\n",
    "db = GraphDB()\n",
    "repo_name = 'zorro'\n",
    "db.create_repo(repo_name).text\n",
    "\n",
    "response = db.load_data(repo_name, 'nlp_graph.ttl', \n",
    "          graph_name = \"https://zorro-project.nl/example/NLPGraph\")\n",
    "print(response.text)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}