1-pattern-exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "be6825f5-fdce-4d98-a7b9-495d177607a2",
   "metadata": {},
   "source": [
    "# Knowledge Graphs Workshop: Pattern exercise\n",
    "\n",
    "First, we'll load the dataset and do some basic pre-processing.\n",
    "Then we'll show the table in a handy interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04a459c8-7e9b-4120-bd2f-06c01161670c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from itables import show\n",
    "df = pd.read_csv('Aircraft_Annotation_DataFile.csv')\n",
    "df.columns = [c.lower() for c in df.columns]\n",
    "df['problem'] = df['problem'].str.strip('.').str.strip()\n",
    "df['action'] = df['action'].str.strip('.').str.strip()\n",
    "show(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf0a1efc-0fec-4c15-a56d-2be34665ceb1",
   "metadata": {},
   "source": [
    "## Define a pattern for Problem strings\n",
    "\n",
    "In this example pattern, we extract the location of the problem, the part, and a problem keyword.\n",
    "\n",
    "Try to modify the pattern to extract more problem types, or make a different kind of problem pattern!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f9b5f8-0547-451e-aa9c-aa366f112ce5",
   "metadata": {},
   "outputs": [],
   "source": [
    "problem_pat = (\n",
    "    '^'\n",
    "    # The location often mentions the engine side and cylinder number\n",
    "    '(?P<location>(?:(?:L|R)/H (?:ENG )?)?(?:CYL ?)?(?:#?\\d(?: ?. \\d)*)(?: CYL ?)?)? ?'\n",
    "    \n",
    "    # A part name ends with a letter, ignore the last \"S\" (for plural words)\n",
    "    '(?P<part>.*?\\w)S? ' \n",
    "\n",
    "    # Match the verb but don't extract it\n",
    "    '(?:IS |ARE |HAS |HAVE )?(?:A )?'\n",
    "\n",
    "    # Some pre-defined problem keywords to match\n",
    "    '(?P<problem>LEAK|LOOSE|TORN|CRACKED|BROKEN|DAMAGED|WORN|BAD|SHEAR|DIRTY)'\n",
    ")\n",
    "problem_extractions = df['problem'].str.extract(problem_pat)\n",
    "\n",
    "# Show the most common problem extractions\n",
    "show(problem_extractions.fillna('').value_counts().rename('count'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c01cbe8b-7d35-4833-a8b5-9db77ec9b88c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show non-matching problems\n",
    "show(df['problem'].loc[problem_extractions.isna().all(axis=1)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6eb572e-482a-4fb3-a2a5-424fce1ec1fc",
   "metadata": {},
   "source": [
    "## Define a pattern for Action strings\n",
    "\n",
    "In this example pattern, we extract the location of the action, the part, and an action keyword.\n",
    "\n",
    "Try to modify the pattern to extract more action types, or make a different kind of action pattern!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fef34c9-7eff-44b4-aba1-3b71781ceced",
   "metadata": {},
   "outputs": [],
   "source": [
    "action_pat = (\n",
    "    '^(?:REMOVED & )?(?:RE)?'\n",
    "    # Pre-defined action keywords\n",
    "    '(?P<action>REPLACED|TIGHTENED|SECURED|ATTACHED|FASTENED|TORQUED|CLEANED|STOP DRILLED) ?'\n",
    "\n",
    "    # The location often mentions the engine side and cylinder number\n",
    "    '(?P<location>(?:(?:L|R)/H (?:ENG )?)?(?:CYL ?)?(?:#?\\d(?: ?. \\d)*)(?: CYL ?)?)? ?'\n",
    "\n",
    "    # Often, replacements mention \"W/ NEW\"; ignore it\n",
    "    '(?:W/ )?(?:NEW )?'\n",
    "\n",
    "    # A part name ends with a letter, ignore the last \"S\" (for plural words)\n",
    "    '(?P<part>.*?\\w)S?'\n",
    "    \n",
    "    '(?: W/ .*)?$'\n",
    ")\n",
    "action_extractions = df['action'].str.extract(action_pat)\n",
    "show(action_extractions.fillna('').value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b132570a-5c1e-4e22-b65f-202357ac1ca5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show non-matching actions\n",
    "show(df['action'].loc[action_extractions.isna().all(axis=1)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c9f291-982d-47db-8492-de9b103626f2",
   "metadata": {},
   "source": [
    "## Loading extractions into graph\n",
    "\n",
    "Now, we'll transform our extractions into graphs and load them into the Knowledge Graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3132cbc2-c99e-4f16-8b4f-1e8511689674",
   "metadata": {},
   "outputs": [],
   "source": [
    "from helperFunctions import obj_to_triples\n",
    "import re\n",
    "from rdflib import Graph, URIRef, BNode, Literal, RDF, RDFS, DC, Namespace\n",
    "ZORRO = Namespace(\"https://zorro-project.nl/example/\")\n",
    "\n",
    "def create_problem_obj(row):\n",
    "    ent = ZORRO[f'problem{row.ident}']\n",
    "    \n",
    "    problem_match = re.search(problem_pat, row.problem)\n",
    "    problem_fields = problem_match.groupdict() if problem_match else {}\n",
    "    action_match = re.search(action_pat, row.action)\n",
    "    action_fields = action_match.groupdict() if action_match else {}\n",
    "\n",
    "    def camelcase(fields, name):\n",
    "        # Convert string into a clean CamelCase name\n",
    "        return re.subn('\\W', '', fields.get(name, '').title())[0]\n",
    "    \n",
    "    return {\n",
    "        '@id': ent,\n",
    "        RDF.type: ZORRO[camelcase(problem_fields, 'problem') + 'Problem'],\n",
    "        DC.description: Literal(row.problem),\n",
    "        \n",
    "        ZORRO.involvedPart: {\n",
    "            RDF.type: ZORRO[camelcase(problem_fields, 'part') + 'Part'],\n",
    "            ZORRO.location: Literal((problem_fields.get('location') or '').strip())\n",
    "        } if problem_fields.get('part') else None,\n",
    "        \n",
    "        ZORRO.requiredAction: {\n",
    "            DC.description: Literal(row.action),\n",
    "            RDF.type: ZORRO[camelcase(action_fields, 'action') + 'Action'],\n",
    "            \n",
    "            ZORRO.involvedPart: {\n",
    "                RDF.type: ZORRO[camelcase(action_fields, 'part') + 'Part'],\n",
    "                ZORRO.location: Literal((action_fields.get('location') or '').strip())\n",
    "            } if action_fields.get('part') else None\n",
    "        }\n",
    "    }\n",
    "\n",
    "# Show the turtle serialization of the first 5 extractions\n",
    "g = Graph()\n",
    "g.namespace_manager.bind('', ZORRO)\n",
    "for obj in df.head(5).apply(create_problem_obj, axis=1):\n",
    "    for t in obj_to_triples(obj):\n",
    "        g.add(t)\n",
    "print(g.serialize())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "688a92c2-d080-41e9-9d99-c971e3835c93",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run on entire dataset, takes a few seconds!\n",
    "g = Graph()\n",
    "g.namespace_manager.bind('', zorro)\n",
    "for obj in df.apply(create_problem_obj, axis=1):\n",
    "    for t in obj_to_triples(obj):\n",
    "        g.add(t)\n",
    "g.serialize('pattern_graph.ttl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f432a6cc-35dd-4901-9b89-2061fe25deb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext ipython_sparql_pandas\n",
    "from helperFunctions import GraphDB\n",
    "\n",
    "db = GraphDB()\n",
    "repo_name = 'zorro'\n",
    "db.create_repo(repo_name).text\n",
    "\n",
    "response = db.load_data(repo_name, 'pattern_graph.ttl', \n",
    "          graph_name = \"https://zorro-project.nl/example/PatternGraph\")\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05d2224-943e-433a-ba46-64f4c2a91510",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sparql http://localhost:{db.port}/repositories/{repo_name} -s result\n",
    "PREFIX : <https://zorro-project.nl/example/>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "\n",
    "select (count(*) as ?c) where { \n",
    "\t?prob a :Problem .\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0afab5e2-5673-4a96-b477-8266cc854e55",
   "metadata": {},
   "source": [
    "### But didn't we add many more instances ..?\n",
    "\n",
    "These are only the ones for which we couldn't extract a more specific problem class!\n",
    "\n",
    "To get our instances, we also load our schema about maintenance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "465c58e7-c3c3-4296-b41e-e15760b3f47b",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = db.load_data(repo_name, 'maintenance.ttl', \n",
    "          graph_name = \"https://zorro-project.nl/example/MaintenanceGraph\")\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e18cea2-499f-4665-aaaf-7ce73958efa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sparql http://localhost:{db.port}/repositories/{repo_name} -s result\n",
    "PREFIX : <https://zorro-project.nl/example/>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "\n",
    "select (count(*) as ?c) where { \n",
    "\t ?prob a :Problem .\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8da80e3f-74a1-4a0e-958f-bb4f54b291de",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sparql http://localhost:{db.port}/repositories/{repo_name} -s result\n",
    "PREFIX : <https://zorro-project.nl/example/>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "\n",
    "select ?problemClass (count(*) as ?count) where { \n",
    "    ?prob a :Problem .\n",
    "    ?prob a ?problemClass . \n",
    "}\n",
    "GROUP BY ?problemClass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3c8a226-e653-4837-b0de-56eb170148b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "result.set_index('problemClass')['count'].plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb5725f3-2de6-49a7-9b65-b10b42aa1f0a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}