cwpk-19-exploring-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#19: Exploring the API to OWL\n",
    "=======================================\n",
    "\n",
    "Let's Get Familiar with Some Basic owlready2 Commands\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In our last installment of this [*Cooking with Python and\n",
    "KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series, we learned some KBpedia terminology and loaded the main [KBpedia](https://kbpedia.org/) knowledge graph file, <code>kbpedia_reference_concepts.owl</code>. (Remember, do **NOT** use the <code>kbpedia_reference_concepts.n3</code> since [owlready2](http://www.lesfleursdunormal.fr/static/informatique/owlready/index_en.html) does not support the [Notation3](https://en.wikipedia.org/wiki/Notation3) syntax.) You will need to navigate back to that file location again. One of the aspects of working with [Jupyter Notebook](https://en.wikipedia.org/wiki/Project_Jupyter) is that, once you leave the application, you must re-load your files again whenever you return to the application. As you also recall, it is also necessary to Run the cells in order for a given notebook to restore the states that each cell needs.\n",
    "\n",
    "In today's installment we will start to get familiar with some basic commands in owlready2, the [API](https://en.wikipedia.org/wiki/Application_programming_interface) to our [OWL2](https://en.wikipedia.org/wiki/Web_Ontology_Language) [ontologies](https://en.wikipedia.org/wiki/Upper_ontology). We will then apply these basic approaches to starting efforts to manage these ontologies, which begins with our next installment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Import the Ontologies\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service (the 'raw' version) or local files. See <a href=\"https://www.mkbergman.com/2347/cwpk-17-choosing-and-installing-an-owl-api/\"><strong>CWPK #17</strong></a> for further details.</div>\n",
    "\n",
    "#### Local File Option\n",
    "As we instructed in the last installment, the best practice is to provide names to absolute file addresses to make sure you are getting and loading the files you need. Again, in this instance, we call name this file <code>main</code> and we also add some code to verify the file was found by listing its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "main = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(main) as fobj:                           \n",
    "    for line in fobj:\n",
    "        print (line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you <code>shift+enter</code> to execute the cell contents, or pick Run from the main menu, you will see a rather long file listing. To clear it from the screen, run Cell &rarr; All Output &rarr; Clear.\n",
    "\n",
    "The KBpedia full ontology also depends on ('imports') two other files: the KBpedia upper ontology, KKO (<code>kko.owl</code>), and the SKOS language that we use for annotations (<code>http://www.w3.org/2004/02/skos/core</code>). As we described in the prior **CWPK** installment, you will need to [download the raw version](https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl) of <code>kko.owl</code>, and store it in the same location as <code>kbpedia_reference_concepts.owl</code> downloaded in the prior installment. There is no need to download the <code>skos</code> file.\n",
    "\n",
    "Again, we give names to their absolute file addresses, only this time we do not echo their file listings to screen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "kko_file = r'C:\\1-PythonProjects\\kbpedia\\sandbox\\kko.owl' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "skos_file = 'http://www.w3.org/2004/02/skos/core' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note one file (<code>kko</code>) is on disk and we use the native Windows file path (with backward slashes) by invoking the '<code>r</code>' (raw) command switch: <code>r'C:\\1-PythonProjects\\kbpedia\\sandbox\\kko.owl'</code>. The other <code>skos</code> file is found off of the Web.\n",
    "\n",
    "#### MyBinder Option\n",
    "If you are running this notebook online, do **NOT** run the above routines, since the load routines vary somewhat. Please use the following instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "main = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view this file, run the next cell, otherwise, run the following cells to load all graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request \n",
    "\n",
    "for line in urllib.request.urlopen(main):\n",
    "    print(line.decode('utf-8'), end='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kko_file = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "skos_file = 'http://www.w3.org/2004/02/skos/core' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Begin the Import\n",
    "Now that our three files have been given logical names, we are ready to import them into our Jupyter Notebook. Run each of these cells (<code>shift+enter</code>) in sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from owlready2 import *\n",
    "onto = get_ontology(main).load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The asterisk (\\*) after the import statement means that we want to load all modules in the owlready2 package. One of those functions is <code>get_ontology</code>. We have also given our main ontology the name of <code>onto</code>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "skos = get_ontology(skos_file).load()\n",
    "onto.imported_ontologies.append(skos) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kko = get_ontology(kko_file).load()\n",
    "onto.imported_ontologies.append(kko) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also place all three commands into a single cell. Note, however, that if you run this consolidated command, that you will see double listings when you invoke <code>print(onto.imported_ontologies)</code> below. In other words, while there is no harm to loading a file again, you are better off only doing the consolidated alone or the three separate loads alone.\n",
    "\n",
    "You can run the command below to see this behavior. You should then log off or Kernal &rarr; Restart & Clear Output and then only load one set before proceeding further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from owlready2 import *\n",
    "onto = get_ontology(main).load()\n",
    "\n",
    "skos = get_ontology(skos_file).load()\n",
    "onto.imported_ontologies.append(skos) \n",
    "\n",
    "kko = get_ontology(kko_file).load()\n",
    "onto.imported_ontologies.append(kko)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now test to make sure the ontologies have been loaded OK by checking to see if the base IRIs for all graphs are recognized by the system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://kbpedia.org/kbpedia/rc#\n"
     ]
    }
   ],
   "source": [
    "print(onto.base_iri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(skos.base_iri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kko.base_iri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also confirm that the two additional ontologies have been properly imported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(onto.imported_ontologies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting Ontology Contents\n",
    "\n",
    "We can now start checking to see if the components of the ontologies have loaded properly. Let's first check on the classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(onto.classes())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, this listing above is interesting. You will see a few warning messages at the top of the listing. This is due to the fact that a very few KBpedia RCs share the same subClasses, making them essentially equivalent classes. The ones listed here are purposeful and do no harm; generally, however, you want to avoid such duplication.\n",
    "\n",
    "Next, if you scroll down to the bottom you will see that the listing is actually truncated at the bottom with the <code>[...]</code> indicator. If you were to copy the output listing to an editor you would see there are exactly 1000 items in this list, the default limit for such listings when you first install Jupyter Python. But we know there are more than 58,000 RCs (classes) in the system. To see a complete listing we will need to make a change to our notebook's configuration file.\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Note:</strong> This option does not apply to the online MyBinder version.</div>\n",
    "\n",
    "First, you need to exit this notebook and at your command line create an IPython config file if you don't already have one:\n",
    "\n",
    "<pre>ipython profile create</pre>\n",
    "\n",
    "Once generated, this config file can be found in your user's directory (<code>C:\\Users\\mike\\.ipython\\profile_default\\ipython_config.py</code> in my case). Make sure and note the dot in the <code>.ipython</code> directory. Navigate to your appropriate directory and edit this config file to remove the comment for this line and to set the value to '0':\n",
    "\n",
    "<pre>c.PlainTextFormatter.max_seq_length = 0</pre>\n",
    "\n",
    "(which, again, is set to 1000 by default.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Restart your notebook instance after making this change and run the cells above again. You will now see the full listing of RCs in KBpedia. (A hat tip to [StackOverflow](https://stackoverflow.com/questions/23388810/ipython-notebook-output-cell-is-truncating-contents-of-my-list) for this snippet.)\n",
    "\n",
    "But, hmmm, where are the classes from the <code>kko</code> and <code>skos</code> imports? All we are showing in this list are the RCs from the default ontology of <code>main</code>.\n",
    "\n",
    "One of the interesting aspects of owlready2 is its use of what it calls 'worlds' to isolate given items into separate namespaces. This allows different projects to be held in memory at one time, including even different open and closed reasoning that may be applied. (A topic for a later installment.)\n",
    "\n",
    "Now, it is the case that we loaded all three of our ontologies without specifying different 'world' namespaces. To see the three combined, let's actually specify the 'world' that is currently active, which is called 'default' if not otherwise specified. Rather than our previous <code>onto</code> namespace, let's now explicitly refer to this default:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(default_world.classes()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the full IRI listings for the <code>kko</code> classes at the bottom of the listing. This signals a bit of a namespace recognition problem, that we will address in a few moments.\n",
    "\n",
    "We can use this same dot approach of <code>namespace.method</code> for other components in our system. (See **Additional Documentation** below for a listing of such methods available in owlready2.)\n",
    "\n",
    "In the two examples below we list the annotation properties for both the main (<code>onto</code>) and <code>skos</code> namespaces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(onto.annotation_properties())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(skos.annotation_properties())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, one of the RCs we see in out listing is for Eutheria, the technical name for mammals. Using the same format as above we want to get a listing of its subclasses, but adding our <code>rc.</code> prefix in fact does not work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(rc.Eutheria.subclasses())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm, it is not immediately clear why we get the error that '<code>rc</code>' is not defined.\n",
    "\n",
    "If we look at the KBpedia Web site we see, in fact, that our prefix to individual RCs is actually <code>http://kbpedia.org/kko/rc/</code> with the trailing '/'. owlready2 documentation indicates it uses the '#' sign as its default separator for vocabulary terms. Since our system uses the '/', perhaps something got screwed up in how owlready2 recognizes our namespace.\n",
    "\n",
    "One way to deal with this potential problem explicitly is to make a direct assignment for what the IRI prefix should be. We do so via this command, where we are directing the prefix '<code>rc</code>' to recognize our standard base IRI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rc = onto.get_namespace('http://kbpedia.org/kko/rc/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this new namespace assignment, we run the command again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(rc.Eutheria.subclasses())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voilà! We now have the proper registry of the namespace in our system.\n",
    "\n",
    "Along with absolute file names that we reference via assigned names, it is also good practice to make sure that our namespaces are properly registered before beginning work with owlready2. (I may discover better tips as we move ahead that I will post when and if discovered.)\n",
    "\n",
    "We can also check out the properties for this concept:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(rc.Eutheria.get_class_properties())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, pull out our definition for the concept:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rc.Eutheria.definition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also the alternate IRI sytax for identifying a class, in this case to obtain all of the children, grandchildren, etc., of our Mammal concept (which produces a long list of mammals!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IRIS['http://kbpedia.org/kko/rc/Mammal'].descendants() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have used the Python <code>list</code> method quite a bit, we can get direct assistance about the method with the standard command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also conduct searches of our RCs, including with wildcards (\\*) in either the pre or post positions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onto.search(iri = \"*Luggage*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another helpful command in owlready2 is to instruct the system what paths to search when it looks for referenced names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onto_path.append(\"C:/1-PythonProjects/kbpedia/sandbox\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, we can get a listing of all such paths registered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(onto_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, for this installment, we can close by saving the ontology we have been working on, being careful in this instance to give a different file name to prevent inadvertent overwriting of our initial input file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "onto.save(file = \"C:/1-PythonProjects/kbpedia/sandbox/owl-test.owl\", format = \"rdfxml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To properly exit the notebook, first pick File &rarr; Save and Checkpoint, then File &rarr; Close and Halt. This will return you to the main Jupyter files screen, from which you Quit and then close the tab. Return to the command window and finish shutting down the service with <code>ctrl+c</code>. That will return you to the command prompt, from which you may again start up a '<code>Jupyter Notebook</code>' instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "Here is a listing of [owlready2 ontology methods](https://owlready2.readthedocs.io/en/latest/onto.html).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}