cwpk-37-organizing-code-base.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#37: Organizing the Code Base\n",
    "=======================================\n",
    "\n",
    "Refining Plans and Directories to Complete the Roundtrip\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "This installment begins a new major part in our [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series. Starting with the format of our extraction files, which can come directly from our prior extraction routines or from other local editing or development efforts, we are now in a position to build a working [KBpedia](https://kbpedia.org/) from scratch, and then to test it for [consistency](https://en.wikipedia.org/wiki/Consistency) and [satisfiability](https://en.wikipedia.org/wiki/Satisfiability). This major part, of all parts in this **CWPK** series, is the one that most closely reflects our traditional build routines of KBpedia using [Clojure](https://en.wikipedia.org/wiki/Clojure).\n",
    "\n",
    "But this current part is also about more than completing our [roundtrip](https://en.wikipedia.org/wiki/Round-trip_format_conversion) back to KBpedia. For, in bringing new assignments to our knowledge graph, we must also test it to ensure it is encoded properly and that it performs as promised. These additional requirements also mean we will be developing more than a build routine module in this part. The way we are structuring this effort will also add a testing module and a cleaning module (for checking encodings and the like). There are a dozen installments, including this one, in this part to cover this ground.\n",
    "\n",
    "The addition of more modules, with the plan of still more to come thereafter, also compels us to look at how we are architecting our code and laying out our files. Thus, besides code development, we need to pay attention to organizational matters as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 id=\"Starting-a-New-Major-Part\">Starting a New Major Part</h3>\n",
    "<p>I am pretty pleased with how the <em>cowpoke</em> extraction module turned out, so will be following something of the same pattern to build KBpedia in this part. Since we made the early call to bootstrap our project from the small, core top-level KBpedia Knowledge Ontology (<a href=\"https://kbpedia.org/docs/kko-upper-structure/\" target=\"_blank\" rel=\"noopener\">KKO</a>), we gained alot of simplification. That is a good trade-off, since KKO itself is a value-neutral top-level ontology built from the <a href=\"https://en.wikipedia.org/wiki/Semiotic_theory_of_Charles_Sanders_Peirce\" target=\"_blank\" rel=\"noopener\">semiotic</a> perspective of <a href=\"https://en.wikipedia.org/wiki/Charles_Sanders_Peirce\" target=\"_blank\" rel=\"noopener\">C.S. Peirce</a> regarding knowledge representation. Our basic design can also be adopted to any other <a href=\"https://en.wikipedia.org/wiki/Upper_ontology\" target=\"_blank\" rel=\"noopener\">top-level ontology</a>. If that is your desire, how to bring in a different TLO is up to you to figure out, though I hope that would be pretty easy following the recipes in these <strong>CWPK</strong> installments.</p>\n",
    "<p>Fortunately as we make the turn to build routines in this part of the roundtrip, we are walking ground that we have been traveling for nearly a decade. We understand the build process and we understand the testing and acceptance criteria necessary to result in quality, publicly-released knowledge graphs. We try to bring these learnings to our functions in this part.</p>\n",
    "<p>But as I caution at the bottom of each of these installments, I am learning <a href=\"https://en.wikipedia.org/wiki/Python_(programming_language)\" target=\"_blank\" rel=\"noopener\">Python</a> myself through this process. I am by no means a knowledgeable programmer, let alone an expert. I am an interested amateur who has had the good fortune to have worked with some of the best developers imaginable, and I only hope I picked up little bits of some good things here and there about how to approach a coding project. Chances are great you can improve on the code offered. I also, unfortunately, do not have the coding experience to provide commercial-grade code. Errors that should be trapped are likely not, cases that need to be accommodated are likely missed, and generalization and code readability is likely not what it could be. I'll take that, if the result is to help others walk these paths at a better and brisker pace. Again, I hope you enjoy . . .</p>\n",
    "<h3 id=\"Organizing-Objectives\">Organizing Objectives</h3>\n",
    "<p>I have learned some common-sense lessons through the years about how to approach a software project. One of those lessons, obvious through this series itself, is captured by John Bytheway's <a href=\"https://www.goodreads.com/quotes/321104-inch-by-inch-life-s-a-cinch-yard-by-yard-life-s\" target=\"_blank\" rel=\"noopener\">quote</a>, \"Inch by inch, life's a cinch. Yard by yard, life's hard.\" Knowing where you want to go and taking bite-sized chunks to get there almost always leads to some degree of success if you are willing to stick with the journey.</p>\n",
    "<p>Another lesson is to conform to community practice. In the case of Python (and most modern languages, I assume), applications need to be 'packaged' in certain ways such that they can be readily brought into the current computing environment. From the source-code perspective, this means conforming to the 'package' standard and the organization of code into importable modules. All of this suggests a code base that is kept separate from any project that uses it, and organized and packaged in a way similar to other applications in that language.</p>\n",
    "<p>An interesting lesson about knowledge graphs is that they are constantly changing -- and need to do so. In this regard, knowledge artifacts are not terribly different than software artifacts. Both need to be updated frequently such that versioning and version control are essential. Versioning tells users the basis for the artifact; version control helps to maintain the versions and track differences between releases. Wherever we decide to store our artifacts, they should be organized and packaged such that different versions may be kept integral. Thus, we organize our project results under version number labels.</p>\n",
    "<p>Then, in terms of this particular project where roundtripping is central and many outputs are driven from the KBpedia knowledge structure, I also thought it made sense to establish separate tracks between inputs (the 'build' side) and outputs (the 'extraction' side) and to seek parallelisms between the tracks where it makes sense. This informs much of the modular architecture put forward.</p>\n",
    "<p>All of this needs to be supplemented with utilities for testing, logging, error trapping and messaging, and statistics. These all occur at the time of build or use, and so belong to this leg of our roundtrip. We will not develop code for all of these aspects in this part of the <strong>CWPK</strong> series, but we will try to organize in anticipation of these requirements for a complete project. We'll fill in many of those pieces in major parts to come.</p>\n",
    "<h3 id=\"Anticipated-Directory-Structures\">Anticipated Directory Structures</h3>\n",
    "<p>These considerations lead to two different directory structures. The first is where the source code resides, and is in keeping with typical approaches to Python packages. Two modules are in-progress, at least at the design level, to complete our roundtripping per the objectives noted above. Two modules in logging and statistics are likely to get started in this current part. And another three are anticipated for efforts with KBpedia to come before we complete this <strong>CWPK</strong> series. Here is that updated directory structure, with these new modules noted in <span style=\"color: #e03e2d;\">red</span>:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>|-- PythonProject                                              \n",
    "          |-- Python                                               \n",
    "               |-- [Anaconda3 distribution]                        \n",
    "               |-- Lib               \n",
    "                    |-- site-packages                              \n",
    "                         |-- [many]                                    \n",
    "                         |-- cowpoke\n",
    "                              |-- __init__.py                                         \n",
    "                              |-- __main__.py\n",
    "                              |-- <span style=\"color: #e03e2d;\">analytics.py</span>        # anticipated new module\n",
    "                              |-- <span style=\"color: #e03e2d;\">build.py</span>            # in-progress module\n",
    "                              |-- <span style=\"color: #e03e2d;\">clean.py</span>            # in-progress module\n",
    "                              |-- config.py                                      \n",
    "                              |-- <span style=\"color: #e03e2d;\">embeddings.py</span>       # anticipated new module<br />                              |-- extract.py\n",
    "                              |-- <span style=\"color: #e03e2d;\">graphics.py</span>         # anticipated new module\n",
    "                              |-- <span style=\"color: #e03e2d;\">logs.py</span>             # likely new module\n",
    "                              |-- <span style=\"color: #e03e2d;\">stats.py</span>            # likely new module\n",
    "                              |-- <span style=\"color: #e03e2d;\">utils.py</span>            # likely new module  \n",
    "               |-- More  \n",
    "          |-- More \n",
    "</pre>\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 1: cowpoke Source Code ('Package') Directory Structure\n",
    "\n",
    "</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast, we need a different directory structure to host our KBpedia project, in which inputs to building KBpedia ('build_ins') are in one main branch, the results of a build ('targets) are in another main branch, 'extractions' from the current version in a third, and a fourth ('outputs') are the results of post-build use of the knowledge graph. Further, these four main branches are themselves listed under their respective version number. This design means individual versions may be readily zipped or shared on [GitHub](https://github.com/) in a versioned repository.\n",
    "\n",
    "(**NB**: The 'sandbox' directory below we have referenced many times in this **CWPK** series, and is unique to it. It houses some of the example starting files needed for this series. We will continue to use the 'sandbox' as one of our main directory options.)\n",
    "\n",
    "As of this current stage in our work, here then is how the project-related directory structures currently look for KBpedia:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>|-- PythonProject \n",
    "          |-- kbpedia\n",
    "               |-- sandbox\n",
    "               |-- v250\n",
    "                    |-- etc.\n",
    "               |-- v300\n",
    "                    |-- build_ins\n",
    "                         |-- classes\n",
    "                              |-- classes_struct.csv\n",
    "                              |-- classes_annot.csv\n",
    "                         |-- fixes\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                         |-- mappings\n",
    "                              |-- etc.\n",
    "                              |-- etc.\n",
    "                         |-- ontologies\n",
    "                              |-- kbpedia-reference-concepts.owl\n",
    "                              |-- kko.owl                              \n",
    "                         |-- properties\n",
    "                              |-- annotation_properties\n",
    "                              |-- data_properties\n",
    "                              |-- object_properties \n",
    "                         |-- stubs\n",
    "                              |-- kbpedia-reference-concepts.owl\n",
    "                              |-- kko.owl\n",
    "                         |-- typologies\n",
    "                              |-- ActionTypes\n",
    "                              |-- Agents\n",
    "                              |-- etc.\n",
    "                         |-- working\n",
    "                              |-- TBD\n",
    "                              |-- TBD                              \n",
    "                    |-- extractions\n",
    "                         |-- classes\n",
    "                              |-- classes_struct.csv\n",
    "                              |-- classes_annot.csv\n",
    "                         |-- mappings\n",
    "                              |-- etc.\n",
    "                              |-- etc.\n",
    "                         |-- properties\n",
    "                              |-- annotation_properties\n",
    "                              |-- data_properties\n",
    "                              |-- object_properties             \n",
    "                         |-- typologies\n",
    "                              |-- ActionTypes\n",
    "                              |-- Agents\n",
    "                              |-- etc.                                        \n",
    "                    |-- outputs\n",
    "                         |-- analytics\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                         |-- embeddings\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                         |-- training_sets\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                   |-- targets\n",
    "                         |-- logs\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                         |-- mappings\n",
    "                              |-- etc.\n",
    "                              |-- etc.\n",
    "                         |-- ontologies\n",
    "                              |-- kbpedia-reference-concepts.owl\n",
    "                              |-- kko.owl                              \n",
    "                         |-- stats\n",
    "                              |-- TBD\n",
    "                              |-- TBD\n",
    "                         |-- typologies\n",
    "                              |-- ActionTypes\n",
    "                              |-- Agents\n",
    "                              |-- etc.\n",
    "          |-- Python                                               \n",
    "               |-- etc.                      \n",
    "               |-- etc.             \n",
    "          |-- More \n",
    "</pre>\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 2: KBpedia Project Directory Structure (by version)\n",
    "\n",
    "</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice that nearly all 'extractions' categories are in the 'build' categories as well, reflecting the roundtrip nature of the design. Some of the output categories remain a bit speculative. This area is likely the one to see further refinement as we proceed.\n",
    "\n",
    "Some of the directories shown, such as 'analytics', 'embeddings', 'mappings', and 'training_sets' are placeholders for efforts to come. One directory, 'working', is a standard one we have adopted over the years to place all of the background working files (some of an intermediate nature leading to the formal build inputs) in one location. Thus, as we progress version-to-version, we can look to this directory to help remind us of the primary activities and changes that were integral to that particular build. When it comes time for a public release, we may remove some of these working or intermediate directories to what is published at GitHub, but we retain this record locally to help document our prior work.\n",
    "\n",
    "Overall, then, what we have is a build that begins with extractions from a prior build or starting raw files, with those files being modified within the 'build_ins' directory during development. Once the build input files are ready, the build processes are initiated to write the new knowledge graph to the 'targets' directory. Once the build has met all logic and build tests, it is then the source for new 'extractions' to be used in a subsequent build, or to conduct analysis or staging of results for other 'outputs'. *Figure 3* is an illustration of this workflow:\n",
    "\n",
    "<div class=\"center-image\">\n",
    "\n",
    "<img src=\"files/workflow-1.png\" title=\"Workflow by Version and Directory Structure\" height=\"800\" alt=\"Workflow by Version and Directory Structure\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"center-caption\">\n",
    "\n",
    "Figure 3: Workflow by Version and Directory Structure\n",
    "\n",
    "</div>\n",
    "\n",
    "A new version thus starts by copying the directory structure to a new version branch, and copying over the extractions and stubs from the prior version.\n",
    "\n",
    "We now have the framework for moving on to the next installment in our **CWPK** series, wherein we begin the return leg of the roundtrip, what is shown as the 'build_ins' &rarr; 'targets' path in *Figure 3*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}