dev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing for Intake\n",
    "\n",
    "Intake has been designed with a plugable structure, to allow adding new data sources and other features as easily as possible. Here we demonstrate making a new plugin to read data from the [GitHub API](https://developer.github.com/v3/), using a small and handy package that already exists, `pygithub`. We will access only one particular part of the API, as a simple example.\n",
    "\n",
    "The directory `intake-github` contains a full package to do the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/mdurant/code/intake-release-blog/intake-github\n"
     ]
    }
   ],
   "source": [
    "cd intake-github/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The contents are typical python package files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "./LICENSE\r\n",
      "./conda\r\n",
      "./conda/meta.yaml\r\n",
      "./setup.py\r\n",
      "./intake_github\r\n",
      "./intake_github/__init__.py\r\n",
      "./intake_github/__pycache__\r\n",
      "./intake_github/__pycache__/__init__.cpython-36.pyc\r\n"
     ]
    }
   ],
   "source": [
    "!find ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `setup.py` and `meta.yaml` are standard distutils and conda package information boilerplate, specifying dependencies and a little metadata. The `LICENSE` file is empty, but in general, all open-source projects should have an appropriate license."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example contains just one DataSource class, although it could have had several related source.\n",
    "\n",
    "The [source code](intake-github/intake_github/__init__.py) has very little code: a subclass of `DataSource` with the following class-level attributes:\n",
    "```python\n",
    "    container = 'dataframe'\n",
    "    version = '0.0.1'\n",
    "    partition_access = False\n",
    "    name = 'github_issues'\n",
    "```\n",
    "These name the new plugin, give it a version and an output data type (Pandas dataframe), and specify that the data will always be loaded in a single shot, without partitioning. Whether or not a dataframe is the right respesentation for this data is left for another discussion.\n",
    "\n",
    "The only real logic is in the read function:\n",
    "```python\n",
    "    def _get_partition(self, _):\n",
    "        from github import Github\n",
    "        import pandas as pd\n",
    "        gh = Github()\n",
    "        repo = gh.get_repo('%s/%s' % (self.organization, self.repo))\n",
    "\n",
    "        raw_data = repo.get_issues(**self.ghkwargs)\n",
    "        data = {}\n",
    "        for column in ['number', 'title', 'user', 'state', 'comments',\n",
    "                            'created_at', 'updated_at', 'body']:\n",
    "            # user is special case\n",
    "            if column == 'user':\n",
    "                data['user'] = [issue.user.login for issue in raw_data]\n",
    "            else:\n",
    "                data[column] = [getattr(issue, column) for issue in raw_data]\n",
    "\n",
    "        return pd.DataFrame(data)\n",
    "```\n",
    "This code is typical of the kind of coersion of data that might be required for different sources. Since we do not attempt partitioning here, the `read()` method is guaranteed to do the right thing.\n",
    "The external module `pygithub` takes care of actually formatting the HTTP REST calls to the github API. We simply pass on keyword arguments, so if a catalog author would like more fine control over what is downloaded for any specific instance of this data source, they would have to read the `pygithub` and/or GitHub API documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can install the package with `pip`, or we could build the conda package and install that instead. To distribute the package, it could be uploaded to PyPI, to a user conda channel on anaconda.org, or added as a feedstock to conda-forge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install with dependencies. Suppress text output. Can take a little while.\n",
    "!pip install -e . > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the class is in the top-level of the package, and the package name starts with `intake_`, it will be scanned when Intake is imported.\n",
    "Now the plugin automatically appears in the set of known plugins in the Intake registry, and an associated `intake.open_github_issues` function is created at import time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import intake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'github_issues' in intake.registry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If this automatic behaviour is not desired, then either catalogs would need to reference the plugin module/location explicitly in the `plugins:` section, or the package would need to insert its class into the Intake registry when *it* is imported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use our new plugin to load data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "source = intake.open_github_issues('ContinuumIO', 'intake')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'datashape': None,\n",
       " 'dtype': {'number': 'int',\n",
       "  'title': 'str',\n",
       "  'user': 'str',\n",
       "  'state': 'str',\n",
       "  'comments': 'int',\n",
       "  'created_at': 'datetime64[ns]',\n",
       "  'updated_at': 'datetime64[ns]',\n",
       "  'body': 'str'},\n",
       " 'shape': (None, 8),\n",
       " 'npartitions': 1,\n",
       " 'metadata': {}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source.discover()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = source.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>title</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>124</td>\n",
       "      <td>Fix failure to find appdir when there is no pr...</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>123</td>\n",
       "      <td>import intake causes UnboundLocalError: local ...</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>119</td>\n",
       "      <td>ENH: json-api support</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>118</td>\n",
       "      <td>[WIP] Caching</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>116</td>\n",
       "      <td>spark datasource support</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>112</td>\n",
       "      <td>implement arrays from server</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>108</td>\n",
       "      <td>make it work for normal pypi / Python-3 users ...</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>107</td>\n",
       "      <td>problem under pip</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>101</td>\n",
       "      <td>_generic_ noarch support</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>99</td>\n",
       "      <td>[WIP] Break up requirements into different fea...</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>97</td>\n",
       "      <td>Slim down requirements</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>95</td>\n",
       "      <td>Develop a jupyter-lab simple data browser pane</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>91</td>\n",
       "      <td>Compare to odo</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>84</td>\n",
       "      <td>Support for caching of data files</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>62</td>\n",
       "      <td>Making container schemas more clear</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>58</td>\n",
       "      <td>Plugin ideas</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>43</td>\n",
       "      <td>RFC: Future container types</td>\n",
       "      <td>open</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    number                                              title state\n",
       "0      124  Fix failure to find appdir when there is no pr...  open\n",
       "1      123  import intake causes UnboundLocalError: local ...  open\n",
       "2      119                              ENH: json-api support  open\n",
       "3      118                                      [WIP] Caching  open\n",
       "4      116                           spark datasource support  open\n",
       "5      112                       implement arrays from server  open\n",
       "6      108  make it work for normal pypi / Python-3 users ...  open\n",
       "7      107                                  problem under pip  open\n",
       "8      101                           _generic_ noarch support  open\n",
       "9       99  [WIP] Break up requirements into different fea...  open\n",
       "10      97                             Slim down requirements  open\n",
       "11      95     Develop a jupyter-lab simple data browser pane  open\n",
       "12      91                                     Compare to odo  open\n",
       "13      84                  Support for caching of data files  open\n",
       "14      62                Making container schemas more clear  open\n",
       "15      58                                       Plugin ideas  open\n",
       "16      43                        RFC: Future container types  open"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# currently open issues and PRs in the Intake repo\n",
    "out[['number', 'title', 'state']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **point** of doing this, is to enable data engineers to specify the plugin in their catalogs or when distributing data packages with conda. Notice that in YAML form, our data source references the name that we gave to the plugin in the source. If this plugin were not found on the system when such a catalog entry is loaded, an error would be raised. The second form also includes the optional plugin section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sources:\n",
      "  github_issues:\n",
      "    args:\n",
      "      organization: ContinuumIO\n",
      "      repo: intake\n",
      "    description: ''\n",
      "    driver: github_issues\n",
      "    metadata: {}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(source.yaml())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "plugins:\n",
      "  source:\n",
      "  - module: intake_github\n",
      "sources:\n",
      "  github_issues:\n",
      "    args:\n",
      "      organization: ContinuumIO\n",
      "      repo: intake\n",
      "    description: ''\n",
      "    driver: github_issues\n",
      "    metadata: {}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(source.yaml(True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "In a small amount of code, we have shown how to wrap a third-party data-loading package, to make it available to Intake as a plugin. The creation of the plugin enables inclusion of data-sets made from this kind of data possible in data catalogs. In this way, we build a uniform interface to access a wide variety of arbirtary data sources."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}