examples/meetup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring RESTful APIs inside Notebooks\n",
    "\n",
    "## Presentation at [PyData Berlin Meetup, 2018-10-17](https://www.meetup.com/PyData-Berlin/events/255361308/)\n",
    "\n",
    "This notebook contains examples for how to explore RESTful APIs inside a [Jupyter](https://jupyter.org) notebook environment. It is somewhat inspired by [Postman](https://www.getpostman.com) and aims at providing more flexibility for customising the UI when entering input and rendering output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ways of Working RESTfully…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Curl and friends"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl https://xkcd.com/552/info.0.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -s https://xkcd.com/552/info.0.json | jq ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! http https://xkcd.com/552/info.0.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -s https://xkcd.com/552/info.0.json | jq .img | sed s/\\\"//g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -s -k $(curl -s 'https://xkcd.com/552/info.0.json' | jq .img | sed s/\\\"//g) --output xkcd.png\n",
    "! open xkcd.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following few examples require an HTTP API token, see https://wit.ai/docs/http/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "WIT_TOKEN = os.getenv('WIT_TOKEN')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -s -H 'Authorization: Bearer {WIT_TOKEN}' \\\n",
    "'https://api.wit.ai/message?q=silly+nonsense' | jq ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -s -H 'Authorization: Bearer {WIT_TOKEN}' \\\n",
    "'https://api.wit.ai/message?'\\\n",
    "'q=\"how+is+the+traffic+around+office+in+chicago\"' | jq ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl 'https://gist.githubusercontent.com/'\\\n",
    "'deeplook/71e9ded257cfc2d8e5e9/raw/f0cfbab5f266fcb8056e8aea046f1f222346b76b/2013.geojson'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "requests.get('http://python.org').headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Postman\n",
    "\n",
    "<img src=\"images/postman.png\" alt=\"Postman UI\" style=\"width: 90%;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enter ipyrest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyrest import Api\n",
    "Api('http://www.apple.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Arguments & Parameters, etc.\n",
    "\n",
    "This requires API access tokens as explained on https://developer.here.com/documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from ipyrest import Api\n",
    "url = 'https://1.{maptype}.maps.api.here.com/' \\\n",
    "      'maptile/2.1/{tiletype}/newest/{scheme}/{zoom}/{xtile}/{ytile}/{size}/{format}'\n",
    "args = dict(\n",
    "    maptype='traffic',\n",
    "    tiletype='traffictile',\n",
    "    scheme='normal.day',\n",
    "    zoom='11',\n",
    "    xtile='525',\n",
    "    ytile='761',\n",
    "    size='256',\n",
    "    format='png8',\n",
    ")\n",
    "params = dict(\n",
    "    app_id=os.getenv('HEREMAPS_APP_ID'), \n",
    "    app_code=os.getenv('HEREMAPS_APP_CODE'),\n",
    "    ppi='320',\n",
    ")\n",
    "Api(url, args=args, params=params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GeoJSON Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl 'https://gist.githubusercontent.com/deeplook/'\\\n",
    "'71e9ded257cfc2d8e5e9/raw/f0cfbab5f266fcb8056e8aea046f1f222346b76b/2013.geojson'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyrest import Api\n",
    "\n",
    "url = 'https://gist.githubusercontent.com/' \\\n",
    "      'deeplook/71e9ded257cfc2d8e5e9/raw/f0cfbab5f266fcb8056e8aea046f1f222346b76b/2013.geojson'\n",
    "\n",
    "def post(resp):\n",
    "    \"Post-process response content-type since gists seem to use text/plain.\"\n",
    "    resp.headers['Content-Type'] = 'application/vnd.geo+json'\n",
    "\n",
    "Api(url, post_process_resp=post)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple custom rendering view"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Textarea, Layout\n",
    "from ipyrest import Api\n",
    "from ipyrest.responseviews import ResponseView\n",
    "\n",
    "class HelloWorldView(ResponseView):\n",
    "    name = 'HelloWorld'\n",
    "    mimetype_pats = ['text/html']\n",
    "    def render(self, resp):\n",
    "        layout = Layout(width='100%', height='100px')\n",
    "        return Textarea(value='Hello World!', layout=layout)\n",
    "\n",
    "url = 'https://python.org'\n",
    "Api(url, additional_views=[HelloWorldView])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced rendering view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example requires API access tokens as explained on https://developer.here.com/documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from ipyleaflet import Map, Marker, Polyline\n",
    "from ipyrest import Api\n",
    "from ipyrest.responseviews import ResponseView, zoom_for_bbox\n",
    "\n",
    "class HereIsolinesView(ResponseView):\n",
    "    \"\"\"\n",
    "    A view for the isolines from the HERE Routing API, see\n",
    "    https://developer.here.com/documentation/routing/topics/request-isoline.html.\n",
    "    \"\"\"\n",
    "    name = 'HereIsolines'\n",
    "    mimetype_pats = ['application/json']\n",
    "    def render(self, resp):\n",
    "        obj = resp.json()\n",
    "        center = obj['response']['center']\n",
    "        lat, lon = center['latitude'], center['longitude']\n",
    "        m = Map(center=(lat, lon))\n",
    "        m += Marker(location=(lat, lon))\n",
    "        mins, maxs = [], []\n",
    "        for isoline in obj['response']['isoline']:\n",
    "            shape = isoline['component'][0]['shape']\n",
    "            path = [tuple(map(float, pos.split(','))) for pos in shape]\n",
    "            m += Polyline(locations=path, color='red', weight=2, fill=True)\n",
    "            mins.append(min(path))\n",
    "            maxs.append(max(path))\n",
    "        m.zoom = zoom_for_bbox(*min(mins), *max(maxs))\n",
    "        self.data = m\n",
    "        return m\n",
    "    \n",
    "url = 'https://isoline.route.api.here.com' \\\n",
    "      '/routing/7.2/calculateisoline.json'\n",
    "lat, lon = 52.5, 13.4\n",
    "params = dict(\n",
    "    app_id=os.getenv('HEREMAPS_APP_ID'), \n",
    "    app_code=os.getenv('HEREMAPS_APP_CODE'),\n",
    "    start=f'geo!{lat},{lon}',\n",
    "    mode='fastest;car;traffic:disabled',\n",
    "    rangetype='time', # time/distance\n",
    "    range='300,600',  # seconds/meters\n",
    "    resolution='20',  # meters\n",
    "    #departure='now', # 2018-07-04T17:00:00+02\n",
    ")\n",
    "Api(url, params=params, additional_views=[HereIsolinesView])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3D Output (Experimental)\n",
    "\n",
    "This might have issues on JupyterLab, but a [classic notebook](http://localhost:8888/notebooks/pysdk/docs/postman/postmanbox-meetup.ipynb) is fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyrest import Api\n",
    "\n",
    "def post(resp):\n",
    "    \"Post-proess response content-type since gists seem to have text/plain.\"\n",
    "    resp.headers['Content-Type'] = 'application/vnd.3d+txt'\n",
    "    \n",
    "url = 'https://gist.githubusercontent.com/deeplook/4568232f2ca9388942aab9830ceeb21f'\\\n",
    "      '/raw/782da3be33080ff7c7d2bd25b7d96b6bb455d570/sample_xyz_1000.txt'\n",
    "Api(url, post_process_resp=post)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an aside, see a crude examples of ipyvolume and pptk using Lidar data in another notebook, [x_point_clouds.ipynb](x_point_clouds.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XKCD Variants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyrest import Api\n",
    "\n",
    "url = 'https://xkcd.com/552/info.0.json'\n",
    "Api(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from ipyrest import Api\n",
    "\n",
    "url = requests.get('https://xkcd.com/552/info.0.json').json()['img']\n",
    "Api(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from ipywidgets import Image\n",
    "from ipyrest import Api\n",
    "from ipyrest.responseviews import ResponseView, builtin_view_classes\n",
    "\n",
    "class XKCDView(ResponseView):\n",
    "    \"Api rendering view for XKCD comics taken from XKCD JSON API.\"\n",
    "    name = 'XKCD'\n",
    "    mimetype_pats = ['application/json']\n",
    "    def render(self, resp):\n",
    "        return Image(value=requests.get(resp.json()['img']).content)\n",
    "\n",
    "url = 'https://xkcd.com/552/info.0.json'\n",
    "Api(url, views=builtin_view_classes + [XKCDView])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's the Latest Comic, BTW?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://xkcd.com/info.0.json'\n",
    "Api(url, views=builtin_view_classes + [XKCDView], click_send=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with [xkcd.com](https://xkcd.com)..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dynamic Views\n",
    "\n",
    "These examples require an API access token as explained on https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "TOKEN = os.environ['GITLAB_TOKEN']\n",
    "server = 'https://gitlab.com/api/v4'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyrest import Api, ResponseView\n",
    "\n",
    "Api(f'{server}/snippets', headers={'PRIVATE-TOKEN': TOKEN},\n",
    "    cassette_path='snippets.yaml',\n",
    "    timeout=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import HBox, VBox, Text, Button, Layout\n",
    "from ipyrest import Api\n",
    "from ipyrest.responseviews import ResponseView\n",
    "\n",
    "class DynamicSnippetView(ResponseView):\n",
    "    \"ResponseView showing snippet IDs with some decent 'UI'.\"\n",
    "    name = 'DynamicSnippetView'\n",
    "    mimetype_pats = ['application/json']\n",
    "    def render(self, resp):\n",
    "        return VBox([\n",
    "                    HBox([Text(str(snippet['id'])), \n",
    "                          Text(snippet['title']), \n",
    "                          Button(description='Delete (dummy)')]\n",
    "                        ) \n",
    "            for snippet in resp.json()])\n",
    "\n",
    "Api(f'{server}/snippets', headers={'PRIVATE-TOKEN': TOKEN},\n",
    "    cassette_path='snippets.yaml',\n",
    "    additional_views=[DynamicSnippetView], timeout=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skipped for Brevity\n",
    "\n",
    "- Protobuf example\n",
    "- Caching\n",
    "- Timeouts\n",
    "- HTTP methods other than GET\n",
    "- Accessing data between response views\n",
    "- Ipywidgets UI **executable w/o browser**\n",
    "- Ipywidgets UI **testable via pytest**\n",
    "- etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test-Suite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run local server for testing the API in a different process locally.\n",
    "import multiprocessing\n",
    "import subprocess\n",
    "import requests\n",
    "\n",
    "def start_test_server():\n",
    "    subprocess.check_call(['python', 'tests/api_server.py'])\n",
    "\n",
    "url = 'http://localhost:5000/'\n",
    "if requests.get(url).status_code >= 400:\n",
    "    print('Starting test server...')\n",
    "    multiprocessing.Process(target=start_test_server).start()\n",
    "else:\n",
    "    print('Test server is already running.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pytest -s -v ../tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ipyrest Under the Hood\n",
    "\n",
    "### Architecture & Documentation\n",
    "\n",
    "**\"It's in a flux!\"**\n",
    "\n",
    "### Dependencies\n",
    "\n",
    "- **requests**\n",
    "- **ipywidgets**\n",
    "- **timeout_decorator**\n",
    "- **typing**\n",
    "- ipyleaflet\n",
    "- ipyvolume\n",
    "- pandas\n",
    "- pytest\n",
    "- ...\n",
    "\n",
    "### To-do-to-do-to-do\n",
    "\n",
    "- make *true* widget\n",
    "- version\n",
    "- package\n",
    "- docker\n",
    "- binder\n",
    "- swagger?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap-Up\n",
    "\n",
    "- http://github.com/deeplook/ipyrest\n",
    "- Ipyrest is an _emerging_ tool for _exploring_ APIs inside _notebooks_.\n",
    "- Best used for _testing_ new APIs _quickly_ and _interactively_.\n",
    "- It _might_ be also useful for providing *executable API examples* online.\n",
    "- It is **#WIP #Alpha #Prototype**!\n",
    "- But already useful!\n",
    "- Try it!\n",
    "- Contribute!\n",
    "  - provide ResponseViews for your use case\n",
    "  - provide examples using other APIs\n",
    "  - push for missing ipywidgets issues like [this on styling](https://github.com/jupyter-widgets/ipywidgets/issues/2206)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q & A & T\n",
    "\n",
    "- Questions?\n",
    "- Answers!\n",
    "- Thank You!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}