glom_pythoncamp_2020_english.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Housekeeping - This cell shortens the traceback of error messages\n",
    "# unless you're on mybinder.org \n",
    "\n",
    "import json \n",
    "from pprint import pprint\n",
    "import sys\n",
    "ipython = get_ipython()\n",
    "unhide_traceback = None\n",
    "\n",
    "def hide_traceback(exc_tuple=None, filename=None, tb_offset=None,\n",
    "                   exception_only=False, running_compiled_code=False):\n",
    "    etype, value, tb = sys.exc_info()\n",
    "    return ipython._showtraceback(etype, value, ipython.InteractiveTB.get_exception_only(etype, value))\n",
    "\n",
    "if not unhide_traceback:\n",
    "    unhide_traceback = ipython.showtraceback\n",
    "\n",
    "ipython.showtraceback = hide_traceback   # or:  ipython.showtraceback = unhide_traceback\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Pythoncamp 2020 Session Martin Borus, Twitter: @mborus\n",
    "\n",
    "Translated - Session originally done in German\n",
    "\n",
    "# GLOM\n",
    "\n",
    "by Mahmoud Hashemi\n",
    "\n",
    "## \"If you have nested data, you need Glom!\"\n",
    "\n",
    "https://github.com/mahmoud/glom \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Main features of Glom\n",
    "\n",
    "- Path based access\n",
    "- Declarative data transformation \n",
    "- Easy to read helpful error messages\n",
    "- comes with debugging features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation for this notebook\n",
    "\n",
    "pip install glom  # current version 20.5.0\n",
    "\n",
    "pip install requests\n",
    "\n",
    "Full docs and tutorial at https://glom.readthedocs.io,\n",
    "some examples from this tutorial are included here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Normal Python\n",
    "\n",
    "These are some examples on how to access nested data with regular Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'a': {'b': {'c': 'd'}}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['a']['b']['c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2 = {'a': {'b': None}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data2['a']['b']['c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Glom \n",
    "\n",
    "Here's how you do the same thing in Glom, using the data above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import glom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, 'a.b.c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this here will product an error message\n",
    "glom(data2, 'a.b.c')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Glom is great for catching errors\n",
    "\n",
    "from glom import GlomError, PathAccessError\n",
    "\n",
    "try:\n",
    "    glom(data2, 'a.b.c')\n",
    "except PathAccessError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    glom(data2, 'a.b.c')\n",
    "except AttributeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    glom(data2, 'a.b.c')\n",
    "except GlomError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can use Glom with lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [1, [2 , 3, 4] , 3, 4 ,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, '1.0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can use Glom with objects !1!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyClass:\n",
    "    def __init__(self):\n",
    "        self.my_hallo = \"hello!\"\n",
    "        self.my_world = \"world!\"\n",
    "       \n",
    "myvar = MyClass()        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(myvar, 'my_world')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Target & Spec\n",
    "- the \"Target\" is the data, (list, dict, object)\n",
    "- the \"Spec\" is the wanted result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = {\n",
    "     'galaxy': {\n",
    "        'system': {\n",
    "            'planet': 'jupiter'\n",
    "         }\n",
    "    }\n",
    "}\n",
    "\n",
    "spec = 'galaxy.system.planet'\n",
    "\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = {\n",
    "    'system': {\n",
    "        'planets': [\n",
    "            {'name': 'earth', 'moons': 1},\n",
    "            {'name': 'jupiter', 'moons': 69}\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "spec = {\n",
    "     'names': ('system.planets', ['name']),\n",
    "     'moons': ('system.planets', ['moons'])\n",
    "}\n",
    "\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = {\n",
    "     'system': {\n",
    "         'planets': [\n",
    "            {\n",
    "                'name': 'earth',\n",
    "                'moons': [\n",
    "                    {'name': 'luna'}\n",
    "                ]\n",
    "            },\n",
    "            {\n",
    "                'name': 'jupiter',\n",
    "                'moons': [\n",
    "                    {'name': 'io'},\n",
    "                    {'name': 'europa'}\n",
    "                ]\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "spec = {\n",
    "    'planet_names': ('system.planets', ['name']),\n",
    "    'moon_names': ('system.planets', [('moons', ['name'],  )])\n",
    "}\n",
    "pprint(glom(target, spec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coalesce: Try the first, if it fails, try the next, and so on....\n",
    "\n",
    "from glom import Coalesce\n",
    "\n",
    "target = {\n",
    "     'system': {\n",
    "         'planets': [\n",
    "             {'name': 'earth', 'moons': 1},\n",
    "             {'name': 'jupiter', 'moons': 69}\n",
    "         ]\n",
    "     }\n",
    "}\n",
    "\n",
    "spec = {\n",
    "     'planets': (Coalesce('system.planets', 'system.dwarf_planets'), ['name']),\n",
    "     'moons': (Coalesce('system.planets', 'system.dwarf_planets'), ['moons'])\n",
    "}\n",
    "\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = {\n",
    "     'system': {\n",
    "         'dwarf_planets': [\n",
    "             {'name': 'pluto', 'moons': 5},\n",
    "             {'name': 'ceres', 'moons': 0},\n",
    "         ]\n",
    "     }\n",
    " }\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use python functions or lambdas inside the code\n",
    "\n",
    "target = {\n",
    "     'system': {\n",
    "         'planets': [\n",
    "             {'name': 'earth', 'moons': 1},\n",
    "             {'name': 'jupiter', 'moons': 69}\n",
    "         ]\n",
    "     }\n",
    "}\n",
    "\n",
    "print(glom(target, {'moon_count': ('system.planets', ['moons'], sum)}))\n",
    "    \n",
    "print(glom(target, {'moon_count': ('system.planets', ['moons'], lambda x: sum(x))}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# glom a class, subclass structure\n",
    "\n",
    "class MySubClass:\n",
    "    def __init__(self):\n",
    "        self.my_hey = \"Hey!\"\n",
    "\n",
    "class MyClass:\n",
    "    def __init__(self):\n",
    "        self.my_hello = \"hello!\"\n",
    "        self.my_world = \"world!\"\n",
    "        self.my_heylist = [MySubClass()] * 6\n",
    "\n",
    "        \n",
    "myvar = MyClass()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Iter\n",
    "spec = {'hello': 'my_hello', 'world': 'my_world', 'heylist': ('my_heylist', ['my_hey'])}\n",
    "\n",
    "glom(myvar, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Flatten and merge data\n",
    "\n",
    "Convert a list of lists & list of dicts to a list or dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Flatten, Merge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [[1, 2], [3], [4], [], [5]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, Flatten())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [{'hello': 'world'}, {'hello2': 'world2'}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, Merge())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. From the tutorial: Convert objects into output.\n",
    "\n",
    "This shows a more complicated example on how nested class objects are converted.\n",
    "Read the official tutorial on what happens here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom.tutorial import * "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contact = Contact('Julian',\n",
    "          emails=[Email(email='jlahey@svtp.info')],\n",
    "                  location='Canada')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contact.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contact.primary_email"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contact.add_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contact.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(Contact.objects.all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Contact.objects.all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# without Glom: You can't dump the object to json\n",
    "\n",
    "json.dumps(Contact.objects.all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Contact.objects.all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# have a look at the data type of the \"add_date\" - this is not dumpable\n",
    "target[0].add_date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: In this spec, the datetime and integers are converted to string.\n",
    "\n",
    "spec = {'results': [{'id': 'id',\n",
    "                      'name': 'name',\n",
    "                      'add_date': ('add_date', str),\n",
    "                      'emails': ('emails', [{'id': 'id',\n",
    "                                            'email': 'email',\n",
    "                                            'type': 'email_type'}]),\n",
    "                      'primary_email': Coalesce('primary_email.email', default=None),\n",
    "                      'pref_name': Coalesce('pref_name', 'name', skip='', default=''),\n",
    "                      'detail': Coalesce('company',\n",
    "                                         'location',\n",
    "                                         ('add_date.year', str),\n",
    "                                         skip='', default='')}]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with Glom: convert your target to something you can dump\n",
    "\n",
    "resp = glom(target, spec)\n",
    "print(json.dumps(resp, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the Flatten command from the previous section, used to get all emails in the result\n",
    "\n",
    "from glom import Flatten\n",
    "\n",
    "glom(resp, ('results', ['emails'],\n",
    "             Flatten(),\n",
    "             ['email'],\n",
    "           )\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. \"T\" - the Stunt-Double\n",
    "\n",
    "T in the spect is a stand in for anything at that position in the spec.\n",
    "It behaves like the python object it matches. \n",
    "\n",
    "In the first example,\n",
    "it behaves like a list, so T[0] is the first list object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import T, Flatten\n",
    "# from the data structure above, show the first entry\n",
    "glom(resp, ('results', T[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "T also works with Objects. Very nice for Namedtuples.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "HelloWorld = namedtuple('HelloWord','hello,world')\n",
    "hw = HelloWorld('hello!', 'world!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(hw, T.hello)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(hw, (T._asdict(), 'hello'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Let's loop with Iter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's prepare a list of the Seven Dwarfs\n",
    "\n",
    "from glom import glom, Iter\n",
    "\n",
    "target = [\n",
    "    'Happy',\n",
    "    'Sneezy',\n",
    "    'Sleepy',\n",
    "    'Doc',\n",
    "    'Bashful',\n",
    "    'Grumpy',\n",
    "    'Dopey'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iter returns a generator type\n",
    "spec = Iter()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the first dwarf only\n",
    "spec = Iter().first()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the generator into a list with \".all\"\n",
    "spec = Iter().all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Chunk the group in teams of 2\n",
    "spec = Iter().chunked(2, fill='Snow White').all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find any dwarf that's not doc with a lambda\n",
    "spec = Iter().filter(lambda x: x != 'Doc').all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hey, 3 dwarfs maximum!\n",
    "spec = Iter().limit(3).all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find dwarfs 2 - 4 in this 0-based list\n",
    "spec = Iter().slice(1, 4).all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only show dwarfs with unique first letters in the names\n",
    "spec = Iter().unique(T[0]).all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take dwarfs until Doc arrives\n",
    "spec = Iter().takewhile(lambda x: x != 'Doc').all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the first dwarf following Doc\n",
    "spec = Iter().dropwhile(lambda x: x != 'Doc').slice(1, 2).first()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the group around Doc\n",
    "spec = Iter().split('Doc').all() \n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a function on all\n",
    "spec = Iter().map(lambda x:x.lower()).all()\n",
    "glom(target, spec)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or instead of lambda, the stunt double T\n",
    "spec = Iter().map(T.upper()).all()\n",
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Literal - assign a fixed value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Literal\n",
    "spec = Iter({'Name': T, 'Size': Literal('Dwarflike')}).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(target, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. Data Driven\n",
    "\n",
    "What to do, if a dictionary key has the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import glom, T, Merge, Iter, Coalesce\n",
    "\n",
    "target = {\n",
    "    \"pluto\": {\"moons\": 6, \"population\": None},\n",
    "    \"venus\": {\"population\": {\"aliens\": 5}},\n",
    "    \"earth\": {\"moons\": 1, \"population\": {\"humans\": 7_700_000_000, \"aliens\": 1}},\n",
    "}\n",
    "\n",
    "spec = {\n",
    "    \"moons\": (\n",
    "           T.items(),\n",
    "           Iter({T[0]: (T[1], Coalesce(\"moons\", default=0))}),\n",
    "           Merge()\n",
    "    )\n",
    "}\n",
    "        \n",
    "glom(target, spec)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Add and remove values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'moons': {'pluto': 6, 'venus': 0, 'earth': 1}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Assign, Delete\n",
    "spec = Assign('moons.saturn', 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec = Delete('moons.earth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(data, spec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec = Delete('moons.mars', ignore_missing=True)\n",
    "\n",
    "glom(data, spec)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11. Scope / Let\n",
    "\n",
    "Scope allows you to collect data which would otherwise not be accessable from within a spec.\n",
    "\n",
    "This is nice if you want to move data inside each element to get a flat structure you can import into a pandas Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import S, glom, Assign, Spec\n",
    "\n",
    "target = {'date': '2020-04-01',\n",
    " 'location': 'A',\n",
    " 'items': [\n",
    "     {'name': 'A', 'id': 'A1'},\n",
    "     {'name': 'B', 'id': 'B1'},\n",
    "     {'name': 'C', 'id': 'C1'}\n",
    "]}\n",
    "\n",
    "spec = ('items', \n",
    "        [Assign('date', Spec(S['date']))], \n",
    "        [Assign('location', Spec(S['location']))]\n",
    "       )\n",
    "\n",
    "glom(target, spec, scope=target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same example with let:\n",
    "Let writes to the scope at runtime.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Fill\n",
    "from glom.core import Let\n",
    "\n",
    "spec = (\n",
    "    # Write outer value to scope\n",
    "    Let(base={\"date\": \"date\", \"location\": \"location\"}),\n",
    "    # select just the items\n",
    "    \"items\",\n",
    "    [\n",
    "        # for every element: add base to element\n",
    "        (Fill([T, S[\"base\"]]), Merge())\n",
    "    ]\n",
    ")\n",
    "\n",
    "glom(target, spec, scope=target)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12. XML\n",
    "\n",
    "Simple example on how to glom XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glom import Ref"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "etree2dicts = Ref('ElementTree',\n",
    "    {\"tag\": \"tag\", \n",
    "     \"text\": \"text\", \n",
    "     \"attrib\": \"attrib\", \n",
    "     \"children\": (iter, [Ref('ElementTree')])})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_text = \"\"\"<html>\n",
    "  <head>\n",
    "    <title>the title</title>\n",
    "  </head>\n",
    "  <body id=\"the-body\">\n",
    "    <p>A paragraph</p>\n",
    "  </body>\n",
    "</html>\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xml.etree import ElementTree\n",
    "etree = ElementTree.fromstring(html_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(etree, etree2dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 13. Out of time...\n",
    "\n",
    "other things worth taking a look at.\n",
    "\n",
    "- Path (A special Path notation for edge cases, where strings don't work)\n",
    "- Invoke / Call (Use a function)\n",
    "- Check -> CheckError (Check for Data consistency!)\n",
    "- Inspect (Debugging helper: Print out whats visible inside the spec)  \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14. Glom, Comments and the code formatter \"Black\"\n",
    "\n",
    "If you use Black on your code (https://pypi.org/project/black/), black may\n",
    "make your glom spec a lot harder to read.\n",
    "\n",
    "If your spec is somewhat complicated, add comments to it.\n",
    "\n",
    "The above used example still works with comments inside\n",
    "\n",
    "    spec = (\n",
    "        # Write outer value to scope\n",
    "        Let(base={\"date\": \"date\", \"location\": \"location\"}),\n",
    "\n",
    "        # select just the items\n",
    "        \"items\",\n",
    "        [\n",
    "            # for every element: add base to element\n",
    "            (Fill([T, S[\"base\"]]), Merge())\n",
    "        ]\n",
    "    )\n",
    "\n",
    "Without comments, Black will try to save space.\n",
    "\n",
    "    spec = (\n",
    "        Let(base={\"date\": \"date\", \"location\": \"location\"}),\n",
    "        \"items\",\n",
    "        [(Fill([T, S[\"base\"]]), Merge())],\n",
    "    )\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hard-ish Real-world-like practice exercises\n",
    "\n",
    "Note: These real two examples are somewhat difficult and show real world use cases where I \n",
    "used glom in production. You can solve them with everything shown in this tutorial.\n",
    "\n",
    "If you have problems, try a partial solution first and work towards the final result. \n",
    "\n",
    "A working example is provided for each exercise, not necessary the best way to do it...\n",
    "\n",
    "### Exercise 1:\n",
    "\n",
    "Extract data from an online booking result\n",
    "http://borus.de/pythoncamp/booking_example.json\n",
    "        \n",
    "Wanted result:\n",
    "\n",
    "    {'departure_harbor_outward_trip': 'DKHNB',\n",
    "     'departure_date_outward_trip': '2019-09-01',\n",
    "     'departure_time_outward_trip': '14:30',\n",
    "     'departure_harbor_return_trip': 'DELIS',\n",
    "     'departure_date_return_trip': '2019-10-02',\n",
    "     'departure_time_return_trip': '19:25',\n",
    "     'tickets': ['CAR', 'AD', 'CH']\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(r'http://borus.de/pythoncamp/booking_example.json')\n",
    "r.json()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write the spec here\n",
    "\n",
    "spec = \"...???\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(r.json(), spec)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For a solution, change this cell to \"code\" and run the code below.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import base64;print(base64.b64decode(b'CnNwZWMgPSB7CiAgICAgICdkZXBhcnR1cmVfaGFyYm9yX291dHdhcmRfdHJpcCc6ICgndHJpcHMnLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSXRlcigpLmZpbHRlcihsYW1iZGEgeDogeFsndHJpcF9wYXJ0J10gPT0gJ09VVCcpLmZpcnN0KCksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2RlcGFydHVyZV9oYXJib3InKSwKICAgICAgICdkZXBhcnR1cmVfZGF0ZV9vdXR3YXJkX3RyaXAnOiAoJ3RyaXBzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEl0ZXIoKS5maWx0ZXIobGFtYmRhIHg6IHhbJ3RyaXBfcGFydCddID09ICdPVVQnKS5maXJzdCgpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdkYXRlJyksCiAgICAgICAnZGVwYXJ0dXJlX3RpbWVfb3V0d2FyZF90cmlwJzogKCd0cmlwcycsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBJdGVyKCkuZmlsdGVyKGxhbWJkYSB4OiB4Wyd0cmlwX3BhcnQnXSA9PSAnT1VUJykuZmlyc3QoKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAndGltZScpLAogICAgICAgJ2RlcGFydHVyZV9oYXJib3JfcmV0dXJuX3RyaXAnOiAoJ3RyaXBzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEl0ZXIoKS5maWx0ZXIobGFtYmRhIHg6IHhbJ3RyaXBfcGFydCddID09ICdSRVQnKS5maXJzdCgpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdkZXBhcnR1cmVfaGFyYm9yJyksCiAgICAgICAnZGVwYXJ0dXJlX2RhdGVfcmV0dXJuX3RyaXAnOiAoJ3RyaXBzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEl0ZXIoKS5maWx0ZXIobGFtYmRhIHg6IHhbJ3RyaXBfcGFydCddID09ICdSRVQnKS5maXJzdCgpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdkYXRlJyksCiAgICAgICAnZGVwYXJ0dXJlX3RpbWVfcmV0dXJuX3RyaXAnOiAoJ3RyaXBzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEl0ZXIoKS5maWx0ZXIobGFtYmRhIHg6IHhbJ3RyaXBfcGFydCddID09ICdSRVQnKS5maXJzdCgpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICd0aW1lJyksCiAgICAgICAndGlja2V0cyc6ICAoJ3RpY2tldHMnLCBJdGVyKCd0eXBlJykuYWxsKCkpCiAgICAgICB9Cg==').decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "### Exercise 2: \n",
    "\n",
    "Take the pythoncamp room list at http://borus.de/pythoncamp/event.json Create a dictionary that has all room names in capital letters as a key and the url as a value,.\n",
    "\n",
    "Wanted result\n",
    "\n",
    "    {\n",
    "     'BERLIN': 'https://bbb01.pythoncamp.online/b/rei-hyz-cgm',\n",
    "     'FLIEGENDER ZIRKUS': 'https://bbb01.pythoncamp.online/b/rei-hyz-cgm',\n",
    "     'TOKIO': 'https://bbb01.pythoncamp.online/b/rei-hyz-cgm'\n",
    "     ...\n",
    "     }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(r'http://borus.de/pythoncamp/event.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write spec here\n",
    "spec = \"... ???\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glom(r.json(), spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For a solution, change this cell to \"code\" and run the code below.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import base64;print(base64.b64decode(b'c3BlYz0gKCdyb29tcycsIAogICAgICAgICAgIChJdGVyKHtUWyduYW1lJ10udXBwZXIoKTogVFsndXJsJ119KSwKICAgICAgICAgICBNZXJnZSgpICAgICAgCiAgICAgICAgICAgKQogICAgICAp'\n",
    ").decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tool - Exercise hider - Encodes a solution\n",
    "import base64\n",
    "solution = \"\"\"my solution\"\"\".encode('utf-8')\n",
    "print(base64.b64encode((solution)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}