4. Accessing Data via APIs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: pip: command not found\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "!pip install -Uq requests\n",
    "import requests\n",
    "!mkdir -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using an API\n",
    "\n",
    "Learning outcomes\n",
    "\n",
    "- difference between API & webscraping\n",
    "- what JSON is (and why it's like a Python `dict`)\n",
    "- how to properly handle files in Python\n",
    "- what a REST API is\n",
    "- how to use the `requests` library\n",
    "\n",
    "## API versus web-scraping\n",
    "\n",
    "**Both are ways to sample data from the internet**\n",
    "\n",
    "API\n",
    "- structured\n",
    "- provided as a service (you are talking to a server via a REST API)\n",
    "- limited data / rate limits / paid / require auth (sometimes)\n",
    "- most will give back JSON (maybe XML or CSV)\n",
    "\n",
    "Web scraping\n",
    "- less structure\n",
    "- parsing HTML meant for your browser\n",
    "\n",
    "Neither is better than the other\n",
    "\n",
    "- API developer can limit what data is accessible through the API\n",
    "- API developer can not maintain the API\n",
    "- website page can change HTML structure\n",
    "- website page can have dynamic (Javascript) content that requires execution (usually done by the browser) before the correct HTML is available\n",
    "\n",
    "Much of the work in using an API is figuring out how to properly construct URLs for `GET` requests\n",
    "- requires looking at their documentation (& ideally a Python example!)\n",
    "\n",
    "## Where to find APIs\n",
    "\n",
    "- [ProgrammableWeb](https://www.programmableweb.com/apis/directory) - a collection of available API's\n",
    "- For the *Developer* or *For Developers* documentation on your favourite website\n",
    "- [public-apis/public-apis](https://github.com/public-apis/public-apis)\n",
    "\n",
    "## Using APIs\n",
    "\n",
    "Most APIs require authentication\n",
    "\n",
    "- so they API developer knows who you are\n",
    "- can charge you\n",
    "- can limit access\n",
    "- commonly via key or OAuth (both of which may be free)\n",
    "\n",
    "All the APIs we use here are unauthenticated - this is to avoid the time of you all signing up\n",
    "\n",
    "If your app requries authentication, it's usually done by passing in your credentials into the request (as a header)\n",
    "\n",
    "```python\n",
    "response = requests.get(url, auth=auth)\n",
    "```\n",
    "\n",
    "## JSON strings\n",
    "\n",
    "JSON (JavaScript Object Notation) is a:\n",
    "- lightweight data-interchange format (text)\n",
    "- easy for humans to read and write \n",
    "- easy for machines to parse and generate\n",
    "- based on key, value pairs\n",
    "\n",
    "You can think of the Python `dict` as JSON like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan-turing'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'name': 'alan-turing'}\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But true JSON is just a string (only text).  We can use `json.dumps` from the standard library to turn the `dict` into a JSON string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"name\": \"alan-turing\"}'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = json.dumps(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use `json.loads` to turn this string back into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan-turing'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.loads(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, reading & writing to files\n",
    "\n",
    "### Reading from a file\n",
    "\n",
    "We open files using the Python `open` builtin function, followed by a `read`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# Introduction to Python at Data Science Retreat \\n\\n## Topics:\\n1. Introduction: \\n  - Data Types \\n  - '"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./readme.md', 'r').read()[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note the `./path` - the `.` refers to the current working directory.  It's not straightforward to know where this is - is it where the notebook is, or where the notebook server is running?\n",
    "\n",
    "If we wanted to read the file as separate lines, we could use `readlines()` (note we would still need to manually strip off the `\\n` characters later)\n",
    "\n",
    "(Note `\\n` is used as a newline indicator in text files - you never see it because your editor interprets it as a line break :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['# Introduction to Python at Data Science Retreat \\n',\n",
       " '\\n',\n",
       " '## Topics:\\n',\n",
       " '1. Introduction: \\n',\n",
       " '  - Data Types \\n',\n",
       " '  - Conditionals \\n',\n",
       " '  - For Loops\\n',\n",
       " '  - PEP8 Coding Standards\\n']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./readme.md', 'r').readlines()[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `open`\n",
    "\n",
    "`open(path, mode)`\n",
    "\n",
    "Common values for the mode:\n",
    "- `r` read\n",
    "- `rb` read binary\n",
    "- `w+` write (`+` to create file if it doesn't exist)\n",
    "- `a` append\n",
    "\n",
    "Note there are options for both reading & writing - we actually use `open` for both reading & writing.\n",
    "\n",
    "We open a file using the Python builtin `open`, which is then followed by either a read or write stage\n",
    "\n",
    "- open the file\n",
    "- read the file OR write to the file\n",
    "\n",
    "Notice that the file is read in as a single string, with the newline character `\\n` separating lines\n",
    "\n",
    "- this is how all text files are structured\n",
    "- your editor does the line splitting for you\n",
    "\n",
    "### Writing to a file (without context management)\n",
    "\n",
    "We can write to a new file using the same `open` builtin\n",
    "\n",
    "- open the file\n",
    "- write to the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./data/output.data', 'w').write('We make this file to show how not to do it\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can do the same file write by explicitly assiging the file object to a variable (note the `a` to append)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('We make this file slightly differently to show how not to do it\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The issue with the code above is that we aren't closing the file - we can fix this by intentionally closing the file.  \n",
    "\n",
    "One way to do this is to use `.close()` when we are done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('This time we close the file manually\\n')\n",
    "fi.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This requires us to remember to close (also an additional line).\n",
    "\n",
    "### Reading files with context management\n",
    "\n",
    "The Pythonic way of handling opening & closing of files is context management:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./readme.md', 'r') as fi:\n",
    "    data = fi.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference with using context management (which essentially just means using the `with` statement) is that it closes the file automatically when finished writing/reading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to a file with context management\n",
    "\n",
    "Using the context management syntax, we can save our `data` dict as JSON, using `json.dump` to write the dict to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'name': 'alan turing'}\n",
    "with open('./data/output.json', 'w') as fi:\n",
    "    json.dump(data, fi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it worked by loading the file, here using `json.load` to load from the file object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan turing'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./data/output.json', 'r') as fi:\n",
    "    data = json.load(fi)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST APIs\n",
    "\n",
    "[REST - Wiki](https://en.wikipedia.org/wiki/Representational_state_transfer)\n",
    "\n",
    "REST is a set of constraints that allow **stateless communication of text data on the internet**\n",
    "\n",
    "- REST = REpresentational State Transfer\n",
    "- API = Application Programming Interface\n",
    "\n",
    "REST\n",
    "- communication of resources (located at URLs / URIs)\n",
    "- requests for a resource are responded to with a text payload (HTML, JSON etc)\n",
    "- these requests are made using HTTP (determines how messages are formatted, what actions (methods) can be taken)\n",
    "- common HTTP methods are `GET` and `POST`\n",
    "\n",
    "HTTP methods\n",
    "- GET - retrieve information about the REST API resource\n",
    "- POST - create a REST API resource\n",
    "- PUT - update a REST API resource\n",
    "- DELETE - delete a REST API resource or related component\n",
    "\n",
    "RESTful APIs enable you to develop any kind of web application having all possible CRUD (create, retrieve, update, delete) operations\n",
    "\n",
    "- can do anything we would want to do with a database\n",
    "\n",
    "*Further reading*\n",
    "- [Web Architecture 101](https://medium.com/storyblocks-engineering/web-architecture-101-a3224e126947) for more detail on how the web works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - sunrise API\n",
    "\n",
    "Docs - https://sunrise-sunset.org/api\n",
    "\n",
    "First we need to form the url\n",
    "- use `?` to separate the API server name from the parameters for our request\n",
    "- use `&` to separate the parameters from each other\n",
    "- use `+` instead of space in the parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'results': {'sunrise': '7:13:04 AM',\n",
       "  'sunset': '3:12:11 PM',\n",
       "  'solar_noon': '11:12:38 AM',\n",
       "  'day_length': '07:59:07',\n",
       "  'civil_twilight_begin': '6:34:38 AM',\n",
       "  'civil_twilight_end': '3:50:38 PM',\n",
       "  'nautical_twilight_begin': '5:50:58 AM',\n",
       "  'nautical_twilight_end': '4:34:17 PM',\n",
       "  'astronomical_twilight_begin': '5:09:37 AM',\n",
       "  'astronomical_twilight_end': '5:15:38 PM'},\n",
       " 'status': 'OK'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting sunrise & sunset for Berlin today\n",
    "res = requests.get(\"https://api.sunrise-sunset.org/json?lat=52.5200&lng=13.4050\")\n",
    "data = res.json()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This response is JSON - `requests.json()` turns it into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common to have a top level heirarchy to dig through to get the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['results', 'status'])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the interesting stuff is in `results`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sunrise': '7:13:04 AM',\n",
       " 'sunset': '3:12:11 PM',\n",
       " 'solar_noon': '11:12:38 AM',\n",
       " 'day_length': '07:59:07',\n",
       " 'civil_twilight_begin': '6:34:38 AM',\n",
       " 'civil_twilight_end': '3:50:38 PM',\n",
       " 'nautical_twilight_begin': '5:50:58 AM',\n",
       " 'nautical_twilight_end': '4:34:17 PM',\n",
       " 'astronomical_twilight_begin': '5:09:37 AM',\n",
       " 'astronomical_twilight_end': '5:15:38 PM'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: \n",
    "Save the parameter `sunrise` as a string value by itself in a variable called `sunrise_time`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - Chronicling America API\n",
    "\n",
    "Docs - https://chroniclingamerica.loc.gov/about/api/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://chroniclingamerica.loc.gov/search/pages/results/?proxtext=germany&format=json'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the `requests` HTTP library to perform a `GET` request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP response\n",
    "\n",
    "What we recieved above is an *HTTP response**\n",
    "\n",
    "[HTTP Response - Wiki](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Response_message)\n",
    "\n",
    "The response message consists of the following:\n",
    "\n",
    "- a status line which includes the status code and reason message (e.g., HTTP/1.1 200 OK, which indicates that the client's request succeeded)\n",
    "- response header fields (e.g., Content-Type: text/html)\n",
    "- an optional message body\n",
    "\n",
    "## What can we do with this HTTP response in `requests`?\n",
    "\n",
    "The Python builtin `dir` gives us all the attributes & methods of a Python object.\n",
    "\n",
    "This also includes all the `__` dunder (literally double-under) methods) - which we filter out using a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['_content',\n",
       " '_content_consumed',\n",
       " '_next',\n",
       " 'apparent_encoding',\n",
       " 'close',\n",
       " 'connection',\n",
       " 'content',\n",
       " 'cookies',\n",
       " 'elapsed',\n",
       " 'encoding',\n",
       " 'headers',\n",
       " 'history',\n",
       " 'is_permanent_redirect',\n",
       " 'is_redirect',\n",
       " 'iter_content',\n",
       " 'iter_lines',\n",
       " 'json',\n",
       " 'links',\n",
       " 'next',\n",
       " 'ok',\n",
       " 'raise_for_status',\n",
       " 'raw',\n",
       " 'reason',\n",
       " 'request',\n",
       " 'status_code',\n",
       " 'text',\n",
       " 'url']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[item for item in dir(response) if '__' not in item]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the HTTP status code (used to communicate things like everything OK (200), stop making requests etc - see [List of HTTP status codes - Wiki](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTTP response headers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Date': 'Fri, 07 Jan 2022 10:52:27 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Last-Modified': 'Fri, 07 Jan 2022 10:52:27 GMT', 'Expires': 'Sat, 08 Jan 2022 10:52:27 GMT', 'Cache-Control': 's-maxage=86400, public, max-age=86400', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'X-requested-with', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Varnish': '375690622', 'Via': '1.1 varnish (Varnish/5.2)', 'CF-Cache-Status': 'MISS', 'Expect-CT': 'max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"', 'Server': 'cloudflare', 'CF-RAY': '6c9ca2e14b0d4528-TXL'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the response body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"totalItems\": 2334262, \"endIndex\": 20, \"startIndex\": 1, \"itemsPerPage\": 20, \"items\": [{\"sequence\": 11, \"county\": [\"Wayne\"], \"edition\": \"Home ed.\", \"frequency\": \"Daily\", \"id\": \"/lccn/sn88063294/1944-09-12/ed-1/seq-11/\", \"subject\": [\"Detroit (Mich.)--Newspapers.\", \"Michigan--Detroit.--fast--(OCoLC)fst01205010\", \"Michigan--Wayne County.--fast--(OCoLC)fst01206628\", \"Wayne County (Mich.)--Newspapers.\"], \"city\": [\"Detroit\"], \"date\": \"19440912\", \"title\": \"Detroit evening times.\", \"end_year\": 1958, \"note\": [\"Archived issues are available in digital format from the Library of Congress Chronicling America online collection.\", \"Description based on: 22nd yr., no. 27 (Nov. 1, 1921).\", \"Issue called: Michigan centennial ed., May 2, 1937.\", \"Most Sunday issues for Aug. 6, 1922-Jan. 12, 1958 called: Detroit Sunday times.\", \"Some issues include supplement called: Goodfellows Special/Old Newsboys ed., published in Dec. of each year <Dce. 16, 1926-Dec. 1948>.\", \"The word \\\\\"evening\\\\\" appears within orna'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.text[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "data = json.loads(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'totalItems': 2334262,\n",
       " 'endIndex': 20,\n",
       " 'startIndex': 1,\n",
       " 'itemsPerPage': 20,\n",
       " 'items': [{'sequence': 11,\n",
       "   'county': ['Wayne'],\n",
       "   'edition': 'Home ed.',\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn88063294/1944-09-12/ed-1/seq-11/',\n",
       "   'subject': ['Detroit (Mich.)--Newspapers.',\n",
       "    'Michigan--Detroit.--fast--(OCoLC)fst01205010',\n",
       "    'Michigan--Wayne County.--fast--(OCoLC)fst01206628',\n",
       "    'Wayne County (Mich.)--Newspapers.'],\n",
       "   'city': ['Detroit'],\n",
       "   'date': '19440912',\n",
       "   'title': 'Detroit evening times.',\n",
       "   'end_year': 1958,\n",
       "   'note': ['Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    'Description based on: 22nd yr., no. 27 (Nov. 1, 1921).',\n",
       "    'Issue called: Michigan centennial ed., May 2, 1937.',\n",
       "    'Most Sunday issues for Aug. 6, 1922-Jan. 12, 1958 called: Detroit Sunday times.',\n",
       "    'Some issues include supplement called: Goodfellows Special/Old Newsboys ed., published in Dec. of each year <Dce. 16, 1926-Dec. 1948>.',\n",
       "    'The word \"evening\" appears within ornament.'],\n",
       "   'state': ['Michigan'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Detroit, Mich',\n",
       "   'start_year': 1921,\n",
       "   'edition_label': 'REDLINE',\n",
       "   'publisher': '[s.n.]',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Detroit Sunday times', 'Detroit times'],\n",
       "   'lccn': 'sn88063294',\n",
       "   'country': 'Michigan',\n",
       "   'ocr_eng': 'Where the Battle of Germany Will Be Fought\\n|DETROir.*KfTIJvIES| -Sea',\n",
       "   'batch': 'mimtptc_holly_ver03',\n",
       "   'title_normal': 'detroit evening times.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn88063294/1944-09-12/ed-1/seq-11.json',\n",
       "   'place': ['Michigan--Wayne--Detroit'],\n",
       "   'page': ''},\n",
       "  {'sequence': 16,\n",
       "   'county': ['San Francisco'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn85066387/1908-11-08/ed-1/seq-16/',\n",
       "   'subject': ['California--San Francisco Bay Area.--fast--(OCoLC)fst01242397',\n",
       "    'California--San Francisco.--fast--(OCoLC)fst01204481',\n",
       "    'San Francisco (Calif.)--Newspapers.',\n",
       "    'San Francisco Bay Area (Calif.)--Newspapers.'],\n",
       "   'city': ['San Francisco'],\n",
       "   'date': '19081108',\n",
       "   'title': 'The San Francisco call. [volume]',\n",
       "   'end_year': 1913,\n",
       "   'note': ['\"San Francisco\" appears above, and later across, masthead ornament.',\n",
       "    'Also issued online.',\n",
       "    'Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    'Issued with a joint ed. of the San Francisco chronicle and the San Francisco examiner on the day after the San Francisco earthquake, Apr. 19, 1906.',\n",
       "    'Master negatives are available for duplication from:',\n",
       "    'Publishers: Charles M. Shortridge, <1896>; John D. Spreckles, <1899>.'],\n",
       "   'state': ['California'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'San Francisco [Calif.]',\n",
       "   'start_year': 1895,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Charles M. Shortridge',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Call', 'Call-chronicle-examiner', 'Sunday call'],\n",
       "   'lccn': 'sn85066387',\n",
       "   'country': 'California',\n",
       "   'ocr_eng': 'JACK— he w.45 11111^^ Germany,\\nI^ll/ riFSIIOVu WaIXS jp',\n",
       "   'batch': 'curiv_llano_ver01',\n",
       "   'title_normal': 'san francisco call.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85066387/1908-11-08/ed-1/seq-16.json',\n",
       "   'place': ['California--San Francisco--San Francisco'],\n",
       "   'page': '16'},\n",
       "  {'sequence': 40,\n",
       "   'county': [None],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn85038615/1905-10-08/ed-1/seq-40/',\n",
       "   'subject': ['Richmond (Va.)--Newspapers.',\n",
       "    'Virginia--Richmond.--fast--(OCoLC)fst01205345'],\n",
       "   'city': ['Richmond'],\n",
       "   'date': '19051008',\n",
       "   'title': 'The times dispatch. [volume]',\n",
       "   'end_year': 1914,\n",
       "   'note': ['Also issued on microfilm from Bell & Howell, Micro Photo Div.',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.'],\n",
       "   'state': ['Virginia'],\n",
       "   'section_label': 'MAGAZINE SECTION',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Richmond, Va.',\n",
       "   'start_year': 1903,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Times-Dispatch Co.',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn85038615',\n",
       "   'country': 'Virginia',\n",
       "   'ocr_eng': 'njuvv/w*r\\nHE GETS SQUARE!\\nHOCH DER7 HOOLIGANS !\\nTHEY REACH GERMANY ALRETTY. YET I',\n",
       "   'batch': 'vi_berea_ver01',\n",
       "   'title_normal': 'times dispatch.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85038615/1905-10-08/ed-1/seq-40.json',\n",
       "   'place': ['Virginia--Richmond'],\n",
       "   'page': ''},\n",
       "  {'sequence': 10,\n",
       "   'county': ['Jefferson'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Monday)',\n",
       "   'id': '/lccn/sn89051168/1919-04-15/ed-1/seq-10/',\n",
       "   'subject': ['Jefferson County (Ark.)--Newspapers.',\n",
       "    'Pine Bluff (Ark.)--Newspapers.'],\n",
       "   'city': ['Pine Bluff'],\n",
       "   'date': '19190415',\n",
       "   'title': 'Pine Bluff daily graphic.',\n",
       "   'end_year': 1942,\n",
       "   'note': ['On Sunday published as: Sunday morning graphic, Oct. 13, 1895-Dec. 5, 1897.',\n",
       "    'Published as: Graphic, Aug. 22, 1941-Mar. 11, 1942.',\n",
       "    'Semiweekly eds.: Pine Bluff graphic, Oct. 9, 1895, and: Semi=weekly graphic, Oct. 12, 1895-Nov. 26, 1902, and: Weekly graphic (Pine Bluff, Ark.), Nov. 29, 1902-<May 28, 1904>, and: Pine Bluff weekly graphic (Pine Bluff, Ark.: 1904), <June 4, 1904-Dec. 29, 1906>.',\n",
       "    'Weekly eds.: Pine Bluff graphic, Oct. 2, 1893-Oct. 4, 1895, and: Pine Bluff weekly graphic (Pine Bluff, Ark. : 1904), <Apr. 6-Sept. 28, 1907>.'],\n",
       "   'state': ['Arkansas'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Pine Bluff, Ark.',\n",
       "   'start_year': 1893,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Graphic Print. Co.',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Graphic', 'Sunday morning graphic'],\n",
       "   'lccn': 'sn89051168',\n",
       "   'country': 'Arkansas',\n",
       "   'ocr_eng': '1?: r\\nSTRAIGHT FROM GERMANY\\nAn advertisement originated and produced\\nfor the Victory Liberty Loan by members of\\nthe American Expeditionary Force.\\nspawn) himiwr-wo\\nUUUUUUC',\n",
       "   'batch': 'arhi_littlefeat_ver01',\n",
       "   'title_normal': 'pine bluff daily graphic.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn89051168/1919-04-15/ed-1/seq-10.json',\n",
       "   'place': ['Arkansas--Jefferson--Pine Bluff'],\n",
       "   'page': 'TEN'},\n",
       "  {'sequence': 60,\n",
       "   'county': ['Jefferson'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn85038485/1915-11-28/ed-1/seq-60/',\n",
       "   'subject': ['Alabama--Birmingham.--fast--(OCoLC)fst01204958',\n",
       "    'Birmingham (Ala.)--Newspapers.'],\n",
       "   'city': ['Birmingham'],\n",
       "   'date': '19151128',\n",
       "   'title': 'The Birmingham age-herald. [volume]',\n",
       "   'end_year': 1950,\n",
       "   'note': ['Also issued on microfilm from Bell & Howell, Micro Photo Div.; the Library of Congress, Photoduplication Service.',\n",
       "    'Also published in a weekly ed.',\n",
       "    'Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    \"Publication suspended with July 12, 1945 issue due to a printers' strike; resumed publication with Aug. 17, 1945 issue.\"],\n",
       "   'state': ['Alabama'],\n",
       "   'section_label': 'ROTOGRAVURE SECTION',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Birmingham, Ala.',\n",
       "   'start_year': 1902,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Age-Herald Co.',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Age-herald', 'Birmingham news, the Birmingham age-herald'],\n",
       "   'lccn': 'sn85038485',\n",
       "   'country': 'Alabama',\n",
       "   'ocr_eng': 'FIGHTERS THAT CANNOT FIGHT: FRENCHMEN IN GERMANY, GERMANS IN FRANCE\\nGerman Soldiers Captured by the French in the Recent Great Drive of the Allies in the Champagne District in France and Now Held as Prisoners of War in France.\\nFrench Priaonen of War Captured by the Gennana and Held at Konigabruck, Germany. Some of the Inatrumenta, aa May Be Seen, Have Been Made by the Men. At the End of Auguat, Germany Hald About 270,000 Frenchmen Priaonera.',\n",
       "   'batch': 'au_ayler_ver01',\n",
       "   'title_normal': 'birmingham age-herald.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85038485/1915-11-28/ed-1/seq-60.json',\n",
       "   'place': ['Alabama--Jefferson--Birmingham'],\n",
       "   'page': ''},\n",
       "  {'sequence': 5,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1916-11-18/ed-1/seq-5/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19161118',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'LAST EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': 'NOTHING DOING!\\n0 0\\nBRITON\\'S CRY VENGEANCE AS\\n\\' GERMANS DEPORT BELGIANS\\nLondon, Nov.\" 18. England is to\\nday seeking some means of reprisal\\nagainst\\' Germany for \\'\"enslavement\"\\nof Belgium\\'s men. Not since the ex\\necution of Miss Edith Cavell has\\nthere been such a wave of popular in-\\'\\ndignation as that which is sweeping\\nthe country over fresh details of\\nTeutonic plan of giving employment\\nto Belgian.civil population by deport\\ning them to Germany.\\nThree hundred thousand pale Bel\\ngians above age of 17 are affected\\nby \"employment\" orders from Ber\\nlin. So far at least 45,000 of these\\nhave been transported from their\\nhomes to Germany.',\n",
       "   'batch': 'iune_india_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1916-11-18/ed-1/seq-5.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 108,\n",
       "   'county': [None],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn83045462/1934-01-28/ed-1/seq-108/',\n",
       "   'subject': ['Washington (D.C.)--fast--(OCoLC)fst01204505',\n",
       "    'Washington (D.C.)--Newspapers.'],\n",
       "   'city': ['Washington'],\n",
       "   'date': '19340128',\n",
       "   'title': 'Evening star. [volume]',\n",
       "   'end_year': 1972,\n",
       "   'note': ['\"From April 25 through May 24, 1861 one sheet issues were published intermittently owing to scarcity of paper.\" Cf. Library of Congress, Photoduplication Service.',\n",
       "    'Also issued on microfilm from Microfilming Corp. of America and the Library of Congress, Photoduplication Service.',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Publisher varies: Noyes, Baker & Co., <1867>; Evening Star Newspaper Co., <1868->',\n",
       "    \"Suspended Jan. 1-6, 1971 because of a machinists' strike.\"],\n",
       "   'state': ['District of Columbia'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Washington, D.C.',\n",
       "   'start_year': 1854,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'W.D. Wallach & Hope',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Star', 'Sunday star'],\n",
       "   'lccn': 'sn83045462',\n",
       "   'country': 'District of Columbia',\n",
       "   'ocr_eng': '“A nation of shopkeepers.” British soldiers traveling toward the front line.\\nA section of the Belgian town of Malines after bombardment by the Germans.\\nA.\\nThese Frenchmen reached Germany—but as prisoners.',\n",
       "   'batch': 'dlc_1duchamp_ver01',\n",
       "   'title_normal': 'evening star.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045462/1934-01-28/ed-1/seq-108.json',\n",
       "   'place': ['District of Columbia--Washington'],\n",
       "   'page': ''},\n",
       "  {'sequence': 39,\n",
       "   'county': [None],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn85038615/1903-03-01/ed-1/seq-39/',\n",
       "   'subject': ['Richmond (Va.)--Newspapers.',\n",
       "    'Virginia--Richmond.--fast--(OCoLC)fst01205345'],\n",
       "   'city': ['Richmond'],\n",
       "   'date': '19030301',\n",
       "   'title': 'The times dispatch. [volume]',\n",
       "   'end_year': 1914,\n",
       "   'note': ['Also issued on microfilm from Bell & Howell, Micro Photo Div.',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.'],\n",
       "   'state': ['Virginia'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Richmond, Va.',\n",
       "   'start_year': 1903,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Times-Dispatch Co.',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn85038615',\n",
       "   'country': 'Virginia',\n",
       "   'ocr_eng': 'Alphonse and Gaston in Germany.\\nU/iOtr ffiriend Xeon &inds a fioar\\nGjub m the SSlack Jfiorest.\\nCopyright, 1608, by W. B.Hearal.\\nDELIArAND MAGGIE?KATZENJAMMER KIDS.',\n",
       "   'batch': 'vi_gualala_ver01',\n",
       "   'title_normal': 'times dispatch.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85038615/1903-03-01/ed-1/seq-39.json',\n",
       "   'place': ['Virginia--Richmond'],\n",
       "   'page': '11'},\n",
       "  {'sequence': 1,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1917-02-03/ed-1/seq-1/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19170203',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'LAST EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': \"2 C LAST EDITION 2 O\\ni\\n- FIRE HORROR REPORT GIVEN HOYNE\\nTHE DAY BOOK\\nAn Adless Newspaper, Daily Except Sunday\\nVOL. 6, NO. 108 Chicago, Saturday, February 3, 1917 3!\\nI. S. BREAKS RELATIONS\\nWITH GERMANY-VON\\nBERNSTORFF OUSTED\\nGerard Is Recalled Swiss Minister Takes Charge of\\nTeuton Affairs President Addresses Congress\\nMay Avoid Actual War Unless Germany\\nSink U. S. Ship-Navy Yards Guarded.\\n' Washington, Feb. 3. President Wilson told congress\\nthis afternoon he had instructed Secretary of State Lan\\nsing to break off diplomatic relations with Germany.\\nAmbassador Gerard and all American consuls have\\nbeen recalled from Germany.\\nGerman Ambassador Von Bernstotff was handed his\\npassports at 1 :57 this afternoon\\nVi-mr-i mitvmmgwmmtmmitmi\",\n",
       "   'batch': 'iune_india_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1917-02-03/ed-1/seq-1.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 15,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1917-02-12/ed-1/seq-15/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19170212',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'LAST EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': \"IN WHITE HOUSE TODAY\\nThe president of Germany's food j sitates the eating of turnips instead\\nregulation board frankly announces Put on turnips we'd yell for peace,\\nthat the shortage of potatoes neces- ourselves, mighty prompt.\",\n",
       "   'batch': 'iune_india_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1917-02-12/ed-1/seq-15.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 24,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1916-01-31/ed-1/seq-24/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19160131',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'NOON EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': 'fnm9q\"ffPVMPPW7PIMPPJP\\nHIS COUNTRY IS IMPERILED\\nM\\nKing Constantine of Greece, Who Is Angry with Allies, but Doesn\\'t Want\\nto Aid Germany.\\nottMtfiftitftfMfttftftflfciiaiftftAiili',\n",
       "   'batch': 'iune_hotel_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1916-01-31/ed-1/seq-24.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 29,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1915-07-03/ed-2/seq-29/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19150703',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'LAST EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': 'WHERE GERMANS GET MUNITIONS OF WAR\\nScenes of and near Essen, Germany, where the Krupps are making\\nguns night and day. At top is a coal mine near Essen, which furnish\\nlifciiiK -j mumm a',\n",
       "   'batch': 'iune_hotel_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1915-07-03/ed-2/seq-29.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 94,\n",
       "   'county': [None],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn83045462/1934-04-22/ed-1/seq-94/',\n",
       "   'subject': ['Washington (D.C.)--fast--(OCoLC)fst01204505',\n",
       "    'Washington (D.C.)--Newspapers.'],\n",
       "   'city': ['Washington'],\n",
       "   'date': '19340422',\n",
       "   'title': 'Evening star. [volume]',\n",
       "   'end_year': 1972,\n",
       "   'note': ['\"From April 25 through May 24, 1861 one sheet issues were published intermittently owing to scarcity of paper.\" Cf. Library of Congress, Photoduplication Service.',\n",
       "    'Also issued on microfilm from Microfilming Corp. of America and the Library of Congress, Photoduplication Service.',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Publisher varies: Noyes, Baker & Co., <1867>; Evening Star Newspaper Co., <1868->',\n",
       "    \"Suspended Jan. 1-6, 1971 because of a machinists' strike.\"],\n",
       "   'state': ['District of Columbia'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Washington, D.C.',\n",
       "   'start_year': 1854,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'W.D. Wallach & Hope',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Star', 'Sunday star'],\n",
       "   'lccn': 'sn83045462',\n",
       "   'country': 'District of Columbia',\n",
       "   'ocr_eng': 'AMERICANS KILLED NOVEMBER 11, 1918. The graves were located near Sedan. I\\nAMERICANS MARCH INTO GERMANY. The first of the United States Army to cross the Rhine.\\nWASTE. German war material destroyed under the terms of the Armistice. ™',\n",
       "   'batch': 'dlc_1ernst_ver01',\n",
       "   'title_normal': 'evening star.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045462/1934-04-22/ed-1/seq-94.json',\n",
       "   'place': ['District of Columbia--Washington'],\n",
       "   'page': ''},\n",
       "  {'sequence': 57,\n",
       "   'county': ['New York'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily',\n",
       "   'id': '/lccn/sn83030214/1915-08-01/ed-1/seq-57/',\n",
       "   'subject': ['New York (N.Y.)--Newspapers.',\n",
       "    'New York (State)--New York County.--fast--(OCoLC)fst01234953',\n",
       "    'New York (State)--New York.--fast--(OCoLC)fst01204333',\n",
       "    'New York County (N.Y.)--Newspapers.'],\n",
       "   'city': ['New York'],\n",
       "   'date': '19150801',\n",
       "   'title': 'New-York tribune. [volume]',\n",
       "   'end_year': 1924,\n",
       "   'note': ['Also available in digital format on the Library of Congress website.',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm from University Microfilms International, and Recordak.',\n",
       "    'Evening ed.: Evening edition of the tribune, 1866.',\n",
       "    'Merged with: New York herald (New York, N.Y. : 1920); to form: New York herald, New York tribune.',\n",
       "    'Semiweekly ed.: New-York tribune (New York, N.Y. : 1866 : Semiweekly), 1866-<1899>.',\n",
       "    'Triweekly eds.: New-York tri-weekly tribune, <1900>-1903, and: New York tribune (New York, N.Y. : 1903), 1903-<1909>.',\n",
       "    'Weekly ed.: New-York tribune (New York, N.Y. : 1866 : Weekly), 1866-<1906>.'],\n",
       "   'state': ['New York'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'New York [N.Y.]',\n",
       "   'start_year': 1866,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'New York Tribune',\n",
       "   'language': ['English'],\n",
       "   'alt_title': ['Combined New York morning newspapers',\n",
       "    'Combined New York Sunday newspapers',\n",
       "    'New-York daily tribune'],\n",
       "   'lccn': 'sn83030214',\n",
       "   'country': 'New York',\n",
       "   'ocr_eng': \"4> -\\n?*nw~?~M\\nW.?\\n? ?\\n?\\n4B\\n? ?>* .**?<_'\\n?}*y\\n- y\\\\:.\\ni-.?.''?'-''?\\n-\\nL^\\nGerman soldiers digging a trench in the Argonne\\nCaptain von Muecke. of the Emden, hailed a hero\\nby Germany. m%^0 OMrrnsnksstrmtims.ee\",\n",
       "   'batch': 'dlc_rupp_ver02',\n",
       "   'title_normal': 'new-york tribune.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83030214/1915-08-01/ed-1/seq-57.json',\n",
       "   'place': ['New York--New York--New York'],\n",
       "   'page': ''},\n",
       "  {'sequence': 5,\n",
       "   'county': ['Apache'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Weekly',\n",
       "   'id': '/lccn/sn85047513/1960-11-01/ed-2/seq-5/',\n",
       "   'subject': ['Apache County (Ariz.)--Newspapers.',\n",
       "    'Arizona--Apache County.--fast--(OCoLC)fst01217372',\n",
       "    'Arizona--Window Rock.--fast--(OCoLC)fst01247418',\n",
       "    'Arizona.--fast--(OCoLC)fst01204820',\n",
       "    'Indians of North America--Arizona--Newspapers.',\n",
       "    'Indians of North America.--fast--(OCoLC)fst00969633',\n",
       "    'Navajo Indians--Arizona--Newspapers.',\n",
       "    'Navajo Indians.--fast--(OCoLC)fst01034799',\n",
       "    'Window Rock (Ariz.)--Newspapers.'],\n",
       "   'city': ['Window Rock'],\n",
       "   'date': '19601101',\n",
       "   'title': 'Navajo times. [volume]',\n",
       "   'end_year': 1984,\n",
       "   'note': ['\"Published for ... Owned by ... The Navajo Tribe.\"',\n",
       "    '\"Voice of Scenic Navajoland.\"',\n",
       "    'A project of the Public Relations and Information Dept. of the Navajo Tribe, Feb. 1962-Dec. 1963; of the Public Relations Dept., 1964-1984.',\n",
       "    'Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm from Southwest Micrographic Pub. Co.',\n",
       "    'Continues the suspended Navajo times (Window Rock, Ariz. : 1959).',\n",
       "    'Danky, J.P. Native American periodicals and newspapers 1828-1982',\n",
       "    'Masthead ill.: Family in wagon on road, sun shining in the west.'],\n",
       "   'state': ['Arizona'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Window Rock, Ariz.',\n",
       "   'start_year': 1960,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Navajo Tribe',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn85047513',\n",
       "   'country': 'Arizona',\n",
       "   'ocr_eng': 'THE NAVAJO CIVIC\\nCENTER PRESENTS\\nHOHNER\\nSYMPHONY\\nFROM GERMANY\\n24 ACCORD IANS\\nAND HARMONICAS\\nFEBRUARY 21, 1961\\nNOVEMBER, 1960 THE NAVAJA TIMES\\nPAGE 5',\n",
       "   'batch': 'az_ironsprings_ver01',\n",
       "   'title_normal': 'navajo times.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85047513/1960-11-01/ed-2/seq-5.json',\n",
       "   'place': ['Arizona--Apache--Window Rock'],\n",
       "   'page': 'PAGE 5'},\n",
       "  {'sequence': 4,\n",
       "   'county': ['Apache'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Weekly',\n",
       "   'id': '/lccn/sn85047513/1961-01-01/ed-1/seq-4/',\n",
       "   'subject': ['Apache County (Ariz.)--Newspapers.',\n",
       "    'Arizona--Apache County.--fast--(OCoLC)fst01217372',\n",
       "    'Arizona--Window Rock.--fast--(OCoLC)fst01247418',\n",
       "    'Arizona.--fast--(OCoLC)fst01204820',\n",
       "    'Indians of North America--Arizona--Newspapers.',\n",
       "    'Indians of North America.--fast--(OCoLC)fst00969633',\n",
       "    'Navajo Indians--Arizona--Newspapers.',\n",
       "    'Navajo Indians.--fast--(OCoLC)fst01034799',\n",
       "    'Window Rock (Ariz.)--Newspapers.'],\n",
       "   'city': ['Window Rock'],\n",
       "   'date': '19610101',\n",
       "   'title': 'Navajo times. [volume]',\n",
       "   'end_year': 1984,\n",
       "   'note': ['\"Published for ... Owned by ... The Navajo Tribe.\"',\n",
       "    '\"Voice of Scenic Navajoland.\"',\n",
       "    'A project of the Public Relations and Information Dept. of the Navajo Tribe, Feb. 1962-Dec. 1963; of the Public Relations Dept., 1964-1984.',\n",
       "    'Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm from Southwest Micrographic Pub. Co.',\n",
       "    'Continues the suspended Navajo times (Window Rock, Ariz. : 1959).',\n",
       "    'Danky, J.P. Native American periodicals and newspapers 1828-1982',\n",
       "    'Masthead ill.: Family in wagon on road, sun shining in the west.'],\n",
       "   'state': ['Arizona'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Window Rock, Ariz.',\n",
       "   'start_year': 1960,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Navajo Tribe',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn85047513',\n",
       "   'country': 'Arizona',\n",
       "   'ocr_eng': \"PAGE 4\\nTHE NAVAJO TIMES JANUARY, 1961\\n. THE NAY AIO CIVIC .\\nCENTER PRESENTS\\nHOHNER '\\nSYMPHONY\\nFROM GERMANY\\n24 ACCORD IANS\\nAND HARMONICAS\\nFEBRUARY 21, 1961\",\n",
       "   'batch': 'az_ironsprings_ver01',\n",
       "   'title_normal': 'navajo times.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85047513/1961-01-01/ed-1/seq-4.json',\n",
       "   'place': ['Arizona--Apache--Window Rock'],\n",
       "   'page': 'PAGE 4'},\n",
       "  {'sequence': 4,\n",
       "   'county': ['Apache'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Weekly',\n",
       "   'id': '/lccn/sn85047513/1961-01-01/ed-2/seq-4/',\n",
       "   'subject': ['Apache County (Ariz.)--Newspapers.',\n",
       "    'Arizona--Apache County.--fast--(OCoLC)fst01217372',\n",
       "    'Arizona--Window Rock.--fast--(OCoLC)fst01247418',\n",
       "    'Arizona.--fast--(OCoLC)fst01204820',\n",
       "    'Indians of North America--Arizona--Newspapers.',\n",
       "    'Indians of North America.--fast--(OCoLC)fst00969633',\n",
       "    'Navajo Indians--Arizona--Newspapers.',\n",
       "    'Navajo Indians.--fast--(OCoLC)fst01034799',\n",
       "    'Window Rock (Ariz.)--Newspapers.'],\n",
       "   'city': ['Window Rock'],\n",
       "   'date': '19610101',\n",
       "   'title': 'Navajo times. [volume]',\n",
       "   'end_year': 1984,\n",
       "   'note': ['\"Published for ... Owned by ... The Navajo Tribe.\"',\n",
       "    '\"Voice of Scenic Navajoland.\"',\n",
       "    'A project of the Public Relations and Information Dept. of the Navajo Tribe, Feb. 1962-Dec. 1963; of the Public Relations Dept., 1964-1984.',\n",
       "    'Archived issues are available in digital format from the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm from Southwest Micrographic Pub. Co.',\n",
       "    'Continues the suspended Navajo times (Window Rock, Ariz. : 1959).',\n",
       "    'Danky, J.P. Native American periodicals and newspapers 1828-1982',\n",
       "    'Masthead ill.: Family in wagon on road, sun shining in the west.'],\n",
       "   'state': ['Arizona'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Window Rock, Ariz.',\n",
       "   'start_year': 1960,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'Navajo Tribe',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn85047513',\n",
       "   'country': 'Arizona',\n",
       "   'ocr_eng': 'PAGE 4\\nTHE NAVAJO TIMES JANUARY, 1961\\nTHE NAVAJO CIVIC\\nCENTER PRESENTS\\nHOHNER\\nSYMPHONY\\nFROM GERMANY\\n24 ACCORD IANS\\nAND HARMONICAS\\nFEBRUARY 21, 1961\\n%\\n-^i^-',\n",
       "   'batch': 'az_ironsprings_ver01',\n",
       "   'title_normal': 'navajo times.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn85047513/1961-01-01/ed-2/seq-4.json',\n",
       "   'place': ['Arizona--Apache--Window Rock'],\n",
       "   'page': 'PAGE 4'},\n",
       "  {'sequence': 9,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1917-04-26/ed-1/seq-9/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19170426',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'NOON EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': 'A WARRIOR OF 1917!\\nRome. Pope Benedict informed j proximate revolution and that gen-\\nby papal nuncios that general strikes eral revolt to back up demand of the\\nin Germany and Austro-Hunearv aD- people might develop\\nmmmtmMmmmmmmmmmmm',\n",
       "   'batch': 'iune_india_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1917-04-26/ed-1/seq-9.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 8,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1912-10-12/ed-1/seq-8/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19121012',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': '',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': 'IN GOOD OLD GERMANY\\nm kv \" \\'\\nl\\'While the crowd\\'s watching the bicycle races\\nQHiiHBBBMVESBliXaBttL!-LaaBnHMaaniaaaK9amB4aMaadmai\\nKeep your ye on\\n\\'-the barrel.\\ni\\ni.',\n",
       "   'batch': 'iune_echo_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1912-10-12/ed-1/seq-8.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''},\n",
       "  {'sequence': 27,\n",
       "   'county': ['Cook County'],\n",
       "   'edition': None,\n",
       "   'frequency': 'Daily (except Sunday and holidays)',\n",
       "   'id': '/lccn/sn83045487/1915-12-08/ed-1/seq-27/',\n",
       "   'subject': ['Chicago (Ill.)--Newspapers.',\n",
       "    'Illinois--Chicago.--fast--(OCoLC)fst01204048'],\n",
       "   'city': ['Chicago'],\n",
       "   'date': '19151208',\n",
       "   'title': 'The day book. [volume]',\n",
       "   'end_year': 1917,\n",
       "   'note': ['\"An adless daily newspaper.\"',\n",
       "    'Archived issues are available in digital format as part of the Library of Congress Chronicling America online collection.',\n",
       "    'Available on microfilm;',\n",
       "    'Description based on: Nov. 1, 1911.',\n",
       "    'Issue for <Nov. 24, 1911> lacks vol., no., and chronological designation.',\n",
       "    'Issue for Nov. 4, 1911 erroneously designated as Oct. 4, 1911.',\n",
       "    'Issue for v. 3, no. 290 (Sept. 7, 1914) erroneously designated as v. 3, no. 300 (Sept. 7, 1914). The error in numbering continues.',\n",
       "    'Issue for v. 5, no. 214 (June 7, 1916) erroneously designated as v. 5, no. 214 (June 6, 1916).',\n",
       "    'Issue for v. 5, no. 7 (Oct. 5, 1915) erroneously designated as v. 5, no. 7 (Sept. 5, 1915).',\n",
       "    'Issues for <May 7-17, 1915> called also \"Moving Picture Edition.\"',\n",
       "    'Issues have no page numbering.',\n",
       "    'Saturdays have Noon and Final editions, Dec. 28, 1912-June 21, 1913; Saturdays have Noon and Last editions, June 28, 1913-<Dec. 13, 1913>; began issuing daily Noon and Last editions, Dec. 20, 1913-July 6, 1917.',\n",
       "    'Vol. 5, no. 36 (Nov. 6, 1915) issue called also \"Garment Workers\\' Special Edition.\"',\n",
       "    'Volume numbering begins with Nov. 20, 1911 issue.'],\n",
       "   'state': ['Illinois'],\n",
       "   'section_label': '',\n",
       "   'type': 'page',\n",
       "   'place_of_publication': 'Chicago, Ill.',\n",
       "   'start_year': 1911,\n",
       "   'edition_label': 'LAST EDITION',\n",
       "   'publisher': 'N.D. Cochran',\n",
       "   'language': ['English'],\n",
       "   'alt_title': [],\n",
       "   'lccn': 'sn83045487',\n",
       "   'country': 'Illinois',\n",
       "   'ocr_eng': \"PREPAREDNESS!\\niJX ' '? mcreasmg n pop- disposal in Europe. Japan, whose\\nulation twice as fast as Germany, it population grows rapidly has little\\nHas four times as much room at its Ijor no room for more.\\nmmmmmtmmtmm\",\n",
       "   'batch': 'iune_hotel_ver01',\n",
       "   'title_normal': 'day book.',\n",
       "   'url': 'https://chroniclingamerica.loc.gov/lccn/sn83045487/1915-12-08/ed-1/seq-27.json',\n",
       "   'place': ['Illinois--Cook County--Chicago'],\n",
       "   'page': ''}]}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the keys of the dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['totalItems', 'endIndex', 'startIndex', 'itemsPerPage', 'items'])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the values using the square bracket indexing with a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2334262"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['totalItems']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While JSON is a simple text format, it can become complex due to\n",
    "\n",
    "- nesting (JSON inside JSON)\n",
    "- lists of JSON\n",
    "\n",
    "An example is our `items`, which has been parsed as a Python `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data['items'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`items` is a list of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['sequence', 'county', 'edition', 'frequency', 'id', 'subject', 'city', 'date', 'title', 'end_year', 'note', 'state', 'section_label', 'type', 'place_of_publication', 'start_year', 'edition_label', 'publisher', 'language', 'alt_title', 'lccn', 'country', 'ocr_eng', 'batch', 'title_normal', 'url', 'place', 'page'])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item = data['items'][0]\n",
    "item.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate over both the keys and values as a pair using `items()`.\n",
    "\n",
    "Below we use a check  a quick check that the value isn't too long before printing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "county: ['Wayne']\n",
      "edition: Home ed.\n",
      "frequency: Daily\n",
      "id: /lccn/sn88063294/1944-09-12/ed-1/seq-11/\n",
      "subject: ['Detroit (Mich.)--Newspapers.', 'Michigan--Detroit.--fast--(OCoLC)fst01205010', 'Michigan--Wayne County.--fast--(OCoLC)fst01206628', 'Wayne County (Mich.)--Newspapers.']\n",
      "city: ['Detroit']\n",
      "date: 19440912\n",
      "title: Detroit evening times.\n",
      "note: ['Archived issues are available in digital format from the Library of Congress Chronicling America online collection.', 'Description based on: 22nd yr., no. 27 (Nov. 1, 1921).', 'Issue called: Michigan centennial ed., May 2, 1937.', 'Most Sunday issues for Aug. 6, 1922-Jan. 12, 1958 called: Detroit Sunday times.', 'Some issues include supplement called: Goodfellows Special/Old Newsboys ed., published in Dec. of each year <Dce. 16, 1926-Dec. 1948>.', 'The word \"evening\" appears within ornament.']\n",
      "state: ['Michigan']\n",
      "section_label: \n",
      "type: page\n",
      "place_of_publication: Detroit, Mich\n",
      "edition_label: REDLINE\n",
      "publisher: [s.n.]\n",
      "language: ['English']\n",
      "alt_title: ['Detroit Sunday times', 'Detroit times']\n",
      "lccn: sn88063294\n",
      "country: Michigan\n",
      "ocr_eng: Where the Battle of Germany Will Be Fought\n",
      "|DETROir.*KfTIJvIES| -Sea\n",
      "batch: mimtptc_holly_ver03\n",
      "title_normal: detroit evening times.\n",
      "url: https://chroniclingamerica.loc.gov/lccn/sn88063294/1944-09-12/ed-1/seq-11.json\n",
      "place: ['Michigan--Wayne--Detroit']\n",
      "page: \n"
     ]
    }
   ],
   "source": [
    "from collections.abc import Iterable\n",
    "\n",
    "for k, v in item.items():\n",
    "    if isinstance(v, Iterable) and len(v) < 100:\n",
    "        print(f'{k}: {v}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish this exercise by only taking articles that appear between two years, and save those to disk.\n",
    "\n",
    "Normally you would apply this kind of filtering in the API request - we are going to filter in memory.\n",
    "\n",
    "First we need a bit of data cleaning of our date, which is an integer representation of time (but is a `str`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'19440912'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here use `strptime` to convert the integer into a proper datetime:\n",
    "- ([Python's strftime directives](http://strftime.org/) is very useful!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(1944, 9, 12, 0, 0)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime as dt\n",
    "dt.strptime(item['date'], \"%Y%m%d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's put this data cleaning & filtering into a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "items = data['items']\n",
    "\n",
    "start = 1900\n",
    "extract = []\n",
    "for item in items:\n",
    "    item['date'] = dt.strptime(item['date'], \"%Y%m%d\")\n",
    "    \n",
    "    if item['date'].year > start:\n",
    "        extract.append(item)\n",
    "        \n",
    "len(extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a list of dictionaries, which plays very nice with `pandas`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: pip: command not found\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>county</th>\n",
       "      <th>edition</th>\n",
       "      <th>frequency</th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>city</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "      <th>end_year</th>\n",
       "      <th>...</th>\n",
       "      <th>language</th>\n",
       "      <th>alt_title</th>\n",
       "      <th>lccn</th>\n",
       "      <th>country</th>\n",
       "      <th>ocr_eng</th>\n",
       "      <th>batch</th>\n",
       "      <th>title_normal</th>\n",
       "      <th>url</th>\n",
       "      <th>place</th>\n",
       "      <th>page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "      <td>[Wayne]</td>\n",
       "      <td>Home ed.</td>\n",
       "      <td>Daily</td>\n",
       "      <td>/lccn/sn88063294/1944-09-12/ed-1/seq-11/</td>\n",
       "      <td>[Detroit (Mich.)--Newspapers., Michigan--Detro...</td>\n",
       "      <td>[Detroit]</td>\n",
       "      <td>1944-09-12</td>\n",
       "      <td>Detroit evening times.</td>\n",
       "      <td>1958</td>\n",
       "      <td>...</td>\n",
       "      <td>[English]</td>\n",
       "      <td>[Detroit Sunday times, Detroit times]</td>\n",
       "      <td>sn88063294</td>\n",
       "      <td>Michigan</td>\n",
       "      <td>Where the Battle of Germany Will Be Fought\\n|D...</td>\n",
       "      <td>mimtptc_holly_ver03</td>\n",
       "      <td>detroit evening times.</td>\n",
       "      <td>https://chroniclingamerica.loc.gov/lccn/sn8806...</td>\n",
       "      <td>[Michigan--Wayne--Detroit]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16</td>\n",
       "      <td>[San Francisco]</td>\n",
       "      <td>None</td>\n",
       "      <td>Daily</td>\n",
       "      <td>/lccn/sn85066387/1908-11-08/ed-1/seq-16/</td>\n",
       "      <td>[California--San Francisco Bay Area.--fast--(O...</td>\n",
       "      <td>[San Francisco]</td>\n",
       "      <td>1908-11-08</td>\n",
       "      <td>The San Francisco call. [volume]</td>\n",
       "      <td>1913</td>\n",
       "      <td>...</td>\n",
       "      <td>[English]</td>\n",
       "      <td>[Call, Call-chronicle-examiner, Sunday call]</td>\n",
       "      <td>sn85066387</td>\n",
       "      <td>California</td>\n",
       "      <td>JACK— he w.45 11111^^ Germany,\\nI^ll/ riFSIIOV...</td>\n",
       "      <td>curiv_llano_ver01</td>\n",
       "      <td>san francisco call.</td>\n",
       "      <td>https://chroniclingamerica.loc.gov/lccn/sn8506...</td>\n",
       "      <td>[California--San Francisco--San Francisco]</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   sequence           county   edition frequency  \\\n",
       "0        11          [Wayne]  Home ed.     Daily   \n",
       "1        16  [San Francisco]      None     Daily   \n",
       "\n",
       "                                         id  \\\n",
       "0  /lccn/sn88063294/1944-09-12/ed-1/seq-11/   \n",
       "1  /lccn/sn85066387/1908-11-08/ed-1/seq-16/   \n",
       "\n",
       "                                             subject             city  \\\n",
       "0  [Detroit (Mich.)--Newspapers., Michigan--Detro...        [Detroit]   \n",
       "1  [California--San Francisco Bay Area.--fast--(O...  [San Francisco]   \n",
       "\n",
       "        date                             title  end_year  ...   language  \\\n",
       "0 1944-09-12            Detroit evening times.      1958  ...  [English]   \n",
       "1 1908-11-08  The San Francisco call. [volume]      1913  ...  [English]   \n",
       "\n",
       "                                      alt_title        lccn     country  \\\n",
       "0         [Detroit Sunday times, Detroit times]  sn88063294    Michigan   \n",
       "1  [Call, Call-chronicle-examiner, Sunday call]  sn85066387  California   \n",
       "\n",
       "                                             ocr_eng                batch  \\\n",
       "0  Where the Battle of Germany Will Be Fought\\n|D...  mimtptc_holly_ver03   \n",
       "1  JACK— he w.45 11111^^ Germany,\\nI^ll/ riFSIIOV...    curiv_llano_ver01   \n",
       "\n",
       "             title_normal                                                url  \\\n",
       "0  detroit evening times.  https://chroniclingamerica.loc.gov/lccn/sn8806...   \n",
       "1     san francisco call.  https://chroniclingamerica.loc.gov/lccn/sn8506...   \n",
       "\n",
       "                                        place page  \n",
       "0                  [Michigan--Wayne--Detroit]       \n",
       "1  [California--San Francisco--San Francisco]   16  \n",
       "\n",
       "[2 rows x 28 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install -q pandas\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(extract)\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - downloading images\n",
    "\n",
    "We can also `requests` to download things other than text - such as images.\n",
    "\n",
    "Below we do a requests and see we get back a binary string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'�PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x02 \\x00\\x00\\x00�\\x08\\x06\\x00\\x00\\x00�#W\\x1b\\x00\\x004�IDATx\\x01��\\x03�%?\\x1e��m۶}����(�Y:��n��z��m۶���w�$=�ח�\\u05fb�^����N�g���|��_'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png'\n",
    "res = requests.get(url)\n",
    "res.text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use context management to dump the contents of the binary string into a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./data/google-logo.png', 'wb') as fi:\n",
    "    fi.write(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the Google logo locally (you may need to re-run this cell)\n",
    "\n",
    "![](./data/google-logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - earthquake API \n",
    "\n",
    "Let's as a group write a program to get data from the USGS Earthquake Catalog - [documentation](https://earthquake.usgs.gov/fdsnws/event/1/#methods)\n",
    "\n",
    "Steps:\n",
    "- Make a folder (from your current working directory) to hold the earthquake data \n",
    "- Investigate the data from the response \n",
    "- Save each earthquake as its own JSON in the folder you created (hint - you will need to decide on a name for each earthquake JSON) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = \"2014-01-01\"\n",
    "end = \"2014-01-02\"\n",
    "url = f\"https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime={start}&endtime={end}\"\n",
    "\n",
    "import os\n",
    "os.makedirs('./data/earthquakes', exist_ok=True)\n",
    "os.makedirs(f'./data/earthquakes/{start}', exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Wikipedia API\n",
    "\n",
    "Now for an open-ended exercise for you! Your task is to:\n",
    "- create a database of countries\n",
    "- in a folder called `countries` (you will need to make the folder)\n",
    "- each country in it's own folder\n",
    "- start with Germany\n",
    "\n",
    "V1 of your program should:\n",
    "- save the url you use to request the data\n",
    "- save the title\n",
    "- save the `line` parameter of each section (`data['parse']['sections']`)\n",
    "- save all in a single JSON\n",
    "\n",
    "V2 of your program should also:\n",
    "- save all '.png' & '.jpg' images as images, with the url as the image name\n",
    "- save all external links as CSV\n",
    "\n",
    "Much of the work will be understanding how the Wikipedia API works - useful resources are below:\n",
    "- [Main API page](https://www.mediawiki.org/wiki/API:Main_page)\n",
    "- [What the actions are](https://www.mediawiki.org/w/api.php)\n",
    "- [Python examples](https://github.com/wikimedia/mediawiki-api-demos/tree/master/python)\n",
    "\n",
    "Please also feel free to work on another API - happy to assist you with this as well :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Etymology',\n",
       "  'number': '1',\n",
       "  'index': '1',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 12477,\n",
       "  'anchor': 'Etymology'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'History',\n",
       "  'number': '2',\n",
       "  'index': '2',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 15058,\n",
       "  'anchor': 'History'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Germanic tribes and Frankish Empire',\n",
       "  'number': '2.1',\n",
       "  'index': '3',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 18483,\n",
       "  'anchor': 'Germanic_tribes_and_Frankish_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'East Francia and Holy Roman Empire',\n",
       "  'number': '2.2',\n",
       "  'index': '4',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 22352,\n",
       "  'anchor': 'East_Francia_and_Holy_Roman_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'German Confederation and Empire',\n",
       "  'number': '2.3',\n",
       "  'index': '5',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 30396,\n",
       "  'anchor': 'German_Confederation_and_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Weimar Republic and Nazi Germany',\n",
       "  'number': '2.4',\n",
       "  'index': '6',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 39074,\n",
       "  'anchor': 'Weimar_Republic_and_Nazi_Germany'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'East and West Germany',\n",
       "  'number': '2.5',\n",
       "  'index': '7',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 52184,\n",
       "  'anchor': 'East_and_West_Germany'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Reunified Germany and the European Union',\n",
       "  'number': '2.6',\n",
       "  'index': '8',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 59735,\n",
       "  'anchor': 'Reunified_Germany_and_the_European_Union'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Geography',\n",
       "  'number': '3',\n",
       "  'index': '9',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 66113,\n",
       "  'anchor': 'Geography'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Climate',\n",
       "  'number': '3.1',\n",
       "  'index': '10',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 68494,\n",
       "  'anchor': 'Climate'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Biodiversity',\n",
       "  'number': '3.2',\n",
       "  'index': '11',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 71987,\n",
       "  'anchor': 'Biodiversity'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Politics',\n",
       "  'number': '4',\n",
       "  'index': '12',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 79164,\n",
       "  'anchor': 'Politics'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Constituent states',\n",
       "  'number': '4.1',\n",
       "  'index': '13',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 84489,\n",
       "  'anchor': 'Constituent_states'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Law',\n",
       "  'number': '4.2',\n",
       "  'index': '14',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 91743,\n",
       "  'anchor': 'Law'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Foreign relations',\n",
       "  'number': '4.3',\n",
       "  'index': '15',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 96244,\n",
       "  'anchor': 'Foreign_relations'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Military',\n",
       "  'number': '4.4',\n",
       "  'index': '16',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 101517,\n",
       "  'anchor': 'Military'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Economy',\n",
       "  'number': '5',\n",
       "  'index': '17',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 109999,\n",
       "  'anchor': 'Economy'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Infrastructure',\n",
       "  'number': '5.1',\n",
       "  'index': '18',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 123671,\n",
       "  'anchor': 'Infrastructure'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Tourism',\n",
       "  'number': '5.2',\n",
       "  'index': '19',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 131764,\n",
       "  'anchor': 'Tourism'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Demographics',\n",
       "  'number': '6',\n",
       "  'index': '20',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 134679,\n",
       "  'anchor': 'Demographics'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Religion',\n",
       "  'number': '6.1',\n",
       "  'index': '21',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 140904,\n",
       "  'anchor': 'Religion'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Languages',\n",
       "  'number': '6.2',\n",
       "  'index': '22',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 146814,\n",
       "  'anchor': 'Languages'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Education',\n",
       "  'number': '6.3',\n",
       "  'index': '23',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 149648,\n",
       "  'anchor': 'Education'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Health',\n",
       "  'number': '6.4',\n",
       "  'index': '24',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 154973,\n",
       "  'anchor': 'Health'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Culture',\n",
       "  'number': '7',\n",
       "  'index': '25',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 158707,\n",
       "  'anchor': 'Culture'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Music',\n",
       "  'number': '7.1',\n",
       "  'index': '26',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 162999,\n",
       "  'anchor': 'Music'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Art and design',\n",
       "  'number': '7.2',\n",
       "  'index': '27',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 166273,\n",
       "  'anchor': 'Art_and_design'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Literature and philosophy',\n",
       "  'number': '7.3',\n",
       "  'index': '28',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 171901,\n",
       "  'anchor': 'Literature_and_philosophy'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Media',\n",
       "  'number': '7.4',\n",
       "  'index': '29',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 176459,\n",
       "  'anchor': 'Media'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Cuisine',\n",
       "  'number': '7.5',\n",
       "  'index': '30',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 181650,\n",
       "  'anchor': 'Cuisine'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Sports',\n",
       "  'number': '7.6',\n",
       "  'index': '31',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 186795,\n",
       "  'anchor': 'Sports'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'See also',\n",
       "  'number': '8',\n",
       "  'index': '32',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 192006,\n",
       "  'anchor': 'See_also'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Notes',\n",
       "  'number': '9',\n",
       "  'index': '33',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 192114,\n",
       "  'anchor': 'Notes'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'References',\n",
       "  'number': '10',\n",
       "  'index': '34',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 192140,\n",
       "  'anchor': 'References'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Sources',\n",
       "  'number': '11',\n",
       "  'index': '35',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 192170,\n",
       "  'anchor': 'Sources'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'External links',\n",
       "  'number': '12',\n",
       "  'index': '36',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 192636,\n",
       "  'anchor': 'External_links'}]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = f\"https://en.wikipedia.org/w/api.php?action=parse&page={term}&format=json\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "\n",
    "data['parse']['sections']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}