2-fun-with-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 6 Lecture - Fun with Files!!!\n",
    "Topics:\n",
    "* Processing unstructured text\n",
    "* Review importing libraries\n",
    "* CSV and Python\n",
    "* JSON and Python\n",
    "* Parsing MARC records\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Processing text - a working example with open text on Project Gutenberg\n",
    "\n",
    "There are a lot of books available for free, in the public domain.  We can use this text to explore working with files in python. For instance, Fitzgerald's Gatsby is available on Project Gutenberg AU\n",
    "\n",
    "http://gutenberg.net.au/ebooks02/0200041.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle = open('gatsby.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle = open('files/gatsby.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "numberOfLines = 0\n",
    "for line in fileHandle :\n",
    "    numberOfLines += 1\n",
    "    \n",
    "print(numberOfLines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an aweful error message, but let's look closely... it's causing an error in a \"codec\" called utf-8.  This is geek speak for specification on how the binary form of the file is encoded... meaning the format of the 1s and 0s of the file.\n",
    "\n",
    "We can find the encoding, on a Mac by calling: \n",
    "\n",
    "file -I {filename} in the terminal.\n",
    "\n",
    "or by doing it directly in the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!file -I files/gatsby.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WTF is [ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) and why do we care?\n",
    "\n",
    "Not everything was... or is unicode.  While it will get better, it will never go away."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(open)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle = open('files/gatsby.txt', encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numberOfLines = 0\n",
    "for line in fileHandle :\n",
    "    numberOfLines += 1\n",
    "    \n",
    "print(numberOfLines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, we need to close the file, especially if we want to use it again from the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the code above we know that there are 6378 lines of code.  If we look at the structure of the file (go in to files folder and open gatsby.txt in Jupyter or a text editor).  You can see the text is formated with a \"Chapter\" line of text at the start of each chapter.  We can use this formatting in code to count the number of chapters by finding the number of times a line in the text contains \"Chapter\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle = open('files/gatsby.txt', encoding='iso-8859-1')\n",
    "\n",
    "numberOfChapters = 0\n",
    "\n",
    "for line in fileHandle :\n",
    "    line = line.lstrip()\n",
    "    if line.startswith('Chapter') :\n",
    "        numberOfChapters += 1\n",
    "\n",
    "print(numberOfChapters)\n",
    "\n",
    "fileHandle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`lstrip()` is new, so let's see what it does exactly (strip white spaces at the beginning of the string):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "someString = \"                        this is a string with a lot of whitespace\"\n",
    "print(someString)\n",
    "print(someString.lstrip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see an example of counting references to characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileHandle = open('files/gatsby.txt', encoding='iso-8859-1')\n",
    "\n",
    "countJay = 0\n",
    "countDaisy = 0\n",
    "\n",
    "fileHandle.seek(0)\n",
    "for line in fileHandle :\n",
    "    if line.find('Jay') != -1 :\n",
    "        countJay += 1\n",
    "    elif line.find('Daisy') != -1 :\n",
    "        countDaisy += 1\n",
    "\n",
    "print(countJay)\n",
    "print(countDaisy)\n",
    "\n",
    "fileHandle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review of Importing Libraries\n",
    "\n",
    "Python has some functions already loaded and ready to use (like `print()`, `len()`, `type()` and `range()`. The full list of built-in functions can be found [on the Python Documentation website](https://docs.python.org/3/library/functions.html).\n",
    "\n",
    "Beyond the built-in functions, Python has a very extensive [standard library](https://docs.python.org/3/library/index.html), which provides additional functionality in a variety of areas including [math](https://docs.python.org/3/library/numeric.html), [text processing](https://docs.python.org/3/library/text.html), and working with unique [file formats](https://docs.python.org/3/library/fileformats.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the time module from the standard library\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the [time module](https://docs.python.org/3/library/time.html) to see how many nanoseconds have elapsed since the [EPOCH](https://en.wikipedia.org/wiki/Unix_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many nanoseconds\n",
    "time.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When is the EPOCH (on this computer)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the gmtime method to see when is time point zero\n",
    "time.gmtime(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The world (of computers) is only 51 years old."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing CSV files\n",
    "\n",
    "* CSV stands for *comma separated values* \n",
    "* Is a file format for tabular data, i.e. Excel Spreadsheets\n",
    "* It is a pretty common data format\n",
    "* So Python has a built-in parser in the [CSV module](https://docs.python.org/3/library/csv.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the csv module into memory\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open the dog names csv file and loop over the contents\n",
    "fileHandle = open('files/pgh-dog-names.csv', 'r')\n",
    "\n",
    "# create the special csv reader object that knows how to parse the file\n",
    "csv_reader = csv.reader(fileHandle)\n",
    "    \n",
    "# initialize a variable with a very short name\n",
    "long_name = \"\"\n",
    "breed = \"\"\n",
    "color = \"\"\n",
    "\n",
    "# loop over each line look for the longest dog name\n",
    "for dog in csv_reader:\n",
    "    # the name is the 4th column\n",
    "    # check to see if that string is longer than long_name\n",
    "    if len(dog[3]) > len(long_name):\n",
    "        # if we have a new long name, save it in the variable long_name\n",
    "        long_name = dog[3]\n",
    "        breed = dog[1]\n",
    "        color = dog[2]\n",
    "\n",
    "\n",
    "# print the name of the dog with the longest name\n",
    "print(\"The dog with the longest name in Pittsburgh is\", long_name)\n",
    "print(\"It is a\", color, breed)\n",
    "\n",
    "fileHandle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for those of us that do not speak German: DE HOUTEN HUISBESCHERMER VAN MEMPHIS is THE WOODEN HOME PROTECTOR OF MEMPHIS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing JSON and Python\n",
    "\n",
    "Python is *batteries included* so there is already a [JSON module](https://docs.python.org/3/library/json.html) in the standard library.\n",
    "\n",
    "The JSON module provides 4 main functions. Two for decoding (parsing) and two for encoding (*serializing*):\n",
    "* `json.load()` - Parse JSON data from a file\n",
    "* `json.loads()` - Parse JSON data from a string\n",
    "* `json.dump()` - Serialize Python data into a JSON file\n",
    "* `json.dumps()` - Serialize Python data into a JSON string\n",
    "\n",
    "Notice a pattern on the naming convention?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding / Parsing JSON data\n",
    "\n",
    "I have included some fun JSON data in the `files` folder. Let's go and see what we have using JupyterLab and then open them with Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_string = \"\"\"\n",
    "{\"something\":\"I was not able to think of any fun data\", \n",
    "\"data\":[1,2,3,4,5,6]}\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "parsed_json = json.loads(json_string)\n",
    "parsed_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What *type* is this parsed data being represented in Python?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(parsed_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON maps very nicely to Python dictionaries. You can see how the JSON data types map to Python data types in [the documentation](https://docs.python.org/3/library/json.html#encoders-and-decoders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually we don't type our json literally as strings...we store it in files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open a JSON file handler and parse it into a python d\n",
    "fileh = open(\"files/stranger.json\")\n",
    "stranger_data = json.load(fileh) #stranger danger\n",
    "#close the file\n",
    "fileh.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# display the data\n",
    "stranger_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fun!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the value at the key summary\n",
    "stranger_data['summary']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the Python dictionary indexing, we can reach into this complex data structure and grab subsets of the tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many episodes\n",
    "len(stranger_data[\"_embedded\"][\"episodes\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, we have to have a pre-existing understanding of the data structure so we can know what keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding / Serializing JSON Data\n",
    "\n",
    "From the [Rick and Morty API](https://rickandmortyapi.com)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileh = open(\"files/rm-characters.json\")\n",
    "rm_characters = json.load(fileh)\n",
    "fileh.close()\n",
    "\n",
    "rm_characters[\"info\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(rm_characters[\"results\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have twenty characters, but maybe we just want to have the aliens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an empty dictionary for our data\n",
    "aliens = {}\n",
    "\n",
    "# loop over all the characters\n",
    "for character in rm_characters[\"results\"]:\n",
    "    # check to see if the character is an alient\n",
    "    if character[\"species\"] == \"Alien\":\n",
    "        #using the name as a key, save the data for alien characters as values\n",
    "        aliens[character[\"name\"]] = character\n",
    "\n",
    "# display the results\n",
    "aliens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save our data to disk\n",
    "with open(\"aliens.json\", \"w\") as fileh:\n",
    "    json.dump(aliens, fileh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ancient aliens](https://i.kym-cdn.com/photos/images/original/000/183/103/alens.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an empty dictionary for our data\n",
    "aliens = []\n",
    "\n",
    "# loop over all the characters\n",
    "for character in rm_characters[\"results\"]:\n",
    "    # check to see if the character is an alient\n",
    "    if character[\"species\"] == \"Alien\":\n",
    "        #using the name as a key, save the data for alien characters as values\n",
    "        aliens.append(character)\n",
    "\n",
    "# display the results\n",
    "aliens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileh = open(\"aliens.json\", \"w\")\n",
    "json.dump(aliens, fileh)\n",
    "fileh.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to get all this data as a JSON string, we can use `dumps`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing MARC records\n",
    "\n",
    "Unfortunately, [MARC records are still a thing](https://www.libraryjournal.com/?detailStory=marc-must-die) and so we need to *deal with it*.\n",
    "\n",
    "Double Unfortunately, Python does not include a MARC parser as part of the standard library.\n",
    "\n",
    "<img src=\"https://images-na.ssl-images-amazon.com/images/I/71uIJYNfJZL._SL1500_.jpg\" alt=\"batteries not included\" style=\"width: 300px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, [a brave soul](https://inkdroid.org/about/) has built this capacity for us in the form of a third-party library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be working with pymarc library for this lecture.  Details about it can be found [here](https://gitlab.com/pymarc/pymarc), which also has some documentation and resources for its use.  There is also a complete library documentation [here](https://readthedocs.org/projects/pymarc/downloads/pdf/latest/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing 3rd party libraries\n",
    "\n",
    "Libraries like Pymarc are not part of the \"standard library,\" which means you have to install them on your computer yourself. There are currently 291,647 on the [PyPI website](https://pypi.org), the Python Package Index, a repository of 3rd party packages. You don't need to install all of them, only the ones you need.\n",
    "\n",
    "In our case, we needed to install the [pymarc](https://pypi.org/project/pymarc/) package. \n",
    "\n",
    "\n",
    "\n",
    "Installing packages is done using a command line tool called `pip` and you can run the code cell below to execute a unix command to install pymarc. NOTE: this may not work on Windows computers AND you don't need to run this if you are on JupyterHub (it just won't do anything)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run the command pip to install pymarc\n",
    "%pip install pymarc --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that pymarc is installed, we can `import` it like any other library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the \n",
    "from pymarc import pymarc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading MARC files\n",
    "\n",
    "MARC files are text files (technically they are MARC8 encoded text files) and they are UGLY. But because they are just text you can open them using the regular Python file machinery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# open MARC file called marc.dat\n",
    "fileh = open(\"files/marc.dat\", 'r')\n",
    "print(fileh.read(1000)) #read first 1000 characters\n",
    "fileh.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot going on in that file and parsing it using Python string methods would be unpleasant. Because somoene did that for us, we can stand on the shoulder of giants!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an empty list to store the records\n",
    "marc_records = []\n",
    "\n",
    "# open the file \n",
    "fileh = open(\"files/marc.dat\", \"rb\")\n",
    "    \n",
    "# create an isntance of the marc reader, like with CSV \n",
    "marc_reader = pymarc.MARCReader(fileh)\n",
    "    \n",
    "# loop over each record\n",
    "for record in marc_reader:\n",
    "    # add record to our list\n",
    "    marc_records.append(record)\n",
    "        \n",
    "fileh.close()\n",
    "        \n",
    "print(\"There are\", len(marc_records), \"MARC records.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does a record look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at the 2nd record at index 1 because...\n",
    "marc_records[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drat! What we are looking at is an complex, custom data structure (technically an \"object\") of the type `Record`. We won't get into the details of object oriented programming. \n",
    "\n",
    "If we want to manipulate are Record, we should [read the documentation](https://pymarc.readthedocs.io/en/latest/#module-pymarc.record)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see what the dictionary representation of the MARC record\n",
    "marc_records[1].as_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ugh, still a mess, but we can use handy methods like `title()` and `author()` to grab information without having to memorize MARC fields "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop over each record\n",
    "for record in marc_records:\n",
    "    # print name and author\n",
    "    print(record.title(), \"by\", record.author())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pymarc includes a bunch of helper functions for grabbing information from a MARC record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the ISBN \n",
    "marc_records[11].isbn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dispay the publisher \n",
    "marc_records[12].publisher()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the publication year\n",
    "marc_records[1].pubyear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the subjects\n",
    "marc_records[-4].subjects()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AH! We have a list of [Fields](https://pymarc.readthedocs.io/en/latest/#module-pymarc.field), we need to do some looping and reading of documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop over each field and display the field value\n",
    "for subject in marc_records[-4].subjects() :\n",
    "    print(subject.value())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop over each field and display the raw MARC\n",
    "for subject in marc_records[-4].subjects() :\n",
    "    print(subject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are wicked smart and have memorized the MARC field numbers by heart, you can always reference them directly using Python indexing syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the value for field 300 from\n",
    "marc_records[1]['300'].value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the 245 field in the original MARC format\n",
    "marc_records[1]['245'].as_marc(\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new blank MARC record\n",
    "record = pymarc.Record() \n",
    "\n",
    "# Create the 245 and 100 fields \n",
    "title = pymarc.Field(\n",
    "        tag = '245',\n",
    "        indicators = ['0','1'], \n",
    "        subfields = [\n",
    "            'a', 'The Beaverkill : ', 'b', 'The History of a Rever and its People /'\n",
    "        ])\n",
    "author = pymarc.Field(\n",
    "        tag = '100',\n",
    "        indicators = ['0','1'], \n",
    "        subfields = [\n",
    "            'a', 'Ed Van Put'\n",
    "        ])\n",
    "record.add_field(title, author)\n",
    "\n",
    "\n",
    "# write the MARC record to disk\n",
    "fileh = open('test_write.dat', 'wb')\n",
    "fileh.write(record.as_marc())\n",
    "fileh.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# what does it look like\n",
    "print(record.as_marc())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ugh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_test = []\n",
    "\n",
    "# open the file \n",
    "with open(\"test_write.dat\", \"rb\") as fileh:\n",
    "    \n",
    "    # create an isntance of the marc reader, like with CSV \n",
    "    marc_reader = pymarc.MARCReader(fileh)\n",
    "    \n",
    "    # loop over each record\n",
    "    for record in marc_reader:\n",
    "        # add record to our list\n",
    "        records_test.append(record)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the title and author of the record we read from disk\n",
    "print(records_test[0].title(), \"by\", records_test[0].author())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Hi MARC](https://media.giphy.com/media/G2vaqcEICxOyA/giphy.gif)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}