Notebooks/other-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading other kinds of files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The best package for reading the contents of modern Word documents (i.e,. files with a  docx exctension) is `docx2txt`. It returns the full text, stripping out all formatting information. \n",
    "\n",
    "To install (from within a notebook):\n",
    "~~~~\n",
    "    %conda install -c conda-forge docx2txt\n",
    "~~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import docx2txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "text = docx2txt.process('data/pandas_wiki.docx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Exercise 1\n",
    "\n",
    "View the contents of one of your Word documents in Python.\n",
    "\n"
   ]
 },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Sample answer code</summary> \n",
    "<code style=\"background-color: white\">\n",
    "text = docx2txt.process('path/to/file.docx')\n",
    "    \n",
    "</code>\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PDF Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The best package for reading the contents of PDF documents is `PyPDF2`. It can be used to read and write PDF documents. It can extract the text, but only when the text is stored in the document, as is commonly the case when PDFs are created on a computer from the original document. It can not extract the text from scanned documents, as the text there is stored as an image.\n",
    "\n",
    "To install (from within a notebook):\n",
    "~~~~\n",
    "    %conda install -c conda-forge PyPDF2\n",
    "~~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import PyPDF2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Extracting the textual elements of a PDF is more complicated than a Word document. After the file is opened, it needs to be parsed, and then the text extracted from each page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pdfFileObj = open('docs/19-1222.pdf', 'rb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pdfReader = PyPDF2.PdfFileReader(pdfFileObj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(pdfReader.numPages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "first_page_text = pdfReader.getPage(0).extractText()\n",
    "\n",
    "print(first_page_text[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you wanted all of the text from a particular PDF, you could iterate over each page. In this case, I add a line break (`\\n`) in between each page. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "full_text = ''\n",
    "\n",
    "for page_number in range(0, pdfReader.numPages):\n",
    "    new_page_text = pdfReader.getPage(page_number).extractText()\n",
    "    full_text = full_text +  '\\n' + new_page_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "len(full_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(full_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PDF Exercise 1\n",
    "\n",
    "View the text of the DOJ press released named 19-1223.pdf (in the docs folder).\n",
    "\n"
   ]
 },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Sample answer code</summary> \n",
    "<code style=\"background-color: white\">\n",
    "pdfFileObj = open('docs/19-1223.pdf', 'rb')\n",
    "pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
    "full_text = ''\n",
    "\n",
    "for page_number in range(0, pdfReader.numPages):\n",
    "    new_page_text = pdfReader.getPage(page_number).extractText()\n",
    "    full_text = full_text +  '\\n' + new_page_text\n",
    "print(full_text)    \n",
    "</code>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This could all be bundled in function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def extract_text(pdf_file_name):\n",
    "    '''Extract text contents from a PDF file'''\n",
    "    \n",
    "    pdfFileObj = open(pdf_file_name, 'rb')\n",
    "        \n",
    "    pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
    "    \n",
    "    full_text = ''\n",
    "\n",
    "    for page_number in range(0, pdfReader.numPages):\n",
    "        new_page_text = pdfReader.getPage(page_number).extractText()\n",
    "        full_text = full_text +  '\\n' + new_page_text\n",
    "    return full_text\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "text = extract_text('docs/19-1222.pdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Alternatively, if you wanted to preserve the pages as individual documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 200)\n",
    "\n",
    "\n",
    "def extract_page(page_number):\n",
    "    text = pdfReader.getPage(page_number).extractText()\n",
    "    return {'page' : page_number + 1,\n",
    "            'text' : text}\n",
    "\n",
    "def extract_pages(pdf_file_name):\n",
    "    '''Extract text contents from a PDF file'''\n",
    "    \n",
    "    pdfFileObj = open(pdf_file_name, 'rb')\n",
    "        \n",
    "    pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
    "    \n",
    "    pages = []\n",
    "\n",
    "    for page_number in range(0, pdfReader.numPages):\n",
    "        new_page_dict = extract_page(page_number)\n",
    "        pages.append(new_page_dict)\n",
    "    df = pd.DataFrame(pages)\n",
    "    df['source'] = pdf_file_name\n",
    "    \n",
    "    return df\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "extract_pages('docs/19-1222.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Plain text files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Plain text files can be opened and contents stored in a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "filename = 'docs/19-1223.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "text = open(filename,'r').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(text[:250])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The above method does not scale well, as Python keeps the file open until you close it. Leaving it open creates memory problems when opening hundreds or thousands of files, so the preferred way to open files is using `with`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "with open(filename, 'r') as infile:\n",
    "    text = infile.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(text[:250])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PDF Exercise 2\n",
    "\n",
    "View the text of the DOJ press released named 19-1224.pdf (in the docs folder).\n",
    "\n"
   ]
 },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Sample answer code</summary> \n",
    "<code style=\"background-color: white\">\n",
    "fn = 'docs/19-1224.txt'\n",
    "with open(fn, 'r') as infile:\n",
    "    text = infile.read() \n",
    "</code>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loading multiple files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from glob import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "glob('docs/*.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "filename_list = glob('docs/*.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pr_list = []\n",
    "for filename in filename_list:\n",
    "    with open(filename, 'r') as infile:\n",
    "        text = infile.read()\n",
    "        \n",
    "    info = {'filename' : filename,\n",
    "            'text'     : text}\n",
    "    \n",
    "    pr_list.append(info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pr_df = pd.DataFrame(pr_list)\n",
    "\n",
    "pr_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pr_df['text'].values[0])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}