W5-4 reading_typical_file_types.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading typical file types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## text files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've done this many times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_f = open(\"corpora/genesis.txt\")\n",
    "txt = text_f.read()\n",
    "txt[:500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(\"\\n\".encode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `csv` module is part of the standard python distribution\n",
    "\n",
    "csv = \"comma separated values\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "csvfile = open('corpora/tekno_flat.csv')\n",
    "tekno_reader = csv.DictReader(csvfile)\n",
    "data_list = []\n",
    "for row in tekno_reader:\n",
    "    data_list.append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install openpyxl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excel spreadsheets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can convert these to `csv` files. But then every sheet will be in a separate file. And you may lose some information.\n",
    "\n",
    "An alternative is the `openpyxl` library, which lets you work directly with spreadsheets and is pretty intuitive.\n",
    "\n",
    "https://openpyxl.readthedocs.io/en/stable/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openpyxl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpyxl import load_workbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb = load_workbook('corpora/tekno_fractions_nona.xlsx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wb.sheetnames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = wb[\"teknoclip7_1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws[\"A1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws.cell(row=1, column=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws[\"A2\"].value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for c in ws[\"C\"]:\n",
    "    print(c.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PDF files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PDF files are generally pretty unpleasant to deal with.\n",
    "\n",
    "`pdfminer` is a standard library that is geared toward extracting the text from pdf files.\n",
    "It's powerful but not very intuitive.\n",
    "\n",
    "https://euske.github.io/pdfminer/\n",
    "\n",
    "The current offical version of pdfminer doesn't work with python 3. \n",
    "But this \"fork\" of it does: \n",
    "\n",
    "https://github.com/pdfminer/pdfminer.six\n",
    "\n",
    "Here's a random page that's slightly helpful:\n",
    "\n",
    "https://www.blog.pythonlibrary.org/2018/05/03/exporting-data-from-pdfs-with-python/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install pdfminer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Command line usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to use pdfminer. One is from the command line in a terminal window.\n",
    "This is easier but it won't always get you what you want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``` \n",
    "pdf2txt.py -o output.txt -c \"ascii\" corpora/mtms2008-09-122a.pdf\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As aside: You can execute command line commands from within python using `subprocess`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "import os\n",
    "def extract_text(fpath, outdir):\n",
    "    newname = os.path.basename(fpath).split(\".\")[0] + \".txt\"\n",
    "    subprocess.call([\"pdf2txt.py\",\"-o\", outdir + \"/\" + newname, \"-c\", \"ascii\", fpath])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fpath = \"corpora/mtms2008-09-122a.pdf\"\n",
    "extract_text(fpath, \"pdf_extracts\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}