1_opening_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Opening files with python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** Ties de Kok ([Personal Website](https://www.tiesdekok.com))  \n",
    "**Last updated:** June 2020  \n",
    "**Conda Environment:** `LearnPythonForResearch`  \n",
    "**Python version:** Python 3.7   \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Some features (like the ToC) will only work if you run it locally, use Binder, or use nbviewer by clicking this link: \n",
    "https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/1_opening_files.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Python you can open and save a wide variety of files.  \n",
    "There are often multiple ways to open a particular file format, the examples below are in my experience the most convenient\n",
    "\n",
    "**Note:** All the sample files are in the `example_data` folder. The code to generate these files is at the end of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Table of Contents* <a id='toc'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Indexing a folder](#indexing)   \n",
    "* [Text files](#text-files)\n",
    "* [Excel files](#excel-files)   \n",
    "* [CSV files](#csv-files) \n",
    "* [Stata files](#stata-files)   \n",
    "* [SAS files](#sas-files) \n",
    "* [JSON files](#json-files)   \n",
    "* [HDF files: Pandas](#hdf-pandas-files)   \n",
    "* [HDF files: General Python objects](#hdf-general-files)\n",
    "* [Code to generate examples files](#example-files) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from glob import glob\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Indexing a folder</span><a id='indexing'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Irrespective of the type of file you are trying to open it is useful to be able to index all files in a folder.  \n",
    "This is necessary if you, for example, want to loop over all files in a folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to go about this, but I will show `os.listdir`, `glob`, and `os.walk`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First define the path to the folder that we want to index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our examples files are in the `example_data` folder so we can set the directory as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = os.path.join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all the files in the root of the folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* this will ignore files in sub-folders!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['311-service-requests.csv', 'auto_df.csv']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filenames = os.listdir(data_path)\n",
    "filenames[:2] # show first two items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\example_data\\\\311-service-requests.csv',\n",
       " 'E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\example_data\\\\auto_df.csv']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepaths = [os.path.join(data_path, filename) for filename in filenames]\n",
    "filepaths[:2] # show first two items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can alternatively use `glob` as this directly allows to include pathname matching.  \n",
    "For example if we only want Excel `.xlsx` files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\example_data\\\\excel_sample.xlsx']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob(os.path.join(data_path, '*.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all files, also those in sub-folders:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the folder contains multiple levels we need to either use `os.walk()` or `glob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\.gitignore',\n",
       " 'E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\0_python_basics.ipynb']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "folder = os.getcwd()\n",
    "filepaths = []\n",
    "for root,dirs,files in os.walk(folder):\n",
    "    for i in files:\n",
    "        filepaths.append(os.path.join(root,i))\n",
    "filepaths[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Personally, using `glob` yields cleaner code although it is a bit harder to understand:  \n",
    "*Note* `glob` will ignore files/folders that start with a dot, which is why the `.gitignore` file is not included"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\0_python_basics.ipynb',\n",
       " 'E:\\\\Dropbox\\\\Work\\\\Programming\\\\active\\\\LearnPythonforResearch\\\\1_opening_files.ipynb']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepaths_glob = glob(os.path.join(folder, '**/*'), recursive=True)\n",
    "filepaths_glob[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Text files</span><a id='text-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Opening text files is done using the default Python library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open a file with different file modes:  \n",
    "w -> write only  \n",
    "r -> read only  \n",
    "w+ -> read and write + completely overwrite file  \n",
    "a+ -> read and write + append at the bottom  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Opening a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'text_sample.txt'), 'r', encoding='utf-8') as file:\n",
    "    file_content = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning Python is great. \n",
      "Good luck!\n"
     ]
    }
   ],
   "source": [
    "print(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'text_sample.txt'), 'w+', encoding='utf-8') as file:\n",
    "    file.write('Learning Python is great. \\nGood luck!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that I am using a `with` statement when opening files.  \n",
    "Another method is to use `open` and `close`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(os.path.join(data_path, 'text_sample.txt'), 'r')\n",
    "file_content = f.read()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `with` method is preferred as it automatically closes the file.  \n",
    "This prevents the file from being 'in use' if you forget to use `.close()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looping over indexed files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_files = glob(os.path.join(data_path, '*.txt'))\n",
    "text_list = []\n",
    "\n",
    "for i in text_files:\n",
    "    with open(i, 'r') as f:\n",
    "        text_list.append(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Learning Python is great. \\nGood luck!']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Excel files</span><a id='excel-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open `Excel`, `csv`, `Stata`, `SAS` files in many ways, but I strongly recommend to use `Pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "excel_file = pd.read_excel(os.path.join(data_path, 'excel_sample.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function has a lot of options, see:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html  \n",
    "\n",
    "*Note:* You often want to specify the encoding to prevent errors, for example: `, encoding='utf-8'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "excel_file.to_excel(os.path.join(data_path, 'excel_sample.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This saves a `Pandas` dataframe object, see the data handling file.  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html  \n",
    "\n",
    "*Note:* You can save as `.xls` but also `.xlsx`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">CSV files</span><a id='csv-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open `Excel`, `csv`, `Stata`, `SAS` files in many ways, but I strongly recommend to use `Pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file = pd.read_csv(os.path.join(data_path, 'csv_sample.csv'), sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file.to_csv(os.path.join(data_path, 'csv_sample.csv'), sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Stata files</span><a id='stata-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Stata file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "stata_file = pd.read_stata(os.path.join(data_path, 'stata_sample.dta'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_stata.html  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Stata file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "stata_file.to_stata(os.path.join(data_path, 'stata_sample.dta'), write_index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_stata.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">SAS files</span><a id='sas-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can only read SAS files but cannot write them:  \n",
    "\n",
    "```\n",
    "sas_file = pd.read_sas(r'C:\\file.sas7bdat', format='sas7bdat')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sas.html  \n",
    "This function works in most cases but files with text are likely to throw hard to fix encoding errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">JSON files</span><a id='json-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files using pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html  \n",
    "\n",
    "*Note:* The path can also be a url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read JSON file to dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_df = pd.read_json(os.path.join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save dataframe to JSON file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_df.to_json(os.path.join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files using the `JSON` module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Read JSON:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'json_sample.json'), 'r', encoding='utf-8') as f:\n",
    "    json_data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Write JSON:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'json_sample.json'), 'w', encoding='utf-8') as f:\n",
    "    json.dump(json_data, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">HDF files for Pandas</span><a id='hdf-pandas-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The traditional formats such as .csv are not very efficient as big-data file formats.  \n",
    "\n",
    "For larger datasets I recommend to use the `Hierarchical Data Format` or `HDF` in short.\n",
    "This `.hdf` file format is designed to store and organize large amounts of data. \n",
    "\n",
    "Writing and reading `.hdf` files is extremely fast compared to `.csv`:\n",
    "\n",
    "**Writing:**\n",
    "\n",
    "```\n",
    "%timeit test_hdf_fixed_write(df)\n",
    "1 loops, best of 3: 237 ms per loop\n",
    "\n",
    "%timeit test_hdf_table_write(df)\n",
    "1 loops, best of 3: 901 ms per loop\n",
    "\n",
    "%timeit test_csv_write(df)\n",
    "1 loops, best of 3: 3.44 s per loop\n",
    "```\n",
    "\n",
    "**Reading:**\n",
    "\n",
    "```\n",
    "%timeit test_hdf_fixed_read()\n",
    "10 loops, best of 3: 19.1 ms per loop\n",
    "\n",
    "%timeit test_hdf_table_read()\n",
    "10 loops, best of 3: 39 ms per loop\n",
    "\n",
    "%timeit test_csv_read()\n",
    "1 loops, best of 3: 620 ms per loop\n",
    "```\n",
    "\n",
    "The downside of `HDF` is that the file sizes tend to be larger.   \n",
    "They can also not be easily inspected as you will need to load them into Python (or some other tool that can read HDF files)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read HDF files using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_df = pd.read_hdf(os.path.join(data_path, 'hdf_sample.h5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_hdf.html  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write HDF files using Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note*: Pandas requires to set a key. You can give it any `key` you like. I recommend to use the filename without `.h5` as `key`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_df.to_hdf(os.path.join(data_path, 'hdf_sample.h5'), 'hdf_sample')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_hdf.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">HDF files for general Python objects</span><a id='hdf-general-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above works great for Python dataframes, but sometimes you want to save general Python objects to HDF files as well.  \n",
    "\n",
    "Traditionally one would use `pickle` in order to do this, however, in my experience HDF files work a lot better. \n",
    "\n",
    "There are couple options, but I recommend the `hickle` library (https://github.com/telegraphic/hickle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hickle as hkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e' : 5}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save data using `hickle`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "hkl.dump(test_data, os.path.join(data_path, 'hkl_example.h5'), mode='w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data using `hickle`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data = hkl.load(os.path.join(data_path, 'hkl_example.h5'))\n",
    "test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Code used to generate example files</span><a id='example-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionary with random data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data = {'foreign':{1:'Domestic',2:'Domestic',3:'Domestic',6:'Domestic',7:'Domestic',8:'Domestic',9:'Domestic',14:'Domestic',21:'Domestic',23:'Domestic',24:'Domestic',30:'Domestic',31:'Domestic',33:'Domestic',37:'Domestic',38:'Domestic',43:'Domestic',48:'Domestic',50:'Domestic',51:'Domestic',53:'Foreign',56:'Foreign',57:'Foreign',66:'Foreign',70:'Foreign'},\n",
    "            'make':{1:'AMCPacer',2:'AMCSpirit',3:'BuickCentury',6:'BuickOpel',7:'BuickRegal',8:'BuickRiviera',9:'BuickSkylark',14:'Chev.Impala',21:'DodgeMagnum',23:'FordFiesta',24:'FordMustang',30:'Merc.Marquis',31:'Merc.Monarch',33:'Merc.Zephyr',37:'OldsDelta88',38:'OldsOmega',43:'Plym.Horizon',48:'Pont.GrandPrix',50:'Pont.Phoenix',51:'Pont.Sunbird',53:'AudiFox',56:'Datsun210',57:'Datsun510',66:'ToyotaCelica',70:'VWDiesel'},\n",
    "            'price': {1:4749,2:3799,3:4816,6:4453,7:5189,8:10372,9:4082,14:5705,21:5886,23:4389,24:4187,30:6165,31:4516,33:3291,37:4890,38:4181,43:4482,48:5222,50:4424,51:4172,53:6295,56:4589,57:5079,66:5899,70:5397},\n",
    "            'weight':{1:3350,2:2640,3:3250,6:2230,7:3280,8:3880,9:3400,14:3690,21:3600,23:1800,24:2650,30:3720,31:3370,33:2830,37:3690,38:3370,43:2200,48:3210,50:3420,51:2690,53:2070,56:2020,57:2280,66:2410,70:2040}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert dictionary to Pandas dataframe for easy saving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>foreign</th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>AMCPacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>3350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>AMCSpirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>2640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickCentury</td>\n",
       "      <td>4816</td>\n",
       "      <td>3250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickOpel</td>\n",
       "      <td>4453</td>\n",
       "      <td>2230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickRegal</td>\n",
       "      <td>5189</td>\n",
       "      <td>3280</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    foreign          make  price  weight\n",
       "1  Domestic      AMCPacer   4749    3350\n",
       "2  Domestic     AMCSpirit   3799    2640\n",
       "3  Domestic  BuickCentury   4816    3250\n",
       "6  Domestic     BuickOpel   4453    2230\n",
       "7  Domestic    BuickRegal   5189    3280"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data = pd.DataFrame(raw_data)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the different files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = os.path.join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'text_sample.txt'), 'w+') as file:\n",
    "    file.write('Learning Python is great. \\nGood luck!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data.to_excel(os.path.join(data_path, 'excel_sample.xlsx'))\n",
    "df_data.to_csv(os.path.join(data_path, 'csv_sample.csv'))\n",
    "df_data.to_stata(os.path.join(data_path, 'stata_sample.dta'))\n",
    "df_data.to_hdf(os.path.join(data_path, 'hdf_sample.h5'), 'hdf_sample')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data.to_json(os.path.join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** pandas does not have a `.to_sas()` function"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}