activity_python_os.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "activity_python_os.ipynb",
      "provenance": [],
      "private_outputs": true,
      "authorship_tag": "ABX9TyMQi68DXggFrbqn810bbaz+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/KenzieAcademy/python-notebooks/blob/master/activity_python_os.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-tB-TaevN9SW",
        "colab_type": "text"
      },
      "source": [
        "<img height=\"100px\" src=\"https://drive.google.com/uc?export=view&id=1Wv8etUe0EkGNpFe2n3lDLA3DsIFxEtAf\" />\n",
        "\n",
        "# Exploring Python's OS Library"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ibfKhwiaPKsa",
        "colab_type": "text"
      },
      "source": [
        "In Python, one of the most heavily used features of the language is the ability to manipulate local files:  Reading, writing, listing, sizing, checking, removing .. you get the idea.  To do this, Python provides a Standard Libary module named `os` (lowercase of course).  There is also a submodule within `os` named `os.path`.  The functionality between these modules can be roughly stated as:\n",
        " - os.path : Working with pathnames (joining directories, filenames, extensions, basenames, absolute paths)\n",
        " - os : Doing things with pathnames (creating, removing, copying, etc)\n",
        "\n",
        "As the name implies, `os` is Python's interface with your local operating system (MacOS, Linux, Windows).  Python’s `os` module does a little bit of everything; it’s sort of a junk drawer for system-related stuff.  In this activity we'll explore some of the capabilities of both of these standard libraries."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEC5JCoNqpxE",
        "colab_type": "text"
      },
      "source": [
        "## References\n",
        "- https://docs.python.org/3/library/os.html\n",
        "- https://docs.python.org/3/library/os.path.html\n",
        "- https://doughellmann.com/blog/the-python-3-standard-library-by-example/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XfQGx48WTr--",
        "colab_type": "text"
      },
      "source": [
        "# Traversing the File System\n",
        "You might recall that different filesystems have different ways of representing paths.\n",
        " - Windows: `C:\\Users\\name`\n",
        " - Mac/Linux: `/home/name`\n",
        "Notice that Windows uses the backslash `\\` as a path delimiter.  MacOS/Linux uses a forward slash `/`.   The `os` module is able to navigate around in the local filesystem, regardless of what OS you are using.  \n",
        "\n",
        "Be careful when constructing your own hard-coded pathnames\n",
        "```python\n",
        "folder = \"home\"\n",
        "user = \"daniel\"\n",
        "pathname = folder + \"/\" + user  # NO\n",
        "pathname = os.path.join(folder, user)  # YES\n",
        "```\n",
        "The first form is \"brittle\" and will break if you try to run your Python program on a Windows machine.  By allowing Python to choose the correct path delimiter with the `os.path.join` function, the second form is now flexible and will work on all platforms."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwMgTzQfXjTi",
        "colab_type": "text"
      },
      "source": [
        "# Exercises\n",
        "In each of the exercises, replace any `???` with an appropriate function call from the os module.  Example:\n",
        "```python\n",
        "# list files\n",
        "os.??? --> os.listdir()\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gUSdBjMneh5O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Before we begin the exercises, let's set up a handy function to print data.\n",
        "import json\n",
        "def print_data(title, data):\n",
        "    print(f\"{title}\\n{json.dumps(data, indent=4)}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lsqC8XPwXstv",
        "colab_type": "text"
      },
      "source": [
        "## 1. Listing items: listdir, scandir, walk\n",
        "Note the title uses the word \"items\".  This is because a directory may contain files as well as more directories.  So we call everything an 'item' in a directory.  This notebook may be running on your local machine if you copied it, but chances are it is running on a cloud server somewhere.  Lets have a look around:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KXuJL6HQH1Pk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "\n",
        "# Where are we?\n",
        "cwd = os.getcwd()\n",
        "print_data(\"Current directory:\", cwd)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTkj59mHc_Xn",
        "colab_type": "text"
      },
      "source": [
        "### Output from cell below should look like this:\n",
        "```console\n",
        "Current Dir Items:\n",
        "[\n",
        "    \".config\",\n",
        "    \"sample_data\"\n",
        "]\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nz4NNwbNI8Zi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# What files and/or directories are here?\n",
        "file_list = ???  # Gets a list of items in the current directory\n",
        "print_data(\"Current Dir Items:\", file_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f861M_iMoTvy",
        "colab_type": "text"
      },
      "source": [
        "If more information is needed than just the names of the files, it is more efficient to use `scandir()` than `listdir()` because more information is collected in one system call when the directory is scanned.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qO19A7AVohxg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get some more details about each item\n",
        "for item in ???:\n",
        "    if item.is_dir():\n",
        "        typ = 'D'\n",
        "    elif item.is_file():\n",
        "        typ = 'F'\n",
        "    elif item.is_symlink():\n",
        "        typ = 'L'\n",
        "    else:\n",
        "        typ = '?'\n",
        "    print(f'{item.name} {typ}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_X9S4uTPd01B",
        "colab_type": "text"
      },
      "source": [
        "### A deeper look\n",
        "We can use a for-loop to probe into the `file_list` that was returned.  Output from cell below should look like this:\n",
        "```console\n",
        "Contents of .config:\n",
        "[\n",
        "    \".last_update_check.json\",\n",
        "    \".last_opt_in_prompt.yaml\",\n",
        "    \"gce\",\n",
        "    \"logs\",\n",
        "    \".last_survey_prompt.yaml\",\n",
        "    \"active_config\",\n",
        "    \"config_sentinel\",\n",
        "    \"configurations\",\n",
        "    \".metricsUUID\"\n",
        "]\n",
        "Contents of sample_data:\n",
        "[\n",
        "    \"README.md\",\n",
        "    \"anscombe.json\",\n",
        "    \"california_housing_train.csv\",\n",
        "    \"california_housing_test.csv\",\n",
        "    \"mnist_train_small.csv\",\n",
        "    \"mnist_test.csv\"\n",
        "]\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F86MaCyvd4E0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Let's go one level deeper: List each item in the list.\n",
        "for item in file_list:\n",
        "    content_list = os.listdir(item)  # content_list is NOT sorted.\n",
        "    print_data(f\"Contents of {item}:\", content_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dmfRvnh9RyB",
        "colab_type": "text"
      },
      "source": [
        "### Walking around\n",
        "The function `walk()` traverses a directory recursively and for each subdirectory generates a 3-tuple containing the directory path, a list of any immediate sub-directories of that path, and a list of the names of any files in that directory.  You have to supply a starting directory as a parameter.\n",
        "\n",
        "Your output data should look something like this:\n",
        ">```console\n",
        "Contents of Directory .:\n",
        "[\n",
        "    \".config\",\n",
        "    \"sample_data\"\n",
        "]\n",
        "Contents of Directory ./.config:\n",
        "[\n",
        "    \"logs\",\n",
        "    \"configurations\",\n",
        "    \".last_update_check.json\",\n",
        "    \".last_opt_in_prompt.yaml\",\n",
        "    \"gce\",\n",
        "    \".last_survey_prompt.yaml\",\n",
        "    \"active_config\",\n",
        "...\n",
        "(and perhaps more)\n",
        "...\n",
        "```  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8RwvG78sipwS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Traverse (iterate over) the entire current directory,\n",
        "# print everything found\n",
        "for dir_name, sub_dirs, files in ???:  # Note the 3-tuple unpacking\n",
        "    # Mix the directory contents together\n",
        "    contents = sub_dirs + files\n",
        "    print_data(f\"Contents of Directory {dir_name}:\", contents)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgHsyWKZV1Dx",
        "colab_type": "text"
      },
      "source": [
        "## 2. Create a new directory\n",
        "Let's now create a SINGLE new directory in our current working folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QTvLSAqTWFBh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "# create one new empty directory in current working directory\n",
        "os.mkdir('new_directory')\n",
        "# Should print something like ['.config', 'new_directory', 'sample_data']\n",
        "print(os.listdir())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N25MhTKAotoR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create a new directory recursively, with multiple parent directories\n",
        "\n",
        "# Leaf directory name.  It's at the end of the path \"branch\"\n",
        "directory = \"leafdir\"\n",
        "# Parent Directory heirarchy\n",
        "parent_dir = \"parent/subdir_1/subdir_2/subdir_3\"\n",
        "# Compose the path \n",
        "path = os.path.join(parent_dir, directory) \n",
        "# Create the directory\n",
        "os.makedirs(path)\n",
        "# Should print something like ['.config', 'parent', 'new_directory', 'sample_data']\n",
        "print(os.listdir())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GTeiEd7FW9on",
        "colab_type": "text"
      },
      "source": [
        "## 3. Rename a directory\n",
        "Change the name of 'new_directory' to 'my_dir'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zqC8U_uEat3N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "os.rename('new_directory', 'my_dir')\n",
        "# Should print ['.config', 'parent', 'my_dir', 'sample_data']\n",
        "print(os.listdir())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8L4vHvbfp-k",
        "colab_type": "text"
      },
      "source": [
        "## 4. Create a new file in `my_dir`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d8i61f5egNcP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# compose path + file\n",
        "filename = os.path.join('my_dir', 'last_night.txt')\n",
        "# write some text to file\n",
        "with open(filename, 'w') as f:\n",
        "    f.write(\"Sag mal was haben wir denn gestern Nacht so getrieben?\")\n",
        "# Should print ['last_night.txt']\n",
        "print(os.listdir('my_dir'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gUhqF0RGj-E",
        "colab_type": "text"
      },
      "source": [
        "## 5. What is an absolute path?\n",
        "*Absolute paths* always start with a `/` character which represents the root of the filesystem tree.  If a path does not start with `/`, it is a *relative path*, which means that in order to navigate to that resource, you need to know where your starting point is.  Find the absolute path of the `last_night.txt` file that we just created."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k7tUQ1ajI-hq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Print the absolute path of the last_night.txt file\n",
        "# Should print \"/content/my_dir/last_night.txt\"\n",
        "print(os.path.abspath(filename))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cs5lsxAZKCco",
        "colab_type": "text"
      },
      "source": [
        "## 6. What is a basename?\n",
        "A file path consists of two components:  The path, and the actual filename.  The basename is the same as the filename.  For example, if the absolute path is `/content/my_dir/last_night.txt`, then the basename is `last_night.txt`.  From the absolute path, determine the basename."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D3mcUA_9LJd4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "full_path = os.path.abspath(filename)\n",
        "print(\"The full (absolute) path is: \", full_path)\n",
        "base_name = os.path.basename(full_path)\n",
        "print(\"The basename is: \", base_name)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9YSeU21Mgv4",
        "colab_type": "text"
      },
      "source": [
        "## 7. Check if a file exists\n",
        "Sometimes you will need to check whether a particular file exists or does not exist, and take some action based on that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BMPj3pVfM020",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Does the last_night.txt file exist?\n",
        "# This illustrates the LBYL way\n",
        "if os.path.exists(filename):\n",
        "    text = open(filename).read()\n",
        "    size = os.path.getsize(filename)\n",
        "    print(f\"File {filename} has size {size} bytes\")\n",
        "    print(f\"Contents of {filename}: {text}\")\n",
        "else:\n",
        "    print(f\"Unable to read {filename}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZ6vH1bF8FFG",
        "colab_type": "text"
      },
      "source": [
        "### Warning about anti-patterns\n",
        "The code snippet in exercise 7 contains an anti-pattern.  Anti-patterns are a common response to a recurring problem that is usually _ineffective_ and **risks being highly counterproductive**.  The problem here is that there is a small but non-zero time gap between checking whether the file exists, and reading it.  Since we are dealing with very fast computers however, this time gap is effectively an eternity.  Anything can happen during that time gap, including having this file deleted right after you thought it was safe to read from it.  A better way is to just try the read, and handle the error if it happens.  This illustrates the differene between LBYL and EAFP.\n",
        "\n",
        "- LBYL = **\"Look Before You Leap\"**  e.g. check first, then do it.\n",
        "- EAFP = **\"Easier to Ask Forgiveness instead of Permission\"** e.g. just try it, and handle errors after the fact"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ufctXZ0vFL4t",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# A better approach than LBYL is EAFP in this case\n",
        "try:\n",
        "    text = open(filename).read()\n",
        "    size = os.path.getsize(filename)\n",
        "    print(f\"File {filename} has size {size} bytes\")\n",
        "    print(f\"Contents of {filename}: {text}\")\n",
        "except OSError:\n",
        "    print(f\"Unable to read {filename}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6mtGFuUnjofy",
        "colab_type": "text"
      },
      "source": [
        "## 8. Check if a path refers to a directory or a file\n",
        "Sometimes it's not obvious whether a path is pointing at a directory or an actual file which can be opened."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Ads48KJkFlY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if os.path.isdir(filename):\n",
        "    print(f\"{filename} is a directory\")\n",
        "else:\n",
        "    print(f\"{filename} is NOT a directory\")\n",
        "\n",
        "if os.path.isdir(path):\n",
        "    print(f\"{path} is a directory\")\n",
        "else:\n",
        "    print(f\"{path} is NOT a directory\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MurHCJvjgEx7",
        "colab_type": "text"
      },
      "source": [
        "## 9. Delete a directory\n",
        "In order to delete a directory with os, the directory must first be empty.\n",
        "Note this operation will raise an OSError exception:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TIYCoChRgOv1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Attempt to remove my_dir\n",
        "os.rmdir('my_dir')\n",
        "print(os.listdir())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHSOTQLikSE3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Delete the file first, then the directory can be removed\n",
        "print('Removing file: ', filename)\n",
        "os.remove(filename)\n",
        "# Now the directory removal will succeed\n",
        "os.rmdir('my_dir')\n",
        "print(os.listdir())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8_LjeR3KPYP",
        "colab_type": "text"
      },
      "source": [
        "## 10. Find current Process ID (PID)\n",
        "Every program running on a computer has a PID.  PIDs are how the Operating System keeps track of all the different running applications"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_nKAvB6dKucW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "my_pid = os.getpid()\n",
        "print(\"My current PID is: \", my_pid)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oNCQrC4hlS3J",
        "colab_type": "text"
      },
      "source": [
        "# Conclusions\n",
        "The OS module in Python Standard Library provides a rich set of functions for interfacing directly with your Operating System, whether it is Windows, Linux, or MacOS. We have only tried out a handful of the available methods in this workbook."
      ]
    }
  ]
}