generate_data_with_Faker.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/groda/big_data/blob/master/generate_data_with_Faker.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwLHeIOc69kD"
      },
      "source": [
        "<a href=\"https://github.com/groda/big_data\"><div><img src=\"https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true\" align=right width=\"90\"></div></a>\n",
        "\n",
        "# Data Generation and Aggregation with Python's Faker Library and PySpark\n",
        "<br>\n",
        "<br>\n",
        "\n",
        "Explore the capabilities of the Python Faker library (https://faker.readthedocs.io/) for dynamic data generation!\n",
        "\n",
        "Whether you're a data scientist, engineer, or analyst, this tutorial will guide you through the process of creating realistic and diverse datasets using Faker and then harnessing the distributed computing capabilities of PySpark to aggregate and analyze the generated data.\n",
        "\n",
        "\n",
        "\n",
        "**Note:** This is not _synthetic_ data as it is generated using simple methods and will most likely not fit any real-life distribution. Still, it serves as a valuable resource for testing purposes when authentic data is unavailable."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install Faker\n",
        "\n",
        "The Python `faker` module needs to be installed. Note that on Google Colab you can use `!pip` as well as just `pip` (no exclamation mark)."
      ],
      "metadata": {
        "id": "BW6yZdpy7run"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install faker"
      ],
      "metadata": {
        "id": "7EOLVOTe7KnP",
        "outputId": "b72af9eb-9fa4-46ed-da58-1d497c43fe8b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting faker\n",
            "  Downloading Faker-23.2.1-py3-none-any.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.4 in /usr/local/lib/python3.10/dist-packages (from faker) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.4->faker) (1.16.0)\n",
            "Installing collected packages: faker\n",
            "Successfully installed faker-23.2.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Generate a Pandas dataframe with fake data"
      ],
      "metadata": {
        "id": "UYLa4gsrB7aS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import `Faker` and set a random seed ($42$)."
      ],
      "metadata": {
        "id": "yKviUSgj7nHP"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7fbGCBoq69kF"
      },
      "outputs": [],
      "source": [
        "from faker import Faker\n",
        "# Set the seed value of the shared `random.Random` object\n",
        "# across all internal generators that will ever be created\n",
        "Faker.seed(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yUEes2gl69kF"
      },
      "source": [
        "`fake` is a fake data generator with `DE_de` locale."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bf7UTx_r69kG"
      },
      "outputs": [],
      "source": [
        "fake = Faker('de_DE')\n",
        "fake.seed_locale('de_DE', 42)\n",
        "# Creates and seeds a unique `random.Random` object for\n",
        "# each internal generator of this `Faker` instance\n",
        "fake.seed_instance(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0bSXBUI469kG"
      },
      "source": [
        "Import Pandas to save data into a dataframe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PI_YqfM169kG"
      },
      "outputs": [],
      "source": [
        "# true if running on Google Colab\n",
        "import sys\n",
        "IN_COLAB = 'google.colab' in sys.modules\n",
        "if not IN_COLAB:\n",
        " !pip install pandas==1.5.3\n",
        "\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LjCsGikw69kG"
      },
      "source": [
        "The function `create_row_faker` creates one row of fake data. Here we choose to generate a row containing the following fields:\n",
        " - `fake.name()`\n",
        " - `fake.postcode()`\n",
        " - `fake.email()`\n",
        " - `fake.country()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f1MiSZl069kG"
      },
      "outputs": [],
      "source": [
        "def create_row_faker(num=1):\n",
        "    output = [{\"name\": fake.name(),\n",
        "               \"age\": fake.random_int(0, 100),\n",
        "               \"postcode\": fake.postcode(),\n",
        "               \"email\": fake.email(),\n",
        "               \"nationality\": fake.country(),\n",
        "              } for x in range(num)]\n",
        "    return output"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TeuZISIh69kH"
      },
      "source": [
        "Generate a single row"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wXP-5uSg69kH",
        "outputId": "e5d284c3-2587-451d-aab8-45f9ba7c99d5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'name': 'Aleksandr Weinhage',\n",
              "  'age': 35,\n",
              "  'postcode': '32181',\n",
              "  'email': 'bbeckmann@example.org',\n",
              "  'nationality': 'Fidschi'}]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "create_row_faker()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4zIjkEOw69kI"
      },
      "source": [
        "Generate a dataframe `df_fake` of 5000 rows using `create_row_faker`.\n",
        "\n",
        "We're using the _cell magic_ `%%time` to time the operation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JtRWDEsT69kI",
        "outputId": "75fc519b-a19c-4fcf-db0e-13d98f394f8c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 358 ms, sys: 1.2 ms, total: 359 ms\n",
            "Wall time: 366 ms\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "df_fake = pd.DataFrame(create_row_faker(5000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meT16YZy69kI"
      },
      "source": [
        "View dataframe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RJK93FxW69kI",
        "outputId": "50d3d1ef-24e6-485c-a43e-40470594cd4a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                 name  age postcode  \\\n",
              "0               Prof. Kurt Bauer B.A.   91    37940   \n",
              "1              Ekkehart Wilms-Kallert   13    61559   \n",
              "2            Annelise Röhrdanz-Hornig   80    93103   \n",
              "3                  Magrit Knappe B.A.   47    34192   \n",
              "4     Univ.Prof. Gotthilf Wirth B.Sc.   29    56413   \n",
              "...                               ...  ...      ...   \n",
              "4995                     Janusz Nette   74    33091   \n",
              "4996               Frau Cathleen Bähr   97    89681   \n",
              "4997                      Ulla Seidel   66    28358   \n",
              "4998                 Janin Speer MBA.   64    76879   \n",
              "4999               Emma Freudenberger   25    98648   \n",
              "\n",
              "                               email          nationality  \n",
              "0          hildaloeffler@example.com            Guatemala  \n",
              "1                 maja07@example.net            Brasilien  \n",
              "2               daniel31@example.com            Guatemala  \n",
              "3       gottliebmitschke@example.com           Guadeloupe  \n",
              "4                heini76@example.net              Litauen  \n",
              "...                              ...                  ...  \n",
              "4995  ann-kathrinsiering@example.net              Belarus  \n",
              "4996      hethurhubertus@example.org       St. Barthélemy  \n",
              "4997        klotzbabette@example.net            St. Lucia  \n",
              "4998   giesskarl-hermann@example.com             Kroatien  \n",
              "4999             andre92@example.net  St. Kitts und Nevis  \n",
              "\n",
              "[5000 rows x 5 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cf796d22-0e4e-4a19-a092-cc107f499ee6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>age</th>\n",
              "      <th>postcode</th>\n",
              "      <th>email</th>\n",
              "      <th>nationality</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Prof. Kurt Bauer B.A.</td>\n",
              "      <td>91</td>\n",
              "      <td>37940</td>\n",
              "      <td>hildaloeffler@example.com</td>\n",
              "      <td>Guatemala</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ekkehart Wilms-Kallert</td>\n",
              "      <td>13</td>\n",
              "      <td>61559</td>\n",
              "      <td>maja07@example.net</td>\n",
              "      <td>Brasilien</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Annelise Röhrdanz-Hornig</td>\n",
              "      <td>80</td>\n",
              "      <td>93103</td>\n",
              "      <td>daniel31@example.com</td>\n",
              "      <td>Guatemala</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Magrit Knappe B.A.</td>\n",
              "      <td>47</td>\n",
              "      <td>34192</td>\n",
              "      <td>gottliebmitschke@example.com</td>\n",
              "      <td>Guadeloupe</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Univ.Prof. Gotthilf Wirth B.Sc.</td>\n",
              "      <td>29</td>\n",
              "      <td>56413</td>\n",
              "      <td>heini76@example.net</td>\n",
              "      <td>Litauen</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4995</th>\n",
              "      <td>Janusz Nette</td>\n",
              "      <td>74</td>\n",
              "      <td>33091</td>\n",
              "      <td>ann-kathrinsiering@example.net</td>\n",
              "      <td>Belarus</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4996</th>\n",
              "      <td>Frau Cathleen Bähr</td>\n",
              "      <td>97</td>\n",
              "      <td>89681</td>\n",
              "      <td>hethurhubertus@example.org</td>\n",
              "      <td>St. Barthélemy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4997</th>\n",
              "      <td>Ulla Seidel</td>\n",
              "      <td>66</td>\n",
              "      <td>28358</td>\n",
              "      <td>klotzbabette@example.net</td>\n",
              "      <td>St. Lucia</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4998</th>\n",
              "      <td>Janin Speer MBA.</td>\n",
              "      <td>64</td>\n",
              "      <td>76879</td>\n",
              "      <td>giesskarl-hermann@example.com</td>\n",
              "      <td>Kroatien</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4999</th>\n",
              "      <td>Emma Freudenberger</td>\n",
              "      <td>25</td>\n",
              "      <td>98648</td>\n",
              "      <td>andre92@example.net</td>\n",
              "      <td>St. Kitts und Nevis</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5000 rows × 5 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cf796d22-0e4e-4a19-a092-cc107f499ee6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-cf796d22-0e4e-4a19-a092-cc107f499ee6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-cf796d22-0e4e-4a19-a092-cc107f499ee6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-629f3186-a0ed-4821-acf8-af095e996579\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-629f3186-a0ed-4821-acf8-af095e996579')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-629f3186-a0ed-4821-acf8-af095e996579 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_fake",
              "summary": "{\n  \"name\": \"df_fake\",\n  \"rows\": 5000,\n  \"fields\": [\n    {\n      \"column\": \"name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4994,\n        \"samples\": [\n          \"Richard Scheuermann-Adler\",\n          \"Karl-Ernst Schaaf\",\n          \"Babette Roskoth-Sontag\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"age\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 29,\n        \"min\": 0,\n        \"max\": 100,\n        \"num_unique_values\": 101,\n        \"samples\": [\n          94,\n          33,\n          78\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"postcode\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4878,\n        \"samples\": [\n          \"67428\",\n          \"31260\",\n          \"82922\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"email\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4972,\n        \"samples\": [\n          \"willystahr@example.com\",\n          \"hsteinberg@example.com\",\n          \"hannilindau@example.net\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"nationality\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 247,\n        \"samples\": [\n          \"Vereinigte Staaten\",\n          \"Griechenland\",\n          \"Bahamas\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "df_fake"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86GX2vpJ69kI"
      },
      "source": [
        "For more fake data generators see Faker's [standard providers](https://faker.readthedocs.io/en/master/providers.html#standard-providers) as well as [community providers](https://faker.readthedocs.io/en/master/communityproviders.html#community-providers)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Generate PySpark dataframe with fake data"
      ],
      "metadata": {
        "id": "6Yv5FGq0CH6D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install PySpark."
      ],
      "metadata": {
        "id": "qw0OUfBE8Aom"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pyspark"
      ],
      "metadata": {
        "id": "PRKNcWR87h7P",
        "outputId": "2d8a788c-2161-4248-9ba3-394464e38706",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pyspark\n",
            "  Downloading pyspark-3.5.0.tar.gz (316.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m316.9/316.9 MB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: py4j==0.10.9.7 in /usr/local/lib/python3.10/dist-packages (from pyspark) (0.10.9.7)\n",
            "Building wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.5.0-py2.py3-none-any.whl size=317425345 sha256=43bca49eea628a32285a1d5b22fa7ba997738b819fdaec8ab315e7075636d9d4\n",
            "  Stored in directory: /root/.cache/pip/wheels/41/4e/10/c2cf2467f71c678cfc8a6b9ac9241e5e44a01940da8fbb17fc\n",
            "Successfully built pyspark\n",
            "Installing collected packages: pyspark\n",
            "Successfully installed pyspark-3.5.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9drMVtgJ69kI"
      },
      "outputs": [],
      "source": [
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession \\\n",
        "    .builder \\\n",
        "    .appName(\"Faker demo\") \\\n",
        "    .getOrCreate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SYIUxAj569kI"
      },
      "outputs": [],
      "source": [
        "df = spark.createDataFrame(create_row_faker(5000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QbBAHBLF69kJ"
      },
      "source": [
        "To avoid getting the warning, either use [pyspark.sql.Row](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Row) and let Spark infer datatypes or create a schema for the dataframe specifying the datatypes of all fields (here's the list of all [datatypes](http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=types#module-pyspark.sql.types))."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qwaQQ2Ud69kJ"
      },
      "outputs": [],
      "source": [
        "from pyspark.sql.types import *\n",
        "schema = StructType([StructField('name', StringType()),\n",
        "                     StructField('age',IntegerType()),\n",
        "                     StructField('postcode',StringType()),\n",
        "                     StructField('email', StringType()),\n",
        "                     StructField('nationality',StringType())])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NM4OqVld69kJ"
      },
      "outputs": [],
      "source": [
        "df = spark.createDataFrame(create_row_faker(5000), schema)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NDPqZqnl69kJ",
        "outputId": "377ebb1c-4dbe-41e1-fc4f-3af97042fc1b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- name: string (nullable = true)\n",
            " |-- age: integer (nullable = true)\n",
            " |-- postcode: string (nullable = true)\n",
            " |-- email: string (nullable = true)\n",
            " |-- nationality: string (nullable = true)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df.printSchema()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HwsIhJxO69kJ"
      },
      "source": [
        "Let's generate some more data (dataframe with $5\\cdot10^4$ rows). The file will be partitioned by Spark."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KZtRuVeR69kJ",
        "outputId": "6f8a7dd0-02a8-4395-c7da-612365c8b082",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 5.9 s, sys: 58 ms, total: 5.95 s\n",
            "Wall time: 7.63 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "n = 5*10**4\n",
        "df = spark.createDataFrame(create_row_faker(n), schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l0ZiQJKp69kJ"
      },
      "source": [
        "It took a long time (~4 sec. for 50000 rows)!\n",
        "\n",
        "Can we do better?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xI7IjQ669kK"
      },
      "source": [
        "The function `create_row_faker()` returns a list. This is not efficient, what we need is a _generator_ instead."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bt2HncFZ69kK",
        "outputId": "347815f8-00bc-4c77-8029-7a5a76f85b71",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "list"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "d = create_row_faker(5)\n",
        "# what type is d?\n",
        "type(d)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EC_IgTtY69kK"
      },
      "source": [
        "Now `d` is a generator"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9YjvEOTV69kK",
        "outputId": "b0400b20-efa8-4feb-9e1f-c956efc222cd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "generator"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "d = ({\"name\": fake.name(),\n",
        "      \"age\": fake.random_int(0, 100),\n",
        "      \"postcode\": fake.postcode(),\n",
        "      \"email\": fake.email(),\n",
        "      \"nationality\": fake.country()} for i in range(5))\n",
        "# what type is d?\n",
        "type(d)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BBdz-PJz69kK",
        "outputId": "7bc09338-b81d-4e33-c87c-01422d1cc749",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 3.67 s, sys: 28 ms, total: 3.7 s\n",
            "Wall time: 3.81 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "n = 5*10**4\n",
        "d = ({\"name\": fake.name(),\n",
        "      \"age\": fake.random_int(0, 100),\n",
        "      \"postcode\": fake.postcode(),\n",
        "      \"email\": fake.email(),\n",
        "      \"nationality\": fake.country()}\n",
        "     for i in range(n))\n",
        "df = spark.createDataFrame(d, schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bd-jNkqk69kK"
      },
      "source": [
        "This wasn't faster.\n",
        "\n",
        "I will look into how one can leverage Hadoop's parallelism to generate dataframes and speed the process."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Filter and aggregate with PySpark"
      ],
      "metadata": {
        "id": "t_NNqOXPCbj3"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DyDn7tqS69kK",
        "outputId": "7ebe42de-a1a8-4933-dd89-128dd4ee530a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 186
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pyspark.sql.dataframe.DataFrame"
            ],
            "text/html": [
              "<div style=\"max-width:800px; border: 1px solid var(--colab-border-color);\"><style>\n",
              "      pre.function-repr-contents {\n",
              "        overflow-x: auto;\n",
              "        padding: 8px 12px;\n",
              "        max-height: 500px;\n",
              "      }\n",
              "\n",
              "      pre.function-repr-contents.function-repr-contents-collapsed {\n",
              "        cursor: pointer;\n",
              "        max-height: 100px;\n",
              "      }\n",
              "    </style>\n",
              "    <pre style=\"white-space: initial; background:\n",
              "         var(--colab-secondary-surface-color); padding: 8px 12px;\n",
              "         border-bottom: 1px solid var(--colab-border-color);\"><b>pyspark.sql.dataframe.DataFrame</b><br/>def __init__(jdf: JavaObject, sql_ctx: Union[&#x27;SQLContext&#x27;, &#x27;SparkSession&#x27;])</pre><pre class=\"function-repr-contents function-repr-contents-collapsed\" style=\"\"><a class=\"filepath\" style=\"display:none\" href=\"#\">/usr/local/lib/python3.10/dist-packages/pyspark/sql/dataframe.py</a>A distributed collection of data grouped into named columns.\n",
              "\n",
              ".. versionadded:: 1.3.0\n",
              "\n",
              ".. versionchanged:: 3.4.0\n",
              "    Supports Spark Connect.\n",
              "\n",
              "Examples\n",
              "--------\n",
              "A :class:`DataFrame` is equivalent to a relational table in Spark SQL,\n",
              "and can be created using various functions in :class:`SparkSession`:\n",
              "\n",
              "&gt;&gt;&gt; people = spark.createDataFrame([\n",
              "...     {&quot;deptId&quot;: 1, &quot;age&quot;: 40, &quot;name&quot;: &quot;Hyukjin Kwon&quot;, &quot;gender&quot;: &quot;M&quot;, &quot;salary&quot;: 50},\n",
              "...     {&quot;deptId&quot;: 1, &quot;age&quot;: 50, &quot;name&quot;: &quot;Takuya Ueshin&quot;, &quot;gender&quot;: &quot;M&quot;, &quot;salary&quot;: 100},\n",
              "...     {&quot;deptId&quot;: 2, &quot;age&quot;: 60, &quot;name&quot;: &quot;Xinrong Meng&quot;, &quot;gender&quot;: &quot;F&quot;, &quot;salary&quot;: 150},\n",
              "...     {&quot;deptId&quot;: 3, &quot;age&quot;: 20, &quot;name&quot;: &quot;Haejoon Lee&quot;, &quot;gender&quot;: &quot;M&quot;, &quot;salary&quot;: 200}\n",
              "... ])\n",
              "\n",
              "Once created, it can be manipulated using the various domain-specific-language\n",
              "(DSL) functions defined in: :class:`DataFrame`, :class:`Column`.\n",
              "\n",
              "To select a column from the :class:`DataFrame`, use the apply method:\n",
              "\n",
              "&gt;&gt;&gt; age_col = people.age\n",
              "\n",
              "A more concrete example:\n",
              "\n",
              "&gt;&gt;&gt; # To create DataFrame using SparkSession\n",
              "... department = spark.createDataFrame([\n",
              "...     {&quot;id&quot;: 1, &quot;name&quot;: &quot;PySpark&quot;},\n",
              "...     {&quot;id&quot;: 2, &quot;name&quot;: &quot;ML&quot;},\n",
              "...     {&quot;id&quot;: 3, &quot;name&quot;: &quot;Spark SQL&quot;}\n",
              "... ])\n",
              "\n",
              "&gt;&gt;&gt; people.filter(people.age &gt; 30).join(\n",
              "...     department, people.deptId == department.id).groupBy(\n",
              "...     department.name, &quot;gender&quot;).agg({&quot;salary&quot;: &quot;avg&quot;, &quot;age&quot;: &quot;max&quot;}).show()\n",
              "+-------+------+-----------+--------+\n",
              "|   name|gender|avg(salary)|max(age)|\n",
              "+-------+------+-----------+--------+\n",
              "|     ML|     F|      150.0|      60|\n",
              "|PySpark|     M|       75.0|      50|\n",
              "+-------+------+-----------+--------+\n",
              "\n",
              "Notes\n",
              "-----\n",
              "A DataFrame should only be created as described above. It should not be directly\n",
              "created via using the constructor.</pre>\n",
              "      <script>\n",
              "      if (google.colab.kernel.accessAllowed && google.colab.files && google.colab.files.view) {\n",
              "        for (const element of document.querySelectorAll('.filepath')) {\n",
              "          element.style.display = 'block'\n",
              "          element.onclick = (event) => {\n",
              "            event.preventDefault();\n",
              "            event.stopPropagation();\n",
              "            google.colab.files.view(element.textContent, 80);\n",
              "          };\n",
              "        }\n",
              "      }\n",
              "      for (const element of document.querySelectorAll('.function-repr-contents')) {\n",
              "        element.onclick = (event) => {\n",
              "          event.preventDefault();\n",
              "          event.stopPropagation();\n",
              "          element.classList.toggle('function-repr-contents-collapsed');\n",
              "        };\n",
              "      }\n",
              "      </script>\n",
              "      </div>"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ],
      "source": [
        "type(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c5X9Bz7g69kL"
      },
      "source": [
        "Show the first five records in the dataframe of fake data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0RYAfYsx69kL",
        "outputId": "8066ee10-f105-42fe-b514-fc750ecb4b13",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------------------+---+--------+------------------------+-----------+\n",
            "|name                 |age|postcode|email                   |nationality|\n",
            "+---------------------+---+--------+------------------------+-----------+\n",
            "|Rudolf Rust          |76 |86566   |kornelia11@example.org  |Indonesien |\n",
            "|Kerstin Putz-Köster  |54 |18953   |qrudolph@example.org    |Kenia      |\n",
            "|Ivo Schinke B.Sc.    |3  |60202   |radischjames@example.net|Chile      |\n",
            "|Hans-Henning Staude  |82 |68552   |ladislaus89@example.com |Thailand   |\n",
            "|Justine Weinhage B.A.|53 |62346   |ytaesche@example.org    |Österreich |\n",
            "+---------------------+---+--------+------------------------+-----------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df.show(n=5, truncate=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ikvl-9Mm69kL"
      },
      "source": [
        "Do some data aggregation:\n",
        " - group by postcode\n",
        " - count the number of persons and the average age for each postcode\n",
        " - filter out postcodes with less than 4 persons\n",
        " - sort by average age descending\n",
        " - show the first 5 entries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1lr4WKxt69kL",
        "outputId": "5a15bd58-23dc-46db-df7c-67ae48af6b42",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+-----+-----------+\n",
            "|postcode|Count|Average age|\n",
            "+--------+-----+-----------+\n",
            "|   18029|    4|      91.75|\n",
            "|   67611|    4|       87.0|\n",
            "|   47898|    4|       85.5|\n",
            "|   45386|    4|      82.75|\n",
            "|   46755|    4|       78.5|\n",
            "+--------+-----+-----------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import pyspark.sql.functions as F\n",
        "df.groupBy('postcode') \\\n",
        "  .agg(F.count('postcode').alias('Count'), F.round(F.avg('age'), 2).alias('Average age')) \\\n",
        "  .filter('Count>3') \\\n",
        "  .orderBy('Average age', ascending=False) \\\n",
        "  .show(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PP4TeBbh69kL"
      },
      "source": [
        "Postcode $18029$ has the highest average age ($91.75$). Show all entries for postcode $18029$ using `filter`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_6NsPWUt69kL",
        "outputId": "0ed58e60-7a25-4d21-ee73-243da451e180",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------------------------+---+--------+------------------------------+------------------+\n",
            "|name                       |age|postcode|email                         |nationality       |\n",
            "+---------------------------+---+--------+------------------------------+------------------+\n",
            "|Univ.Prof. Roderich Liebelt|89 |18029   |anne-katrinscholtz@example.com|Grönland          |\n",
            "|Herwig Matthäi B.A.        |90 |18029   |steinberggerta@example.com    |Amerikanisch-Samoa|\n",
            "|Univ.Prof. Mijo Weihmann   |92 |18029   |hoevelantonius@example.net    |Niederlande       |\n",
            "|Aynur Karz B.Eng.          |96 |18029   |cschleich@example.com         |Puerto Rico       |\n",
            "+---------------------------+---+--------+------------------------------+------------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df.filter('postcode==18029').show(truncate=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ib139jd69kL"
      },
      "source": [
        "# Another example with multiple locales and weights\n",
        "\n",
        "We are going to use multiple locales with weights (following the [examples](https://faker.readthedocs.io/en/master/fakerclass.html#examples) in the documentation).\n",
        "\n",
        "Here's the [list of all available locales](https://faker.readthedocs.io/en/master/locales.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eXBHg-x269kL"
      },
      "outputs": [],
      "source": [
        "from faker import Faker\n",
        "# set a seed for the random generator\n",
        "Faker.seed(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generate data with locales `de_DE` and `de_AT` with weights respectively $5$ and $2$.\n",
        "\n",
        "The distribution of locales will be:\n",
        " - `de_DE` - $71.43\\%$ of the time ($5 / (5+2)$)\n",
        " - `de_AT` - $28.57\\%$ of the time ($2 / (5+2)$)\n"
      ],
      "metadata": {
        "id": "ssmNP9zR_Ldb"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j0XFMtOf69kT",
        "outputId": "ac8da24a-aac3-4973-8ca2-53e89cecdc00",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['de_DE', 'de_AT']"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "from collections import OrderedDict\n",
        "locales = OrderedDict([\n",
        "    ('de_DE', 5),\n",
        "    ('de_AT', 2),\n",
        "])\n",
        "fake = Faker(locales)\n",
        "fake.seed_instance(42)\n",
        "fake.locales"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LHCsMtbR69kU"
      },
      "outputs": [],
      "source": [
        "fake.seed_locale('de_DE', 0)\n",
        "fake.seed_locale('de_AT', 0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gUS9h0A669kU",
        "outputId": "70fa2cd9-98c9-4dd0-d441-052b1cf5e1e6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'current_location': (Decimal('26.547114'), Decimal('-10.243190')),\n",
              " 'blood_group': 'B-',\n",
              " 'name': 'Axel Jung',\n",
              " 'sex': 'M',\n",
              " 'mail': 'claragollner@gmail.com',\n",
              " 'birthdate': datetime.date(2003, 6, 10)}"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ],
      "source": [
        "fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
        "                     'mail', 'current_location'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PiSKooLQ69kU"
      },
      "outputs": [],
      "source": [
        "from pyspark.sql.types import *\n",
        "location = StructField('current_location',\n",
        "                       StructType([StructField('lat', DecimalType()),\n",
        "                                   StructField('lon', DecimalType())])\n",
        "                      )\n",
        "schema = StructType([StructField('name', StringType()),\n",
        "                     StructField('birthdate', DateType()),\n",
        "                     StructField('sex', StringType()),\n",
        "                     StructField('blood_group', StringType()),\n",
        "                     StructField('mail', StringType()),\n",
        "                     location\n",
        "                     ])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8j35IGPd69kU",
        "outputId": "22ba1c61-b965-4c1d-9c77-8b4fc024cf40",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'current_location': (Decimal('79.153888'), Decimal('-0.003034')),\n",
              " 'blood_group': 'B-',\n",
              " 'name': 'Dr. Anita Suppan',\n",
              " 'sex': 'F',\n",
              " 'mail': 'schauerbenedict@kabsi.at',\n",
              " 'birthdate': datetime.date(1980, 2, 21)}"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ],
      "source": [
        "fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
        "                     'mail', 'current_location'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mBHUAB8Z69kU"
      },
      "outputs": [],
      "source": [
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession \\\n",
        "    .builder \\\n",
        "    .appName(\"Faker demo - part 2\") \\\n",
        "    .getOrCreate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gjkL0XM69kU"
      },
      "source": [
        "Create dataframe with $5\\cdot10^3$ rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pKxvxwDK69kU",
        "outputId": "eddad2ac-a161-4b37-e650-5926d26329c8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 2.52 s, sys: 19.1 ms, total: 2.54 s\n",
            "Wall time: 2.81 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "n = 5*10**3\n",
        "d = (fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
        "                          'mail', 'current_location'])\n",
        "     for i in range(n))\n",
        "df = spark.createDataFrame(d, schema)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XbsEgC1d69kY",
        "outputId": "445f3b9f-cc72-4478-990f-76b7e34896c3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- name: string (nullable = true)\n",
            " |-- birthdate: date (nullable = true)\n",
            " |-- sex: string (nullable = true)\n",
            " |-- blood_group: string (nullable = true)\n",
            " |-- mail: string (nullable = true)\n",
            " |-- current_location: struct (nullable = true)\n",
            " |    |-- lat: decimal(10,0) (nullable = true)\n",
            " |    |-- lon: decimal(10,0) (nullable = true)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df.printSchema()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-SQ-dsb69kY"
      },
      "source": [
        "Note how `location` represents a _tuple_ data structure (a `StructType` of `StructField`s)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rut1uSF069kY",
        "outputId": "3acfe637-7745-4a01-8ebe-c497b7c57bf0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
            "|name                   |birthdate |sex|blood_group|mail                         |current_location|\n",
            "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
            "|Prof. Valentine Noack  |1979-03-26|F  |B-         |maricagotthard@aol.de        |{74, 164}       |\n",
            "|Magrit Graf            |1943-01-27|F  |A-         |hartungclaudio@web.de        |{-86, -34}      |\n",
            "|Harriet Weller-Lindau  |1959-12-06|F  |AB+        |heserhilma@gmail.com         |{20, 126}       |\n",
            "|Ing. Walfried Roskoth  |1912-04-15|M  |B-         |kambsliane@hotmail.de        |{73, 169}       |\n",
            "|Alexa Loidl-Schönberger|1934-01-11|F  |O-         |hannafroehlich@gmail.com     |{-23, -117}     |\n",
            "|Hans-Erich Hartmann    |1971-07-14|M  |O-         |vadimkostolzin@gmx.de        |{39, -118}      |\n",
            "|Ing. Sofia Fritsch B.A.|1966-12-12|F  |A-         |weinhagehans-christian@gmx.de|{-11, 73}       |\n",
            "|Rahel Zahn MBA.        |1999-09-27|F  |O-         |kathrin37@gmx.de             |{-77, -126}     |\n",
            "|Prof. Violetta Eberl   |1913-01-21|F  |B+         |lars24@chello.at             |{80, -135}      |\n",
            "|Gudula Täsche-Barth    |1970-01-02|F  |O-         |hans-eberhard91@aol.de       |{22, -164}      |\n",
            "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df.show(n=10, truncate=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Save to Parquet"
      ],
      "metadata": {
        "id": "KSCL83BVC3xu"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fw7FSGff69kY"
      },
      "source": [
        "[Write to parquet](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=write#pyspark.sql.DataFrameWriter.parquet) file ([Parquet](http://parquet.apache.org/) is a compressed, efficient columnar data representation compatible with all frameworks in the Hadoop ecosystem):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VcTsZQXI69kZ"
      },
      "outputs": [],
      "source": [
        "df.write.mode(\"overwrite\").parquet(\"fakedata.parquet\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2LjMN4zy69kZ"
      },
      "source": [
        "Check the size of parquet file (it is actually a directory containing the partitions):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q3mJjr6q69kZ",
        "outputId": "181e5836-2a5a-4e30-8327-3bfb53da0e39",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "188K\tfakedata.parquet\n"
          ]
        }
      ],
      "source": [
        "!du -h fakedata.parquet"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -lh fakedata.parquet"
      ],
      "metadata": {
        "id": "Pju4GyKL-DAn",
        "outputId": "90d5de7c-7ec2-46ef-d01b-fdae2507efc0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 172K\n",
            "-rw-r--r-- 1 root root 71K Feb 25 22:02 part-00000-8c909152-e1e6-4c4c-8a86-04e0a29cfdfc-c000.snappy.parquet\n",
            "-rw-r--r-- 1 root root 98K Feb 25 22:02 part-00001-8c909152-e1e6-4c4c-8a86-04e0a29cfdfc-c000.snappy.parquet\n",
            "-rw-r--r-- 1 root root   0 Feb 25 22:02 _SUCCESS\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stop Spark session"
      ],
      "metadata": {
        "id": "JAwKdWmcC-ib"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBG-CEbV69kZ"
      },
      "source": [
        "Don't forget to close the Spark session when you're done!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MsCutnmw69kZ"
      },
      "outputs": [],
      "source": [
        "spark.stop()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}