generate_data_with_Faker.nbconvert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/groda/big_data/blob/master/generate_data_with_Faker.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hwLHeIOc69kD"
   },
   "source": [
    "<a href=\"https://github.com/groda/big_data\"><div><img src=\"https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true\" align=right width=\"90\"></div></a>\n",
    "\n",
    "# Data Generation and Aggregation with Python's Faker Library and PySpark\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "Explore the capabilities of the Python Faker library (https://faker.readthedocs.io/) for dynamic data generation!\n",
    "\n",
    "Whether you're a data scientist, engineer, or analyst, this tutorial will guide you through the process of creating realistic and diverse datasets using Faker and then harnessing the distributed computing capabilities of PySpark to aggregate and analyze the generated data.\n",
    "\n",
    "\n",
    "\n",
    "**Note:** This is not _synthetic_ data as it is generated using simple methods and will most likely not fit any real-life distribution. Still, it serves as a valuable resource for testing purposes when authentic data is unavailable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BW6yZdpy7run"
   },
   "source": [
    "# Install Faker\n",
    "\n",
    "The Python `faker` module needs to be installed. Note that on Google Colab you can use `!pip` as well as just `pip` (no exclamation mark)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:30.148990Z",
     "iopub.status.busy": "2024-03-24T21:16:30.148795Z",
     "iopub.status.idle": "2024-03-24T21:16:32.937275Z",
     "shell.execute_reply": "2024-03-24T21:16:32.936486Z"
    },
    "id": "7EOLVOTe7KnP",
    "outputId": "b72af9eb-9fa4-46ed-da58-1d497c43fe8b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting faker\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading Faker-24.3.0-py3-none-any.whl.metadata (15 kB)\r\n",
      "Requirement already satisfied: python-dateutil>=2.4 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from faker) (2.9.0.post0)\r\n",
      "Requirement already satisfied: typing-extensions>=3.10.0.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from faker) (4.10.0)\r\n",
      "Requirement already satisfied: six>=1.5 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from python-dateutil>=2.4->faker) (1.16.0)\r\n",
      "Downloading Faker-24.3.0-py3-none-any.whl (1.8 MB)\r\n",
      "\u001b[?25l"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.8 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.7/1.8 MB\u001b[0m \u001b[31m21.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m34.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25h"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: faker\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed faker-24.3.0\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install faker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UYLa4gsrB7aS"
   },
   "source": [
    "# Generate a Pandas dataframe with fake data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yKviUSgj7nHP"
   },
   "source": [
    "Import `Faker` and set a random seed ($42$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:32.940788Z",
     "iopub.status.busy": "2024-03-24T21:16:32.940375Z",
     "iopub.status.idle": "2024-03-24T21:16:32.987123Z",
     "shell.execute_reply": "2024-03-24T21:16:32.986527Z"
    },
    "id": "7fbGCBoq69kF"
   },
   "outputs": [],
   "source": [
    "from faker import Faker\n",
    "# Set the seed value of the shared `random.Random` object\n",
    "# across all internal generators that will ever be created\n",
    "Faker.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yUEes2gl69kF"
   },
   "source": [
    "`fake` is a fake data generator with `DE_de` locale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:32.989855Z",
     "iopub.status.busy": "2024-03-24T21:16:32.989424Z",
     "iopub.status.idle": "2024-03-24T21:16:33.015444Z",
     "shell.execute_reply": "2024-03-24T21:16:33.014947Z"
    },
    "id": "Bf7UTx_r69kG"
   },
   "outputs": [],
   "source": [
    "fake = Faker('de_DE')\n",
    "fake.seed_locale('de_DE', 42)\n",
    "# Creates and seeds a unique `random.Random` object for\n",
    "# each internal generator of this `Faker` instance\n",
    "fake.seed_instance(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0bSXBUI469kG"
   },
   "source": [
    "Import Pandas to save data into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:33.017809Z",
     "iopub.status.busy": "2024-03-24T21:16:33.017504Z",
     "iopub.status.idle": "2024-03-24T21:16:38.451687Z",
     "shell.execute_reply": "2024-03-24T21:16:38.451075Z"
    },
    "id": "PI_YqfM169kG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas==1.5.3\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading pandas-1.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\r\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas==1.5.3) (2.9.0.post0)\r\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas==1.5.3) (2024.1)\r\n",
      "Requirement already satisfied: numpy>=1.20.3 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas==1.5.3) (1.24.4)\r\n",
      "Requirement already satisfied: six>=1.5 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas==1.5.3) (1.16.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading pandas-1.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/12.2 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.7/12.2 MB\u001b[0m \u001b[31m19.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.2/12.2 MB\u001b[0m \u001b[31m75.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[32m10.5/12.2 MB\u001b[0m \u001b[31m130.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m152.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m106.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25h"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: pandas\r\n",
      "  Attempting uninstall: pandas\r\n",
      "    Found existing installation: pandas 2.0.3\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Uninstalling pandas-2.0.3:\r\n",
      "      Successfully uninstalled pandas-2.0.3\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed pandas-1.5.3\r\n",
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "# true if running on Google Colab\n",
    "import sys\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "if not IN_COLAB:\n",
    " !pip install pandas==1.5.3\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LjCsGikw69kG"
   },
   "source": [
    "The function `create_row_faker` creates one row of fake data. Here we choose to generate a row containing the following fields:\n",
    " - `fake.name()`\n",
    " - `fake.postcode()`\n",
    " - `fake.email()`\n",
    " - `fake.country()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:38.455094Z",
     "iopub.status.busy": "2024-03-24T21:16:38.454452Z",
     "iopub.status.idle": "2024-03-24T21:16:38.458894Z",
     "shell.execute_reply": "2024-03-24T21:16:38.458256Z"
    },
    "id": "f1MiSZl069kG"
   },
   "outputs": [],
   "source": [
    "def create_row_faker(num=1):\n",
    "    output = [{\"name\": fake.name(),\n",
    "               \"age\": fake.random_int(0, 100),\n",
    "               \"postcode\": fake.postcode(),\n",
    "               \"email\": fake.email(),\n",
    "               \"nationality\": fake.country(),\n",
    "              } for x in range(num)]\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TeuZISIh69kH"
   },
   "source": [
    "Generate a single row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:38.461463Z",
     "iopub.status.busy": "2024-03-24T21:16:38.461111Z",
     "iopub.status.idle": "2024-03-24T21:16:38.467847Z",
     "shell.execute_reply": "2024-03-24T21:16:38.467153Z"
    },
    "id": "wXP-5uSg69kH",
    "outputId": "e5d284c3-2587-451d-aab8-45f9ba7c99d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Aleksandr Weinhage',\n",
       "  'age': 35,\n",
       "  'postcode': '32181',\n",
       "  'email': 'bbeckmann@example.org',\n",
       "  'nationality': 'Fidschi'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_row_faker()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4zIjkEOw69kI"
   },
   "source": [
    "Generate a dataframe `df_fake` of 5000 rows using `create_row_faker`.\n",
    "\n",
    "We're using the _cell magic_ `%%time` to time the operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:38.470343Z",
     "iopub.status.busy": "2024-03-24T21:16:38.469999Z",
     "iopub.status.idle": "2024-03-24T21:16:38.753851Z",
     "shell.execute_reply": "2024-03-24T21:16:38.753168Z"
    },
    "id": "JtRWDEsT69kI",
    "outputId": "75fc519b-a19c-4fcf-db0e-13d98f394f8c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 279 ms, sys: 1.35 ms, total: 280 ms\n",
      "Wall time: 280 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_fake = pd.DataFrame(create_row_faker(5000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "meT16YZy69kI"
   },
   "source": [
    "View dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:38.756446Z",
     "iopub.status.busy": "2024-03-24T21:16:38.756237Z",
     "iopub.status.idle": "2024-03-24T21:16:38.766383Z",
     "shell.execute_reply": "2024-03-24T21:16:38.765694Z"
    },
    "id": "RJK93FxW69kI",
    "outputId": "50d3d1ef-24e6-485c-a43e-40470594cd4a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>age</th>\n",
       "      <th>postcode</th>\n",
       "      <th>email</th>\n",
       "      <th>nationality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Prof. Kurt Bauer B.A.</td>\n",
       "      <td>91</td>\n",
       "      <td>37940</td>\n",
       "      <td>hildaloeffler@example.com</td>\n",
       "      <td>Guatemala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ekkehart Wilms-Kallert</td>\n",
       "      <td>13</td>\n",
       "      <td>61559</td>\n",
       "      <td>maja07@example.net</td>\n",
       "      <td>Brasilien</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Annelise Röhrdanz-Hornig</td>\n",
       "      <td>80</td>\n",
       "      <td>93103</td>\n",
       "      <td>daniel31@example.com</td>\n",
       "      <td>Guatemala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Magrit Knappe B.A.</td>\n",
       "      <td>47</td>\n",
       "      <td>34192</td>\n",
       "      <td>gottliebmitschke@example.com</td>\n",
       "      <td>Guadeloupe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Univ.Prof. Gotthilf Wirth B.Sc.</td>\n",
       "      <td>29</td>\n",
       "      <td>56413</td>\n",
       "      <td>heini76@example.net</td>\n",
       "      <td>Litauen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>Janusz Nette</td>\n",
       "      <td>74</td>\n",
       "      <td>33091</td>\n",
       "      <td>ann-kathrinsiering@example.net</td>\n",
       "      <td>Belarus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>Frau Cathleen Bähr</td>\n",
       "      <td>97</td>\n",
       "      <td>89681</td>\n",
       "      <td>hethurhubertus@example.org</td>\n",
       "      <td>St. Barthélemy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>Ulla Seidel</td>\n",
       "      <td>66</td>\n",
       "      <td>28358</td>\n",
       "      <td>klotzbabette@example.net</td>\n",
       "      <td>St. Lucia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>Janin Speer MBA.</td>\n",
       "      <td>64</td>\n",
       "      <td>76879</td>\n",
       "      <td>giesskarl-hermann@example.com</td>\n",
       "      <td>Kroatien</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>Emma Freudenberger</td>\n",
       "      <td>25</td>\n",
       "      <td>98648</td>\n",
       "      <td>andre92@example.net</td>\n",
       "      <td>St. Kitts und Nevis</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 name  age postcode  \\\n",
       "0               Prof. Kurt Bauer B.A.   91    37940   \n",
       "1              Ekkehart Wilms-Kallert   13    61559   \n",
       "2            Annelise Röhrdanz-Hornig   80    93103   \n",
       "3                  Magrit Knappe B.A.   47    34192   \n",
       "4     Univ.Prof. Gotthilf Wirth B.Sc.   29    56413   \n",
       "...                               ...  ...      ...   \n",
       "4995                     Janusz Nette   74    33091   \n",
       "4996               Frau Cathleen Bähr   97    89681   \n",
       "4997                      Ulla Seidel   66    28358   \n",
       "4998                 Janin Speer MBA.   64    76879   \n",
       "4999               Emma Freudenberger   25    98648   \n",
       "\n",
       "                               email          nationality  \n",
       "0          hildaloeffler@example.com            Guatemala  \n",
       "1                 maja07@example.net            Brasilien  \n",
       "2               daniel31@example.com            Guatemala  \n",
       "3       gottliebmitschke@example.com           Guadeloupe  \n",
       "4                heini76@example.net              Litauen  \n",
       "...                              ...                  ...  \n",
       "4995  ann-kathrinsiering@example.net              Belarus  \n",
       "4996      hethurhubertus@example.org       St. Barthélemy  \n",
       "4997        klotzbabette@example.net            St. Lucia  \n",
       "4998   giesskarl-hermann@example.com             Kroatien  \n",
       "4999             andre92@example.net  St. Kitts und Nevis  \n",
       "\n",
       "[5000 rows x 5 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_fake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "86GX2vpJ69kI"
   },
   "source": [
    "For more fake data generators see Faker's [standard providers](https://faker.readthedocs.io/en/master/providers.html#standard-providers) as well as [community providers](https://faker.readthedocs.io/en/master/communityproviders.html#community-providers)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Yv5FGq0CH6D"
   },
   "source": [
    "# Generate PySpark dataframe with fake data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qw0OUfBE8Aom"
   },
   "source": [
    "Install PySpark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:38.768965Z",
     "iopub.status.busy": "2024-03-24T21:16:38.768621Z",
     "iopub.status.idle": "2024-03-24T21:16:40.084847Z",
     "shell.execute_reply": "2024-03-24T21:16:40.084162Z"
    },
    "id": "PRKNcWR87h7P",
    "outputId": "2d8a788c-2161-4248-9ba3-394464e38706"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pyspark in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (3.5.1)\r\n",
      "Requirement already satisfied: py4j==0.10.9.7 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pyspark) (0.10.9.7)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install pyspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:40.087943Z",
     "iopub.status.busy": "2024-03-24T21:16:40.087525Z",
     "iopub.status.idle": "2024-03-24T21:16:43.073997Z",
     "shell.execute_reply": "2024-03-24T21:16:43.073223Z"
    },
    "id": "9drMVtgJ69kI"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/03/24 21:16:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .appName(\"Faker demo\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:43.077143Z",
     "iopub.status.busy": "2024-03-24T21:16:43.076925Z",
     "iopub.status.idle": "2024-03-24T21:16:44.941962Z",
     "shell.execute_reply": "2024-03-24T21:16:44.941228Z"
    },
    "id": "SYIUxAj569kI"
   },
   "outputs": [],
   "source": [
    "df = spark.createDataFrame(create_row_faker(5000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QbBAHBLF69kJ"
   },
   "source": [
    "To avoid getting the warning, either use [pyspark.sql.Row](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Row) and let Spark infer datatypes or create a schema for the dataframe specifying the datatypes of all fields (here's the list of all [datatypes](http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=types#module-pyspark.sql.types))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:44.945265Z",
     "iopub.status.busy": "2024-03-24T21:16:44.944785Z",
     "iopub.status.idle": "2024-03-24T21:16:44.950051Z",
     "shell.execute_reply": "2024-03-24T21:16:44.949300Z"
    },
    "id": "qwaQQ2Ud69kJ"
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "schema = StructType([StructField('name', StringType()),\n",
    "                     StructField('age',IntegerType()),\n",
    "                     StructField('postcode',StringType()),\n",
    "                     StructField('email', StringType()),\n",
    "                     StructField('nationality',StringType())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:44.953144Z",
     "iopub.status.busy": "2024-03-24T21:16:44.952908Z",
     "iopub.status.idle": "2024-03-24T21:16:45.272401Z",
     "shell.execute_reply": "2024-03-24T21:16:45.271554Z"
    },
    "id": "NM4OqVld69kJ"
   },
   "outputs": [],
   "source": [
    "df = spark.createDataFrame(create_row_faker(5000), schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:45.276406Z",
     "iopub.status.busy": "2024-03-24T21:16:45.275974Z",
     "iopub.status.idle": "2024-03-24T21:16:45.292852Z",
     "shell.execute_reply": "2024-03-24T21:16:45.292201Z"
    },
    "id": "NDPqZqnl69kJ",
    "outputId": "377ebb1c-4dbe-41e1-fc4f-3af97042fc1b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- name: string (nullable = true)\n",
      " |-- age: integer (nullable = true)\n",
      " |-- postcode: string (nullable = true)\n",
      " |-- email: string (nullable = true)\n",
      " |-- nationality: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HwsIhJxO69kJ"
   },
   "source": [
    "Let's generate some more data (dataframe with $5\\cdot10^4$ rows). The file will be partitioned by Spark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:45.295453Z",
     "iopub.status.busy": "2024-03-24T21:16:45.295090Z",
     "iopub.status.idle": "2024-03-24T21:16:48.210371Z",
     "shell.execute_reply": "2024-03-24T21:16:48.209712Z"
    },
    "id": "KZtRuVeR69kJ",
    "outputId": "6f8a7dd0-02a8-4395-c7da-612365c8b082"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.84 s, sys: 52 ms, total: 2.89 s\n",
      "Wall time: 2.91 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "n = 5*10**4\n",
    "df = spark.createDataFrame(create_row_faker(n), schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l0ZiQJKp69kJ"
   },
   "source": [
    "It took a long time (~4 sec. for 50000 rows)!\n",
    "\n",
    "Can we do better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6xI7IjQ669kK"
   },
   "source": [
    "The function `create_row_faker()` returns a list. This is not efficient, what we need is a _generator_ instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:48.213415Z",
     "iopub.status.busy": "2024-03-24T21:16:48.212919Z",
     "iopub.status.idle": "2024-03-24T21:16:48.217636Z",
     "shell.execute_reply": "2024-03-24T21:16:48.217027Z"
    },
    "id": "Bt2HncFZ69kK",
    "outputId": "347815f8-00bc-4c77-8029-7a5a76f85b71"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = create_row_faker(5)\n",
    "# what type is d?\n",
    "type(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EC_IgTtY69kK"
   },
   "source": [
    "Now `d` is a generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:48.220200Z",
     "iopub.status.busy": "2024-03-24T21:16:48.219816Z",
     "iopub.status.idle": "2024-03-24T21:16:48.224600Z",
     "shell.execute_reply": "2024-03-24T21:16:48.224023Z"
    },
    "id": "9YjvEOTV69kK",
    "outputId": "b0400b20-efa8-4feb-9e1f-c956efc222cd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "generator"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = ({\"name\": fake.name(),\n",
    "      \"age\": fake.random_int(0, 100),\n",
    "      \"postcode\": fake.postcode(),\n",
    "      \"email\": fake.email(),\n",
    "      \"nationality\": fake.country()} for i in range(5))\n",
    "# what type is d?\n",
    "type(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:48.227056Z",
     "iopub.status.busy": "2024-03-24T21:16:48.226725Z",
     "iopub.status.idle": "2024-03-24T21:16:51.188463Z",
     "shell.execute_reply": "2024-03-24T21:16:51.187850Z"
    },
    "id": "BBdz-PJz69kK",
    "outputId": "7bc09338-b81d-4e33-c87c-01422d1cc749"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.91 s, sys: 20.3 ms, total: 2.93 s\n",
      "Wall time: 2.96 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "n = 5*10**4\n",
    "d = ({\"name\": fake.name(),\n",
    "      \"age\": fake.random_int(0, 100),\n",
    "      \"postcode\": fake.postcode(),\n",
    "      \"email\": fake.email(),\n",
    "      \"nationality\": fake.country()}\n",
    "     for i in range(n))\n",
    "df = spark.createDataFrame(d, schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bd-jNkqk69kK"
   },
   "source": [
    "This wasn't faster.\n",
    "\n",
    "I will look into how one can leverage Hadoop's parallelism to generate dataframes and speed the process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t_NNqOXPCbj3"
   },
   "source": [
    "## Filter and aggregate with PySpark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 186
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:51.191285Z",
     "iopub.status.busy": "2024-03-24T21:16:51.190962Z",
     "iopub.status.idle": "2024-03-24T21:16:51.195101Z",
     "shell.execute_reply": "2024-03-24T21:16:51.194419Z"
    },
    "id": "DyDn7tqS69kK",
    "outputId": "7ebe42de-a1a8-4933-dd89-128dd4ee530a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pyspark.sql.dataframe.DataFrame"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c5X9Bz7g69kL"
   },
   "source": [
    "Show the first five records in the dataframe of fake data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:51.197656Z",
     "iopub.status.busy": "2024-03-24T21:16:51.197365Z",
     "iopub.status.idle": "2024-03-24T21:16:53.445273Z",
     "shell.execute_reply": "2024-03-24T21:16:53.444429Z"
    },
    "id": "0RYAfYsx69kL",
    "outputId": "8066ee10-f105-42fe-b514-fc750ecb4b13"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 0:>                                                          (0 + 1) / 1]\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------+---+--------+------------------------+-----------+\n",
      "|name                 |age|postcode|email                   |nationality|\n",
      "+---------------------+---+--------+------------------------+-----------+\n",
      "|Rudolf Rust          |76 |86566   |kornelia11@example.org  |Indonesien |\n",
      "|Kerstin Putz-Köster  |54 |18953   |qrudolph@example.org    |Kenia      |\n",
      "|Ivo Schinke B.Sc.    |3  |60202   |radischjames@example.net|Chile      |\n",
      "|Hans-Henning Staude  |82 |68552   |ladislaus89@example.com |Thailand   |\n",
      "|Justine Weinhage B.A.|53 |62346   |ytaesche@example.org    |Österreich |\n",
      "+---------------------+---+--------+------------------------+-----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show(n=5, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ikvl-9Mm69kL"
   },
   "source": [
    "Do some data aggregation:\n",
    " - group by postcode\n",
    " - count the number of persons and the average age for each postcode\n",
    " - filter out postcodes with less than 4 persons\n",
    " - sort by average age descending\n",
    " - show the first 5 entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:53.448956Z",
     "iopub.status.busy": "2024-03-24T21:16:53.448681Z",
     "iopub.status.idle": "2024-03-24T21:16:55.154488Z",
     "shell.execute_reply": "2024-03-24T21:16:55.153643Z"
    },
    "id": "1lr4WKxt69kL",
    "outputId": "5a15bd58-23dc-46db-df7c-67ae48af6b42"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 1:>                                                          (0 + 4) / 4]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-----+-----------+\n",
      "|postcode|Count|Average age|\n",
      "+--------+-----+-----------+\n",
      "|   18029|    4|      91.75|\n",
      "|   67611|    4|       87.0|\n",
      "|   47898|    4|       85.5|\n",
      "|   45386|    4|      82.75|\n",
      "|   46755|    4|       78.5|\n",
      "+--------+-----+-----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "import pyspark.sql.functions as F\n",
    "df.groupBy('postcode') \\\n",
    "  .agg(F.count('postcode').alias('Count'), F.round(F.avg('age'), 2).alias('Average age')) \\\n",
    "  .filter('Count>3') \\\n",
    "  .orderBy('Average age', ascending=False) \\\n",
    "  .show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PP4TeBbh69kL"
   },
   "source": [
    "Postcode $18029$ has the highest average age ($91.75$). Show all entries for postcode $18029$ using `filter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.158806Z",
     "iopub.status.busy": "2024-03-24T21:16:55.157951Z",
     "iopub.status.idle": "2024-03-24T21:16:55.528879Z",
     "shell.execute_reply": "2024-03-24T21:16:55.528182Z"
    },
    "id": "_6NsPWUt69kL",
    "outputId": "0ed58e60-7a25-4d21-ee73-243da451e180"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------+---+--------+------------------------------+------------------+\n",
      "|name                       |age|postcode|email                         |nationality       |\n",
      "+---------------------------+---+--------+------------------------------+------------------+\n",
      "|Univ.Prof. Roderich Liebelt|89 |18029   |anne-katrinscholtz@example.com|Grönland          |\n",
      "|Herwig Matthäi B.A.        |90 |18029   |steinberggerta@example.com    |Amerikanisch-Samoa|\n",
      "|Univ.Prof. Mijo Weihmann   |92 |18029   |hoevelantonius@example.net    |Niederlande       |\n",
      "|Aynur Karz B.Eng.          |96 |18029   |cschleich@example.com         |Puerto Rico       |\n",
      "+---------------------------+---+--------+------------------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.filter('postcode==18029').show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_Ib139jd69kL"
   },
   "source": [
    "# Another example with multiple locales and weights\n",
    "\n",
    "We are going to use multiple locales with weights (following the [examples](https://faker.readthedocs.io/en/master/fakerclass.html#examples) in the documentation).\n",
    "\n",
    "Here's the [list of all available locales](https://faker.readthedocs.io/en/master/locales.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.532429Z",
     "iopub.status.busy": "2024-03-24T21:16:55.532144Z",
     "iopub.status.idle": "2024-03-24T21:16:55.536109Z",
     "shell.execute_reply": "2024-03-24T21:16:55.535255Z"
    },
    "id": "eXBHg-x269kL"
   },
   "outputs": [],
   "source": [
    "from faker import Faker\n",
    "# set a seed for the random generator\n",
    "Faker.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ssmNP9zR_Ldb"
   },
   "source": [
    "Generate data with locales `de_DE` and `de_AT` with weights respectively $5$ and $2$.\n",
    "\n",
    "The distribution of locales will be:\n",
    " - `de_DE` - $71.43\\%$ of the time ($5 / (5+2)$)\n",
    " - `de_AT` - $28.57\\%$ of the time ($2 / (5+2)$)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.539630Z",
     "iopub.status.busy": "2024-03-24T21:16:55.539372Z",
     "iopub.status.idle": "2024-03-24T21:16:55.603612Z",
     "shell.execute_reply": "2024-03-24T21:16:55.602940Z"
    },
    "id": "j0XFMtOf69kT",
    "outputId": "ac8da24a-aac3-4973-8ca2-53e89cecdc00"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['de_DE', 'de_AT']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "locales = OrderedDict([\n",
    "    ('de_DE', 5),\n",
    "    ('de_AT', 2),\n",
    "])\n",
    "fake = Faker(locales)\n",
    "fake.seed_instance(42)\n",
    "fake.locales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.606569Z",
     "iopub.status.busy": "2024-03-24T21:16:55.606098Z",
     "iopub.status.idle": "2024-03-24T21:16:55.609705Z",
     "shell.execute_reply": "2024-03-24T21:16:55.609145Z"
    },
    "id": "LHCsMtbR69kU"
   },
   "outputs": [],
   "source": [
    "fake.seed_locale('de_DE', 0)\n",
    "fake.seed_locale('de_AT', 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.612125Z",
     "iopub.status.busy": "2024-03-24T21:16:55.611738Z",
     "iopub.status.idle": "2024-03-24T21:16:55.616816Z",
     "shell.execute_reply": "2024-03-24T21:16:55.616187Z"
    },
    "id": "gUS9h0A669kU",
    "outputId": "70fa2cd9-98c9-4dd0-d441-052b1cf5e1e6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'current_location': (Decimal('26.547114'), Decimal('-10.243190')),\n",
       " 'blood_group': 'B-',\n",
       " 'name': 'Axel Jung',\n",
       " 'sex': 'M',\n",
       " 'mail': 'claragollner@gmail.com',\n",
       " 'birthdate': datetime.date(2003, 7, 8)}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
    "                     'mail', 'current_location'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.619429Z",
     "iopub.status.busy": "2024-03-24T21:16:55.618977Z",
     "iopub.status.idle": "2024-03-24T21:16:55.624159Z",
     "shell.execute_reply": "2024-03-24T21:16:55.623508Z"
    },
    "id": "PiSKooLQ69kU"
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "location = StructField('current_location',\n",
    "                       StructType([StructField('lat', DecimalType()),\n",
    "                                   StructField('lon', DecimalType())])\n",
    "                      )\n",
    "schema = StructType([StructField('name', StringType()),\n",
    "                     StructField('birthdate', DateType()),\n",
    "                     StructField('sex', StringType()),\n",
    "                     StructField('blood_group', StringType()),\n",
    "                     StructField('mail', StringType()),\n",
    "                     location\n",
    "                     ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.626752Z",
     "iopub.status.busy": "2024-03-24T21:16:55.626528Z",
     "iopub.status.idle": "2024-03-24T21:16:55.631457Z",
     "shell.execute_reply": "2024-03-24T21:16:55.630808Z"
    },
    "id": "8j35IGPd69kU",
    "outputId": "22ba1c61-b965-4c1d-9c77-8b4fc024cf40"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'current_location': (Decimal('79.153888'), Decimal('-0.003034')),\n",
       " 'blood_group': 'B-',\n",
       " 'name': 'Dr. Anita Suppan',\n",
       " 'sex': 'F',\n",
       " 'mail': 'schauerbenedict@kabsi.at',\n",
       " 'birthdate': datetime.date(1980, 3, 20)}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
    "                     'mail', 'current_location'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.633834Z",
     "iopub.status.busy": "2024-03-24T21:16:55.633451Z",
     "iopub.status.idle": "2024-03-24T21:16:55.640898Z",
     "shell.execute_reply": "2024-03-24T21:16:55.640292Z"
    },
    "id": "mBHUAB8Z69kU"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/03/24 21:16:55 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .appName(\"Faker demo - part 2\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6gjkL0XM69kU"
   },
   "source": [
    "Create dataframe with $5\\cdot10^3$ rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:55.643302Z",
     "iopub.status.busy": "2024-03-24T21:16:55.642956Z",
     "iopub.status.idle": "2024-03-24T21:16:57.261260Z",
     "shell.execute_reply": "2024-03-24T21:16:57.260610Z"
    },
    "id": "pKxvxwDK69kU",
    "outputId": "eddad2ac-a161-4b37-e650-5926d26329c8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.55 s, sys: 3.7 ms, total: 1.55 s\n",
      "Wall time: 1.61 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "n = 5*10**3\n",
    "d = (fake.profile(fields=['name', 'birthdate', 'sex', 'blood_group',\n",
    "                          'mail', 'current_location'])\n",
    "     for i in range(n))\n",
    "df = spark.createDataFrame(d, schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:57.263872Z",
     "iopub.status.busy": "2024-03-24T21:16:57.263668Z",
     "iopub.status.idle": "2024-03-24T21:16:57.267718Z",
     "shell.execute_reply": "2024-03-24T21:16:57.267078Z"
    },
    "id": "XbsEgC1d69kY",
    "outputId": "445f3b9f-cc72-4478-990f-76b7e34896c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- name: string (nullable = true)\n",
      " |-- birthdate: date (nullable = true)\n",
      " |-- sex: string (nullable = true)\n",
      " |-- blood_group: string (nullable = true)\n",
      " |-- mail: string (nullable = true)\n",
      " |-- current_location: struct (nullable = true)\n",
      " |    |-- lat: decimal(10,0) (nullable = true)\n",
      " |    |-- lon: decimal(10,0) (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f-SQ-dsb69kY"
   },
   "source": [
    "Note how `location` represents a _tuple_ data structure (a `StructType` of `StructField`s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:57.270187Z",
     "iopub.status.busy": "2024-03-24T21:16:57.269986Z",
     "iopub.status.idle": "2024-03-24T21:16:57.405999Z",
     "shell.execute_reply": "2024-03-24T21:16:57.405195Z"
    },
    "id": "Rut1uSF069kY",
    "outputId": "3acfe637-7745-4a01-8ebe-c497b7c57bf0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
      "|name                   |birthdate |sex|blood_group|mail                         |current_location|\n",
      "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
      "|Prof. Valentine Noack  |1979-04-23|F  |B-         |maricagotthard@aol.de        |{74, 164}       |\n",
      "|Magrit Graf            |1943-02-24|F  |A-         |hartungclaudio@web.de        |{-86, -34}      |\n",
      "|Harriet Weller-Lindau  |1960-01-03|F  |AB+        |heserhilma@gmail.com         |{20, 126}       |\n",
      "|Ing. Walfried Roskoth  |1912-05-13|M  |B-         |kambsliane@hotmail.de        |{73, 169}       |\n",
      "|Alexa Loidl-Schönberger|1934-02-08|F  |O-         |hannafroehlich@gmail.com     |{-23, -117}     |\n",
      "|Hans-Erich Hartmann    |1971-08-11|M  |O-         |vadimkostolzin@gmx.de        |{39, -118}      |\n",
      "|Ing. Sofia Fritsch B.A.|1967-01-09|F  |A-         |weinhagehans-christian@gmx.de|{-11, 73}       |\n",
      "|Rahel Zahn MBA.        |1999-10-25|F  |O-         |kathrin37@gmx.de             |{-77, -126}     |\n",
      "|Prof. Violetta Eberl   |1913-02-18|F  |B+         |lars24@chello.at             |{80, -135}      |\n",
      "|Gudula Täsche-Barth    |1970-01-30|F  |O-         |hans-eberhard91@aol.de       |{22, -164}      |\n",
      "+-----------------------+----------+---+-----------+-----------------------------+----------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show(n=10, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KSCL83BVC3xu"
   },
   "source": [
    "# Save to Parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fw7FSGff69kY"
   },
   "source": [
    "[Write to parquet](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=write#pyspark.sql.DataFrameWriter.parquet) file ([Parquet](http://parquet.apache.org/) is a compressed, efficient columnar data representation compatible with all frameworks in the Hadoop ecosystem):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:57.408825Z",
     "iopub.status.busy": "2024-03-24T21:16:57.408607Z",
     "iopub.status.idle": "2024-03-24T21:16:58.646224Z",
     "shell.execute_reply": "2024-03-24T21:16:58.645423Z"
    },
    "id": "VcTsZQXI69kZ"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 7:>                                                          (0 + 4) / 4]\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "df.write.mode(\"overwrite\").parquet(\"fakedata.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2LjMN4zy69kZ"
   },
   "source": [
    "Check the size of parquet file (it is actually a directory containing the partitions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:58.649027Z",
     "iopub.status.busy": "2024-03-24T21:16:58.648809Z",
     "iopub.status.idle": "2024-03-24T21:16:58.781024Z",
     "shell.execute_reply": "2024-03-24T21:16:58.780243Z"
    },
    "id": "Q3mJjr6q69kZ",
    "outputId": "181e5836-2a5a-4e30-8327-3bfb53da0e39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "212K\tfakedata.parquet\r\n"
     ]
    }
   ],
   "source": [
    "!du -h fakedata.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:58.784025Z",
     "iopub.status.busy": "2024-03-24T21:16:58.783806Z",
     "iopub.status.idle": "2024-03-24T21:16:58.910871Z",
     "shell.execute_reply": "2024-03-24T21:16:58.910172Z"
    },
    "id": "Pju4GyKL-DAn",
    "outputId": "90d5de7c-7ec2-46ef-d01b-fdae2507efc0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 188K\r\n",
      "-rw-r--r-- 1 runner docker   0 Mar 24 21:16 _SUCCESS\r\n",
      "-rw-r--r-- 1 runner docker 39K Mar 24 21:16 part-00000-2888bc4d-4b82-43a6-93f7-11e194d555b1-c000.snappy.parquet\r\n",
      "-rw-r--r-- 1 runner docker 40K Mar 24 21:16 part-00001-2888bc4d-4b82-43a6-93f7-11e194d555b1-c000.snappy.parquet\r\n",
      "-rw-r--r-- 1 runner docker 39K Mar 24 21:16 part-00002-2888bc4d-4b82-43a6-93f7-11e194d555b1-c000.snappy.parquet\r\n",
      "-rw-r--r-- 1 runner docker 67K Mar 24 21:16 part-00003-2888bc4d-4b82-43a6-93f7-11e194d555b1-c000.snappy.parquet\r\n"
     ]
    }
   ],
   "source": [
    "!ls -lh fakedata.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAwKdWmcC-ib"
   },
   "source": [
    "# Stop Spark session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JBG-CEbV69kZ"
   },
   "source": [
    "Don't forget to close the Spark session when you're done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:16:58.914201Z",
     "iopub.status.busy": "2024-03-24T21:16:58.913557Z",
     "iopub.status.idle": "2024-03-24T21:16:59.616104Z",
     "shell.execute_reply": "2024-03-24T21:16:59.615404Z"
    },
    "id": "MsCutnmw69kZ"
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}