notebooks/1_Generate_data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "is_executing": true
   },
   "outputs": [],
   "source": [
    "# install presidio via pip if not yet installed\n",
    "\n",
    "#!pip install presidio-analyzer\n",
    "#!pip install presidio-evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "is_executing": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pprint\n",
    "from collections import Counter\n",
    "from pathlib import Path\n",
    "from typing import Dict, List\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tqdm\n",
    "\n",
    "from presidio_evaluator import InputSample\n",
    "from presidio_evaluator.data_generator import PresidioDataGenerator\n",
    "from presidio_evaluator.data_generator.faker_extensions import (\n",
    "    FakerSpansResult,\n",
    "    RecordsFaker,\n",
    "    IpAddressProvider,\n",
    "    NationalityProvider,\n",
    "    OrganizationProvider,\n",
    "    UsDriverLicenseProvider,\n",
    "    AgeProvider,\n",
    "    AddressProviderNew,\n",
    "    PhoneNumberProviderNew,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate fake PII data using Presidio's data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Presidio's data generator is based on the [Python Faker tool](https://faker.readthedocs.io/en/master/)\n",
    "and allows you to generate a synthetic dataset from sentence templates.\n",
    "It features wrappers for Faker which allows you to sample from existing sources of fake data.\n",
    "\n",
    "Example templates:\n",
    "\n",
    "> I live at {{address}}\n",
    "\n",
    "> You can email me at {{email}}. Thanks, {{first_name}}\n",
    "\n",
    "> What's your last name? It's {{last_name}}\n",
    "\n",
    "> Every time I see you falling I get down on my knees and pray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple example\n",
    "This uses all the default values to generate 10 samples based on three templates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "is_executing": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sampling: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 9149.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My name is Joshua Jackson\n",
      "[{\"value\": \"Joshua Jackson\", \"start\": 11, \"end\": 25, \"type\": \"name\"}]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "sentence_templates = [\n",
    "    \"My name is {{name}}\",\n",
    "    \"Please send it to {{address}}\",\n",
    "    \"I just moved to {{city}} from {{country}}\",\n",
    "]\n",
    "\n",
    "\n",
    "data_generator = PresidioDataGenerator()\n",
    "fake_records = data_generator.generate_fake_data(\n",
    "    templates=sentence_templates, n_samples=10\n",
    ")\n",
    "\n",
    "fake_records = list(fake_records)\n",
    "\n",
    "# Print the spans of the first sample\n",
    "print(fake_records[0].fake)\n",
    "print(fake_records[0].spans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate a full dataset\n",
    "\n",
    "In this example we customize the data generator to:\n",
    "1. Accept more types of entities (by adding more providers to Faker. see [Faker's documentation](https://faker.readthedocs.io/en/master/index.html#how-to-create-a-provider)\n",
    "2. Handle records of multiple PII entities per fake person for a more realistic dataset\n",
    "\n",
    "We then translate the generated entity types to match Presidio's, and save the new dataset in json and CONLL03 formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Specify parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "is_executing": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "number_of_samples = 1500\n",
    "cur_time = datetime.date.today().strftime(\"%B_%d_%Y\")\n",
    "\n",
    "raw_data_path = Path(\"../presidio_evaluator/data_generator/raw_data\")\n",
    "output_file = f\"../data/generated_size_{number_of_samples}_date_{cur_time}.json\"\n",
    "output_conll = f\"../data/generated_size_{number_of_samples}_date_{cur_time}.tsv\"\n",
    "\n",
    "templates_file_path = Path(raw_data_path, \"templates.txt\").resolve()\n",
    "fake_name_generator_file = Path(raw_data_path, \"FakeNameGenerator.com_3000.csv\").resolve()\n",
    "\n",
    "lower_case_ratio = 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "b. Read [FakeNameGenerator](https://www.fakenamegenerator.com/) data (optional, extends the set of fake values)\n",
    "and create a `RecordsFaker` which returns a fake person record (with multiple values) instead of one value,\n",
    "allowing dependencies between values belonging to the same fake person\n",
    "(e.g. name = Michael Smith with the email michael.smith@gmail.com).\n",
    "\n",
    "The `fake_name_generator_file` can be downloaded from https://www.fakenamegenerator.com/order.php\n",
    "\n",
    "> Note that you can create fake records for multiple name sets, allowing you to adapt the fake data to the real data if needed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>gender</th>\n",
       "      <th>nationality</th>\n",
       "      <th>prefix</th>\n",
       "      <th>first_name</th>\n",
       "      <th>middle_initial</th>\n",
       "      <th>last_name</th>\n",
       "      <th>street_name</th>\n",
       "      <th>city</th>\n",
       "      <th>state_abbr</th>\n",
       "      <th>...</th>\n",
       "      <th>company</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>person</th>\n",
       "      <th>name</th>\n",
       "      <th>first_name_female</th>\n",
       "      <th>first_name_male</th>\n",
       "      <th>prefix_female</th>\n",
       "      <th>prefix_male</th>\n",
       "      <th>last_name_female</th>\n",
       "      <th>last_name_male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>Czech</td>\n",
       "      <td>Mrs.</td>\n",
       "      <td>Marie</td>\n",
       "      <td>J</td>\n",
       "      <td>Hamanová</td>\n",
       "      <td>P.O. Box 255</td>\n",
       "      <td>Kangerlussuaq</td>\n",
       "      <td>QE</td>\n",
       "      <td>...</td>\n",
       "      <td>Simple Solutions</td>\n",
       "      <td>MarathonDancing.gl</td>\n",
       "      <td>Marie J Hamanová</td>\n",
       "      <td>Marie J Hamanová</td>\n",
       "      <td>Marie</td>\n",
       "      <td></td>\n",
       "      <td>Mrs.</td>\n",
       "      <td></td>\n",
       "      <td>Hamanová</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>French</td>\n",
       "      <td>Ms.</td>\n",
       "      <td>Patricia</td>\n",
       "      <td>G</td>\n",
       "      <td>Desrosiers</td>\n",
       "      <td>Avenida Noruega 42</td>\n",
       "      <td>Vila Real</td>\n",
       "      <td>VR</td>\n",
       "      <td>...</td>\n",
       "      <td>Formula Gray</td>\n",
       "      <td>LostMillions.com.pt</td>\n",
       "      <td>Patricia Desrosiers</td>\n",
       "      <td>Patricia Desrosiers</td>\n",
       "      <td>Patricia</td>\n",
       "      <td></td>\n",
       "      <td>Ms.</td>\n",
       "      <td></td>\n",
       "      <td>Desrosiers</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>American</td>\n",
       "      <td>Ms.</td>\n",
       "      <td>Debra</td>\n",
       "      <td>O</td>\n",
       "      <td>Neal</td>\n",
       "      <td>1659 Hoog St</td>\n",
       "      <td>Brakpan</td>\n",
       "      <td>GA</td>\n",
       "      <td>...</td>\n",
       "      <td>Dahlkemper's</td>\n",
       "      <td>MediumTube.co.za</td>\n",
       "      <td>Debra O Neal</td>\n",
       "      <td>Debra O Neal</td>\n",
       "      <td>Debra</td>\n",
       "      <td></td>\n",
       "      <td>Ms.</td>\n",
       "      <td></td>\n",
       "      <td>Neal</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>male</td>\n",
       "      <td>French</td>\n",
       "      <td>Mr.</td>\n",
       "      <td>Peverell</td>\n",
       "      <td>C</td>\n",
       "      <td>Racine</td>\n",
       "      <td>183 Epimenidou Street</td>\n",
       "      <td>Limassol</td>\n",
       "      <td>LI</td>\n",
       "      <td>...</td>\n",
       "      <td>Quickbiz</td>\n",
       "      <td>ImproveLook.com.cy</td>\n",
       "      <td>Peverell Racine</td>\n",
       "      <td>Peverell Racine</td>\n",
       "      <td></td>\n",
       "      <td>Peverell</td>\n",
       "      <td></td>\n",
       "      <td>Mr.</td>\n",
       "      <td></td>\n",
       "      <td>Racine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>female</td>\n",
       "      <td>Slovenian</td>\n",
       "      <td>Mrs.</td>\n",
       "      <td>Iolanda</td>\n",
       "      <td>S</td>\n",
       "      <td>Tratnik</td>\n",
       "      <td>Karu põik 61</td>\n",
       "      <td>Pärnu</td>\n",
       "      <td>PR</td>\n",
       "      <td>...</td>\n",
       "      <td>Dubrow's Cafeteria</td>\n",
       "      <td>PostTan.com.ee</td>\n",
       "      <td>Iolanda Tratnik</td>\n",
       "      <td>Iolanda Tratnik</td>\n",
       "      <td>Iolanda</td>\n",
       "      <td></td>\n",
       "      <td>Mrs.</td>\n",
       "      <td></td>\n",
       "      <td>Tratnik</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   number  gender nationality prefix first_name middle_initial   last_name  \\\n",
       "0       1  female       Czech   Mrs.      Marie              J    Hamanová   \n",
       "1       2  female      French    Ms.   Patricia              G  Desrosiers   \n",
       "2       3  female    American    Ms.      Debra              O        Neal   \n",
       "3       4    male      French    Mr.   Peverell              C      Racine   \n",
       "4       5  female   Slovenian   Mrs.    Iolanda              S     Tratnik   \n",
       "\n",
       "             street_name           city state_abbr  ...             company  \\\n",
       "0           P.O. Box 255  Kangerlussuaq         QE  ...    Simple Solutions   \n",
       "1     Avenida Noruega 42      Vila Real         VR  ...        Formula Gray   \n",
       "2           1659 Hoog St        Brakpan         GA  ...        Dahlkemper's   \n",
       "3  183 Epimenidou Street       Limassol         LI  ...            Quickbiz   \n",
       "4           Karu põik 61          Pärnu         PR  ...  Dubrow's Cafeteria   \n",
       "\n",
       "           domain_name               person                 name  \\\n",
       "0   MarathonDancing.gl     Marie J Hamanová     Marie J Hamanová   \n",
       "1  LostMillions.com.pt  Patricia Desrosiers  Patricia Desrosiers   \n",
       "2     MediumTube.co.za         Debra O Neal         Debra O Neal   \n",
       "3   ImproveLook.com.cy      Peverell Racine      Peverell Racine   \n",
       "4       PostTan.com.ee      Iolanda Tratnik      Iolanda Tratnik   \n",
       "\n",
       "  first_name_female first_name_male prefix_female prefix_male  \\\n",
       "0             Marie                          Mrs.               \n",
       "1          Patricia                           Ms.               \n",
       "2             Debra                           Ms.               \n",
       "3                          Peverell                       Mr.   \n",
       "4           Iolanda                          Mrs.               \n",
       "\n",
       "   last_name_female last_name_male  \n",
       "0          Hamanová                 \n",
       "1        Desrosiers                 \n",
       "2              Neal                 \n",
       "3                           Racine  \n",
       "4           Tratnik                 \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read FakeNameGenerator CSV\n",
    "fake_name_generator_df = pd.read_csv(fake_name_generator_file)\n",
    "\n",
    "# Update to match existing templates\n",
    "fake_name_generator_df = PresidioDataGenerator.update_fake_name_generator_df(fake_name_generator_df)\n",
    "fake_name_generator_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Create a Faker object (in this case, a `RecordsFaker`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "is_executing": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create RecordsFaker (extension which handles records instead of independent values) and add additional specific providers\n",
    "fake = RecordsFaker(records=fake_name_generator_df, locale=\"en_US\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Add more providers, not part of the original Faker package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "is_executing": true
   },
   "outputs": [],
   "source": [
    "fake.add_provider(IpAddressProvider)  # Both Ipv4 and IPv6 IP addresses\n",
    "fake.add_provider(NationalityProvider)  # Read countries + nationalities from file\n",
    "fake.add_provider(OrganizationProvider)  # Read organization names from file\n",
    "fake.add_provider(UsDriverLicenseProvider)  # Read US driver license numbers from file\n",
    "fake.add_provider(AgeProvider)  # Age values (unavailable on Faker)\n",
    "fake.add_provider(AddressProviderNew)  # Extend the default address formats\n",
    "fake.add_provider(PhoneNumberProviderNew)  # Extend the default phone number formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "e. Create the Presidio Data Generator object and add provider aliases if the templates have a different entity name than the Faker object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Create Presidio Data Generator\n",
    "data_generator = PresidioDataGenerator(\n",
    "    custom_faker=fake, lower_case_ratio=lower_case_ratio\n",
    ")\n",
    "\n",
    "# Create entity aliases (e.g. if faker supports \"name\" but templates contain \"person\").\n",
    "data_generator.add_provider_alias(provider_name=\"name\", new_name=\"person\")\n",
    "data_generator.add_provider_alias(\n",
    "    provider_name=\"credit_card_number\", new_name=\"credit_card\"\n",
    ")\n",
    "data_generator.add_provider_alias(provider_name=\"date_of_birth\", new_name=\"birthday\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "scrolled": true
   },
   "source": [
    "f. Generate data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sampling: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:00<00:00, 17987.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"fake\": \"Title VII of the Civil Rights Act of 2005 protects individuals against employment discrimination on the basis of race and color as well as national origin, sex, or religion.\\n\", \"spans\": [{\"value\": \"2005\", \"start\": 37, \"end\": 41, \"type\": \"year\"}], \"template\": \"Title VII of the Civil Rights Act of {{year}} protects individuals against employment discrimination on the basis of race and color as well as national origin, sex, or religion.\\n\", \"template_id\": 190}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "sentence_templates = PresidioDataGenerator.read_template_file(templates_file_path)\n",
    "fake_records = data_generator.generate_fake_data(\n",
    "    templates=sentence_templates, n_samples=number_of_samples\n",
    ")\n",
    "\n",
    "fake_records = list(fake_records)\n",
    "pprint.pprint(fake_records[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verify randomness of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "is_executing": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total: 1500\n",
      "Avg # of records per template: 7.142857142857143\n",
      "Median # of records per template: 7.0\n",
      "Std: 2.5872528966106905\n"
     ]
    }
   ],
   "source": [
    "count_per_template_id = Counter([sample.template_id for sample in fake_records])\n",
    "\n",
    "print(f\"Total: {sum(count_per_template_id.values())}\")\n",
    "print(f\"Avg # of records per template: {np.mean(list(count_per_template_id.values()))}\")\n",
    "print(f\"Median # of records per template: {np.median(list(count_per_template_id.values()))}\")\n",
    "print(f\"Std: {np.std(list(count_per_template_id.values()))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Which entities did we generate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'organization': 257,\n",
       "         'first_name': 244,\n",
       "         'person': 238,\n",
       "         'city': 235,\n",
       "         'address': 209,\n",
       "         'street_name': 164,\n",
       "         'name': 162,\n",
       "         'country': 154,\n",
       "         'credit_card_number': 152,\n",
       "         'phone_number': 121,\n",
       "         'last_name': 119,\n",
       "         'building_number': 110,\n",
       "         'age': 72,\n",
       "         'secondary_address': 64,\n",
       "         'year': 58,\n",
       "         'nationality': 55,\n",
       "         'postcode': 49,\n",
       "         'zipcode': 45,\n",
       "         'url': 39,\n",
       "         'email': 39,\n",
       "         'name_female': 37,\n",
       "         'job': 33,\n",
       "         'first_name_male': 31,\n",
       "         'name_male': 29,\n",
       "         'prefix_male': 28,\n",
       "         'date_of_birth': 24,\n",
       "         'iban': 22,\n",
       "         'date_time': 21,\n",
       "         'prefix_female': 21,\n",
       "         'day_of_week': 16,\n",
       "         'state_abbr': 15,\n",
       "         'last_name_male': 15,\n",
       "         'prefix': 12,\n",
       "         'ip_address': 11,\n",
       "         'ssn': 11,\n",
       "         'nation_plural': 9,\n",
       "         'nation_woman': 8,\n",
       "         'first_name_nonbinary': 6,\n",
       "         'us_driver_license': 6,\n",
       "         'first_name_female': 3,\n",
       "         'last_name_female': 3})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_per_entity = Counter()\n",
    "for record in fake_records:\n",
    "    count_per_entity.update(Counter([span.type for span in record.spans]))\n",
    "\n",
    "count_per_entity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Translate tags from Faker's to Presidio's (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "translator = {\n",
    "    \"person\": \"PERSON\",\n",
    "    \"ip_address\": \"IP_ADDRESS\",\n",
    "    \"us_driver_license\": \"US_DRIVER_LICENSE\",\n",
    "    \"organization\": \"ORGANIZATION\",\n",
    "    \"name_female\": \"PERSON\",\n",
    "    \"address\": \"STREET_ADDRESS\",\n",
    "    \"country\": \"GPE\",\n",
    "    \"state\": \"GPE\",\n",
    "    \"credit_card_number\": \"CREDIT_CARD\",\n",
    "    \"city\": \"GPE\",\n",
    "    \"street_name\": \"STREET_ADDRESS\",\n",
    "    \"building_number\": \"STREET_ADDRESS\",\n",
    "    \"name\": \"PERSON\",\n",
    "    \"iban\": \"IBAN_CODE\",\n",
    "    \"last_name\": \"PERSON\",\n",
    "    \"last_name_male\": \"PERSON\",\n",
    "    \"last_name_female\": \"PERSON\",\n",
    "    \"first_name\": \"PERSON\",\n",
    "    \"first_name_male\": \"PERSON\",\n",
    "    \"first_name_female\": \"PERSON\",\n",
    "    \"phone_number\": \"PHONE_NUMBER\",\n",
    "    \"url\": \"DOMAIN_NAME\",\n",
    "    \"ssn\": \"US_SSN\",\n",
    "    \"email\": \"EMAIL_ADDRESS\",\n",
    "    \"date_time\": \"DATE_TIME\",\n",
    "    \"date_of_birth\": \"DATE_TIME\",\n",
    "    \"day_of_week\": \"DATE_TIME\",\n",
    "    \"year\": \"DATE_TIME\",\n",
    "    \"name_male\": \"PERSON\",\n",
    "    \"prefix_male\": \"TITLE\",\n",
    "    \"prefix_female\": \"TITLE\",\n",
    "    \"prefix\": \"TITLE\",\n",
    "    \"nationality\": \"NRP\",\n",
    "    \"nation_woman\": \"NRP\",\n",
    "    \"nation_man\": \"NRP\",\n",
    "    \"nation_plural\": \"NRP\",\n",
    "    \"first_name_nonbinary\": \"PERSON\",\n",
    "    \"postcode\": \"STREET_ADDRESS\",\n",
    "    \"secondary_address\": \"STREET_ADDRESS\",\n",
    "    \"job\": \"TITLE\",\n",
    "    \"zipcode\": \"ZIP_CODE\",\n",
    "    \"state_abbr\": \"GPE\",\n",
    "    \"age\": \"AGE\",\n",
    "}\n",
    "\n",
    "def update_entity_types(dataset:List[FakerSpansResult], entity_mapping:Dict[str,str]):\n",
    "    \"\"\"Replace entity types using a translator dictionary.\"\"\"\n",
    "\n",
    "    for sample in dataset:\n",
    "        # update entity types on spans\n",
    "        for span in sample.spans:\n",
    "            span.type = entity_mapping[span.type]\n",
    "        # update entity types on the template string\n",
    "        for key, value in entity_mapping.items():\n",
    "            sample.template = sample.template.replace(\"{{\" + key + \"}}\", \"{{\" + value + \"}}\")\n",
    "\n",
    "update_entity_types(fake_records, entity_mapping=translator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "is_executing": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\"fake\": \"Title VII of the Civil Rights Act of 2005 protects individuals against employment discrimination on the basis of race and color as well as national origin, sex, or religion.\\n\", \"spans\": [{\"value\": \"2005\", \"start\": 37, \"end\": 41, \"type\": \"DATE_TIME\"}], \"template\": \"Title VII of the Civil Rights Act of {{DATE_TIME}} protects individuals against employment discrimination on the basis of race and color as well as national origin, sex, or religion.\\n\", \"template_id\": 190}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fake_records[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frequency of new entity types after mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('PERSON', 887),\n",
       " ('STREET_ADDRESS', 596),\n",
       " ('GPE', 404),\n",
       " ('ORGANIZATION', 257),\n",
       " ('CREDIT_CARD', 152),\n",
       " ('PHONE_NUMBER', 121),\n",
       " ('DATE_TIME', 119),\n",
       " ('TITLE', 94),\n",
       " ('NRP', 72),\n",
       " ('AGE', 72),\n",
       " ('ZIP_CODE', 45),\n",
       " ('DOMAIN_NAME', 39),\n",
       " ('EMAIL_ADDRESS', 39),\n",
       " ('IBAN_CODE', 22),\n",
       " ('IP_ADDRESS', 11),\n",
       " ('US_SSN', 11),\n",
       " ('US_DRIVER_LICENSE', 6)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "count_per_entity_new = Counter()\n",
    "for record in fake_records:\n",
    "    for span in record.spans:\n",
    "        count_per_entity_new[span.type] += 1\n",
    "\n",
    "count_per_entity_new.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenize and transform the fake samples to a list of `InputSample` objects (Common data structure for this package)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the spaCy tokenizer model if missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                                                   | 0/1500 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading model en_core_web_sm\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:06<00:00, 215.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.76 s, sys: 33.8 ms, total: 6.8 s\n",
      "Wall time: 6.96 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "input_samples = [\n",
    "    InputSample.from_faker_spans_result(faker_spans_result=fake_record)\n",
    "    for fake_record in tqdm.tqdm(fake_records)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Save as json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "InputSample.to_json(dataset=input_samples, output_file=output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Create a CONLL like data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:00<00:00, 76888.23it/s]\n"
     ]
    }
   ],
   "source": [
    "conll = InputSample.create_conll_dataset(input_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "is_executing": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conll.to_csv(output_conll, sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps\n",
    "\n",
    "- Evaluate Presidio using this fake data. [Sample](4_Evaluate_Presidio_Analyzer.ipynb)\n",
    "- Split to train/test/validation while ensuring sentences originiating from the same template are all on the same subset. [Sample](3_Split_by_pattern_number.ipynb)\n",
    "- Conduct a small exploratory data analysis on the generated data. [Sample](2_PII_EDA.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copyright notice:\n",
    "\n",
    "\n",
    "Data generated for evaluation was created using Fake Name Generator.\n",
    "\n",
    "Fake Name Generator identities by the [Fake Name Generator](https://www.fakenamegenerator.com/) \n",
    "are licensed under a [Creative Commons Attribution-Share Alike 3.0 United States License](http://creativecommons.org/licenses/by-sa/3.0/us/). Fake Name Generator and the Fake Name Generator logo are trademarks of Corban Works, LLC."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2509fbe9adc3579fd0ef23e6a2c6fb50cb745caa174aafdf017283479e60bc43"
  },
  "kernelspec": {
   "display_name": "presidio_evaluator",
   "language": "python",
   "name": "presidio_evaluator"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}