1.fair_unfair_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness in Biometrics: a figure of merit to assess biometric verification system\n",
    "\n",
    "This notebook is an extension of the paper `Fairness in Biometrics: a figure of merit to assess biometric verification systems`.\n",
    "The goal is to give a little bit more intuition about the issues with respect to fairness in biometric verification and how the Fairness Discrepancy Rate can be useful.\n",
    "\n",
    "## Fair example\n",
    "\n",
    "The cell below presents the **fair** synthetic biometric verification system example from section 3.1\n",
    "Play with the variables below to sense the problem of having one decision threshold that is fair to **all demographic groups**.\n",
    "\n",
    "Play also with `ALPHA` variable.\n",
    "\n",
    "You can extend the number of demographics by adding/removing elements in `DEMOGRAPHIC_MEAN_DISTANCE` and `DEMOGRAPHIC_STD`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f44edd53ef6549799b8784580b4e4f36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##############################################\n",
      "################## FMR #######################\n",
      "##############################################\n",
      "       0.1    0.01    0.001    0.0001    1e-05    1e-06\n",
      " 0  0.0804  0.0084   0.0008    0.0001        0        0\n",
      " 1  0.0821  0.0085   0.0008    0.0001        0        0\n",
      " 2  0.0804  0.0086   0.0008    0             0        0\n",
      "##############################################\n",
      "################## FNMR #######################\n",
      "##############################################\n",
      "      0.1    0.01    0.001    0.0001    1e-05    1e-06\n",
      " 0      0  0        0.0027    0.0389   0.186    0.4577\n",
      " 1      0  0.0001   0.0041    0.0403   0.191    0.4599\n",
      " 2      0  0        0.0044    0.0415   0.1756   0.4346\n",
      "##############################################\n",
      "################## FDR #######################\n",
      "##############################################\n",
      "    0.1     0.01    0.001    0.0001    1e-05    1e-06\n",
      "0.99915  0.99985  0.99915   0.99865   0.9923  0.98735\n"
     ]
    }
   ],
   "source": [
    "%matplotlib widget\n",
    "\n",
    "#################################################\n",
    "## PLAYGROUND: PLAY WITH THE VALUES OF VARIABLES BELOW TO CHANGE THE BEHAVIOUR OF THE FAIR VERIFICATION SYSTEM\n",
    "\n",
    "ALPHA = 0.5\n",
    "\n",
    "\n",
    "# RANGE OF DECISION THRESHOLDS USED IN THE DEVELOPMENT SET\n",
    "TEN_POWER_OF_X = [0.1, 0.01, 0.001, 0.0001, 0.00001, 0.000001] # BASICALLY, THE FMR THRESHOLDS\n",
    "\n",
    "# NUMBER OF IDENTITIES PER DEMOGRAPHIC\n",
    "IDENTITIES_PER_DEMOGRAPHIC = 300\n",
    "\n",
    "# REPRESENTS HOW A SAMPLES ARE SAMPLED\n",
    "# EACH COLUMN REPRESENTS ONE DEMOGRAPHIC.\n",
    "# THE VALUES REPRESENTS THE AVERAGE EUCLIDEAN DISTANCE BETWEEN EACH IDENTITY\n",
    "# EACH IDENTITY IS SAMPLED IN THE CORNER OF A HYPERCUBE\n",
    "DEMOGRAPHIC_MEAN_DISTANCE = [1, 1, 1]\n",
    "# EACH COLUMN REPRESENTS ONE DEMOGRAPHIC.\n",
    "# THE VALUES REPRESENTS THE STANDARD DEVIATIONS OF THE SAMPLES SAMPLED IN THE CORNER OF A HYPERCUBE\n",
    "DEMOGRAPHIC_STD = [0.2, 0.2, 0.2]\n",
    "#################################################\n",
    "\n",
    "\n",
    "\n",
    "import plots\n",
    "import experiments\n",
    "import scoring\n",
    "import fairness_metric\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# fetching the samples\n",
    "impostors_per_demographic_dev_fair, genuines_per_demographic_dev_fair, \\\n",
    "impostors_per_demographic_eval_fair, genuines_per_demographic_eval_fair = \\\n",
    "experiments.generate_uncalibrated_experiment(DEMOGRAPHIC_DISPLACEMENTS=DEMOGRAPHIC_MEAN_DISTANCE,\n",
    "                                             DEMOGRAPHIC_STD=DEMOGRAPHIC_STD,\n",
    "                                             DEV_EVAL_IDENTITIES_PER_DEMOGRAPHIC=IDENTITIES_PER_DEMOGRAPHIC)\n",
    "\n",
    "n_demographics_fair = len(DEMOGRAPHIC_MEAN_DISTANCE)\n",
    "\n",
    "# computing TAU USING THE ZEROTH-EFFORT IMPOSTORS from the development set\n",
    "taus_fair = scoring.compute_thresholds(impostors_per_demographic_dev_fair, TEN_POWER_OF_X)\n",
    "\n",
    "# Plot Figure 1 using data from evaluation set\n",
    "plots.plot_cohort_box_plot(impostors_per_demographic_eval_fair, \n",
    "                     genuines_per_demographic_eval_fair, \n",
    "                     [f\"{i}\" for i in range(n_demographics_fair)],\n",
    "                     thresholds=taus_fair,\n",
    "                     far_thresholds=[10**(i+1) for i,_ in enumerate(TEN_POWER_OF_X)]\n",
    "                    )\n",
    "\n",
    "plots.compute_FMR_FNMR_FDR(impostors_per_demographic_eval_fair,\n",
    "                           genuines_per_demographic_eval_fair,\n",
    "                           thresholds=taus_fair,\n",
    "                           far_thresholds=TEN_POWER_OF_X,\n",
    "                           tablefmt=\"plain\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unfair example\n",
    "\n",
    "The cell below presents the **UNFAIR** synthetic biometric verification system example from section 3.1\n",
    "Play with the variables below to sense the problem of having one decision threshold that is fair to **all demographic groups**.\n",
    "\n",
    "Play also with `ALPHA` variable.\n",
    "\n",
    "You can extend the number of demographics by adding/removing elements in `DEMOGRAPHIC_MEAN_DISTANCE` and `DEMOGRAPHIC_STD`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51c25081ed5f492e979f84a39ea01cd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##############################################\n",
      "################## FMR #######################\n",
      "##############################################\n",
      "       0.1    0.01    0.001    0.0001    1e-05    1e-06\n",
      " 0  0.2255  0.0256   0.0024    0.0002        0        0\n",
      " 1  0.0024  0        0         0             0        0\n",
      " 2  0       0        0         0             0        0\n",
      "##############################################\n",
      "################## FNMR #######################\n",
      "##############################################\n",
      "       0.1    0.01    0.001    0.0001    1e-05    1e-06\n",
      " 0  0       0        0         0        0        0\n",
      " 1  0       0.006    0.0876    0.2471   0.4596   0.6482\n",
      " 2  0.0016  0.5701   0.8576    0.9438   0.981    0.9909\n",
      "##############################################\n",
      "################## FDR #######################\n",
      "##############################################\n",
      "    0.1     0.01    0.001    0.0001    1e-05    1e-06\n",
      "0.88645  0.70215     0.57     0.528   0.5095  0.50455\n"
     ]
    }
   ],
   "source": [
    "#################################################\n",
    "## PLAYGROUND: PLAY WITH THE VALUES OF VARIABLES BELOW TO CHANGE THE BEHAVIOUR OF THE FAIR VERIFICATION SYSTEM\n",
    "\n",
    "# NUMBER OF IDENTITIES PER DEMOGRAPHIC\n",
    "IDENTITIES_PER_DEMOGRAPHIC = 300\n",
    "\n",
    "# REPRESENTS HOW A SAMPLES ARE SAMPLED\n",
    "# EACH COLUMN REPRESENTS ONE DEMOGRAPHIC.\n",
    "# THE VALUES REPRESENTS THE AVERAGE EUCLIDEAN DISTANCE BETWEEN EACH IDENTITY\n",
    "# EACH IDENTITY IS SAMPLED IN THE CORNER OF A HYPERCUBE\n",
    "DEMOGRAPHIC_MEAN_DISTANCE = [0.8, 1.5, 2.3]\n",
    "# EACH COLUMN REPRESENTS ONE DEMOGRAPHIC.\n",
    "# THE VALUES REPRESENTS THE STANDARD DEVIATIONS OF THE SAMPLES SAMPLED IN THE CORNER OF A HYPERCUBE\n",
    "DEMOGRAPHIC_STD = [0.1, 0.2, 0.3]\n",
    "#################################################\n",
    "\n",
    "n_demographics_unfair = len(DEMOGRAPHIC_MEAN_DISTANCE)\n",
    "\n",
    "impostors_per_demographic_dev_unfair, genuines_per_demographic_dev_unfair, \\\n",
    "impostors_per_demographic_eval_unfair, genuines_per_demographic_eval_unfair = \\\n",
    "experiments.generate_uncalibrated_experiment(DEMOGRAPHIC_DISPLACEMENTS=DEMOGRAPHIC_MEAN_DISTANCE,\n",
    "                                             DEMOGRAPHIC_STD=DEMOGRAPHIC_STD,\n",
    "                                             DEV_EVAL_IDENTITIES_PER_DEMOGRAPHIC=IDENTITIES_PER_DEMOGRAPHIC)\n",
    "                       \n",
    "# computing TAU USING THE ZEROTH-EFFORT IMPOSTORS from the development set\n",
    "taus_unfair = scoring.compute_thresholds(impostors_per_demographic_dev_unfair, TEN_POWER_OF_X)\n",
    "\n",
    "# Plot Figure 1 using data from evaluation set\n",
    "plots.plot_cohort_box_plot(impostors_per_demographic_eval_unfair,\n",
    "                           genuines_per_demographic_eval_unfair,\n",
    "                           [f\"{i}\" for i in range(n_demographics_unfair)],\n",
    "                           thresholds=taus_unfair,\n",
    "                           far_thresholds=[10**(i+1) for i,_ in enumerate(taus_unfair)])\n",
    "\n",
    "\n",
    "plots.compute_FMR_FNMR_FDR(impostors_per_demographic_eval_unfair,\n",
    "                           genuines_per_demographic_eval_unfair,\n",
    "                           thresholds=taus_unfair,\n",
    "                           far_thresholds=TEN_POWER_OF_X,\n",
    "                           tablefmt=\"plain\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairness Discrepancy Curve\n",
    "\n",
    "It computes **FDR** for the synthetic fair and unfair systems and computes the Area Under FDR.\n",
    "\n",
    "Play around with the value of `ALPHA` to sense how this impacts the FDR Curve and the Area Under FDR.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c163eb37a1b42689df9d9ad9fdff156",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AU-FDR from fair system 0.999493051121444\n",
      "AU-FDR from unfair system 0.7775781073589396\n"
     ]
    }
   ],
   "source": [
    "import fairness_metric\n",
    "\n",
    "### FAIR\n",
    "f_tau_fair = fairness_metric.compute_fairness_biometric_verification(impostors_per_demographic_eval_fair, \n",
    "                                                                     genuines_per_demographic_eval_fair,\n",
    "                                                                     taus_fair,\n",
    "                                                                     len(taus_fair),\n",
    "                                                                     n_demographics_fair,\n",
    "                                                                     alpha=ALPHA)\n",
    "\n",
    "f_tau_unfair = fairness_metric.compute_fairness_biometric_verification(impostors_per_demographic_eval_unfair,\n",
    "                                                                       genuines_per_demographic_eval_unfair,\n",
    "                                                                       taus_unfair,\n",
    "                                                                       len(taus_unfair),\n",
    "                                                                       n_demographics_unfair,\n",
    "                                                                       alpha=ALPHA)\n",
    "\n",
    "fairness_metric.fairness_plot_clean([f_tau_fair,f_tau_unfair],\n",
    "                                    labels=[\"Fair Biom. System\",\"Unfair Biom. System\"],\n",
    "                                    far_thresholds=TEN_POWER_OF_X,\n",
    "                                    title=\"\")\n",
    "\n",
    "\n",
    "\n",
    "print(f\"AU-FDR from fair system {fairness_metric.fpr_auc_score(f_tau_fair, TEN_POWER_OF_X)}\")\n",
    "print(f\"AU-FDR from unfair system {fairness_metric.fpr_auc_score(f_tau_unfair, TEN_POWER_OF_X)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why ROC curves are not fairness proxies?\n",
    "\n",
    "In this example we demonstrate why ROC curves are not proxies for fairness.\n",
    "\n",
    "ROC curves measures the True and False Positive Rates (TPR and FPR respectively) trade-offs.\n",
    "Although this seems sensible; it assumes that verification decision thresholds are demographic-specific.\n",
    "Hence, TPR and FPR is computed under different decision thresholds depending on the demographic.\n",
    "This doesn’t represent operational conditions where one single threshold is set, and this operational point has to be fair with respect to different demographics.\n",
    "The plot below demonstrates that by showing the ROC curve of the fair/unfair biometric systems of the three demographics.\n",
    "It's possible to notice that all ROC curves are perfectly aligned and the Area Under the ROC is equals to 1 for every single demographic for both fair/unfair systems.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31ab87b9d0b24e3498860e0a35e5b097",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import bob.measure\n",
    "\n",
    "plt.figure()\n",
    "for i in range(3):\n",
    "            \n",
    "    bob.measure.plot.roc(impostors_per_demographic_eval_fair[i],\n",
    "                         genuines_per_demographic_eval_fair[i],\n",
    "                         tpr=True,\n",
    "                         label = None if i>0 else \"Fair Biom. Systems\",\n",
    "                         alpha=0.60)\n",
    "\n",
    "    bob.measure.plot.roc(impostors_per_demographic_eval_unfair[i],\n",
    "                         genuines_per_demographic_eval_unfair[i],\n",
    "                         tpr=True,\n",
    "                         label = None if i>0 else \"Unfair Biom. Systems\",\n",
    "                         linestyle=\"--\"\n",
    "                         )\n",
    "\n",
    "    \n",
    "plt.ylabel(\"True Positive Rate\", fontsize=18)\n",
    "plt.xlabel(\"False Positive Rate\", fontsize=18)\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}