Curva de Benefício-Alunos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercício: Curva de Benefício\n",
    "\n",
    "Assuma que estamos lidando com um problema de churn e gostaríamos de entender qual o threshold ótimo da minha classificação. Considere que:\n",
    "- `y_pred` é o output do meu classificador, indicando a probabilidade de um cliente dar churn, obtido em um backtest\n",
    "- `y` é o valor real medido, no backtest\n",
    "\n",
    "Nesta ação, os clientes identificados com alta probabilidade de churn irão receber um voucher de R$30,00 de compras, através de uma ação no próprio site e divulgação por push. Assuma ainda que:\n",
    "\n",
    "- Cada cliente tem um LTV potencial futuro de R\\$ 200,00 (ou seja, perder um cliente custa esse valor para a empresa)\n",
    "- A ação comercial é 100% eficaz para evitar o churn: O cliente impactado pela iniciativa tem 100% de chance de permanecer na empresa\n",
    "\n",
    "\n",
    "\n",
    "Obs.: Para cálculo dos valores financeiros, a sugestão é sempre comparar com um caso de referência (ex. caso não houvesse nenhum modelo de churn)\n",
    "\n",
    "- Valor Monetário de **TN: 0,00** - clientes que não deram churn e que meu modelo indicou corretamente\n",
    "- Valor Monetário de **FP: -30,00** - clientes que não dariam churn, mas o modelo apontou que dariam. Neste caso, minha ação de evitar churn foi em vão\n",
    "- Valor Monetário de **FN: 0,00** - clientes que deram churn e meu modelo não detectou. Neste caso, perco o cliente, mas perderia de qualquer forma se não houvesse o modelo (importância de definir um caso de referência!)\n",
    "- Valor Monetário de **TP: 170,00** - clientes que eu corretamente identifiquei como churn e consegui evitar com a ação comercial. Neste caso, mantenho um cliente avaliado em 200,00, gastando um total de 30,00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\Borba\\anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import pandas as pd\n",
    "from sklearn.metrics import confusion_matrix, roc_auc_score, f1_score\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "%matplotlib inline\n",
    "\n",
    "N = 10000\n",
    "y_pred = np.random.poisson(10, N)\n",
    "y_pred = y_pred/max(y_pred)\n",
    "\n",
    "# Create y\n",
    "y = y_pred -0.2 + np.random.randn(N)/5\n",
    "y = y.round(0)\n",
    "y = np.clip(y, 0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAALwklEQVR4nO3dfYxld13H8fenu0V8qGLcwZA+OJKAsanRkknFYFSKkKY17R8aUpL6kDRuwGgwmpga/lH8p/whUZMmusEGfODJB8wGRK3QZgNhC7O0lHYLptRVtzbuVKS2MSKFr3/cu2W7zO6c7b3nznfufb+SSe/sPXvv97d3+94zZ865k6pCktTXRbs9gCTp/Ay1JDVnqCWpOUMtSc0Zaklqbv8YD3rgwIFaX18f46ElaSkdO3bsiapa2+6+UUK9vr7O5ubmGA8tSUspyb+c6z4PfUhSc4Zakpoz1JLU3KBj1ElOAE8BXwWeqaqNMYeSJH3dhXwz8dVV9cRok0iStuWhD0lqbmioC/iHJMeSHBxzIEnScw099PGjVfVYkhcDdyX5XFUdOXODacAPAlxxxRVzHlOSVtegPeqqemz631PAB4BrttnmUFVtVNXG2tq2F9dIkp6HHfeok3wrcFFVPTW9/TrgraNPptGt3/ah5/17T9x+wxwnkXQ+Qw59fDfwgSSnt393Vf3dqFNJkp61Y6ir6lHgBxcwiyRpG56eJ0nNGWpJas5QS1JzhlqSmjPUktScoZak5gy1JDVnqCWpOUMtSc0ZaklqzlBLUnOGWpKaM9SS1JyhlqTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4Zakpoz1JLUnKGWpOYMtSQ1Z6glqTlDLUnNGWpJas5QS1JzhlqSmjPUktScoZak5gy1JDVnqCWpOUMtSc0NDnWSfUnuS/LBMQeSJD3XhexRvxl4eKxBJEnbGxTqJJcBNwDvGHccSdLZhu5R/x7wG8DXzrVBkoNJNpNsbm1tzWM2SRIDQp3kp4BTVXXsfNtV1aGq2qiqjbW1tbkNKEmrbsge9auAG5OcAN4LXJvkz0adSpL0rB1DXVW/WVWXVdU6cDPw0aq6ZfTJJEmA51FLUnv7L2TjqroHuGeUSSRJ23KPWpKaM9SS1JyhlqTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4Zakpoz1JLUnKGWpOYMtSQ1Z6glqTlDLUnNGWpJas5QS1JzhlqSmjPUktScoZak5gy1JDVnqCWpOUMtSc0ZaklqzlBLUnOGWpKaM9SS1JyhlqTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4ZakprbMdRJXpjkk0k+k+ShJL+9iMEkSRP7B2zzZeDaqno6ycXAx5J8uKqOjjybJIkBoa6qAp6efnrx9KPGHEqS9HWDjlEn2ZfkfuAUcFdV3bvNNgeTbCbZ3NramvOYkrS6BoW6qr5aVT8EXAZck+SqbbY5VFUbVbWxtrY25zElaXVd0FkfVfUl4G7gulGmkSR9gyFnfawledH09jcDrwU+N/JckqSpIWd9vAR4V5J9TML+/qr64LhjSZJOG3LWxwPA1QuYRZK0Da9MlKTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4Zakpoz1JLU3JArE6VvsH7bh2b6/Sduv2FOk0jLzz1qSWrOUEtSc4Zakpoz1JLUnKGWpOYMtSQ1Z6glqTlDLUnNGWpJas5QS1JzhlqSmvO9PrQrZnmvEN8nRKvGPWpJas5QS1JzhlqSmjPUktScoZak5gy1JDVnqCWpOUMtSc0ZaklqzlBLUnOGWpKaM9SS1NyOoU5yeZK7kxxP8lCSNy9iMEnSxJB3z3sG+PWq+nSSS4BjSe6qquMjzyZJYsAedVU9XlWfnt5+CngYuHTswSRJExd0jDrJOnA1cO829x1Msplkc2tra07jSZIGhzrJtwF/BfxqVf332fdX1aGq2qiqjbW1tXnOKEkrbdBPeElyMZNI/3lV/fW4I0nn50+H0aoZctZHgD8GHq6qt48/kiTpTEMOfbwK+Fng2iT3Tz+uH3kuSdLUjoc+qupjQBYwiyRpG/4U8j1uluO1kvYGLyGXpOYMtSQ1Z6glqTlDLUnNGWpJas5QS1JzhlqSmjPUktScoZak5gy1JDVnqCWpOUMtSc0ZaklqznfP00rxp8NoL3KPWpKaM9SS1JyhlqTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4Zakpoz1JLUnKGWpOYMtSQ155syNTDLGwVJWn7uUUtSc4Zakprz0Ic0kO9lrd3iHrUkNWeoJam5HUOd5M4kp5I8uIiBJEnPNWSP+p3AdSPPIUk6hx1DXVVHgC8uYBZJ0jbmdow6ycEkm0k2t7a25vWwkrTy5hbqqjpUVRtVtbG2tjavh5WkledZH5LUnKGWpOaGnJ73HuATwPclOZnk1vHHkiSdtuMl5FX1hkUMIknanoc+JKk5Qy1JzRlqSWrOtzmdA39Ci6QxuUctSc25Ry3tAf7QgtVmqKUF8PCYZuGhD0lqzlBLUnOGWpKaM9SS1JyhlqTmDLUkNWeoJak5Qy1JzRlqSWrOUEtSc4Zakpoz1JLUnKGWpOZ897wp391MUlfuUUtSc4Zakpoz1JLUnKGWpOYMtSQ1Z6glqTlDLUnNeR61tORmuUbgxO03zHESPV/uUUtSc0u1R+3VhZKWkXvUktScoZak5gYd+khyHfD7wD7gHVV1+1gDefhCkp5rx1An2QfcAbwWOAl8Ksnhqjo+9nCSdpdnjPQwZI/6GuCRqnoUIMl7gZsAQy3pnFbxq+Ox/nEaEupLgX874/OTwA+fvVGSg8DB6adPJ/k8cAB4YtYh96hVXjus9vpXee2wwuvP22Za+/ec6465nZ5XVYeAQ2f+WpLNqtqY13PsJau8dljt9a/y2mG11z/W2oec9fEYcPkZn182/TVJ0gIMCfWngJcl+d4kLwBuBg6PO5Yk6bQdD31U1TNJfhn4eyan591ZVQ8NfPxDO2+ytFZ57bDa61/ltcNqr3+UtaeqxnhcSdKceGWiJDVnqCWpuZlDneS6JJ9P8kiS27a5/5uSvG96/71J1md9zk4GrP/XkhxP8kCSjyQ557mSe81Oaz9ju59OUkmW6pStIetP8vrp6/9QkncvesaxDPh7f0WSu5PcN/27f/1uzDmGJHcmOZXkwXPcnyR/MP2zeSDJK2Z+0qp63h9Mvrn4BeClwAuAzwBXnrXNLwF/OL19M/C+WZ6z08fA9b8a+Jbp7Tcty/qHrH263SXAEeAosLHbcy/4tX8ZcB/wndPPX7zbcy9w7YeAN01vXwmc2O2557j+HwNeATx4jvuvBz4MBHglcO+szznrHvWzl5dX1f8Bpy8vP9NNwLumt/8SeE2SzPi8Xey4/qq6u6r+Z/rpUSbnoS+DIa89wO8AbwP+d5HDLcCQ9f8icEdV/RdAVZ1a8IxjGbL2Ar59evs7gH9f4HyjqqojwBfPs8lNwJ/UxFHgRUleMstzzhrq7S4vv/Rc21TVM8CTwHfN+LxdDFn/mW5l8i/tMthx7dMv+S6vqmV804chr/3LgZcn+XiSo9N3oVwGQ9b+W8AtSU4Cfwv8ymJGa+FCu7CjpfoJL50luQXYAH58t2dZhCQXAW8HfmGXR9lN+5kc/vgJJl9JHUnyA1X1pd0cakHeALyzqn43yY8Af5rkqqr62m4PthfNukc95PLyZ7dJsp/Jl0H/OePzdjHo8vokPwm8Bbixqr68oNnGttPaLwGuAu5JcoLJsbrDS/QNxSGv/UngcFV9par+GfgnJuHe64as/Vbg/QBV9QnghUzerGkVzP1tN2YN9ZDLyw8DPz+9/TPAR2t6xH0J7Lj+JFcDf8Qk0styjBJ2WHtVPVlVB6pqvarWmRyfv7GqNndn3Lkb8nf/b5jsTZPkAJNDIY8ucMaxDFn7vwKvAUjy/UxCvbXQKXfPYeDnpmd/vBJ4sqoen+kR5/Ad0OuZ7Cl8AXjL9NfeyuR/Spi8QH8BPAJ8Enjpbn/Xdp4fA9b/j8B/APdPPw7v9syLWvtZ297DEp31MfC1D5PDP8eBzwI37/bMC1z7lcDHmZwRcj/wut2eeY5rfw/wOPAVJl813Qq8EXjjGa/7HdM/m8/O4++9l5BLUnNemShJzRlqSWrOUEtSc4Zakpoz1JLUnKGWpOYMtSQ19/9dAspFKOe54wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "count, bins, ignored = plt.hist(y_pred, 20, density=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11.77"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100*sum(y)/len(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPyElEQVR4nO3cf4xlZX3H8fdHVtoUqaA7RQR0bYukSAuSyaqppVCUwkqgP4xlUytY2lWKTW1NG1oTMfqPxqiJYt2uskEbRWpb7CaAQKwN2oA6IOCCIitdZRfKDqKoxdaufvvHnE3H8V7m7j13Zphn36/kZs55nuee5/vszH7mzLnn3lQVkqR2PWmlC5AkLS2DXpIaZ9BLUuMMeklqnEEvSY1bs9IFDLJ27dpat27dSpchSavGrbfe+nBVTQ3qe0IG/bp165iZmVnpMiRp1Ujy9WF9XrqRpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxi95Hn2QrcDawp6pO6NquAo7rhhwGfLuqThrw3J3Ad4EfAnuranoiVUuSRjbKG6auAC4DPryvoap+b992kncCjz7O80+rqofHLVCS1M+iQV9VNyVZN6gvSYBXAL8x4brGtu6Sa8Z+7s63vWyClUjSE0Pfa/S/BjxUVfcO6S/ghiS3Jtn0eAdKsinJTJKZ2dnZnmVJkvbpG/QbgSsfp//FVXUycBZwcZJThg2sqi1VNV1V01NTAz+XR5I0hrGDPska4HeAq4aNqard3dc9wNXA+nHnkySNp88Z/UuAr1TVrkGdSQ5Jcui+beAMYHuP+SRJY1g06JNcCdwMHJdkV5ILu67zWHDZJskzk1zb7R4BfDbJHcDngWuq6pOTK12SNIpR7rrZOKT9ggFtDwAbuu37gBN71idJ6sl3xkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1btGgT7I1yZ4k2+e1vTnJ7iS3d48NQ557ZpJ7kuxIcskkC5ckjWaUM/orgDMHtL+7qk7qHtcu7ExyEPA+4CzgeGBjkuP7FCtJ2n+LBn1V3QQ8Msax1wM7quq+qvoB8DHg3DGOI0nqoc81+tclubO7tHP4gP6jgPvn7e/q2gZKsinJTJKZ2dnZHmVJkuYbN+jfD/wCcBLwIPDOvoVU1Zaqmq6q6ampqb6HkyR1xgr6qnqoqn5YVT8CPsDcZZqFdgPHzNs/umuTJC2jsYI+yZHzdn8b2D5g2BeAY5M8J8nBwHnAtnHmkySNb81iA5JcCZwKrE2yC7gUODXJSUABO4HXdGOfCXywqjZU1d4krwOuBw4CtlbVXUuxCEnScIsGfVVtHNB8+ZCxDwAb5u1fC/zErZeSpOXjO2MlqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4RYM+ydYke5Jsn9f2jiRfSXJnkquTHDbkuTuTfCnJ7UlmJli3JGlEo5zRXwGcuaDtRuCEqvoV4KvAXz/O80+rqpOqanq8EiVJfSwa9FV1E/DIgrYbqmpvt3sLcPQS1CZJmoBJXKP/Q+C6IX0F3JDk1iSbHu8gSTYlmUkyMzs7O4GyJEnQM+iTvBHYC3xkyJAXV9XJwFnAxUlOGXasqtpSVdNVNT01NdWnLEnSPGMHfZILgLOB36+qGjSmqnZ3X/cAVwPrx51PkjSesYI+yZnAXwHnVNVjQ8YckuTQfdvAGcD2QWMlSUtnlNsrrwRuBo5LsivJhcBlwKHAjd2tk5u7sc9Mcm331COAzya5A/g8cE1VfXJJViFJGmrNYgOqauOA5suHjH0A2NBt3wec2Ks6SVJvvjNWkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNGynok2xNsifJ9nltT0tyY5J7u6+HD3nu+d2Ye5OcP6nCJUmjGfWM/grgzAVtlwCfqqpjgU91+z8mydOAS4EXAOuBS4f9QpAkLY2Rgr6qbgIeWdB8LvChbvtDwG8NeOpvAjdW1SNV9S3gRn7yF4YkaQn1uUZ/RFU92G3/J3DEgDFHAffP29/Vtf2EJJuSzCSZmZ2d7VGWJGm+ibwYW1UFVM9jbKmq6aqanpqamkRZkiT6Bf1DSY4E6L7uGTBmN3DMvP2juzZJ0jLpE/TbgH130ZwP/MuAMdcDZyQ5vHsR9oyuTZK0TEa9vfJK4GbguCS7klwIvA14aZJ7gZd0+ySZTvJBgKp6BHgr8IXu8ZauTZK0TNaMMqiqNg7pOn3A2Bngj+btbwW2jlWdJKk33xkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1Lixgz7JcUlun/f4TpLXLxhzapJH5415U++KJUn7Zc24T6yqe4CTAJIcBOwGrh4w9DNVdfa480iS+pnUpZvTga9V1dcndDxJ0oRMKujPA64c0veiJHckuS7J84YdIMmmJDNJZmZnZydUliSpd9AnORg4B/j4gO7bgGdX1YnAe4FPDDtOVW2pqumqmp6amupbliSpM4kz+rOA26rqoYUdVfWdqvpet30t8OQkaycwpyRpRJMI+o0MuWyT5BlJ0m2v7+b75gTmlCSNaOy7bgCSHAK8FHjNvLbXAlTVZuDlwEVJ9gLfB86rquozpyRp//QK+qr6L+DpC9o2z9u+DLiszxySpH58Z6wkNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrXO+iT7EzypSS3J5kZ0J8k70myI8mdSU7uO6ckaXRrJnSc06rq4SF9ZwHHdo8XAO/vvkqSlsFyXLo5F/hwzbkFOCzJkcswrySJyQR9ATckuTXJpgH9RwH3z9vf1bX9mCSbkswkmZmdnZ1AWZIkmEzQv7iqTmbuEs3FSU4Z5yBVtaWqpqtqempqagJlSZJgAkFfVbu7r3uAq4H1C4bsBo6Zt3901yZJWga9gj7JIUkO3bcNnAFsXzBsG/Cq7u6bFwKPVtWDfeaVJI2u7103RwBXJ9l3rI9W1SeTvBagqjYD1wIbgB3AY8Cre84pSdoPvYK+qu4DThzQvnnedgEX95lHkjQ+3xkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGjR30SY5J8ukkdye5K8mfDRhzapJHk9zePd7Ur1xJ0v5a0+O5e4E3VNVtSQ4Fbk1yY1XdvWDcZ6rq7B7zSJJ6GPuMvqoerKrbuu3vAl8GjppUYZKkyZjINfok64DnA58b0P2iJHckuS7J8x7nGJuSzCSZmZ2dnURZkiQmEPRJngL8E/D6qvrOgu7bgGdX1YnAe4FPDDtOVW2pqumqmp6amupbliSp0+caPUmezFzIf6Sq/nlh//zgr6prk/xtkrVV9XCfeSXpiWrdJdeM/dydb3vZBCv5f33uuglwOfDlqnrXkDHP6MaRZH033zfHnVOStP/6nNH/KvAHwJeS3N61/Q3wLICq2gy8HLgoyV7g+8B5VVU95pQk7aexg76qPgtkkTGXAZeNO4ckqT/fGStJjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUuF5Bn+TMJPck2ZHkkgH9P5Xkqq7/c0nW9ZlPkrT/xg76JAcB7wPOAo4HNiY5fsGwC4FvVdUvAu8G3j7ufJKk8fQ5o18P7Kiq+6rqB8DHgHMXjDkX+FC3/Y/A6UnSY05J0n5a0+O5RwH3z9vfBbxg2Jiq2pvkUeDpwMMLD5ZkE7Cp2/1eknvGrGvtoOOPIqv3742x17yKHWhrPtDWCwfgmvP2Xmt+9rCOPkE/UVW1BdjS9zhJZqpqegIlrRquuX0H2nrBNU9Sn0s3u4Fj5u0f3bUNHJNkDfBU4Js95pQk7ac+Qf8F4Ngkz0lyMHAesG3BmG3A+d32y4F/rarqMackaT+Nfemmu+b+OuB64CBga1XdleQtwExVbQMuB/4+yQ7gEeZ+GSy13pd/ViHX3L4Dbb3gmicmnmBLUtt8Z6wkNc6gl6TGrdqgP9A+fmGE9f5FkruT3JnkU0mG3lO7Wiy25nnjfjdJJVn1t+KNsuYkr+i+13cl+ehy1zhpI/xsPyvJp5N8sfv53rASdU5Kkq1J9iTZPqQ/Sd7T/XvcmeTk3pNW1ap7MPfi79eAnwcOBu4Ajl8w5k+Azd32ecBVK133Eq/3NOBnuu2LVvN6R11zN+5Q4CbgFmB6petehu/zscAXgcO7/Z9b6bqXYc1bgIu67eOBnStdd881nwKcDGwf0r8BuA4I8ELgc33nXK1n9Afaxy8sut6q+nRVPdbt3sLc+xpWs1G+xwBvZe4zlP57OYtbIqOs+Y+B91XVtwCqas8y1zhpo6y5gJ/ttp8KPLCM9U1cVd3E3F2Iw5wLfLjm3AIcluTIPnOu1qAf9PELRw0bU1V7gX0fv7AajbLe+S5k7oxgNVt0zd2ftMdU1TXLWdgSGuX7/FzguUn+PcktSc5ctuqWxihrfjPwyiS7gGuBP12e0lbM/v5/X9QT5iMQNBlJXglMA7++0rUspSRPAt4FXLDCpSy3NcxdvjmVub/abkryy1X17ZUsaoltBK6oqncmeRFz7805oap+tNKFrRar9Yz+QPv4hVHWS5KXAG8Ezqmq/1mm2pbKYms+FDgB+LckO5m7lrltlb8gO8r3eRewrar+t6r+A/gqc8G/Wo2y5guBfwCoqpuBn2buA89aNdL/9/2xWoP+QPv4hUXXm+T5wN8xF/Kr/botLLLmqnq0qtZW1bqqWsfc6xLnVNXMypQ7EaP8XH+CubN5kqxl7lLOfctY46SNsuZvAKcDJPkl5oJ+dlmrXF7bgFd1d9+8EHi0qh7sc8BVeemmnrgfv7AkRlzvO4CnAB/vXnP+RlWds2JF9zTimpsy4pqvB85IcjfwQ+Avq2q1/qU66prfAHwgyZ8z98LsBav4pI0kVzL3y3pt97rDpcCTAapqM3OvQ2wAdgCPAa/uPecq/veSJI1gtV66kSSNyKCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9Jjfs/UC9wXFpua1cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "count, bins, ignored = plt.hist(y, 20, density=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Métrica - AUC\n",
    "\n",
    "Visualizando métricas simples, que independem do theshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8027915857902479"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix\n",
    "Pela matriz de confusão, obtemos os valores de TN, FN, TP, FP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6593, 2230],\n",
       "       [ 346,  831]], dtype=int64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold = 0.5\n",
    "y_pred_round = [1 if x >= threshold else 0 for x in y_pred]\n",
    "CM = confusion_matrix(y, y_pred_round)\n",
    "CM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.39216611609249646"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(y, y_pred_round)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "TN = CM[0][0] # 0 -> nao fiz acao, nao deu churn\n",
    "FN = CM[1][0] # 0 -> nao fiz acao, e cliente deu churn, mas ja daria mesmo sem o modelo de predicao (entao, em relacao ao caso de refenrencia, custo 0)\n",
    "TP = CM[1][1] # +++ -> gastei com a acao, mas conseguir reter o cliente (+LTV)\n",
    "FP = CM[0][1] # -- -> gastei com a acao, mas cliente nao daria churn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2714799085266253"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# de todos que eu falo que são positivos, acerto precision%\n",
    "precision = TP / (TP + FP)\n",
    "precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7060322854715378"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# de todos que de fato são positivos, consigo identificar recall%\n",
    "recall = TP / (TP + FN)\n",
    "recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Curva de Benefício\n",
    "\n",
    "Construa a Curva de benefício. \n",
    "- Qual o melhor threshold? \n",
    "- Esta ação vale a pena? \n",
    "- O que você acha que conteceria se o modelo perder performance? Custos da ação aumentarem ou o LTV do cliente diminuir? Se as classes deste problema forem mais ou menos desbalanceadas?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FPcost = -30\n",
    "TPcost = 170\n",
    "value_total_vec = []\n",
    "array_thresh = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0, 1.1]\n",
    "for threshold in array_thresh:\n",
    "    # define 1/0's\n",
    "    # create confusion matrix\n",
    "    # compute total value (based on costs defined above)\n",
    "    # save the results in value_total_vec"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}