CS_loss.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.losses import CostSensitiveLoss, CostSensitiveRegularizedLoss\n",
    "\n",
    "import numpy as np\n",
    "import torch, torch.nn as nn\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a simple idea behind a Cost-Sensitive Loss: we impose a different penalty to different kind of errors. For illustration purposes, imagine you have a three-class classification problem and you wish to impose a greater penalty to errors in misclassifying class 2 as class 0, but you don't care about the other errors. \n",
    "\n",
    "We first instantiate a CS loss function by just passing the number of classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = 3\n",
    "criterion = CostSensitiveLoss(n_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you want to add your constraints encoded as a penalty/confusion matrix, that you store inside the loss object. Note the heavy penalty on the \"it was class 2 but you predicted class 0\" kind of error, and the lack of penalty on any other kind of errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = np.array([[0,0,0],\n",
    "              [0,0,0],\n",
    "              [10,0,0]], dtype=float)\n",
    "M /= M.max()\n",
    "criterion.M = torch.from_numpy(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essentially what we do here is taking the scalar product between a one-hot encoded prediction and the l-th row of the penalty matrix, where l is the correct label associated to that prediction.\n",
    "\n",
    "For a particular example, let us imagine the associated label is 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "label = torch.tensor([2]).unsqueeze(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a terrible prediction (we use logits here, so this is **pre-softmax activations**) we obtain a relatively large loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.9999, dtype=torch.float64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits_terrible = torch.tensor([10.0, 0.0, 0.0]).unsqueeze(0)\n",
    "criterion(logits_terrible, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us look at better predictions. In this case, predicting either class 1 or class 2 is perfect, since we are not penalizing neither of those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4.5396e-05, dtype=torch.float64)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits_good = torch.tensor([0.0, 10.0, 0.0]).unsqueeze(0)\n",
    "criterion(logits_good, label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4.5396e-05, dtype=torch.float64)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits_good = torch.tensor([0.0, 00.0, 10.0]).unsqueeze(0)\n",
    "criterion(logits_good, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Default Penalty Matrix\n",
    "If you do not indicate anything, our Cost-Sensitive loss implementation will use a penalty that is suitable for image grading problems. In this context, we impose more penalty on labels that are \"further away\" from the true category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 0.5000, 1.0000],\n",
       "        [0.5000, 0.0000, 0.5000],\n",
       "        [1.0000, 0.5000, 0.0000]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs_criterion = CostSensitiveLoss(n_classes)\n",
    "cs_criterion.M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, if your example is of class `[0,0,1]` and your model predicts `[1,0,0]`, that should be more penalized than the (better) prediction `[0,1,0]`. Note that standard losses like cross-entropy will penalize the same both cases, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = torch.tensor([2])\n",
    "worse_logits = torch.tensor([10.0, 0.0, 0.0]).unsqueeze(0)\n",
    "better_logits = torch.tensor([0.0, 10.0, 0.0]).unsqueeze(0)\n",
    "perfect_logits = torch.tensor([0.0, 0.0, 10.0]).unsqueeze(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross Entropy:\n",
      "Worst prediction loss = 10.0001\n",
      "Better prediction loss = 10.0001\n",
      "Perfect prediction loss = 0.0001\n",
      "-----------------------------------\n",
      "Cost Sensitive loss:\n",
      "Worst prediction loss = 0.9999\n",
      "Better prediction loss = 0.5000\n",
      "Perfect prediction loss = 0.0001\n"
     ]
    }
   ],
   "source": [
    "print('Cross Entropy:')\n",
    "ce_criterion = torch.nn.CrossEntropyLoss()\n",
    "print('Worst prediction loss = {:.4f}'.format(ce_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(ce_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(ce_criterion(perfect_logits, labels)))\n",
    "print(35*'-')\n",
    "print('Cost Sensitive loss:')\n",
    "ce_criterion = torch.nn.CrossEntropyLoss()\n",
    "print('Worst prediction loss = {:.4f}'.format(cs_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(cs_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(cs_criterion(perfect_logits, labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cost-Sensitive Loss Regularization\n",
    "In our experiments (and elsewhere) we found out that simply using a CS loss leads to lots of trouble in terms of CNNs staying at local minima where they will predict a trivial configuration (all the time the same category), which seems to be very satisfying for this kind of losses. For this reason, we recommend using this as a regularizer for other standard classification losses (just a weighted sum of both, with a weighing parameter `lambd`). \n",
    "\n",
    "In our implementation we provide a wrapper for doing this, where you specify a `base_loss` and the regularization parameter `lambd`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = 3 \n",
    "base_loss = 'ce'\n",
    "lambd = 10\n",
    "cs_regularized_criterion = CostSensitiveRegularizedLoss(n_classes=n_classes, base_loss=base_loss, lambd=lambd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost Sensitive-Regularized CE loss:\n",
      "Worst prediction loss = 19.9993\n",
      "Better prediction loss = 12.5003\n",
      "Perfect prediction loss = 0.0007\n"
     ]
    }
   ],
   "source": [
    "print('Cost Sensitive-Regularized CE loss:')\n",
    "print('Worst prediction loss = {:.4f}'.format(cs_regularized_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(cs_regularized_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(cs_regularized_criterion(perfect_logits, labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also provide other base losses beyond cross-entropy: if you have `kornia` installed, you can use focal loss as the base loss as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# need to install kornia for using this\n",
    "# base_loss = 'focal_loss'\n",
    "# cs_regularized_criterion = CostSensitiveRegularizedLoss(n_classes=n_classes, base_loss=base_loss, lambd=lambd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost Sensitive-Regularized Focal loss:\n",
      "Worst prediction loss = 19.9993\n",
      "Better prediction loss = 12.5003\n",
      "Perfect prediction loss = 0.0007\n"
     ]
    }
   ],
   "source": [
    "print('Cost Sensitive-Regularized Focal loss:')\n",
    "print('Worst prediction loss = {:.4f}'.format(cs_regularized_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(cs_regularized_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(cs_regularized_criterion(perfect_logits, labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, you can use cross-entropy with Label Smoothing (LS):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_loss = 'ls'\n",
    "cs_regularized_criterion = CostSensitiveRegularizedLoss(n_classes=n_classes, base_loss=base_loss, lambd=lambd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost Sensitive-Regularized CE loss with Label Smoothing:\n",
      "Worst prediction loss = 19.6660\n",
      "Better prediction loss = 12.1670\n",
      "Perfect prediction loss = 0.6673\n"
     ]
    }
   ],
   "source": [
    "print('Cost Sensitive-Regularized CE loss with Label Smoothing:')\n",
    "print('Worst prediction loss = {:.4f}'.format(cs_regularized_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(cs_regularized_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(cs_regularized_criterion(perfect_logits, labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our paper we found that a variant of LS known as Gaussian Label Smoothing was particularly well suited for the grading problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_loss = 'gls'\n",
    "cs_regularized_criterion = CostSensitiveRegularizedLoss(n_classes=n_classes, base_loss=base_loss, lambd=lambd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost Sensitive-Regularized CE loss with Label Smoothing:\n",
      "Worst prediction loss = 11.0818\n",
      "Better prediction loss = 10.4818\n",
      "Perfect prediction loss = 9.0593\n"
     ]
    }
   ],
   "source": [
    "print('Cost Sensitive-Regularized CE loss with Label Smoothing:')\n",
    "print('Worst prediction loss = {:.4f}'.format(cs_regularized_criterion(worse_logits, labels)))\n",
    "print('Better prediction loss = {:.4f}'.format(cs_regularized_criterion(better_logits, labels)))\n",
    "print('Perfect prediction loss = {:.4f}'.format(cs_regularized_criterion(perfect_logits, labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gaussian Label Smoothing was introduced here:\n",
    "```\n",
    "Galdran A, Chelbi J, Kobi R, Dolz J, Lombaert H, ben Ayed I, Chakor H. Nonuniform label smoothing for diabetic  retinopathy grading from retinal fundus images with deep neural networks. \n",
    "Trans Vis Sci Tech. 2020;0(0):1933, https://doi.org/10.1167/tvst.0.0.1933\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Penalty Matrix: Modeling Inter-Observer Variability\n",
    "In our paper we used the above approach to better model the problem of Diabetic Retinopathy grading from retinal images (we have five categories here). In our case, we had an external source of data stating that ophthalmologists tend to mislabel examples differently depending on the category of the image. This is reflected in the following confusion matrix:\n",
    "\n",
    "\n",
    "$$\n",
    "M_{opht} = \\begin{bmatrix}\n",
    "1469 &  4 &   5 &  0 &  0\\\\\n",
    "58   & 62 &   5 &  0 &  0\\\\\n",
    "22   &  3 & 118 &  1 &  0\\\\\n",
    " 0   &  0 &  13 & 36 &  1\\\\\n",
    " 0   &  0 &   0 &  1 & 15\\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "\n",
    "which was extracted from:\n",
    "```\n",
    "Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G.S., Peng,\n",
    "L., Webster, D.R.: Grader Variability and the Importance of Reference Standards\n",
    "for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology\n",
    "125(8), 1264–1272 (Aug 2018)\n",
    "```\n",
    "\n",
    "After row-normalization, we can interpret the $(i,j)$ entry of this matrix as $P(D_j\\vert D_i) = $  the probability that an annotator diagnoses an image as grade $D_j$ when it actually was of grade $D_i$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.99391069, 0.00270636, 0.00338295, 0.        , 0.        ],\n",
       "       [0.464     , 0.496     , 0.04      , 0.        , 0.        ],\n",
       "       [0.15277778, 0.02083333, 0.81944444, 0.00694444, 0.        ],\n",
       "       [0.        , 0.        , 0.26      , 0.72      , 0.02      ],\n",
       "       [0.        , 0.        , 0.        , 0.0625    , 0.9375    ]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_opht = np.array([[1469,  4,   5,  0,  0],\n",
    "                    [58, 62,   5,  0,  0],\n",
    "                    [22,  3, 118,  1,  0],\n",
    "                    [0,   0,  13, 36,  1],\n",
    "                    [0,   0,   0,  1, 15]\n",
    "                    ], dtype=np.float)\n",
    "# M_oph = M_oph.T\n",
    "# Normalize M_oph and subtract from Id. to obtain M_difficulty:\n",
    "M_opht = np.divide(M_opht, np.sum(M_opht, axis=1)[:, None])\n",
    "M_opht"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this matrix we can draw some conclusiosn on how ophthalmologist could have labeled our dataset, such as:\n",
    "\n",
    "* Annotators are likely to be greatly accurate when grading $D_0$ and $D_4$ images, as derived from $t_{00}\\approx 1$ and $t_{44}\\approx 0.94$.\n",
    "* Around $50\\%$ of $D_1$ images are likely to be incorrectly labeled ($t_{11}\\approx0.5$). \n",
    "* Only $8\\%$ of the total of incorrectly labeled $D_1$ images are likely to be labeled as $D_2$.\n",
    "* Approximately $93\\%$ of those incorrectly labeled $D_1$ images are likely to be labeled as $D_0$.\n",
    "\n",
    "Under the hypothesis that in a dataset labeled by a single annotator the reliability of the annotations will follow a distribution similar to the above, we can assume, for instance, that such dataset will contain reliable labels concerning $D_0$ grades. However, we may also assume that when an image has been annotated as of grade $D_1$, this is quite likely to be incorrect, and it may well be the case that such image is actually of grade $D_0$, since the corresponding entry $t_{10}=P(D_0\\vert D_1)=0.464$ holds value comparable to $t_{11}=P(D_1\\vert D_1)=0.496$. Similarly, it is quite unlikely that the image is of grade $D_2$, as $t_{12}=P(D_2\\vert D_1)=0.04$, and there is no chance it is of grades $D_3$ or $D_4$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, how do we use this matrix to penalize more intelligently the predictions of our models? Well, since we think an image graded as $D_1$ could be of grade $D_0$ or $D_1$, but not $D_2$, $D_3$, or $D_4$, we want to insert lower values in $t_{10}$ and $t_{11}$, and higher values in $t_{12},t_{13},t_{14}$. We can accomplish that simply using $1-M_{opht}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00608931, 0.99729364, 0.99661705, 1.        , 1.        ],\n",
       "       [0.536     , 0.504     , 0.96      , 1.        , 1.        ],\n",
       "       [0.84722222, 0.97916667, 0.18055556, 0.99305556, 1.        ],\n",
       "       [1.        , 1.        , 0.74      , 0.28      , 0.98      ],\n",
       "       [1.        , 1.        , 1.        , 0.9375    , 0.0625    ]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1-M_opht"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note however that some entries in our matrix are non-informative (those with ones). For that reason, in our paper we use an averaged penalty matrix as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes, exp = 5, 2\n",
    "x = np.abs(np.arange(n_classes, dtype=np.float32))\n",
    "M = np.abs((x[:, np.newaxis] - x[np.newaxis, :])) ** exp\n",
    "\n",
    "# average M and 1-M_opht:\n",
    "M = 0.5 * M + 0.5 * (1-M_opht)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "vessels",
   "language": "python",
   "name": "build_central"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}