AnomalyThresholds

This is the code for the paper "Empirical Thresholding on Spatio-temporal Autoencoders Trained on Surveillance Videos in a Dementia Care Unit". The paper has been accepted at 20th Conference on Robots and Vision. The code gives different thresholds to choose from in anomaly detection problems in the absence of a validation set.

Data

Due to ethical considerations, the data used in the paper cannot be made publicly available. Here, we have provided dummy reconstruction error files in the data folder for the purpose of running the code.

Creating Validation Set from Normal Data

We first create two sets from the reconstruction error of full training samples (N) – Inliers (I) and Outliers (O), using IQR analysis with Ω = 1.5. In the absence of samples from anomalous class, O can serve as a proxy for unseen anomalous events. Then, I and O are further divided with 90%-10% split into: training (I_t, O_t) and validation (I_v, O_v) sets (see above figure).

Process flow

We train an anomaly detection model (a 3D Convolutional Autoencoder in paper) on the full normal data (N), and obtain reconstruction error on all the training samples. Then, we can apply the following thresholding methods:

• MaxRE: maximum value of reconstruction error on all the training samples is considered as a threshold.
• StdRE: The threshold is set as 3 standard deviations (σ) away from the mean of the reconstruction error.
• RedMaxRE: To tighten the above thresholds, we perform IQR analysis on the reconstruction error of the training samples, and extreme reconstruction values are removed based on a rejection rate, ω. The maximum of the remaining reconstruction error values is considered as a threshold. Please refer to the paper for more details.

The above threshold mechanisms don't need a proxy validation set. However, the below threshold approaches need a proxy validation set and were observed to perform better than the previous approaches. For the below approaches, first we train an anomaly detection model on I_t, and obtain reconstruction error on all the samples in I_t.

• OptRE: We choose each reconstruction error as a threshold and find the corresponding gmean using I_v and O_v. The reconstruction error with the highest gmean is chosen as the threshold.
• OptROC: We make a receiver operating characteristic curve using I_v and O_v. This will give various true positive rates and false positive rates along with corresponding thresholds on the proxy validation set (by using "roc curve" function). The corresponding threshold value that gives the highest gmean is chosen as the threshold.
• OptPR: We make a precision-recall curve using I_v and O_v. This will give various precision and recall values along with corresponding thresholds on the proxy validation set (by using "precision recall curve" function). The corresponding threshold value that gives the highest gmean is chosen as the threshold.

During testing, any sample with a reconstruction error greater than the threshold will be considered an anomalous event.

Usage of scripts

Below is the description of different scripts:

• setSplits.py: Returns indexes for I_t, I_v, O_t, O_v sets.
  
• thresholds.py: Calculates threshold using one of many different thresholding approaches of choice.
  
• sample.py: Contains the sample code on how to use setSplits.py and thresholds.py scripts.