doc/orca_tutorial_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Naive approaches and decomposition methods in orca](#naive-approaches-and-decomposition-methods-in-orca)\n",
    "\t1. [Loading the dataset and performing some preliminary experiments](#loading-the-dataset-and-performing-some-preliminary-experiments)\n",
    "\t2. [Naive approaches](#naive-approaches)\n",
    "\t\t1. [Regression (SVR)](#regression-svr)\n",
    "\t\t2. [Nominal classification (SVC1V1 and SVC1VA)](#nominal-classification-svc1v1-and-svc1va)\n",
    "\t\t3. [Cost sensitive classification (CSSVC)](#cost-sensitive-classification-cssvc)\n",
    "\t\t4. [Summary of results for naive approaches](#summary-of-results-for-naive-approaches)\n",
    "\t3. [Binary decomposition methods](#binary-decomposition-methods)\n",
    "\t\t1. [SVM with ordered partitions (SVMOP)](#svm-with-ordered-partitions-svmop)\n",
    "\t\t2. [Neural network approaches (ELMOP and NNOP)](#neural-network-approaches-elmop-and-nnop)\n",
    "\t\t3. [Summary of results for binary decompositions](#summary-of-results-for-binary-decompositions)\n",
    "\t4. [Ternary decomposition](#ternary-decomposition)\n",
    "2. [References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive approaches and decomposition methods in orca\n",
    "\n",
    "This tutorial covers how to apply naive approaches and decomposition methods in ORCA. It is highly recommended to have previously completed the ['how to' tutorial (Notebook)](orca_tutorial_1.ipynb)/['how to' tutorial (HTML)](http://htmlpreview.github.io/?https://github.com/ayrna/orca/blob/master/doc/orca_tutorial_1.html).\n",
    "\n",
    "We are going to test these methods using a melanoma diagnosis dataset based on dermatoscopic images. Melanoma is a type of cancer that develops from the pigment-containing cells known as melanocytes. Usually occurring on the skin, early detection and diagnosis is strongly related to survival rates. The dataset is aimed at predicting the severity of the lesion:\n",
    "- A total of `100` image descriptors are used as input features, including features related to shape, colour, pigment network and texture.\n",
    "- The severity is assessed in terms of melanoma thickness, measured by the Breslow index. The problem is tackled as a five-class classification problem, where the first class represents benign lesions, and the remaining four classes represent the different stages of the melanoma (0, I, II and III, where III is the thickest one and the most dangerous).\n",
    "\n",
    "![Graphical representation of the Breslow index](tutorial/images/diagram-melanoma-stages.png)\n",
    "*Graphical representation of the Breslow index (source [1])*\n",
    "\n",
    "The dataset from [1] is included in this repository, in a specific [folder](/exampledata/10-fold/melanoma-5classes-abcd-100/matlab). The corresponding script for this tutorial, ([exampleMelanoma.m](../src/code-examples/exampleMelanoma.m)), can be found and run in the [code example](../src/code-examples) folder.\n",
    "\n",
    "## Loading the dataset and performing some preliminary experiments\n",
    "\n",
    "First, we will load the dataset and examine the label for some of the patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainMelanoma = load('../exampledata/10-fold/melanoma-5classes-abcd-100-fs/matlab/train_melanoma-5classes-abcd-100-fs.2');\n",
    "testMelanoma = load('../exampledata/10-fold/melanoma-5classes-abcd-100-fs/matlab/test_melanoma-5classes-abcd-100-fs.2');\n",
    "trainMelanoma([1:5 300:305],end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the data is prepared to perform a 10-fold experimental design, we are going to examine the properties of the whole set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "melanoma = [trainMelanoma; testMelanoma];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is quite imbalanced, as you can check with this code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = {'Non-melanoma', 'In Situ', 'I', 'II', 'III-IV'};\n",
    "nn = hist(melanoma(:,end),5);\n",
    "bar(nn)\n",
    "set (gca, 'xticklabel',labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "***Exercise 1***: obtain the average imbalanced ratio for this dataset, where the imbalanced ratio of each class is the sum of the number of patterns of the rest of classes divided by the number of classes times the number of patterns of the class.\n",
    "\n",
    "---\n",
    "We can apply a simple method, [POM](../src/Algorithms/POM.m) [2], to check the accuracy (CCR) and MAE obtained for this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.patterns = trainMelanoma(:,1:(end-1));\n",
    "train.targets = trainMelanoma(:,end);\n",
    "test.patterns = testMelanoma(:,1:(end-1));\n",
    "test.targets = testMelanoma(:,end);\n",
    "addpath('../src/Algorithms/');\n",
    "algorithmObj = POM();\n",
    "info = algorithmObj.fitpredict(train,test);\n",
    "addpath('../src/Measures/');\n",
    "CCR.calculateMetric(info.predictedTest,test.targets)\n",
    "MAE.calculateMetric(info.predictedTest,test.targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following code, we try to improve the results by considering standardization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addpath('../src/Utils/');\n",
    "[trainStandarized,testStandarized] = DataSet.standarizeData(train,test);\n",
    "disp('Some patterns before standarizing')\n",
    "train.patterns(1:10,2:5)\n",
    "disp('The same patterns after standarizing')\n",
    "trainStandarized.patterns(1:10,2:5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info = algorithmObj.fitpredict(trainStandarized,testStandarized);\n",
    "CCR.calculateMetric(info.predictedTest,test.targets)\n",
    "MAE.calculateMetric(info.predictedTest,test.targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results have not improved in this specific case. The static method `DataSet.standarizeData(train,test)` transforms the training and test datasets and returns a copy where all the input variables have zero mean and unit standard deviation. There are other pre-processing methods in the `DataSet` class which delete constant input attributes or non numeric attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[train,test] = DataSet.deleteConstantAtributes(train,test);\n",
    "[train,test] = DataSet.standarizeData(train,test);\n",
    "[train,test] = DataSet.deleteNonNumericValues(train,test);\n",
    "info = algorithmObj.fitpredict(train,test);\n",
    "CCR.calculateMetric(info.predictedTest,test.targets)\n",
    "MAE.calculateMetric(info.predictedTest,test.targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the results have not changed, as there were no attributes with these characteristics. However, in general, it is a good idea to apply standardisation of the input variables.\n",
    "\n",
    "---\n",
    "\n",
    "***Exercise 2***: construct a function (`preprocess.m`) applying these three pre-processing steps (standarisation, removal of constant features and removal of non numeric values) for future uses.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive approaches\n",
    "\n",
    "The first thing we will do is applying standard approaches for this ordinal regression dataset. This includes applying regression, classification and cost-sensitive classification. In this section we will use kernel methods, that are more suitable for high-dimensional data, so that we can use melanoma dataset with the full set of 100 features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainMelanoma = load('../exampledata/10-fold/melanoma-5classes-abcd-100/matlab/train_melanoma-5classes-abcd-100.2');\n",
    "testMelanoma = load('../exampledata/10-fold/melanoma-5classes-abcd-100/matlab/test_melanoma-5classes-abcd-100.2');\n",
    "train.patterns = trainMelanoma(:,1:(end-1));\n",
    "train.targets = trainMelanoma(:,end);\n",
    "test.patterns = testMelanoma(:,1:(end-1));\n",
    "test.targets = testMelanoma(:,end);\n",
    "[train,test] = DataSet.deleteConstantAtributes(train,test);\n",
    "[train,test] = DataSet.standarizeData(train,test);\n",
    "[train,test] = DataSet.deleteNonNumericValues(train,test);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Regression (SVR)\n",
    "\n",
    "A very simple way of solving an ordinal classification problem is to apply regression. This is, we train a regressor to predict the number of the category (where categories are coded with real consecutive values, `1`, `2`, ..., `Q`, which are scaled between 0 and 1, `0/(Q-1)=0`, `1/(Q-1)`, ..., `(Q-1)/(Q-1)`). Then, to predict categories, we round the real values predicted by the regressor to the nearest integer.\n",
    "\n",
    "ORCA includes one algorithm following this approach based on support vector machines: [Support Vector Regression (SVR)](../src/Algorithms/SVR.m). Note that SVR considers the epsilon-SVR model with an RBF kernel, involving three different parameters:\n",
    "- Parameter `C`, importance given to errors.\n",
    "- Parameter `k`, inverse of the width of the RBF kernel.\n",
    "- Parameter `e`, epsilon. It specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.\n",
    "\n",
    "We can check the performance of this model in the melanoma dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = SVR();\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001,'e',0.01));\n",
    "fprintf('\\nSupport Vector Regression\\n---------------\\n');\n",
    "fprintf('SVR Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('SVR MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object info also contains the projection values, which, in this case, are the real values without being rounded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, poor performance is obtained. We can try different parameter values by using a `for` loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fprintf('\\nSupport Vector Regression parameters\\n---------------\\n');\n",
    "bestAccuracy=0;\n",
    "for C=10.^(-3:1:3)\n",
    "   for k=10.^(-3:1:3)\n",
    "       for e=10.^(-3:1:3)\n",
    "           param = struct('C',C,'k',k,'e',e);\n",
    "           info = algorithmObj.fitpredict(train,test,param);\n",
    "           accuracy = CCR.calculateMetric(test.targets,info.predictedTest);\n",
    "           if accuracy > bestAccuracy\n",
    "               bestAccuracy = accuracy;\n",
    "               bestParam = param;\n",
    "           end\n",
    "           fprintf('SVR C %f, k %f, e %f --> Accuracy: %f, MAE: %f\\n' ...\n",
    "               , C, k, e, accuracy, MAE.calculateMetric(test.targets,info.predictedTest));\n",
    "       end\n",
    "   end\n",
    "end\n",
    "fprintf('Best Results SVR C %f, k %f, e %f --> Accuracy: %f\\n', bestParam.C, bestParam.k, bestParam.e, bestAccuracy);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can check, the best configuration leads to almost a 70% of accuracy, which is not very bad considering that we have 5 classes.\n",
    "\n",
    "This way of adjusting the parameters is not fair, as we can be overfitting the test set. The decision of the optimal parameters should be taken without checking test results. This can be done by using nested crossvalidation.\n",
    "\n",
    "---\n",
    "\n",
    "***Exercise 3*** : complete the code of the script ([crossvalide.m](tutorial/scripts/crossvalide.m)) for automatising hyper-parameter selection in this problem. The idea is to have a function like this:\n",
    "``` MATLAB\n",
    ">> param = crossvalide(algorithmObj,train,5);\n",
    ">> param\n",
    "\n",
    "param =\n",
    "\n",
    "    C: 0.0100\n",
    "    k: 0.0100\n",
    "    e: 0.0100\n",
    ">> fprintf('\\nSupport Vector Regression with cross validated parameters\\n---------------\\n');\n",
    "fprintf('SVR Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('SVR MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));\n",
    "\n",
    "Support Vector Regression with cross validated parameters\n",
    "---------------\n",
    "SVR Accuracy: 0.589286\n",
    "SVR MAE: 0.732143\n",
    "```\n",
    "Although the results are worse, we can be sure that here there is no overfitting.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, this can be easily done in ORCA by using the `ini` files with the correct format. [svrMelanoma.ini](tutorial/config-files/svrMelanoma.ini) is a configuration file with the following contents:\n",
    "```ini\n",
    ";SVR experiments for melanoma\n",
    ";\n",
    "; Experiment ID\n",
    "[svr-mae-tutorial-melanoma]\n",
    "{general-conf}\n",
    "seed = 1\n",
    "; Datasets path\n",
    "basedir = ../exampledata/10-fold\n",
    "; Datasets to process (comma separated list or all to process all)\n",
    "datasets = melanoma-5classes-abcd-100\n",
    "; Activate data standardization\n",
    "standarize = true\n",
    "; Number of folds for the parameters optimization\n",
    "num_folds = 5\n",
    "; Crossvalidation metric\n",
    "cvmetric = ccr\n",
    "\n",
    "; Method: algorithm and parameter\n",
    "{algorithm-parameters}\n",
    "algorithm = SVR\n",
    ";kernelType = rbf\n",
    "\n",
    "; Method's hyper-parameter values to optimize\n",
    "{algorithm-hyper-parameters-to-cv}\n",
    "C = 10.^(-2:1:2)\n",
    "k = 10.^(-2:1:2)\n",
    "e = 10.^(-3:1:0)\n",
    "```\n",
    "In this way, we will obtain the results for the 10 partitions. This `ini` file can be run by using the following code (to be run from the `doc` folder):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Utilities.runExperiments('tutorial/config-files/svrMelanomafs.ini')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the number of experiments is high, so the execution can take a considerable amount of time. To accelerate the experiments you can use multiple cores of your CPU with `Utilities.runExperiments('experiments.ini','parallel', true)` (see this [page](orca_parallel.md))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nominal classification (SVC1V1 and SVC1VA)\n",
    "\n",
    "We can also approach ordinal classification by considering nominal classification, i.e. by ignoring the ordering information. It has been shown that this can make the classifier need more data to learn.\n",
    "\n",
    "ORCA includes two approaches to perform ordinal classification by nominal classification, both based on the Support Vector Classifier:\n",
    "- [One-Vs-One (SVC1V1)](../src/Algorithms/SVC1V1.m) [3], where all pairs of classes are compared in different binary SVCs. The prediction is based on majority voting.\n",
    "- [One-Vs-All (SVC1VA)](../src/Algorithms/SVC1VA.m) [3], where each class is compared against the rest. The class predicted is that with the largest decision function value.\n",
    "\n",
    "Both methods consider an RBF kernel with the following two parameters:\n",
    "- Parameter `C`, importance given to errors.\n",
    "- Parameter `k`, inverse of the width of the RBF kernel.\n",
    "\n",
    "Now, we run the SVC1V1 method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = SVC1V1();\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001));\n",
    "fprintf('\\nSVC1V1\\n---------------\\n');\n",
    "fprintf('SVC1V1 Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('SVC1V1 MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In SVC1V1, the decision values have `(Q(Q-1))/2` (the number of combinations of two classes from the set of `Q` possibilities) columns and majority voting is applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also try SVC1VA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = SVC1VA();\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001));\n",
    "fprintf('\\nSVC1VA\\n---------------\\n');\n",
    "fprintf('SVC1VA Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('SVC1VA MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Five decision values are obtained for each pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, SVC1V1 obtains better results.\n",
    "\n",
    "### Cost sensitive classification (CSSVC)\n",
    "\n",
    "This is a special case of approaching ordinal classification by nominal classifiers. We can include different misclassification costs in the optimization function, in order to penalize more those mistakes which involve several categories in the ordinal scale. ORCA implements this method using again SVC and specifically the SVC1VA alternative. The costs are included as weights in the patterns, in such a way that, when generating the `Q` binary problems, the patterns of the negative class are given a weight according to the absolute difference (in number of categories) between the positive class and the specific negative class.\n",
    "\n",
    "The method is called [Cost Sensitive SVC (CSSVC)](../src/Algorithms/CSSVC.m) [3] and considers an RBF kernel with the following two parameters:\n",
    "- Parameter `C`, importance given to errors.\n",
    "- Parameter `k`, inverse of the width of the RBF kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = CSSVC();\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001));\n",
    "fprintf('\\nCSSVC\\n---------------\\n');\n",
    "fprintf('CSSVC Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('CSSVC MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the structure of decision values is the same than for SVC1VA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of results for naive approaches\n",
    "\n",
    "We can compare all the results obtained by naive methods in the third partition of the melanoma dataset:\n",
    "- SVR Accuracy: 0.678571\n",
    "- SVC1V1 Accuracy: 0.678571\n",
    "- SVC1VA Accuracy: 0.660714\n",
    "- CSSVC Accuracy: 0.660714\n",
    "- SVR MAE: 0.392857\n",
    "- SVC1V1 MAE: 0.517857\n",
    "- SVC1VA MAE: 0.535714\n",
    "- CSSVC MAE: 0.571429\n",
    "\n",
    "In this case, SVR has definitely obtained the best results. As can be checked, SVC1V1 accuracy is quite high, but it is masking a not so good MAE value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary decomposition methods\n",
    "\n",
    "These methods decompose the original problem in several binary problems (as SVC1V1 and SVC1VA do) but they binary subproblems are organised in such a way that the ordinal structure of the targets is maintained. Specifically, patterns of two non-consecutive categories in the ordinal scale will never be included in the same class against a pattern of an intermediate category. ORCA includes three methods with this structure:\n",
    "- One based on SVMs. Because of the way SVM is formulated, the binary subproblems are trained with **multiple models**.\n",
    "- Two based on neural networks. The flexibility of NN training makes possible learn all binary subproblems with one **single model**.\n",
    "All of them are based on an ordered partition decomposition, where the binary subproblems have the following structure:\n",
    "\n",
    "| Class | Problem1 | Problem2 | Problem3 | Problem4 |\n",
    "| --- | --- | --- | --- | --- |\n",
    "| C1 | 0 | 0 | 0 | 0 |\n",
    "| C2 | 1 | 0 | 0 | 0 |\n",
    "| C3 | 1 | 1 | 0 | 0 |\n",
    "| C4 | 1 | 1 | 1 | 0 |\n",
    "| C5 | 1 | 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM with ordered partitions (SVMOP)\n",
    "\n",
    "[SVMOP](../src/Algorithms/SVMOP) method is based on applying the ordered partition binary decomposition, together different weights according to the absolute distance between the class of the binary problem and the specific category being examined [4,5]. The models are trained independently and final prediction is based on the first model (in the ordinal scale) predicting a positive class. Again, the parameters of this model are:\n",
    "- Parameter `C`, importance given to errors.\n",
    "- Parameter `k`, inverse of the width of the RBF kernel.\n",
    "\n",
    "The same parameter values are considered for all subproblems, although the results could be improved by considering different `C` and `k` for each subproblem (resulting in a significantly higher computational cost). Here, we can check the performance of SVMOP on the partition of melanoma we have been studying:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = SVMOP();\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001));\n",
    "fprintf('\\nSVMOP\\n---------------\\n');\n",
    "fprintf('SVMOP Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('SVMOP MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, decision values include the independent values obtained for all subproblems:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural network approaches (ELMOP and NNOP)\n",
    "\n",
    "Neural networks allow solving all the binary subproblems using a single model with several output nodes. Two neural network models are considered in ORCA:\n",
    "- [Extreme learning machines with ordered partitions (ELMOP)](../src/Algorithms/ELMOP.m) [6].\n",
    "- [Neural network with ordered partitions (NNOP)](../src/Algorithms/NNOP.m) [7].\n",
    "\n",
    "ELMOP model is based on ELM, which are a quite popular type of neural network. In ELMs, the hidden weights are randomly set and the output weights are analytically set. The implementation in ORCA consider the ordered partition decomposition in the output layer. The prediction phase is tackled using an exponential loss based decoding process, where the class predicted is that with the minimum exponential loss with respect to the decision values.\n",
    "\n",
    "The algorithm can be configured using different activation functions for the hidden layer ('sig, 'sin', 'hardlim','tribas', 'radbas', 'up','rbf'/'krbf' or 'grbf'). During training, the user has to specify the following parameter in the `param` structure:\n",
    "- Parameter `hiddenN`: number of hidden nodes of the model. This is a decisive parameter for avoiding overfitting.\n",
    "\n",
    "Now, we perform a test for training ELMOP (note that ELMOP is not deterministic, this is, the results may vary among different runs of the algorithm):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = ELMOP('activationFunction','sig');\n",
    "info = algorithmObj.fitpredict(train,test,struct('hiddenN',20));\n",
    "fprintf('\\nELMOP\\n---------------\\n');\n",
    "fprintf('ELMOP Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('ELMOP MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the decision values for ELMOP:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "***Exercise 4***: compare all different activation functions for ELM trying to find the most appropriate one. Check the source code of [ELMOP](../src/Algorithms/ELMOP.m) to understand the different functions.\n",
    "\n",
    "---\n",
    "\n",
    "The other neural network model is NNOP. In this case, a standard neural network is considered, training all its parameters (hidden and output weights). In the output layer, a standard sigmoidal function is used, and the mean squared error with respect to the ordered partition targets is used for gradient descent. The algorithm used for gradient descent is the iRProp+ algorithm.\n",
    "\n",
    "The prediction rule is based on checking which is the first class whose output value is higher than a predefined threshold (0.5 in this case).\n",
    "\n",
    "Three parameters have to be specified in this case:\n",
    "- Parameter `hiddenN`, number of hidden neurons of the model.\n",
    "- Parameter `iter`, number of iterations for gradient descent.\n",
    "- Parameter `lambda`, regularization parameter in the error function (L2 regularizer), in order to avoid overfitting.\n",
    "\n",
    "This is an example of execution of NNOP (note that results may vary among different runs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = NNOP();\n",
    "info = algorithmObj.fitpredict(train,test,struct('hiddenN',20,'iter',500,'lambda',0.1));\n",
    "fprintf('\\nNNOP\\n---------------\\n');\n",
    "fprintf('NNOP Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('NNOP MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the decision values are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of results for binary decompositions\n",
    "\n",
    "As a summary, the results obtained for the third partition of melanoma dataset are:\n",
    "- SVMOP Accuracy: 0.678571\n",
    "- ELMOP Accuracy: 0.535714\n",
    "- NNOP Accuracy: 0.642857\n",
    "- SVMOP MAE: 0.517857\n",
    "- ELMOP MAE: 0.678571\n",
    "- NNOP MAE: 0.571429\n",
    "\n",
    "In this case, the best classifier is SVMOP, although parameter values can be influencing these results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ternary decomposition\n",
    "\n",
    "The last method considered in the tutorial is a projection similar to One-Vs-All but generating three class subproblems, instead of binary ones [8]. The subproblems are solved considering independent classifiers, and the prediction phase is performed under a probabilistic approach (which firstly obtain a `Q` class probability distribution for each ternary classifier and then fuse all the distributions).\n",
    "\n",
    "The base algorithm used can be configured by the user in the constructor, but it is necessary to use a one-dimensional projection method (threshold model). The parameters of OPBE are the same than the base algorithm, all subproblems being solved using the same parameter values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithmObj = OPBE('base_algorithm','SVORIM');\n",
    "info = algorithmObj.fitpredict(train,test,struct('C',10,'k',0.001));\n",
    "fprintf('\\nOPBE\\n---------------\\n');\n",
    "fprintf('OPBE Accuracy: %f\\n', CCR.calculateMetric(test.targets,info.predictedTest));\n",
    "fprintf('OPBE MAE: %f\\n', MAE.calculateMetric(test.targets,info.predictedTest));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the decision values only include the maximum probability after considering the weights given for each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info.projectedTest(1:10,:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "***Exercise 5***: in this tutorial, we have considered a total of 8 classifiers with different parameter values for one of the folds of the melanoma dataset. In this exercise, you should generalise these results over the `10` partitions and interpret the results, trying to search for the best method. Apart from the two metrics considered in the tutorial (CCR and MAE), include metrics more sensitive to minority classes (for example, MS and MMAE). Construct a table with the average of these four metrics over the 10 folds. You can use the parameter values given in this tutorial or try to tune a bit them.\n",
    "\n",
    "---\n",
    "\n",
    "***Exercise 6***: now you should consider cross-validation to tune hyper parameters. In order to limit the computational time, do not include too many values for each parameter and only use the three first partitions of the dataset (by deleting or moving the files for the rest of partitions). Check again the conclusions about the methods. **Hyper parameters are decisive for performance!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# References\n",
    "\n",
    "1. J. Sánchez-Monedero, M. Pérez-Ortiz, A. Sáez, P.A. Gutiérrez, and C. Hervás-Martínez. \"Partial order label decomposition approaches for melanoma diagnosis\". Applied Soft Computing. Volume 64, March 2018, Pages 341-355. https://doi.org/10.1016/j.asoc.2017.11.042\n",
    "1. P. McCullagh, \"Regression models for ordinal data\",  Journal of the Royal Statistical Society. Series B (Methodological), vol. 42, no. 2, pp. 109–142, 1980.\n",
    "1. C.-W. Hsu and C.-J. Lin. \"A comparison of methods for multi-class support vector machines\", IEEE Transaction on Neural Networks,vol. 13, no. 2, pp. 415–425, 2002. https://doi.org/10.1109/72.991427\n",
    "1. E. Frank and M. Hall, \"A simple approach to ordinal classification\", in Proceedings of the 12th European Conference on Machine Learning, ser. EMCL'01. London, UK: Springer-Verlag, 2001, pp. 145–156. https://doi.org/10.1007/3-540-44795-4_13\n",
    "1. W. Waegeman and L. Boullart, \"An ensemble of weighted support vector machines for ordinal regression\", International Journal of Computer Systems Science and Engineering, vol. 3, no. 1, pp. 47–51, 2009.\n",
    "1. W.-Y. Deng, Q.-H. Zheng, S. Lian, L. Chen, and X. Wang, \"Ordinal extreme learning machine\", Neurocomputing, vol. 74, no. 1-3, pp. 447-456, 2010.         http://dx.doi.org/10.1016/j.neucom.2010.08.022\n",
    "1. J. Cheng, Z. Wang, and G. Pollastri, \"A neural network  approach to ordinal regression,\" in Proc. IEEE Int. Joint Conf. Neural Netw. (IEEE World Congr. Comput. Intell.), 2008, pp. 1279-1284.\n",
    "1. M. Pérez-Ortiz, P. A. Gutiérrez y C. Hervás-Martínez. \"Projection based ensemble learning for ordinal regression\", IEEE Transactions on Cybernetics, Vol. 44, May, 2014, pp. 681-694."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Octave",
   "language": "octave",
   "name": "octave"
  },
  "language_info": {
   "file_extension": ".m",
   "help_links": [
    {
     "text": "GNU Octave",
     "url": "https://www.gnu.org/software/octave/support.html"
    },
    {
     "text": "Octave Kernel",
     "url": "https://github.com/Calysto/octave_kernel"
    },
    {
     "text": "MetaKernel Magics",
     "url": "https://metakernel.readthedocs.io/en/latest/source/README.html"
    }
   ],
   "mimetype": "text/x-octave",
   "name": "octave",
   "version": "4.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}