Experiments on Small Datasets with Deep Learning
This repository contains experiments that use neural networks to learn classifiers using small datasets.
The datasets are:
• Breast Cancer Wisconsin Data Set (reference to UCI ML Repository description)
• Diabetes Data Set (reference to UCI Machine Learning Repository description)
• Ionosphere Data Set (reference to UCI Machine Learning Repository description)
• Mushroom Data Set (reference to UCI Machine Learning Repository description)
For each dataset there are three groups of notebooks:
• Data Analysis performs basic data analysis, visualizations of the data sets, outlier detection and feature engineering.
• ML model search uses traditional sklearn Grid Search to select a traditional sklearn estimator (e.g. Linear Regression) and allows tuning hyper-parameters.
• ML classifier trains the final estimator and validates its performance using metrics and visualizations.
• DL model search uses traditional sklearn Grid Search to select a MLP implemented in pytorch and allows tuning hyper-parameters.
• DL classifier trains the final estimator and validates its performance using metrics and visualizations.
The notebooks implement the following data analysis, model selection and tuning flow for both traditional ML estimators and NNs:
Notes:
• All process steps have been configured for a 12 core CPU and a 12Mb GPU running OSX and need to be re-configured if executed on significantly different HW (especially the n_jobs parameter of sklearn functions).
• All functions make heavy use of python multi-processing and might hang is a sub-process fails. Use the verbose parameter to get log information from sklearn for debugging purposes.
• Sklearn GridSearchCV is used but RandomizedSearchCV and BayesSearchCV can be plugged in but did not perform in this experiment.
• A wrapper called Classifier is used to plug pytorch NNs into the sklearn functions. The code is in the lib directory. A supported version of similar functionality can be found here.
• The focus of the notebooks is to understand the behavior of traditional ML models versus neural networks if one uses small datasets (i.e. overtraining, overfitting, runtime, metrics). Only minimal feature engineering and individual tuning has been done.
• If code has been included form other places the URL of the source blog, repository, etc. has been included above the code.