"Machine Learning Resources"

Articles

• 50 years of Data Science

• Ten Simple Rules for Better Figures

• The Foundations of Algorithmic Bias

• Rules of Machine Learning: Best Practices for ML Engineering

• The Unreasonable Effectiveness of Recurrent Neural Networks

• Vanilla LSTM with numpy

• http://colah.github.io/

  • Understanding LSTM Networks

• An overview of gradient descent optimization algorithms

• Vanilla LSTM with numpy

• LSTM

• MIT Deep Learning Basics: Introduction and Overview with TensorFlow

• Deep Learning in a nutshell [part 1][part 2][part 3][part 4]

• Neural Machine Translation (seq2seq) Tutorial

• Non negative matrix factorization

• design patters in scikit learn

• Attention:

  • A Beginner's Guide to Attention Mechanisms and Memory Networks
  • Google AI Blog: Transformer: A Novel Neural Network Architecture for Language Understanding
  • Jay Alammar: The Illustrated Transformer
  • The Annotated Transformer - Python Code

• Recommender Systems

  • Collaborative Filtering
  • Content Based
  • Social Recommenders

• Bagging (Bootstrap Aggregating)

• Boosting Methods

  • Adaboost AKA Adaptive Boosting:

    • Types:
      • Discrete AdaBoost AKA AdaBoost.M1
      • AdaBoost.SAMME
      • AdaBoost.SAMME.R
      • AdaBoost.R2
    • Weights are adaptive. Incorrect predictions get larger and larger weights, and correct weights get smaller and smaller weights
    • Each classifier has an adaptive weight as well
    • Uses decision stumps, or decision trees of height 1 that are weak learners (accuracy of > 50%)
    • Steps:
      1. Initialize sample weights to be 1/N
      2. Fit m classifier h_m on training data weighted by w_i
         1. compute the classifier's error e_m = sum( w_i * (predicted - actual) ) / sum(w_i)
         2. compute the classifier's weight a_m = log( (1-e_m)/e_m )
         3. w_i <- w_i * exp(a_m * e_m)
      3. FINAL CLASSIFIER = H(x) = sign( sum(a_m x h_m(x) ) )
    • Caveats:
      • weak learner too complex -> overfitting
      • weak learner too weak -> underfitting
      • susceptible to noise
    • can be viewed as a coordinate based gradient descent algorithm
    • A Step by Step Adaboost Example: With numerical examples and python code

  • Gradient Boosting:

    • Regression and Classification
    • Optimizes a loss function (in contrast to AdaBoost)
    • ensemble of weak models
    • Next Model = Previous Model + learning_rate * some_multiplier * h(x) -- learning rate is to apply shrinkage
    • The goal is to make Next Model = Previous Model + h(x) = actual value, hence, h(x) is fit on the residual of (y-Previous Model)
    • Example Algorithms:
      • Gradient Tree Boosting:
        • Sequential built of trees (unlike random forests)
        • Week Trees built non randomly.
        • Prediction is fast and memory efficient
        • Learning rate parameter (RF don't have it.) Higher LR: more emphasis on correcting the previous tree (more complex) and vice versa
        • clf = GradientBoostingClassifier(learning_rate=0.01, max_depth=2).fit(X_train, y_train)
        • To avoid overfitting reduce max depth and learning rate and n_estimators
        • No feature scaling needed
        • Hard to interpret; training computationally expensive; tunning hard; bad for text classification and other problems with high dimensional sparse features

  • Fast scalable GBM:

    • XGBoost
    • LightGBM
    • CatBoost

  • Naive Bayes Classifiers

  • Random Forests

  • Manifold Learning

Books

• R Graphics Cookbook, 2nd Edition

• R for Data Science

• Deep Learning

Courses

• Google Machine Learning Crash Course
• 6.S191: Introduction to Deep Learning
• Geoffrey Hinton's Neural Networks for Machine Learning
• Data Visualization
• Modern Regression - CUM
  • Lecture13 Linear Regression

Repos

• Natural Language Processing Best Practices & Examples

• Pytorch Implementation

• lucid

  • Lucid is a collection of infrastructure and tools for research in neural network interpretability.

• oxford-cs-deepnlp-2017

Twitter

• Gabriel Goh

Datasets

• Quick Draw

Videos

• MIT 6.S094: Recurrent Neural Networks for Steering Through Time

• Arxiv Insights

Websites

• distill
• skymind A.I. Wiki
• Tensorflow Playground
• Shan Carter's website
• Kevin Quealy's website
• Pandas Trick by Kevin Markham

Other directories

• Awesome Deep Learning
• Data Science Learning Directory

Checklist

• How to Use a Machine Learning Checklist to Get Accurate Predictions, Reliably
• Machine Learning Project Review Checklist

Lecture Notes

• Regression