Machine- and Deep Learning resources

Machine and deep learning and data analysis resources. Please, contribute and get in touch! See MDmisc notes for other programming and genomics-related notes.

Table of content

• Cheatsheets
• Machine learning
  • ML Books
  • ML Courses & Tutorials
  • ML Videos
  • ML Papers
  • ML Tools
  • ML Misc
• Deep Learning
  • Keras, Tensorflow
  • PyTorch
  • Graph Neural Networks
  • DL Books
  • DL Courses & Tutorials
  • DL Videos
  • DL Papers
    • DL Papers Genomics
  • DL Tools
    • Auto ML
    • DL models
    • DL projects
  • DL Misc)
• JAX

Cheatsheets

• Artificial-intelligence - Awesome AI Learning with +100 AI Cheat-Sheets, Free online Books, Top Courses, Best Videos and Lectures, Papers, Tutorials, +99 Researchers, Premium Websites, +121 Datasets, Conferences, Frameworks, Tools

• Over 200 of the Best Machine Learning, NLP, and Python Tutorials — 2018 Edition, Source

• 101 Machine Learning Algorithms for Data Science with Cheat Sheets - Brief description and R/Python examples of algorithms, categorized into several categories: classification, regression, neural networks, anomaly detection, dimensionality reduction, ensemble learning, clusterint, association rule analysis, regularization

• Machine Learning Cheatsheet - Brief visual explanations of machine learning concepts with diagrams, code examples and links to resources for learning more.

• Data Science Cheatsheet - Data Science and ML Cheat Sheet, by Maverick Lin. Source

• Data Science Cheatsheet - A helpful 4-page data science cheatsheet to assist with exam reviews, interview prep, and anything in-between

• cheatsheets-ai - Essential Cheat Sheets for deep learning and machine learning researchers

• machine-learning-cheat-sheet - 30-page MachineLearning cheat sheet with classical equations & diagrams, Tweet by Kirk Borne

• ml_cheatsheet - A 5-pages only Machine Learning cheatsheet focusing on the most popular algorithms under the hood. Online version

• stanford-cs-229-machine-learning - VIP cheatsheets for Stanford's CS 229 Machine Learning. Online version

• Machine Learning 101 - Machine and deep learning overview in 100 slides, or 35-min video by Jason Mayers. Tweet

• Mathematics-for-ML - A collection of resources to learn mathematics for machine learning. Linka to books, videos.

• Here are 450 Ivy League courses you can take online right now for free blog post by Dhawal Shah with links to free courses in Computer Science, Data Science, Programming, Humanities, Business, Art & Design, Science, Social Sciences, Health & Medicine, Engineering, Mathematics, Education & Teaching, and Personal Development

• TOP 10 GitHub Repositories for Data Science by Analysics Vidhya (Ayushi Gupta)

• machine-learning-resource - Machine- and deep learning notes by Ming Tang

Machine learning

• awesome-machine-learning - A curated list of awesome Machine Learning frameworks, libraries and software

• awesome-machine-learning-interpretability - A curated list of awesome machine learning interpretability resources

• awesome-machine-learning-operations - A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

• awesome-courses - List of awesome university courses for learning Computer Science

• Best-of Machine Learning with Python - A ranked list of awesome machine learning Python libraries. Updated weekly.

• data-science - "Path to a free self-taught education in Data Science!" - Open Source Society University, a collection of free online courses in logical order of learning data science. Massive list of courses, from linear algebra and calculus to R/Python programming/machine learning

• Data-science-best-resources - Carefully curated resource links for data science in one place

• free-data-science - Thematic list of high-quality data science resources. R, Python, Shell, Regular Expressions, Git, Docker, Markdown/Latex, Statistics, Machine/Deep learning, Visualization, Time Series, Spatial Analysis, more.

• machine_learning - Machine learning in R notes by Dave Tang

• machine-learning-interview - Machine Learning Interviews from FAANG, Snapchat, LinkedIn. More info at mlengineer.io

• machine-learning-notes - Collection of useful machine learning codes and snippets, Jupyter notebooks, by Sebastian Raschka.

ML Books

• Probabilistic Machine Learning: An Introduction by Kevin Patrick Murphy, 2022 edition. Intro (probability, statistics, decision theory, information theory, linear algebra), linear models, nonparametric modeling, deep neural networks, dimensionality reduction, clustering, more. GitHub, Tweet1, Tweet2

• Mathematics for Machine Learning, 2020 by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong.

• Linear_Algebra_With_Python - Lecture Notes for Linear Algebra Featuring Python. These lecture notes are intended for introductory linear algebra courses, suitable for university students, programmers, data analysts, algorithmic traders and etc.

• Mathematics for Machine Learning by Garrett Thomas. Tweet

• A Machine Learning Primer by Mihail Eric @mihail_eric. Tweet

• 100+ Free Data Science Books

• ciml - book "A Course in Machine Learning". Online version

• Interpretable Machine Learning book by Christoph Molnar, A Guide for Making Black Box Models Explainable. LearnPub

• Introduction to Machine Learning book by Nils Nilsson, free PDF

• hands-on-machine-learning-with-r - Hands-on Machine Learning with R: An applied book covering the fundamentals of machine learning with R. Supplementary material, Online version

• mit-deep-learning-book-pdf - MIT Deep Learning Book, PDF of the original http://www.deeplearningbook.org/ book.

• ML_for_Hackers - Code accompanying the book "Machine Learning for Hackers"

• rtemis - Advanced Machine Learning and Visualization in R. Book

• Feature Engineering and Selection: A Practical Approach for Predictive Models by Kuhn and Johnson, GitHub

ML Courses & Tutorials

• Machine Learning 2021 - a seven-module course covering basics of machine learning, by Bioinformatics.ca. Youtube playlist, Course material on Google Drive

• Full Stack Deep Learning - from development to deployment of machine learning methods

• 40+ Modern Tutorials Covering All Aspects of Machine Learning, Tweet

• 100-Days-Of-ML-Code - 100 Days of Machine Learning Coding as proposed by Siraj Raval. Illustrated step-by-step guides with code and data. Links to videos.

• Code for Workshop: Introduction to Machine Learning with R by Shirin Glander. More in her blog posts, twitter etc.

• aml-london-2019 - Course materials for Applied Machine Learning course in 2019 in London, by Max Kuhn

• aml-training - The most recent version of the Applied Machine Learning notes, related to the parsnip R package by Max Kuhn

• cs-video-courses - List of 800+ Computer Science courses with video lectures, Tweet

• Data-Analysis-and-Machine-Learning-Projects - Randy Olson's data analysis and machine learning projects

• google-interview-university - List of ML/CS courses. A complete daily plan for studying to become a Google software engineer

• H2O-3 - The third version of H2OAI - Open Source Fast Scalable Machine Learning Platform For Smarter Applications: Deep Learning, Gradient Boosting & XGBoost, Random Forest, Generalized Linear Modeling (Logistic Regression, Elastic Net), K-Means, PCA, Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

• machine-learning-for-software-engineers - A complete daily plan for studying to become a machine learning engineer

• Machine-Learning-in-R - Workshop (6 hours): preprocessing, cross-validation, lasso, decision trees, random forest, xgboost, superlearner ensembles

• LatinR-2019-h2o-tutorial - H2O Machine Learning Tutorial in R

• lecture_i2ml - Introduction to Machine Learning (regression/classification, performance evaluation, parameter tuning, random forests), Python

• mlcourse.ai - Open Machine Learning course mlcourse.ai, 2018 English version. Online version, Video

• MTH594_MachineLearning - The materials for the course MTH 594 Advanced data mining: theory and applications (Dmitry Efimov, American University of Sharjah)

• pattern_classification - A collection of tutorials and examples for solving and understanding machine learning and pattern classification tasks

• sklearn-classification - Data Science Notebook on a Classification Task, using sklearn and Tensorflow. Jupyter Notebook, the Census Income Dataset to predict whether an individual's income exceeds $50K/yr based on census data. Docker-wrapped

• supervised-ML-case-studies-course - Supervised machine learning case studies in R. Book

• useR-machine-learning-tutorial - useR! 2016 Tutorial: Machine Learning Algorithmic Deep Dive. IPython notebooks running R kernel

ML Videos

• Probabilistic Machine Learning — Philipp Hennig, 2021, Tweet

• 10 Powerful YouTube Channels for Data Science Aspirants - Analytics Vidhya's post. Sentdex, 3Blue1Brown, freeCodeCamp.org, StatQuest, Krish Naik, Python Programmer, Corey Schafer, Tech With Tim, Brandon Foltz, 365 Data Science

• NC ASA Webinar: Introduction to Machine Learning, by Dr. Funda Gunes, Part 1, Part 2. A one hour overview of the main machine learning concepts

• Learning from data - Statistical learning theory course from Caltech, taught by Feynman Prize winner Professor Yaser Abu-Mostafa. Videos, slides

• Statistical Machine Learning: Spring 2017 by Ryan Tibshirani, Larry Wasserman, Carnegie Mellon University.

ML Papers

• Domingos, Pedro. “A Few Useful Things to Know about Machine Learning.” Communications of the ACM 55, no. 10 (October 1, 2012): 78. https://doi.org/10.1145/2347736.2347755. Twelve lessons for machine learning. Overview of machine learning problems and algorithms, problem of overfitting, causes and solutions, curse of dimensionality, issues with high-dimensional data, feature engineering, bagging, boosting, stacking, model sparsity. Video lectures

ML Tools

• mlr3 - Machine learning in R R package, the unified interface to classification, regression, survival analysis, and other machine learning tasks. GitHub repo, mlr3gallery - Examples of problems and code solutions, mlr3 Manual - mlr3 bookdown. More on the mlr3 package site, including videos

ML Misc

• The Algorithms - R - GitHub repo with code examples of main machine learning algorithms

• algorithms_in_ipython_notebooks - A repository with IPython notebooks of algorithms implemented in Python. [https://github.com/rasbt/algorithms_in_ipython_notebooks]

• awesome-decision-tree-papers - A collection of research papers on decision, classification and regression trees with implementations

• Understanding the Bias-Variance Tradeoff - bias, variance, total error, classic figures and explanation by Scott Fortmann-Roe.

• lares - R Library for Analytics and Machine Learning

• ml_techniques - R code for performing typical ML tasks and techniques, e.g., naive Bayes, random forest, by Shirin Glander

• ML-From-Scratch - Bare bones Python implementations of some of the fundamental Machine Learning models and algorithms

• Gradient Boosting Essentials in R Using XGBOOST

• MLPB - Machine Learning Problem Bible, problems and solutions in R. XGBoost, SVM, neural networks, and other methods

• Best XGBoost settings: "a second xgboost version (xgboost_best) with the best parameter settings that I obtained in on of my publications. These are: nrounds=500, eta=0.0518715, subsample=0.8734055, booster=”gbtree”, max_depth=11, min_child_weight=1.750185, colsample_bytree=0.7126651, colsample_bylevel=0.6375492." From Is catboost the best gradient boosting R package? post on r-bloggers.com

Deep Learning

• Awesome Deep Learning - A curated list of awesome Deep Learning tutorials, projects and communities

• Awesome - Most Cited Deep Learning Papers - the most cited deep learning papers

• AI and DeepRL - source code, links and other learning materials related to Artificial Intelligence, especially focused on Deep Reinforcement Learning

• THE NEURAL NETWORK ZOO - infographics of different neural network architectures, explanation of each, references to the original papers

• Over 150 of the Best Machine Learning, NLP, and Python Tutorials, Tweet by Andrew Trask

Keras, Tensorflow

• Deep learning with R by François Chollet (the creator of Keras) with J. J. Allaire (the founder of RStudio and the author of the R interfaces to Keras and TensorFlow), R notebooks, Python notebooks

• Deep Learning with Keras and TensorFlow in R Workflow by Brad Boehmke. GutHub repo with Rmd files for data download, code examples, lectures.

• dlaicourse - Deep learning course, TensorFlow, Jupyter notebooks, by Laurence Moroney, Google

• easy-tensorflow - Simple and comprehensive tutorials in TensorFlow, by Jahandar Jahanipour. Online version

• handson-ml2 - A series of Jupyter notebooks that walk you through the fundamentals of Machine Learning and Deep Learning in Python using Scikit-Learn, Keras and TensorFlow 2. Example code and solutions for the Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow book by Aurélien Géron. Run on Google Colab

• Introduction to Deep Learning - Deep Learning basics with Python, TensorFlow and Keras. Several posts, each ncludes video, text and code tutorial

• image_classification_keras_tf - Workshop material for Image Classification & Natural Language Processing with Python, Keras and TensorFlow, by Shirin Glander

• keras-workshop - Keras R workshop, by Doug Ashton. slides, simple examples

• Machine Learning with TensorFlow and TensorFlow-Book - GitHub with the source code

• Machine Learning Foundations - Machine Learning Foundations is a free training course where you’ll learn the fundamentals of building machine learned models using TensorFlow with Laurence Moroney. Computer vision-focused

• Tensorflow-101 - Tensorflow Tutorials using Jupyter Notebook with data

• TensorFlow-Course - Simple and ready-to-use tutorials for TensorFlow. Step-by-step instructions with screenshots. By Amirsina Torfi

• TensorFlow-Examples - TensorFlow Tutorial and Examples for Beginners with Latest APIs, by Aymeric Damien

• TensorFlow-LiveLessons - "Deep Learning with TensorFlow" LiveLessons, Jupyter notebooks, by Jon Krohn

• TensorFlow 2.0 Complete Course - Python Neural Networks for Beginners - 7 hours of walk-through programming with Tim Ruscica. Links to Google Colaboratory Notebooks are in the description

• Text Classification with TensorFlow - Python TensorFlow for Machine Learning – Neural Network Text Classification Tutorial, by Kylie Ying. 1h 54m

• useR! 2020: Deep Learning with Keras and TensorFlow (S. Elsinghorst), tutorial 2h 07m video, and the GitLab repo keras_tutorial_user2020

PyTorch

• Awesome-Pytorch-list - A comprehensive list of pytorch related content on github,such as different models,implementations,helper libraries, tutorials etc. Tweet

• DEEP LEARNING with PyTorch by Yann LeCun & Alfredo Canziani. Videos, transcripts, slides, practicals. YouTube playlist

• pytorch-tutorial - PyTorch Tutorial for Deep Learning Researchers. Basic, Intermediate, and Advanced code examples, by Yunjey Choi

• the-incredible-pytorch - The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

• tutorials - official PyTorch tutorials, with videos. Website

• Zero to GANs - PyTorch, video course and Jupyter notebooks

Graph Neural Networks

• CS224W: Machine Learning with Graphs - Youtube playlist with course videos, by Jure Leskovec. Main concepts and deep neural networks training on graphs. Course website

• Deep Learning on Graphs book by Yao Ma and Jiliang Tang. Basics, methods, applications, and more. English and Chinese versions. Tweet

• GNNPapers - Must-read papers on graph neural networks (GNN). Tweet

• GNNs-Recipe - A recipe to study Graph Neural Networks (GNNs), by omarsar

DL Books

• Deep Learning Interviews book by Shlomo Kashani. Hundreds of fully solved job interview questions from a wide range of key topics in AI. GitHub repo has link to free PDF.

• Deep Learning for Molecules and Materials, Tweet

• The Deep Learning textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville. Includes lectures in .key and .pdf formats, videos discussing different chapters. https://www.deeplearningbook.org/

• Fundamentals-of-Deep-Learning-Book - Python code companion to the O'Reilly "Fundamentals of Deep Learning" book

• Dive into Deep Learning - An interactive deep learning book with code, math, and discussions, based on MXNet, useful as general learning material. https://d2l.ai/

• Grokking-Deep-Learning - Python code for the "Grokking Deep Learning" book by Andrew Trask

• neural-networks-and-deep-learning - Code samples for "Neural Networks and Deep Learning" book. Python/Theano examples, theory, and practice of deep learning by Michael Nielsen

• python-machine-learning-book - "Python Machine Learning (3rd Ed.) Code Repository" book by Sebastian Raschka, iPython notebooks

• reinforcement-learning-an-introduction - Python code for Sutton & Barto's book "Reinforcement Learning: An Introduction (2nd Edition)"

• The Matrix Calculus You Need For Deep Learning paper by Terence Parr and Jeremy Howard

• Algorithms for Convex Optimization, by Nisheeth K. Vishnoi. PDF, Tweet

DL Courses & Tutorials

• AI-For-Beginners - 12 Weeks, 24 Lessons, AI for All! PyTorch, Keras/Tensorflow, notebooks. Computer vision, natural language processing, extras. GitHub

• The Ultimate FREE Machine Learning Study Plan - A complete study plan to become a Machine Learning Engineer with links to all FREE resources.

• dive-into-machine-learning - Dive into Machine Learning with Python Jupyter notebook and scikit-learn! Links to many resources

• The Matrix Calculus You Need For Deep Learning by Terence Parr and Jeremy Howard.

• CS231n - Convolutional Neural Networks for Visual Recognition course, by Fei-Fei Li, Ranjay Krishna, Danfei Xu, at Stanford. Lecture slides, additional material, Colab notebooks. GitHub

• t81_558_deep_learning - Washington University (in St. Louis) Course T81-558: Applications of Deep Neural Networks

• CS685: Advanced Natural Language Processing, Tweet

• 2020 - 2021: Machine-Learning / Deep-Learning / AI -Tutorials - A comprehensive list of Deep Learning / Artificial Intelligence and Machine Learning tutorials - rapidly expanding into areas of AI/Deep Learning / Machine Vision / NLP and industry specific areas such as Climate / Energy, Automotives, Retail, Pharma, Medicine, Healthcare, Policy, Ethics and more

• MIT 6.874 Computational Systems Biology: Deep Learning in the Life Sciences - machine/deep learning, genomics, systems biology MIT course, Spring 2020. Taught by David Gifford, Manolis Kellis, Sachit Dinesh Saksena, Corban Swain, Timothy Fei Truong Jr. Lecture videos, slides, reading references. GitHub repo

• Colah's blog, articles on neural networks, visualization - Illustrated and highly informative posts on types of neural networks and their applications by Christopher Olah

• Introduction to Deep Learning course, D2L, Berkeley STAT 157, Jupyter notebooks, GitHub repository with slides and notebooks, Video course

  • d2l.ai - Dive into Deep Learning: An interactive deep learning book with code, math, and discussions, based on the NumPy interface, Jupyter notebooks
  • Mathematics for Deep Learning, d2l.ai - systematic deep learning math, linear algebra and matrix operations, eigendecomposition, single- and multivariable calculus, integral calculus, maximum likelihood and optimization, statistics (random variables, distributions, naive Bayes), information theory

• Practical Deep Learning for Coders, v3 - FAST.AI main course. Introduction to Machine Learning for Coders - another course by Jeremy Howard, with videos

• Step-by-step guides to learn Applied Machine Learning - Machine Learning Mastery web site aggregating structured posts for beginner and intermediate machine learning users, deep learning

• Stanford Computer Science courses CS221/229/230 ― Several GitBook-formatted courses on Artificial Intelligence, machine learnint, deep learning

• Machine Learning courses by Thorsten Joachims - Thorsten Joachims' home page with links to courses and more. CS4780/CS5780 Machine Learning for Intelligent Systems, CS6780 Advanced Machine Learning, and more. Videos and slides

• Machine and deep learning courses by Google - a collection of Google Developers courses

• Azure Machine Learning Python SDK notebooks - Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK, Microsoft

• Deep Learning in Computer Vision with Prof. Kosta Derpanis (Ryerson University)

• DeepLearningProject - An in-depth machine learning tutorial introducing readers to a whole machine learning pipeline from scratch, by Spandan Madan,Visual Computing Group, Harvard University. Python

• homemade-machine-learning - Python examples of popular machine learning algorithms with interactive Jupyter demos and math being explained, by Oleksii Trekhleb. Medium blog post

• nn-from-scratch - Implementing a Neural Network from Scratch – An Introduction, by Denny Britz. Notes

• Practical_DL - Deep learning course, Python notebooks, PDF lectures, videos. DL course co-developed by YSDA, HSE and Skoltech

• stat453-deep-learning-ss20 - Intro to Deep Learning, UW-Madison (Spring 2020) by Sebastian Raschka, videos

• stat479-machine-learning-fs19 - Course material for STAT 479: Machine Learning (FS 2019) taught by Sebastian Raschka at University Wisconsin-Madison, pdf slides

• stat479-deep-learning-ss19 - Course material for STAT 479: Deep Learning (SS 2019) taught by Sebastian Raschka at University Wisconsin-Madison, pdf slides

DL Videos

• ML-YouTube-Courses - A repository to index and organize the latest machine learning courses found on YouTube. Tweet

• Applied Deep Learning playlist, short course lectures by Maziar Raissi. GitHub.

• 3blue1brown Neural Networks playlist, and other 3blue1brown playlists

• MIT Introduction to Deep Learning | 6.S191 - MIT video course by Alexander Amini, Ava Soleimani, and guests. Dense and informative ~45min lectures covering various topics of deep learning. introtodeeplearning.com - course web site with slides, video, and other material. GitHub

• Deep Learning Crash Course for Beginner - a 1h 25m overview of deep learning techniques, highly informative narrative by Jason Dsouza

• Series of eight video lectures on the math of machine learning by Tinnam Ganesh. "Elements of Neural Networks & Deep Learning", Part1,2,3, Parts 4,5, Parts 6,7,8

• Coursera Neural Networks for Machine Learning — Geoffrey Hinton - Video course of short lectures introducing theoretical foundations of machine learning

• Introduction to Deep Learning course, D2L, Berkeley STAT 157, video lectures by Alex Smola. Accompanies the https://d2l.ai/ book

• Machine Learning & Deep Learning Fundamentals, by DeepLizard - information-dense short videos about fundamentals and math behind neural networks. Blog posts

• Brandon Rohrer's YouTube channel - short videos about basics of deep learning and neural networks

• Undergraduate machine learning at UBC 2012 by Nando de Freitas. Slides

• Deep learning at Oxford 2015 by Nando de Freitas. Slides

• Undergraduate machine learning at UBC 2012 by Nando de Freitas. Slides

• Deep learning at Oxford 2015 by Nando de Freitas. Slides

• Heroes of Deep Learning, Interviews by Andrew Ng.

• Advanced Deep Learning & Reinforcement Learning - a video-course on deep RL taught at @UCL by DeepMind researchers

• Weights & Biases video and code tutorials - Short videos and text with Python code for individual topics, by Lukas Biewald. GitHub repo with code. Weights & Biases Youtube channel

• UCL Course on Reinforcement Learning by David Silver. Slides and video lectures

• Deep Reinforcement Learning: CS 285 Fall 2020 - Lectures for UC Berkeley CS 285: Deep Reinforcement Learning.

DL Papers

• DeepMind Research - implementations and illustrative code to accompany DeepMind publications. Jupyter notebooks and data, list of projects

• Lee, Benjamin D, Anthony Gitter, Casey S Greene, Sebastian Raschka, Finlay Maguire, Alexander J Titus, Michael D Kessler, et al. “Ten Quick Tips for Deep Learning in Biology.” ArXiv 29 May 2021 - 1. Use appropriate method; 2. Establish baseline; 3. Train reproducibly; 4. Know your data; 5. Select sensible architecture; 6. Optimize hyperparameters; 7. Mitigate overfitting; 8. Maximize interpretability; 9. Avoid over-interpretation; 10. Prioritize research ethics. Summary in Figure 1. References. Latest version

• Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention Is All You Need,” arXiv:1706.03762, 6 Dec 2017 - Transformer paper. Illustrated Guide to Transformers Neural Network: A step by step explanation - 15 min video. The Annotated Transformer - PyTorch implementation of the original transformer paper. The Illustrated Transformer - blog post by Jay Alammar explaining Transformer architecture. Tweet - best resources to learn Transformers

• Sebastian Ruder, “An Overview of Gradient Descent Optimization Algorithms.” June 15, 2017 - Gradient descent optimization algorithm review, by . Definitions, intuitive progression of algorithm improvements. Gradient descent variants: Batch, Stochastic, Mini-batch. Gradient descent algorithme: Momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, Adam, AdaMax, Nadam. Visualizattion. Parallel implementations.

• eugeneyan/applied-ml - Papers & tech blogs by companies sharing their work on data science & machine learning in production.

• 2020: A Year Full of Amazing AI papers- A Review - A curated list of the latest breakthroughs in AI by release date with a clear video explanation, link to a more in-depth article, and code

• awesome-deepbio - A curated list of awesome deep learning publications in the field of computational biology

• Deep Learning Papers Reading Roadmap - Deep Learning papers reading roadmap for anyone who are eager to learn this amazing tech!

• Papers with code - Systematic collection of machine- and deep learning papers with code, State-of-the-art, Methods, Datasets

• Deep_learning_examples - Examples of using deep learning in Bioinformatics. Deep Learning in Bioinformatics

• deep_learning_papers - A place to collect papers that are related to deep learning and computational biology, by Harold Pimentel

• Deep-Learning-Papers-Reading-Roadmap - Deep Learning papers reading roadmap for anyone who are eager to learn this amazing tech!

• deeplearning-biology - A list of papers on deep learning implementations in biology

• Machine-learning-for-proteins - List of papers about machine learning for proteins

• Key Papers in Deep Reinforcement Learning, Twitter

• LeCun, Bengio, and Hinton, “Deep Learning.” - Classical deep learning review. Areas of application, historical development, principles of supervised learning, stochastic gradient descent (Figure 1 - illustration of forward and backpropagation, with equations), convolutional neural networks for image recognition and in other areas, language processing, recurrent neural networks, LSTMs

• Vincent et al., “Extracting and Composing Robust Features with Denoising Autoencoders.” - Denoising autoencoder paper, statistical formulations

• Schmidhuber, “Deep Learning in Neural Networks.” - Deep overview of deep learning history. Year-by-year description of types of DL, approaches, algorithmic (backpropagation) improvements, problems, and solutions

DL Papers Genomics

• Avsec, Žiga, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka Grabska-Barwinska, Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. “Effective Gene Expression Prediction from Sequence by Integrating Long-Range Interactions.” Nature Methods, (October 2021) - Enformer - a deep learning model (transformers) to predict epigenetic and gene expression profiles (128bp resolution) from human and mouse cell types using only the DNA sequence as input, incorporating information from up to 100 kb on either side of the target locus (200kb of input DNA sequence). Transformers, allow to increase the receptive field up to 100kb, in contrast to 20kb for Basenji2 or ExPecto. Significant increase in performance. Predictions using CAGE data. Improves variant effect prediction on eQTL data. Excellent transformer description.

• Greener, Joe G., Shaun M. Kandathil, Lewis Moffat, and David T. Jones. “A Guide to Machine Learning for Biologists.” Nature Reviews Molecular Cell Biology, September 13, 2021. - Introduction to machine/deep learning, focusing on biology applications. General terms, supervised/unsupervised learning, loss function, parameters and hyperparameters, training/validation/testing, overfitting, bias-variance tradeoff, Classification, regression, clustering, dimensionality reduction, neural networks (CNN, LSTM/RNN/transformers, autoencoders), network training, data leakage. Evaluation of machine learning reports. References on each topic.

• Deep Review: Opportunities and obstacles for deep learning in biology and medicine - A collaboratively written review paper on deep learning, genomics, and precision medicine led by Casey Greene and many others

• Deep Learning Genomics Primer - This tutorial is a supplement to the manuscript, A Primer on Deep Learning in Genomics (Nature Genetics, 2018) by James Zou, Mikael Huss, Abubakar Abid, Pejman Mohammadi, Ali Torkamani & Amalio Telentil. Box 1 and 2 - concepts and definitions. Box 3 - online resources (cloud platforms, GPU services, software libraries, educational resources, more). Python tutorial on detecting DNA motifs.

• Eraslan et al., “Deep Learning.” - Deep learning in genomics review. Big data description, evolution of machine learning into deep learning with the help of GPUs. Supervised learning - Four major classes of neural networks (fully connected, convolutional, recurrent and graph convolutional). Two unsupervised learning techniques, autoencoders and generative adversarial networks (GANs). From basic logistic regression to each network architecture illustrated on figures, theory descriptions, examples of applications in genomics. Transfer learning, model zoos, interpretation/feature importance.

• Angermueller et al., “Deep Learning for Computational Biology.” - Review on machine learning, (epi)genomics examples. Supervised vs. unsupervised learning. Deep neural networks. Box 1 - network basics. Box 2 - convolutional NN. TOOLS: Caffe, Theano, Torch7, TensorFlow. Data preparation, model training and optimization

• Min, Lee, and Yoon, “Deep Learning in Bioinformatics.” - Deep neural networks in bioinformatics. Overview of deep learning development, programming libraries, basic structure of neural networks, convolutional NNs, recurrent NNs. Table 4 - Omics applications, biomedical imaging, biomedical signal processing. References. Code examples (Jupyter notebooks) of eight bioinformatics deep learning applications

• Zou et al., “A Primer on Deep Learning in Genomics.” - Deep learning in genomics overview (feed-forward, convolutional, recurrent) and a Python tutorial on detecting DNA motifs. Box 1 and 2 - concepts and definitions. Box 3 - online resources (cloud platforms, GPU services, software libraries, educational resources, more). GitHub repo and Colab notebook with Interactive tutorial to build a convolutional neural network to discover DNA-binding motifs

• Pérez-Enciso, and Zingaretti. “A Guide for Using Deep Learning for Complex Trait Genomic Prediction.” Genes, 2019 - Deep learning for predicting phenotypes from genomics data. Deep learning basics, definitions

• Sakellaropoulos, Theodore, Konstantinos Vougas, Sonali Narang, Filippos Koinis, Athanassios Kotsinas, Alexander Polyzos, Tyler J. Moss, et al. “A Deep Learning Framework for Predicting Response to Therapy in Cancer.” Cell Reports, December 2019 - Drug response prediction from gene expression data. Deep Neural Network (DNN, H2O.ai framework) compared with Elastic Net, Random Forest. Trained on highly variable (by MAD) gene expression in 1001 cell lines and 251 drugs pharmacogenomic dataset (GDSC. CCLP) to predict IC50. Hyper-parameter optimization using 5-fold cross-validation and minimizing Mean Square Error. Batch correction between the datasets Tested on unseen patient cohorts (OCCAMS, MD Anderson, TCGA, Multiple Myeloma Consortium) to predict IC50 and test low, medium, high IC50 groups for survival differences. RDS files data, R code

• M. Jannesari, M. Habibzadeh, H. Aboulkheyr, P. Khosravi, O. Elemento, M. Totonchi, and I. Hajirasouliha. “Breast Cancer Histopathological Image Classification: A Deep Learning Approach.” In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2018 - Breast cancer image classification. Data from Stanford Tissue Microarray Database (TMAD) and Breast Cancer Histopathological Database (BreakHis), >6K images. Different variants of ResNet and Inception architectures. Data augmentation (resizing, rotation, cropping, flipping). Training details. Classification into malignant and benign, or into subtypes. Can handle images at different magnifications. ResNet performs better. GitHub repository includes crawler to get images

DL Tools

• Interactive_Tools - Interactive Tools for Machine Learning, Deep Learning and Math. Play with deep neural network in browser

• ivy - The Unified Machine Learning Framework supporting JAX, TensorFlow, PyTorch, MXNet, and Numpy. Python module. Documentation

• keras - Deep Learning for humans http://keras.io/

• MXNet-Gluon-Style-Transfer - neural artistic style transfer using MXNet. PyTorch and Torch implementations available

• openai.com - GPT-3 Access Without the Wait (API access to GPT-3)

• pathology_learning - Using traditional machine learning and deep learning methods to predict stuff from TCGA pathology slides

• ruta - Unsupervised Deep Architechtures in R, autoencoders. Requires Keras and TensorFlow. Book

• tensor2tensor - Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research

• Janggu - deep learning interface to genomic data (FASTA, BAM, BigWig, BED, GFF). Numpy-like Bioseq and Cover objects accessable by Keras. Includes model evaluation and interpretation features. Pypi, Docs, Janggu - Deep learning for genomics

• maui - Multi-omics Autoencoder Integration. Latent factors from different data types (stacked variational autoencoders), and their clustering, testing for association with survival. Tested vs. latent factors extracted using Multifactor Analysis (MFA) and iCluster+, on TCGA colorectal cancer RNA-seq, SNPs, CNVs. Evaluation of Colorectal Cancer Subtypes and Cell Lines Using Deep Learning

• Ludwig is a toolbox built on top of TensorFlow that allows to train and test deep learning models without the need to write code. GitHub

• Mask_RCNN - Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow

• PennAI - AI-Driven Data Science, entry-level machine learning interface for non-experts. A System for Accessible Artificial Intelligence

Auto ML

• ClearML - Auto-Magical Suite of tools (Python) to streamline your ML workflow Experiment Manager, MLOps and Data-Management. Documentation, Youtube channel

• TPOT - A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming. Simplified interface to many machine learning algorithms. Scaling Tree-Based Automated Machine Learning to Biomedical Big Data with a Feature Set Selector

DL models

• PyTorch model zoos

• Keras model zoos

• Tensorflow model zoos

• Kipoi - a model zoo for genomics. Examples of transfer learning, predicting pathogenic variants, TFBSs. Avsec et al., “The Kipoi Repository Accelerates Community Exchange and Reuse of Predictive Models for Genomics.”, GitHub repo

• Caffe Model Zoo

• DNABERT - pre-trained Bidirectional Encoder Representation from Transformers (BERT) model for DNA sequence, captures contextual information. Input - k-mer tokens (6-mer perform best). Architecture details, comparison with RNN, CNN, Transformers. Applied to promoter region prediction, transcription factor binding sites, splicing sites, genetic variants. visualization of learned attention patterns.Supplementary material contains detailed description of the model's architecture, input data, training, evaluation, various case scenarios, visualization. Bringing BERT to the field blog post contains more details and links to transformer resources.

  • Ji, Yanrong, Zhihan Zhou, Han Liu, and Ramana V Davuluri. “DNABERT: Pre-Trained Bidirectional Encoder Representations from Transformers Model for DNA-Language in Genome.” Bioinformatics, (August 9, 2021)

• BERT, Bidirectional Encoder Representations from Transformers, for natural language processing tasks. Model architecture, implemented using TensorFlow. Applications - Masked Language Model, next sentence prediction. Excels in several benchmarks. Pretrained models and code. See also BioBert

  • Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”

• progressive_growing_of_gans - Progressive Growing of GANs for Improved Quality, Stability, and Variation — Official TensorFlow implementation of the ICLR 2018 paper

DL projects

• 500-AI-Machine-learning-Deep-learning-Computer-vision-NLP-Projects-with-code - links to many ML/DL projects and resources

• awesome-colab-notebooks - Collection of google colaboratory notebooks for fast and easy experiments

• DALL-E-2 - a new AI system that can create realistic images and art from a description in natural language. Tweet. How This A.I. Draws Anything You Describe 16m video by ColdFusion. HOW DALL-E COULD POWER A CREATIVE REVOLUTION, The Verge. DALLE2-pytorch - Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch.

• CogView2 - official code repo for paper "CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers" (ArXiv), Example

• JoJoGAN - Official PyTorch repo for JoJoGAN: One Shot Face Stylization. Arxiv

• Bringing-Old-Photos-Back-to-Life - Old Photo Restoration (Official PyTorch Implementation)

• Real-Time-Voice-Cloning - Clone a voice in 5 seconds to generate arbitrary speech in real-time. Learn voice characteristics from a short audio clip and perform text-to-speech conversion using this voice.

• SpeechBrain - A PyTorch-based Speech Toolkit for speech/speaker recognition, speech enhancement, processing, and more. GitHub repo

• bert-finetuning-catalyst - Code for BERT classifier finetuning for multiclass text classification, code and video, by Yury Kashnitsky

• 18 All-Time Classic Open Source Computer Vision Projects for Beginners by Analytics Vidhya

• Project DeepSpeech - A TensorFlow implementation of Baidu's DeepSpeech architecture. Transcribe audio data, English model available. Documentation

• Deeplearning-digital-pathology - Python code demonstrating image classification using Keras with Caffe or TensorFlow backend, image manipulation utilities

• Weights & Biases Gallery of Curated machine learning reports - selected examples with code

• Machine learning lessons and teaching projects designed for engineers - GitHub repo by Lukas Biewald, the founder of Weights and Biases. Code and video tutorials

• Weights & Biases GitHub examples of deep learning projects

• neuralart_tensorflow - Implementation of "A Neural Algorithm of Artistic Style" by Tensorflow

• Jukebox - music generation neural network. Hierarchical Vector Quantised-Variational AutoEncoder (VQ-VAE) architecture, three separate temporal resolutions. Able to generate singing from lyrics, extend music examples. Dhariwal et al., “Jukebox: A Generative Model for Music.”, Blog post with examples of generated music

• Magenta - Music and Art Generation with Machine Intelligence

• Selfie2Anime online tool and a GitHub repo

• NVIDIA Deep Learning Examples for Tensor Cores - This repository provides State-of-the-Art Deep Learning examples that are easy to train and deploy, achieving the best reproducible accuracy and performance with NVIDIA CUDA-X software stack running on NVIDIA Volta, Turing and Ampere GPUs.

DL Misc

• app.wombo.art - deep generative model dreaming awesome images from text, Android and iOS apps available. Tweet describing the VQGAN+CLIP technology behind it

• Elvis Saravia - machine learning researcher at Facebook AI, Twitter account to follow.

• traingenerator.jrieke.com - A web app to generate template code for machine learning. GitHub, Tweet

• Deep-learning-in-cloud - List of deep learning cloud providers

• Deep learning resources - (cloud) platforms, software, educational resources. From Zou et al., “A Primer on Deep Learning in Genomics.”

• Collections of GitHub repositories of deep learning projects, Analytics Vidhya

• Google Colab Free GPU Tutorial

• tpu-starter - Everything you want to know about Google Cloud TPU

• How to use R with Google Colaboratory?, direct link to a new R-notebook

• Deep-Reinforcement-Learning-Algorithms-with-PyTorch - PyTorch implementations of deep reinforcement learning algorithms and environments

• ML Visuals - Visuals contains figures and templates which you can reuse and customize to improve your scientific writing. Google Slides

• Machine-Learning-Figures - images of most representative concepts and diagrams for machine- and deep learning.

JAX

JAX is a combination of Automatic Differentiation and XLA (Accelerated Linear ALgebra). XLA is a compiler developed by Google to work on TPU units. Jax has Numpy as its higher layer of abstraction, and works the same way on CPU, GPU, and TPU (much faster).

• awesome-jax - JAX - A curated list of resources

• JAX - Jupyter (Colab) notebooks introducing JAX basic (jit, vmap, pmap, grad, and other) and advanced concepts, by @yvrjsharma

Material in Russian

• Scientific_graphics_in_python - matplotlib for scientific graphics. 3 parts, 13 chapters. By Pavel Shabanov

• ml-course-hse - machine learning course at the Computer Sciences Department, High Schoool of Economy. Multiple years, videos

• mlcourse_open - OpenDataScience Machine Learning course (Both in English and Russian). Python-based ML course, with video lectures. Video

• DL_CSHSE_spring2018 - Deep learning, Anton Osokin, Higher School of Economics, Computer Sciences Department (Russian), course material, and video lectures

• Ordinary Differential Equations - Обыкновенные дифференциальные уравнения, Интерактивный учебник, Илья Щуров (НИУ ВШЭ)

• Calculus - Математический анализ, Записки лекций, Илья Щуров (НИУ ВШЭ). Tweet

• mathprofi.ru - Высшая математика – просто и доступно. Mirror