Deep Learning

Professor Curro's Website: http://ee.cooper.edu/~curro/cgml/

Differentiable directed acyclic graphs covering applications in unsupervised learning, as well as generative and discriminative modeling. Gradient-based methods for optimization (stochastic gradient descent, Nesterov momentum, adam). Fast gradient computation for arbitrary computational graphs (automatic differentiation). Exploding and vanishing gradient problems. Convolutional networks. Arbitrary graphs for regression, classification and ranking. Autoencoders, adversarial networks and variations for unsupervised representation learning, generative modeling and other applications. Focus on applications in computer vision, speech processing and research problems in communication theory.

Submissions

Four Assignments - respective folder contains assignment and submission
Midterm Project - reimplement an existing research paper
Final Project - improve on existing research - see https://github.com/yuvalofek/DefensiveLayer

Weekly Reading/Discussion based on the following papers:

Week 1:

• Automatic Differentiation in Machine Learning: a Survey, https://arxiv.org/abs/1502.05767
• Stochastic Gradient Descent Tricks, https://arxiv.org/abs/1502.05767v4

Week 2:

• ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, https://arxiv.org/abs/1412.6980
• Why Momentum Really Works, https://distill.pub/2017/momentum/

Week 3:

• Understanding intermediate layers using linear classifier probes, https://arxiv.org/abs/1610.01644v3
• REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS, https://arxiv.org/abs/1701.06548
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models, https://arxiv.org/abs/1702.03275v2

Week 4:

• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, https://arxiv.org/abs/1502.01852
• Going deeper with convolutions, https://arxiv.org/abs/1409.4842
• Identity Mappings in Deep Residual Networks, https://arxiv.org/pdf/1603.05027.pdf

Week 5:

• SQUEEZENET: ALEXNET-LEVEL ACCURACY WITH 50X FEWER PARAMETERS AND <0.5MB MODEL SIZE, https://arxiv.org/abs/1602.07360
• Densely Connected Convolutional Networks, https://arxiv.org/abs/1608.06993
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, https://arxiv.org/abs/1704.04861
• UNDERSTANDING DEEP LEARNING REQUIRES RETHINKING GENERALIZATION, https://openreview.net/pdf?id=Sy8gdB9xx

Week 6:

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, https://arxiv.org/abs/1703.03400
• METRIC LEARNING WITH ADAPTIVE DENSITY DISCRIMINATION, https://arxiv.org/abs/1511.05939
• Neural Ordinary Differential Equations, https://arxiv.org/pdf/1806.07366

Week 7:

• Spectral Normalization for Generative Adversarial Networks, https://arxiv.org/pdf/1802.05957
• Analyzing and Improving the Image Quality of StyleGAN, https://arxiv.org/pdf/1912.04958.pdf
• SMOOTHNESS AND STABILITY IN GANS, https://openreview.net/pdf?id=HJeOekHKwr

Week 8:

• DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT, https://arxiv.org/pdf/1912.02292.pdf
• Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates, https://arxiv.org/pdf/1708.07120.pdf

Week 9:

• WAVENET: A GENERATIVE MODEL FOR RAW AUDIO, https://arxiv.org/abs/1609.03499
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders, https://arxiv.org/abs/1704.01279
• HIERARCHICAL GENERATIVE MODELING FOR CONTROLLABLE SPEECH SYNTHESIS, https://arxiv.org/abs/1810.07217
• FFTNET: A REAL-TIME SPEAKER-DEPENDENT NEURAL VOCODER, https://gfx.cs.princeton.edu/pubs/Jin_2018_FAR/fftnet-jin2018.pdf
• WAVEGLOW: A FLOW-BASED GENERATIVE NETWORK FOR SPEECH SYNTHESIS, https://arxiv.org/abs/1811.00002v1

Week 10:

• Big Bird: Transformers for Longer Sequences, https://arxiv.org/pdf/2007.14062.pdf
• Language Models are Few-Shot Learners, https://arxiv.org/pdf/2005.14165.pdf
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/pdf/1810.04805.pdf

Week 11:

• Hybrid computing using a neural network with dynamic external memory, https://www.nature.com/articles/nature20101
• Neural Arithmetic Logic Units, https://arxiv.org/pdf/1808.00508.pdf

Week 12:

• Fourier Spectrum Discrepancies in Deep Network Generated Images, https://arxiv.org/pdf/1911.06465.pdf