Topics Course on Deep Learning for Spring 2016
by Joan Bruna, UC Berkeley, Statistics Department
##Syllabus
- Invariance, stability.
- Variability models (deformation model, stochastic model).
- Scattering
- Extensions
- Group Formalism
- Supervised Learning: classification.
- Properties of CNN representations: invertibility, stability, invariance.
- covariance/invariance: capsules and related models.
- Connections with other models: dictionary learning, LISTA, Random Forests.
- Other tasks: localization, regression.
- Embeddings (DrLim), inverse problems
- Extensions to non-euclidean domains.
- Dynamical systems: RNNs and optimal control.
- Guest Lecture: Wojciech Zaremba (OpenAI)
- Autoencoders (standard, denoising, contractive, etc.)
- Variational Autoencoders
- Adversarial Generative Networks
- Maximum Entropy Distributions
- Open Problems
- Guest Lecture: Ian Goodfellow (Google)
- Non-convex optimization theory for deep networks
- Stochastic Optimization
- Attention and Memory Models
- Guest Lecture: Yann Dauphin (Facebook AI Research)
-
Lec1 Jan 19: Intro and Logistics
-
Lec2 Jan 21: Representations for Recognition : stability, variability. Kernel approaches / Feature extraction. Properties.
recommended reading:
- Elements of Statistical Learning, chapt. 12, Hastie, Tibshirani, Friedman.
- Understanding Deep Convolutional Networks, S. Mallat.
-
Lec3 Jan 26: Groups, Invariants and Filters.
recommended reading
-
Lec4 Jan 28: Scattering Convolutional Networks.
recommended reading
further reading
-
Lec5 Feb 2: Further Scattering: Properties and Extensions.
recommended reading
-
Lec6 Feb 4: Convolutional Neural Networks: Geometry and first Properties.
recommended reading
- Deep Learning Y. LeCun, Bengio & Hinton.
- Understanding Deep Convolutional Networks, S. Mallat.
-
Lec7 Feb 9: Properties of learnt CNN representations: Covariance and Invariance, redundancy, invertibility
recommended reading
- Deep Neural Networks with Random Gaussian Weights: A universal Classification Strategy?, R. Giryes, G. Sapiro, A. Bronstein.
- Intriguing Properties of Neural Networks C. Szegedy et al.
- Geodesics of Learnt Representations O. Henaff & E. Simoncelli.
- Inverting Visual Representations with Convolutional Networks, A. Dosovitskiy, T. Brox.
- Visualizing and Understanding Convolutional Networks M. Zeiler, R. Fergus.
-
Lec8 Feb 11: Connections with other models (DL, Lista, Random Forests, CART)
recommended reading
-
Proximal Splitting Methods in Signal Processing Combettes & Pesquet.
-
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems Beck & Teboulle
-
Learning Fast Approximations of Sparse Coding K. Gregor & Y. LeCun
-
Task Driven Dictionary Learning J. Mairal, F. Bach, J. Ponce
-
Exploiting Generative Models in Discriminative Classifiers T. Jaakkola & D. Haussler
-
Improving the Fisher Kernel for Large-Scale Image Classification F. Perronnin et al.
-
NetVLAD R. Arandjelovic et al.
-
Lec9 Feb 16: Other high level tasks: localization, regression, embedding, inverse problems. Extensions to non-Euclidean domain.
-
Lec10 Feb 18: Representations of stationary processes. Properties.
-
Lec11 Feb 23: Sequential Data: Recurrent Neural Networks.
-
Lec12 Feb 25: Guest Lecture ( W. Zaremba, OpenAI )
-
Lec13 Mar 1: Unsupervised Learning: autoencoders. Density estimation. Parzen estimators. Curse of dimensionality
-
Lec14 Mar 3: Variational Autoencoders
-
Lec15 Mar 8: Adversarial Generative Networks
-
Lec16 Mar 10: Maximum Entropy Distributions
-
Lec17 Mar 29: Self-supervised models (analogies, video prediction, text, word2vec).
-
Lec18 Mar 31: Guest Lecture ( I. Goodfellow, Google Brain )
-
Lec19 Apr 5: Non-convex Optimization: parameter redundancy, spin-glass, optimiality certificates. stability
-
Lec20 Apr 7: Tensor Decompositions
-
Lec21 Apr 12: Stochastic Optimization, Batch Normalization, Dropout
-
Lec22 Apr 14: Reasoning, Attention and Memory: New trends of the field and challenges. limits of sequential representations (need for attention and memory). modern enhancements (NTM, Memnets, Stack/RNNs, etc.)
-
Lec23 Apr 19: Guest Lecture (Y. Dauphin, Facebook AI Research)
-
Lec24-25: Oral Presentations