COS 495


This course is an elementary introduction to a machine learning technique called deep learning (also called deep neural nets), as well as its applications to a variety of domains, including image classification, speech recognition, and natural language processing. Along the way the course also provides an intuitive introduction to basic notions such as supervised vs unsupervised learning, linear and logistic regression, continuous optimization (especially variants of gradient descent), generalization theory and overfitting, regularizers, and probabilistic modeling. The homeworks explore key concepts and simple applications, and the final project allows an indepth exploration of a particular application area.
Instructor: Yingyu Liang, CS building 103b, Reception hours: Thu 3:004:00
Teaching Assistant: Bochao Wang, Electrical Engineering, C319B, Reception hours: Mon + Tue, 11:0012:00
Requirements:
COS 126 General Computer Science (or equivalent) and COS 340 Reasoning About Computation.
Mostly need linear algebra, calculus, probability, and some programming knowledge.
Attendance and the use of electronic devices: Attendance is expected at all lectures. The use of laptops and similar devices for notetaking is permitted.
#  Date  Topic  Lecture notes  Extra reading  Problem sets 

1  02/01  Motivation and logistics of the course  slides  Nature review on deep learning  
2  02/03  Machine learning basics 1: linear regression  slides  Math background: Chapter 24 of the textbook  
3  02/08  Machine learning basics 2: linear classification  slides  
4  02/10  Machine learning basics 3: Perceptron and SGD  slides  Chapter 4.1.7 and 4.2 in Pattern Recognition and Machine Learning  homework 1 
5  02/15  Machine learning basics 4: SVM I  slides  Andrew Ng's note on SVM  
6  02/18  Machine learning basics 5: SVM II  slides  Appendix B (Convex Optimization) in Foundations of Machine Learning  
7  02/22  Machine learning basics 6: overfitting  slides  Chapter 5.15.4 of the textbook  
8  02/24  Machine learning basics 7: multiclass classification  slides  homework 2  
9  02/29  Deep learning 1: feedforward neural networks  slides  Chapter 6 of the textbook  
10  03/02  Deep learning 2: backpropagation  slides  
11  03/07  Deep learning 3: regularization I  slides  Chapter 7.17.3 of the textbook  
12  03/09  Deep learning 4: regularization II  slides  Paper on dropout regularization  homework 3 
13  03/21  Deep learning 5: convolution  slides  Chapter 9.69.10 of the textbook  
14  03/23  Deep learning 6: convolutional neural networks  slides  Chapter 8.38.4 of the textbook  
15  03/28  Deep learning 7: factor analysis  slides  Chapter 13 of the textbook  
16  03/30  Deep learning 8: autoencoder and DBM  slides  Chapter 14.314.5 of the textbook  homework 4 
17  04/04  Deep learning 9: recurrent neural networks  slides  Chapter 10.510.7 of the textbook  
18  04/11  Deep learning 10: natural language processing  slides  
19  04/13  Guest lecture by Tengyu Ma: word embedding theory  slides  Paper on arXiv  
20  04/18  Deep learning 11: practical methodology  slides  Chapter 11 of the textbook; Practical Recommendations for GradientBased Training of Deep Architectures from Yoshua Bengio 
homework 5 