COS 495
Introduction to Deep Learning
Yingyu Liang

Spring 2016

Course Summary

This course is an elementary introduction to a machine learning technique called deep learning (also called deep neural nets), as well as its applications to a variety of domains, including image classification, speech recognition, and natural language processing. Along the way the course also provides an intuitive introduction to basic notions such as supervised vs unsupervised learning, linear and logistic regression, continuous optimization (especially variants of gradient descent), generalization theory and overfitting, regularizers, and probabilistic modeling. The homeworks explore key concepts and simple applications, and the final project allows an in-depth exploration of a particular application area.

Administrative Information

Lectures: Mon + Wed, 3:00-4:20 in Friend Center, room 109.

Instructor: Yingyu Liang, CS building 103b, Reception hours: Thu 3:00-4:00

Teaching Assistant: Bochao Wang, Electrical Engineering, C319B, Reception hours: Mon + Tue, 11:00-12:00

Requirements: COS 126 General Computer Science (or equivalent) and COS 340 Reasoning About Computation.
Mostly need linear algebra, calculus, probability, and some programming knowledge.

Attendance and the use of electronic devices: Attendance is expected at all lectures. The use of laptops and similar devices for note-taking is permitted.

Textbook and readings

Deep Learning: A recent book on deep learning by leading researchers in the field. The book is the most complete and the most up-to-date textbook on deep learning, and can be used as a reference and further-reading materials. Not all topics in the book will be covered in class. The book can be downloaded from the link for academic purpose. It is still in preparation (updated frequently), but any recent version should be sufficient for the course. For convenience, we will use the version on 2016-01-30.

We'll post additional optional reading as the course progresses.

Grading and collaboration

Grading:
Problem Sets: 70%
Projects: 30% (Oral Presentation: 5%, Project report: 25%)

Turning in assignments and late policy: Coordinate submission of assignments with the TA. Every late day reduces the attainable credit for the exercise by 10%.

Projects: Students should form groups of 2-4 members. A list of candidate papers will be posted, and each group should pick one from the list (you can also propose your own choice but need to coordinate with the TA). Each group is required to give an oral presentation about the content of the paper in the last two weeks, and submit a report at the end. The report should include at the minimum a summary of the method/framework, and experimental results obtained by playing the code published along with the paper. Division of work should be determined by the members.

Collaboration: Students are expected to finish the homeworks by himself/herself, but discussion on the problem sets is allowed. The people you discussed with on assignments should be clearly detailed: before the solution to each question, list all people that you discussed with on that particlar question.

Lecture notes and schedule

# Date Topic Lecture notes Extra reading Problem sets
  1     02/01     Motivation and logistics of the course     slides     Nature review on deep learning  
  2     02/03     Machine learning basics 1: linear regression     slides     Math background: Chapter 2-4 of the textbook  
  3     02/08     Machine learning basics 2: linear classification     slides      
  4     02/10     Machine learning basics 3: Perceptron and SGD     slides     Chapter 4.1.7 and 4.2 in Pattern Recognition and Machine Learning    homework 1  
  5     02/15     Machine learning basics 4: SVM I     slides     Andrew Ng's note on SVM      
  6     02/18     Machine learning basics 5: SVM II     slides     Appendix B (Convex Optimization) in Foundations of Machine Learning      
  7     02/22     Machine learning basics 6: overfitting     slides     Chapter 5.1-5.4 of the textbook      
  8     02/24     Machine learning basics 7: multiclass classification     slides         homework 2  
  9     02/29     Deep learning 1: feedforward neural networks     slides     Chapter 6 of the textbook      
  10     03/02     Deep learning 2: backpropagation     slides          
  11     03/07     Deep learning 3: regularization I     slides     Chapter 7.1-7.3 of the textbook        
  12     03/09     Deep learning 4: regularization II     slides     Paper on dropout regularization       homework 3  
  13     03/21     Deep learning 5: convolution     slides     Chapter 9.6-9.10 of the textbook            
  14     03/23     Deep learning 6: convolutional neural networks     slides     Chapter 8.3-8.4 of the textbook            
  15     03/28     Deep learning 7: factor analysis     slides     Chapter 13 of the textbook            
  16     03/30     Deep learning 8: autoencoder and DBM     slides     Chapter 14.3-14.5 of the textbook          homework 4  
  17     04/04     Deep learning 9: recurrent neural networks     slides     Chapter 10.5-10.7 of the textbook           
  18     04/11     Deep learning 10: natural language processing     slides              
  19     04/13     Guest lecture by Tengyu Ma: word embedding theory     slides     Paper on arXiv           
  20     04/18     Deep learning 11: practical methodology     slides     Chapter 11 of the textbook;
  Practical Recommendations for Gradient-Based Training of Deep Architectures from Yoshua Bengio        		  homework 5