Prof: | Barbara Engelhardt |
barbara.engelhardt@duke.edu | OH: Fri 2-3 | Gross 318 | |

TAs: | Kevin Luo | kevinluo@cs.duke.edu | OH: | Friday 10-11 | North 225 |

Jordan Hashimi | jordan.hashemi@duke.edu | OH: | Thurs 3-4 | Gross Hall, 318 | |

Nick Jarrett | nicholas.jarrett@duke.edu | OH: | Wed 7-9pm | SECC (Old Chemistry 211A) | |

Wenzhao Lian | wl89@duke.edu | OH: | Thurs 1-2 | Gross Hall, 351 | |

Class: | M/W 10:05-11:20am | Westbrook 0012, Divinity School |

- Homework 4 is not due until 10/7. However, we are writing up a part B of the homework and we have posted it on Sakai; both parts are now due 10/7/2013 before class.
- The take-home midterm will go out on 10/7.
- Scribes for the next part of the class: please sign up again on Sakai. We are putting three people on each set of notes.
- Survey sent around about which poster session you will go to.
- The project proposal (single page PDF) is due on Sakai on Wednesday October 2nd before class. We have outlined exactly what should go into this proposal. Please use this template and talk to us if you have questions or ideas!

**
All students: we will have two poster sessions: December 4th (Wednesday) from 2-5pm and December 14th (Saturday) from 2-5pm (both in Gross Hall 3rd floor East Meeting Space). Come to one and only one of these sessions. I highly recommend coming to the first. If you are in the above category of CS students taking this class for credit, you will be able to take the final exam during either time period. We will send around a survey so you can sign up for one of the two dates. If you are auditing the course, we'd love to have you at the first poster session (bring your research groups too!).
**

Statistics at the level of STA611 (Introduction to Statistical Methods) is encouraged, along with knowledge of linear algebra and multivariate calculus.

Course grade is based on homeworks (45%), take home midterm (15%), a final project (30%), and class participation and scribe notes (10%). Homeworks are due to me exactly one week after they are handed out at the beginning of class. Solutions should be uploaded to SAKAI before class on the due date; they should be each a single PDF document, and additional files will not be considered. I recommend using LaTeX to write up your homeworks (here is a homework template for you to use); if you have never used LaTeX before, you should consider this course an excellent opportunity to learn how to use it. Late homeworks will not be accepted, although you are allowed one late homework (maximum one week) for the course. Students may (and should) collaboratively discuss the homework assignments; however, I expect each student to program and write up their own homework solutions. Please write the names of the students you discussed the homework assignment with at the top of your solutions, and cite any material used in the preparation of the homeworks.

There is a Piazza course discussion page. Please direct questions about homeworks and other matters to that page. Otherwise, you can email the instructors (TAs and professor) at sta561-ta@duke.edu. Note that we are more likely to respond to the Piazza questions than to the email, and your classmates may respond too, so that is a good place to start.

Each lecture will have up to four scribes, who will type up notes in the LaTeX template. Within a week of class, the scribes should send the TAs the LaTeX file, at which point we will edit them and post them to the website. It is best to discuss beforehand with your fellow scribes what role you will take in the write-up: Careful note-taker, tex writer, figure maker, and copy editor, or whether you will split these roles among the four of you. If you have never used LaTeX before, there are online tutorials, Mac GUIs, and even online compilers that might help you. Here is an example of well-done scribe notes and associated figure.

The course project will include a project proposal due mid-semester, a four page writeup of the project at the end of the semester, and an all-campus poster session where you will present your work. This is the most important part of the course; we strongly encourage you to come and discuss project ideas with us early and often throughout the semester. We expect some of these projects to become publications. You are absolutely permitted to use your current rotation or research project as course projects.

A second set of references for R may be useful. First, you can download R from the CRAN website. There are many resources, such as R Studio, that can help with the programming interface, and tutorials on R are all over the place. If you are getting bored with the standard graphics package, I really like using ggplot2 for beautiful graphics and figures. Finally, you can integrate R code and output with plain text using KNITR, but that might be going a bit too far if you are a beginner.

The course will follow Kevin Murphy's Machine Learning: a probabilistic approach book. I may include optional readings or videos as appropriate. Some other texts and notes that may be useful include:

- Michael Lavine, Introduction to Statistical Thought (an introductory statistical textbook with plenty of R examples, and it's online too)
- Chris Bishop, Pattern Recognition and Machine Learning
- Daphne Koller & Nir Friedman, Probabilistic Graphical Models
- Hastie, Tibshirani, Friedman, Elements of Statistical Learning (ESL) (PDF available online)
- David J.C. MacKay Information Theory, Inference, and Learning Algorithms (PDF available online)

The final project TeX template and final project style file should be used in preparation of your final project report. Please follow the instructions and let me know if you have questions. We will have a poster session where you present your research project in lieu of a final exam.

This syllabus is *tentative*, and will almost surely be modified. Reload your browser for the current version.

- (August 26th) Introduction: concepts in probability and statistics [Scribe notes]
- Required: (reading) MLPP Ch 1, 2
- Optional: (video) Christopher Bishop Embracing Uncertainty: The New Machine Intelligence
- Optional: (video) Sam Roweis Machine Learning, Probability and Graphical Models, Part 1

- (August 28th) Introduction: MLE, MAP, Bayesian reasoning [Scribe notes]
- Required: (reading) MLPP Ch 3,5
- Optional: (video) Michael Jordan Bayesian or Frequentist: Which Are You?

- (September 2nd) Introduction: exponential family, conjugacy, and sufficiency [Scribe notes]
- Required: (reading) MLPP Ch 9
- Optional: (video) Alex Smola -- Exponential Families, Part I

- (September 4th) Simple discrete models: chains, trees, hierarchical models [Scribe notes]
- Required: (reading) MLPP Ch 3, 10
- Optional: (paper) Michael I Jordan -- Graphical Models. Statistical Science 19(1):140-155, 2004.
- Optional: (video) Zoubin Ghahramani -- Graphical Models
- Optional: Metacademy -- Bayesian Networks

- (September 9th) Gaussian models [Scribe notes]
- Required: (reading) MLPP Ch 4
- Optional: (paper) Sam Roweis -- Gaussian Identities.
- Optional: Metacademy -- Multivariate Gaussian Distribution

- (September 11th) Linear Regression [Scribe notes]
- Required: (reading) MLPP Ch 7
- Optional: Metacademy -- Linear Regression (I particularly recommend the chapter of Hastie, Tibshirani, Friedman)

- (September 16th) Generalized linear models [Scribe notes]
- Required: (reading) MLPP Ch 8, 9
- Optional: (reading) Elements of Statistical Learning (ESL) Chapter 4
- Optional: Metacademy -- Binary linear classifiers
- Optional: Metacademy -- Generalized linear models

- (September 18th) Mixture models & K-means & Expectation Maximization [Scribe notes]
- Required: (reading) MLPP Ch 11
- Optional: Metacademy -- K-means
- Optional: (video) EM algorithm and mixtures of Gaussians
- Optional: Metacademy -- Expectation Maximization

- (September 23th) Hidden Markov models [Scribe notes]
- Required: (reading) MLPP Ch 17
- Optional: (reading) A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition (Rabiner)
- Optional: Metacademy -- Hidden Markov Models

- (September 25th) Hidden Markov models (continued) and State Space models [Scribe notes]
- Required: (reading) MLPP Ch 18.1-18.4
- Optional: (reading) The unscented Kalman filter for nonlinear estimation (Wan & van der Merwe)
- Optional: Metacademy -- Dynamical linear systems

- (September 30th) Exact inference [Scribe notes]
- Required: (reading) MLPP Ch 20
- Optional: (reading) MacKay: Chapter 16 (Message passing), Chapter 21 (Exact Inference by Complete Enumeration), Chapter 24 (Exact Marginalization), Chapter 26 (Exact Marginalization in Graphs)
- Optional: Metacademy -- Variable elimination

- (October 2nd) Factor analysis [Scribe notes]
- Required: (reading) MLPP Ch 12
- Optional: (reading) Sam Roweis and Zoubin Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation 11(2), 1999.
- Optional: (reading) MacKay: Chapter 34 (Independent Component Analysis and Latent Variable Modelling)
- Optional: Metacademy -- Factor analysis
- Optional: Metacademy -- Principal components analysis

- (October 7th) Sparse linear models [Scribe notes]
- Required: (reading) MLPP Ch 13
- Optional: (reading) Elements of Statistical Learning (ESL) Chapter 3: Linear models for regression
- Optional: Metacademy -- LASSO

- (October 9th) Kernels and kernel methods [Scribe notes]
- Required: (reading) MLPP Ch 14
- Optional: (reading) ESL Chapter 12: Support vector machines and flexible discriminants
- Optional: (video) Introduction to Kernel Methods Partha Niyogi
- Optional: (video) Kernel Methods and Support Vector Machines Alex Smola
- Optional: Metacademy -- The kernel trick
- Optional: Metacademy -- Kernel Support Vector Machine

- (October 16th) Gaussian processes [Scribe notes]
- Required: (reading) MLPP Ch 15
- Optional: (reading) GPML, Chapter 3: Gaussian process regression
- Optional: (reading) GPML, Chapter 4: Gaussian process classification
- Optional: (video) Gaussian processes Karl Rasmussen
- Optional: (video) Gaussian Process Basics David MacKay
- Optional: Metacademy -- Gaussian Processes

- (October 21st) Markov random fields
- Required: (reading) MLPP Ch 19
- Optional: (reading) ESL Chapter 17:
*Undirected Graphical Models* - Optional: Metacademy -- Markov Random Fields

- (October 23rd) MCMC introduction: importance sampling, rejection sampling, simulated annealing, MCs
- Required: (reading) MLPP Ch 23
- Optional: (reading) MacKay Chapter 30:
*Efficient Monte Carlo Methods* - Optional: (reading) Christophe Andrieu, Nando de Freitas, Arnaud Doucet and Michael I. Jordan. An Introduction to MCMC for Machine Learning. Machine Learning 50:5-43, 2003.
- Optional: (video) Iain Murray, Markov Chain Monte Carlo
- Optional: (video) Nando de Freitas: Monte Carlo Simulation for Statistical Inference
- Optional: (video) Christian Robert: Markov Chain Monte Carlo Methods
- Optional: Metacademy -- Markov Chain Monte Carlo

- (October 28th) MCMC: Metropolis hastings, Gibbs sampling
- Required: (reading) MLPP Ch 24
- Optional: (reading) MacKay Chapter 32:
*Exact Monte Carlo Sampling* - Optional: Metacademy -- Metropolis-Hastings
- Optional: Metacademy -- Gibbs Sampling
- Optional: Metacademy -- Slice Sampling
- Optional: Metacademy -- Hamiltonian Monte Carlo

- (October 30th) Variational algorithms: duality, mean field
- Required: (reading) MLPP Ch 21
- Optional: (reading) MacKay Chapter 33:
*Variational Methods* - Optional: (reading) Wainwright & Jordan Graphical Models, Exponential Families, and Variational Inference
- Optional: (video) Tom Minka Approximate Inference
- Optional: (video) Martin Wainwright Graphical Models, Variational Methods and Message Passing
- Optional: (video) Christopher Bishop Graphical Models and Variational Methods
- Optional: Metacademy -- Variational Inference

- (November 4th) Variational algorithms: BP, EP, convex relaxations, VB [Scribe notes]
- Required: (reading) MLPP Ch 22
- Optional: (reading) John Winn and Christopher Bishop.
*Variational Message Passing*. Journal of Machine Learning Research 6:661-694, 2005. - Optional (video) David Sontag Approximate Inference in Graphical Models using LP relaxations
- Optional: Metacademy -- Loopy Belief Propagation

- (November 6th) Latent Dirichlet allocation & topic models [no scribe notes; see slides in Sakai and David Blei's website for details]
- Required: (reading) MLPP Ch 27
- Optional: (reading) David M. Blei, Andrew Ng and Michael I. Jordan.
*Latent Dirichlet allocation*. Journal of Machine Learning Research 3:993-1022, 2003. - Optional: (reading) David M. Blei and John D. Lafferty.
*Topic Models*. Text Mining: Classification, Clustering and Applications, 2009. - Optional (reading) Thomas L. Griffiths and Mark Steyvers.
*Finding Scientific Topics*. Proceedings of the National Academy of Sciences 101:5228-5235, 2004. - Optional: (video) David M. Blei Topic Models
- Optional: Metacademy -- Latent Dirichlet Allocation

- (November 11th) Dirichlet process mixture models
- Required: (reading) MLPP Ch 25
- Optional: (reading) Peter Orbanz and Yee Whye Teh.
*Bayesian Nonparametric Models*. Encyclopedia of Machine Learning, 2010. - Optional: (reading) Carl Edward Rasmussen.
*The Infinite Gaussian Mixture Model*. NIPS, 1999. - Optional (video) Michael Jordan -- Dirichlet Processes, Chinese Restaurant Processes and All That
- Optional: (video) Yee Whye Teh -- Dirichlet Processes: Tutorial and Practical Course
- Optional (video) Tom Griffiths -- Inferring Structure from Data
- Optional: Metacademy -- Dirichlet Process
- Optional: Metacademy -- Chinese Restaurant Process

- (November 13th) Review class [Unedited lecture notes]
- (November 18th) Adaptive basis function learning (CART and Boosting)
- Required: (reading) MLPP Ch 16
- Optional: (reading) ESL Chapter 11:
*Boosting and additive trees* - Optional: (video) Robert Schapire Theory and Applications of Boosting

- (November 20th) Deep belief learning
- Required: (reading) MLPP Ch 28
- Optional: (reading) MacKay Chapter 44:
*Supervised Learning in Multilayer Networks* - Optional: (reading) ESL Chapter 11:
*Neural Networks* - Optional (video) Geoffrey Hinton Deep Belief Networks
- Optional: (video) Geoffrey Hinton A Tutorial on Deep Learning
- Optional (video) Yoshua Benigo and Yann LeCun Tutorial on Deep Learning Architectures
- Optional (video) Yann LeCun Visual Perception with Deep Learning
- Optional: Metacademy -- Feedforward Neural Networks

- (November 25th) Graphical model structure learning
- Required: (reading) MLPP Ch 26
- Optional: (reading) Zoubin Ghahramani Inference and Structure learning
- Optional: Metacademy -- Structure Learning