Introduction to nonparametric Bayesian statistics

• Please look at Michael Jordan's NIPS tutorial slides and Zoubin Ghahramani's UAI tutorial slides before class.
• Optionally, read Chapter 2 of Erik Sudderth's PhD dissertation.

MCMC Sampling for Dirichlet process mixtures

Assigned reading:

• M. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90:577-588, 1995. [JSTOR]
• R. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9(2):249-265, 2000. [JSTOR]

Additional background reading (optional):

• R. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto, 1993. [PDF]

Original constructions: Ferguson, Antoniak, and Urn schemes

Assigned reading:

• J. Ghosh and R. Ramamoorthi. Bayesian Nonparametrics. Springer, 2003. (Chapter 3.) [PDF part1] [PDF part2]

Optional reading:

• T. Ferguson. A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1:209-230, 1973. [JSTOR]
• C. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2(6):1152-1174, 1974. [JSTOR]
• D. Blackwell and J. MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics, 1(2):353-355, 1973. [JSTOR]

Sethuraman's stick-breaking construction

• J. Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 4:639-650, 1994. [PDF]

Variational inference for Dirichlet process mixtures

• D. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Journal of Bayesian Analysis, 1(1):121-144, 2005. [PDF]
• K. Kurihara, M. Welling, and Y. Teh. Collapsed variational Dirichlet process mixture models. 2007. [PDF]

Hierarchical Dirichlet processes and document modeling

• Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476):1566-1581, 2007. [PDF]
• P. Liang, S. Petrov, D. Klein, and M. Jordan. The infinite PCFG using Hierarchical Dirichlet processes. In Empirical Methods in Natural Language Processing, 2007. [PDF]

NP Bayes extensions to sequential modeling

Guest discussant: Emily Fox (MIT)

Reading:

• Fox, E., Sudderth, E., Jordan, M., and Willsky, A. (2007). Developing a tempered HDP-HMM for systems with state persistence. Technical report, MIT Laboratory for Information and Decision Systems. [PDF]

The Indian Buffet Process

• T. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian buffet process. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems 18, pages 475-482. MIT Press, Cambridge, MA, 2006. [PDF] (tech report version)
• R. Thibaux and M. Jordan. Hierarchical beta processes and the Indian buffet process. In 11th Conference on Artificial Intelligence and Statistics, 2007. [PDF] (tech report version)

NP Bayes and computer vision

• Kivinen, J., Sudderth, E., and Jordan, M. (2007). Learning multiscale representations of natural scenes using Dirichlet processes. In IEEE Conference on Computer Vision and Pattern Recognition. [PDF]
• Sudderth, E., Torralba, A., Freeman, W., and Willsky, A. (2005). Describing visual scenes using transformed Dirichlet processes. In Advances in Neural Information Processing Systems 18. [PDF]

Random Combinatorial Structures

Pitman, J. Some Developments of the Blackwell-MacQueen Urn Scheme . [PS]

Dirichlet processes and consistency

• J. Ghosh and R. Ramamoorthi. Bayesian Nonparametrics. Springer, 2003. (1-1.3, 2.3-2.5, and chapter 4)

Class presentations and course summary