Boosting

Boosting is a general method of producing a very accurate prediction rule by combining rough and moderately inaccurate "rules of thumb." Much recent work has been on the "AdaBoost" boosting algorithm and its extensions.

Overviews

• Robert E. Schapire.
  The boosting approach to machine learning: An overview.
  In D. D. Denison, M. H. Hansen, C. Holmes, B. Mallick, B. Yu, editors, Nonlinear Estimation and Classification. Springer, 2003.
  Postscript or gzipped postscript.

• Ron Meir and Gunnar Rätsch.
  An introduction to boosting and leveraging.
  In Advanced Lectures on Machine Learning (LNAI2600), 2003.
  Pdf.

• Peter Bühlmann and Torsten Hothorn.
  Boosting algorithms: regularization, prediction and model fitting.
  Statistical Science, 22(4):477-505, 2007.
  Pdf.

• Yoav Freund and Robert E. Schapire.
  A short introduction to boosting.
  Journal of Japanese Society for Artificial Intelligence, 14(5):771-780, September, 1999. (Appearing in Japanese, translation by Naoki Abe.)
  Postscript or gzipped postscript.

• Robert E. Schapire.
  A brief introduction to boosting.
  In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, 1999.
  Postscript or gzipped postscript.

• Robert E. Schapire.
  Theoretical views of boosting and applications.
  In Tenth International Conference on Algorithmic Learning Theory, 1999.
  Postscript or gzipped postscript.

• Robert E. Schapire.
  Theoretical views of boosting.
  In Computational Learning Theory: Fourth European Conference, EuroCOLT'99, pages 1-10, 1999.
  Postscript or gzipped postscript.

• Thomas G. Dietterich.
  Ensemble learning.
  In The Handbook of Brain Theory and Neural Networks, Second Edition, 2002.
  Gzipped postscript.

Software

(Partial) Bibliography

• David Mease and Abraham Wyner.
  Evidence contrary to the statistical view of boosting, with responses and rejoinder.
  Journal of Machine Learning Research, 9(Feb):131--201, 2008.
  pdf.

• Joseph K. Bradley and Robert E. Schapire.
  FilterBoost: Regression and classification on large datasets.
  In Advances in Neural Information Processing Systems 20, 2008.
  Pdf.

• Cynthia Rudin, Robert E. Schapire and Ingrid Daubechies.
  Precise statements of convergence for AdaBoost and arc-gv.
  AMS-IMS-SIAM Joint Summer Research Conference on Machine and Statistical Learning, Prediction and Discovery, pages 131-145, 2007.
  Pdf.

• Cynthia Rudin, Robert E. Schapire and Ingrid Daubechies.
  Analysis of boosting algorithms using the smooth margin function.
  The Annals of Statistics, 35(6):2723-2768, 2007.
  Pdf.

• Lev Reyzin and Robert E. Schapire.
  How boosting the margin can also boost classifier complexity.
  In Proceedings of the 23rd International Conference on Machine Learning, 2006.
  Pdf.

• Aurélie C. Lozano, Sanjeev R. Kulkarni and Robert E. Schapire.
  Convergence and consistency of regularized boosting algorithms with stationary beta-mixing observations.
  In Advances in Neural Information Processing Systems 18, 2006.
  Pdf.

• Cynthia Rudin, Corinna Cortes, Mehryar Mohri and Robert E. Schapire.
  Margin-based ranking meets boosting in the middle.
  In 18th Annual Conference on Computational Learning Theory, 2005.
  Pdf.

• Robert E. Schapire, Marie Rochery, Mazin Rahim and Narendra Gupta.
  Boosting with prior knowledge for call classification.
  IEEE Transactions on Speech and Audio Processing, 13(2), March, 2005.
  Pdf.

• Cynthia Rudin, Ingrid Daubechies and Robert E. Schapire.
  The dynamics of AdaBoost: Cyclic behavior and convergence of margins.
  Journal of Machine Learning Research, 5: 1557-1595, 2004.
  Pdf.

• Cynthia Rudin, Robert E. Schapire and Ingrid Daubechies.
  Boosting based on a smooth margin.
  In 17th Annual Conference on Computational Learning Theory, 2004.
  Postscript or gzipped postscript.

• Cynthia Rudin, Ingrid Daubechies and Robert E. Schapire.
  On the dynamics of boosting.
  In Advances in Neural Information Processing Systems 16, 2004.
  Postscript or gzipped postscript.

• Yoav Freund and Robert E. Schapire.
  A discussion of ``Process consistency for AdaBoost'' by Wenxin Jiang, ``On the Bayes-risk consistency of regularized boosting methods'' by Gábor Lugosi and Nicolas Vayatis, ``Statistical behavior and consistency of classification methods based on convex risk minimization'' by Tong Zhang.
  The Annals of Statistics, 32(1), 2004.
  Postscript or gzipped postscript.

• Gokhan Tur, Robert E. Schapire and Dilek Hakkani-Tür.
  Active learning for spoken language understanding.
  In IEEE International Conference on Acoustics, Speech and Signal Processing, 2003.
  Pdf.

• Peter Stone, Robert E. Schapire, Michael L. Littman, János A. Csirik and David McAllester.
  Decision-theoretic bidding based on learned density models in simultaneous, interacting auctions.
  Journal of Artificial Intelligence Research, 19:209-242, 2003.
  Postscript or gzipped postscript.

• Robert E. Schapire.
  Advances in boosting.
  In Uncertainty in Artificial Intelligence: Proceedings of the Eighteenth Conference, 2002.
  Postscript or gzipped postscript.

• Robert E. Schapire, Peter Stone, David McAllester, Michael L. Littman and János A. Csirik.
  Modeling auction price uncertainty using boosting-based conditional density estimation.
  In Machine Learning: Proceedings of the Nineteenth International Conference, 2002.
  Postscript or gzipped postscript.

• Robert E. Schapire, Marie Rochery, Mazin Rahim and Narendra Gupta.
  Incorporating prior knowledge into boosting.
  In Machine Learning: Proceedings of the Nineteenth International Conference, 2002.
  Postscript or gzipped postscript.

• M. Rochery, R. Schapire, M. Rahim, N. Gupta, G. Riccardi, S. Bangalore, H. Alshawi and S. Douglas.
  Combining prior knowledge and boosting for call classification in spoken language dialogue.
  In International Conference on Accoustics, Speech and Signal Processing, 2002.
  Postscript or gzipped postscript.

• Marie Rochery, Robert Schapire, Mazin Rahim and Narendra Gupta.
  BoosTexter for text categorization in spoken language dialogue.
  Accepted to Automatic Speech Recognition and Understanding Workshop, 2001 (but withdrawn due to travel restrictions following September 11).
  Postscript or gzipped postscript.

• Michael Collins, Robert E. Schapire and Yoram Singer.
  Logistic regression, AdaBoost and Bregman distances.
  Machine Learning, 48(1/2/3), 2002.
  Postscript or gzipped postscript.

• Raj D. Iyer, David D. Lewis, Robert E. Schapire, Yoram Singer and Amit Singhal.
  Boosting for document routing.
  In Proceedings of the Ninth International Conference on Information and Knowledge Management, 2000.
  Postscript or gzipped postscript.

• Erin L. Allwein, Robert E. Schapire and Yoram Singer.
  Reducing multiclass to binary: A unifying approach for margin classifiers.
  Journal of Machine Learning Research, 1:113-141, 2000.
  Postscript or gzipped postscript.

• Robert E. Schapire.
  Drifting games.
  Machine Learning, 43(3):265-291, 2001.
  Postscript or gzipped postscript.

• Llew Mason, Jonathan Baxter, Peter Bartlett and Marcus Frean.
  Functional gradient techniques for combining hypotheses.
  In Advances in Large Margin Classifiers, MIT Press, 1999.
  PDF.

• Jerome Friedman, Trevor Hastie and Robert Tibshirani.
  Additive logistic regression: a statistical view of boosting.
  The Annals of Statistics, 38(2):337-374, April, 2000.
  Postscript of the paper.
  Postscript or gzipped postscript of our discussion of the paper.

• Steven Abney and Robert E. Schapire and Yoram Singer.
  Boosting applied to tagging and PP attachment.
  In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, 1999.
  Postscript or gzipped postscript.

• Robert E. Schapire and Yoram Singer.
  BoosTexter: A boosting-based system for text categorization.
  Machine Learning, 39(2/3):135-168, 2000.
  Postscript or compressed postscript.

• Yoav Freund, Raj Iyer, Robert E. Schapire and Yoram Singer.
  An efficient boosting algorithm for combining preferences.
  Journal of Machine Learning Research, 4:933-969, 2003.
  Postscript or compressed postscript.

• Robert E. Schapire and Yoram Singer.
  Improved boosting algorithms using confidence-rated predictions.
  Machine Learning, 37(3):297-336, 1999.
  Postscript or compressed postscript.

• Robert E. Schapire, Yoram Singer and Amit Singhal.
  Boosting and Rocchio applied to text filtering.
  In SIGIR '98: Proceedings of the 21st Annual International Conference on Research and Development in Information Retrieval, pages 215-223, 1998.
  Postscript or compressed postscript.

• Llew Mason, Peter Bartlett and Jonathan Baxter
  Direct optimization of margins improves generalization in combined classifiers.
  In Advances in Neural Information Processing Systems 11, pages 288-294, 1999.
  PDF.

• Leo Breiman.
  Arcing the edge.
  Technical Report 486, Statistics Department, University of California at Berkeley, 1997.
  Compressed postscript.

• Leo Breiman.
  Prediction games and arcing classifiers.
  Technical Report 504, Statistics Department, University of California at Berkeley, 1997.
  Compressed postscript.

• Thomas G. Dietterich.
  An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization.
  Machine Learning, 40(2):139-158, 2000.
  Gzipped postscript.

• Adam J. Grove and Dale Schuurmans.
  Boosting in the limit: Maximizing the margin of learned ensembles.
  In Proceedings of the Fifteenth National Conference on Artificial Intelligence, 1998.
  Gzipped postscript.

• Robert E. Schapire.
  Using output codes to boost multiclass learning problems.
  In Machine Learning: Proceedings of the Fourteenth International Conference, 1997.
  Postscript or compressed postscript.

• Robert E. Schapire, Yoav Freund, Peter Bartlett and Wee Sun Lee.
  Boosting the margin: A new explanation for the effectiveness of voting methods.
  The Annals of Statistics, 26(5):1651-1686, 1998.
  Postscript or compressed postscript.

• Leo Breiman.
  Arcing classifiers.
  The Annals of Statistics, 26(3):801-849, 1998.
  Compressed postscript of the paper.
  Postscript or compressed postscript of our discussion of the paper.

• Eric Bauer and Ron Kohavi.
  An empirical comparison of voting classification algorithms: Bagging, boosting, and variants.
  Machine Learning, 36(1/2):105-139, 1999.

• Holger Schwenk and Yoshua Bengio.
  Training methods for adaptive boosting of neural networks for character recognition.
  In Advances in Neural Information Processing Systems 10. Morgan Kaufmann, 1998.

• Richard Maclin and David Opitz.
  An empirical evaluation of bagging and boosting.
  In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pages 546-551, 1997.

• Yoav Freund and Robert E. Schapire.
  Game theory, on-line prediction and boosting.
  In Proceedings of the Ninth Annual Conference on Computational Learning Theory, pages 325-332, 1996.
  Postscript or compressed postscript.

• Yoav Freund and Robert E. Schapire.
  Experiments with a new boosting algorithm.
  In Machine Learning: Proceedings of the Thirteenth International Conference, pages 148-156, 1996.
  Postscript or compressed postscript.

• J. Ross Quinlan.
  Bagging, boosting, and C4.5.
  In Proceedings, Fourteenth National Conference on Artificial Intelligence, 1996.
  Postscript.

• Harris Drucker and Corinna Cortes.
  Boosting decision trees.
  NIPS'95.

• Yoav Freund and Robert E. Schapire.
  A decision-theoretic generalization of on-line learning and an application to boosting.
  Journal of Computer and System Sciences, 55(1):119-139, 1997.
  Postscript or compressed postscript.

• Yoav Freund.
  Boosting a weak learning algorithm by majority.
  Information and Computation, 121(2):256-285, 1995.

• Robert E. Schapire.
  The strength of weak learnability.
  Machine Learning, 5(2):197-227, 1990.
  Pdf.

Back to my home page.
Last update: December 18, 2008.

Free software for viewing postscript files is available here.

.