PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

On Smoothing and Inference for Topic Models
Arthur Asuncion, Max Welling, Padraic Smyth and Yee Whye Teh
In: UAI 2009, 18-21 Jul 2009, Montreal, Canada.


Latent Dirichlet Analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Paper)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Learning/Statistics & Optimisation
Natural Language Processing
Theory & Algorithms
Information Retrieval & Textual Information Access
ID Code:6729
Deposited By:Yee Whye Teh
Deposited On:08 March 2010