Large scale topic modeling made practical
Bjarne Ørum Wahlgreen and Lars Kai Hansen
Machine Learning for Signal Processing (MLSP), 2011 IEEE International Workshop on pp. 1-6, 2011. ISSN 1551-2541

Abstract

Topic models are of broad interest. They can be used for query expansion and result structuring in information retrieval and as an important component in services such as recommender systems and user adaptive advertising. In large scale applications both the size of the database (number of documents) and the size of the vocabulary can be significant challenges. Here we discuss two mechanisms that can make scalable solutions possible in the face of large document databases and large vocabularies. The first issue is addressed by a parallel distributed implementation, while the vocabulary problem is reduced by use of large and carefully curated term set. We demonstrate the performance of the proposed system and in the process break a previously claimed ’world record’ announced April 2010 both by speed and size of problem. We show that the use of a WordNet derived vocabulary can identify topics at par with a much larger case specific vocabulary.

PDF - PASCAL Members only - Requires Adobe Acrobat Reader or other PDF viewer.

[Edit]