PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

A linear-time algorithm for computing the multinomial stochastic complexity
Petri Kontkanen and Petri Myllymäki
Information Processing Letters Volume 103, Number 6, pp. 227-233, 2007. ISSN 0020-0190


The minimum description length (MDL) principle is a theoretically well-founded, general framework for performing model class selection and other types of statistical inference. This framework can be applied for tasks such as data clustering, density estimation and image denoising. The MDL principle is formalized via the so-called normalized maximum likelihood (NML) distribution, which has several desirable theoretical properties. The codelength of a given sample of data under the NML distribution is called the stochastic complexity, which is the basis for MDL model class selection. Unfortunately, in the case of discrete data, straightforward computation of the stochastic complexity requires exponential time with respect to the sample size, since the definition involves an exponential sum over all the possible data samples of a fixed size. As a main contribution of this paper, we derive an elegant recursion formula which allows efficient computation of the stochastic complexity in the case of n observations of a single multinomial random variable with K values. The time complexity of the new method is O(n+K) as opposed to O(n log n log K) obtained with the previous results.

EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:3539
Deposited By:Petri Kontkanen
Deposited On:11 February 2008