Kolmogorov's Structure functions and model selection ## AbstractIn 1974 Kolmogorov proposed a non-probabilistic approach to statistics, an individual combinatorial relation between the data and its model, expressed by the so-called ``structure function'' of the data. We show that the structure function determines all stochastic properties of the data in the sense of determining the best-fitting model at every model-complexity level. A consequence is this: minimizing the data-to-model code length (finding the ML estimator or MDL estimator), in a class of contemplated models of prescribed maximal (Kolmogorov) complexity, {\em always} results in a model of best fit, irrespective of whether the source producing the data is in the model class considered. In this setting, code minimization {\em always} separates optimal model information from the remaining accidental information, and not only with high probability. The function that maps the maximal allowed model complexity to the goodness-of-fit (expressed as minimal ``randomness deficiency'') of the best model cannot itself be monotonically approximated. However, the shortest one-part or two-part code above can---implicitly optimizing this elusive goodness-of-fit. We show that---within the obvious constraints---every graph is realized by the structure function of some data. We determine the (un)computability properties of the various functions contemplated and of the ``algorithmic minimal sufficient statistic.''
[Edit] |