The Minimum Description Length Principle
This book provides a comprehensive introduction and reference guide to the minimum description length (MDL) Principle, a powerful method of inductive inference that holds that the best explanation, given a limited set of observed data, is the one that permits the greatest compression of the data. The central concepts of this theory are explained in great detail. The book should be accessible to researchers dealing with inductive inference in diverse areas including statistics, machine learning, data mining, biology, econometrics, and experimental psychology, as well as philosophers interested in the foundations of statistics. The book consists of four parts. Part I provides a basic introduction to MDL and an overview of the concepts in statistics and information theory needed to understand MDL. Part II treats universal coding, the information-theoretic notion on which MDL is built, and Part III gives a formal treatment of MDL theory as a theory of inductive inference based on universal coding. Part IV provides a comprehensive overview of the statistical theory of exponential families with an emphasis on their information-theoretic properties. The book contains some new results that have not been published elsewhere.