Discussion of V.Koltchinskii's 2004 IMS Medallion Lecture paper, "Local Rademacher complexities and oracle inequalities in risk minimization"
Gilles Blanchard and Pascal Massart
Annals of Statistics Volume 34, Number 6, 2006.

Abstract

Discussion paper for V. Koltchinskii's paper in the Annals of Statistics. We discuss the current state of model selection and the gap between theory and practice. We show that the Hold-out procedure is adaptive to Tsybakov's noise condition and this fact has an elementary proof. Therefore, theoretical results for model selection face a double adversary: from a theoretical point of view, Hold-out is a simple and efficient challenger to more elaborate methods. From a practical point of view, the problem is that strongly data-dependent penalization schemes such as cross-validation do not enter in the framework of the current theory. We conclude by proposing some possible promising directions for dealing with data-driven penalties that are both ready to be used in practice and theoretically efficient.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.

[Edit]