Aggregation for Gaussian regression.
Florentina Bunea, Alexandre Tsybakov and Marten Wegkamp
Annals of Statistics Volume 35, Number 4, pp. 1674-1697, 2007.

## Abstract

This paper studies statistical aggregation procedures in the regression setting. A motivating factor is the existence of many different methods of estimation, leading to possibly competing estimators. We consider here three different types of aggregation: model selection (MS) aggregation, convex (C) aggregation and linear (L) aggregation. The objective of (MS) is to select the optimal single estimator from the list; that of (C) is to select the optimal convex combination of the given estimators; and that of (L) is to select the optimal linear combination of the given estimators. We are interested in evaluating the rates of convergence of the excess risks of the estimators obtained by these procedures. Our approach is motivated by recent minimax results in \citep{nem00,tsy03}. There exist competing aggregation procedures achieving optimal convergence rates for each of the (MS), (C) and (L) cases separately. Since these procedures are not directly comparable with each other, we suggest an alternative solution. We prove that all the three optimal rates, as well as those for the newly introduced (S) aggregation (subset selection), are nearly achieved via a single universal" aggregation procedure. The procedure consists in mixing the initial estimators with weights obtained by penalized least squares. Two different penalties are considered: one of them is of the BIC type, the second one is a data dependent $\ell_1$-type penalty.