PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Agnostic Online Learning
Shai Ben-David, David Pal and Shai Shalev-Shwartz
In: COLT 2009, 18-20 Jun 2009, Montreal.


We study learnability of hypotheses classes in agnostic online prediction models. The analogous question in the PAC learning model \citep{Valiant84} was addressed by \cite{Haussler92} and others, who showed that the VC dimension characterization of the sample complexity of learnability extends to the agnostic (or "unrealizable") setting. In his influential work, \cite{Littlestone88} described a combinatorial characterization of hypothesis classes that are learnable in the online model. We extend Littlestone's results in two aspects. First, while Littlestone only dealt with the realizable case, namely, assuming there exists a hypothesis in the class that perfectly explains the entire data, we derive results for the non-realizable (agnostic) case as well. In particular, we describe several models of non-realizable data and derive upper and lower bounds on the achievable regret. Second, we extend the theory to include margin-based hypothesis classes, in which the prediction of each hypothesis is accompanied by a confidence value. We demonstrate how the newly developed theory seamlessly yields novel online regret bounds for the important class of large margin linear separators.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Paper)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Theory & Algorithms
ID Code:5421
Deposited By:Shai Shalev-Shwartz
Deposited On:02 July 2009