PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Individual Sequence Prediction using Memory-efficient Context Trees
Ofer Dekel, Shai Shalev-Shwartz and Yoram Singer
IEEE IT 2009.

Abstract

Context trees are a popular and effective tool for tasks such as compression, sequential prediction, and language modeling. We present an algebraic perspective of context trees for the task of individual sequence prediction. Our approach stems from a generalization of the notion of margin used for linear predictors. By exporting the concept of margin to context trees, we are able to cast the individual sequence prediction problem as the task of finding a linear separator in a Hilbert space, and to apply techniques from machine learning and online optimization to this problem. Our main contribution is a memory efficient adaptation of the Perceptron algorithm for individual sequence prediction. We name our algorithm the \emph{Shallow Perceptron} and prove a \emph{shifting} mistake bound, which relates its performance with the performance of any sequence of context trees. We also prove that the Shallow Perceptron grows a context tree at a rate that is upper-bounded by its mistake-rate, which imposes an upper-bound on the size of the trees grown by our algorithm.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Theory & Algorithms
ID Code:5417
Deposited By:Shai Shalev-Shwartz
Deposited On:02 July 2009