A second-order Perceptron algorithm
Nicolò Cesa-Bianchi, Alex Conconi and Claudio Gentile
SIAM Journal on Computing
Kernel-based linear-threshold algorithms, such as Support Vector Machines and Perceptron-like algorithms, are among the best available techniques for solving pattern classification problems.
In this paper, we describe an extension of the classical Perceptron algorithm, called second-order Perceptron, and analyze its performance within the mistake bound model of on-line learning. The bound achieved by our algorithm depends on the sensitivity to second-order data information, and is the best known mistake bound for (efficient) kernel-based linear-threshold classifiers to date.
This mistake bound, which strictly generalizes the well-known Perceptron
bound, is expressed in terms of the eigenvalues of the empirical data correlation matrix and depends on a parameter controlling the sensitivity of the algorithm to the distribution of these eigenvalues.
Since the optimal setting of this parameter is not known a priori, we also analyze two variants of the second-order Perceptron algorithm: one that adaptively sets the value of the parameter in terms of the number of mistakes made so far, and one parameterless, based on pseudoinverses.