PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

On the ultimate convergence rates for isotropic algorithms and the best choices among various forms of isotropy
Sylvain Gelly, Jeremie Mary and Olivier Teytaud
In: 10th International Conference on Parallel Problem Solving from Nature (PPSN 2006), 9-13 Sep 2006, Reykjavik.

Abstract

In this paper, we show universal lower bounds for isotropic algorithms, that hold for any algorithm such that each new point is the sum of one already visited point plus one random isotropic direction multiplied by any step size (whenever the step size is chosen by an oracle with arbitrarily high computational power). The bound is 1 − O(1/d) for the constant in the linear convergence (i.e. the constant C such that the distance to the optimum after n steps is upper bounded by C n ), as already seen for some families of evolution strategies in [19, 12], in contrast with 1 − O(1) for the reverse case of a random step size and a direction chosen by an oracle with arbitrary high computational power. We then recall that isotropy does not uniquely determine the distribution of a sample on the sphere and show that the convergence rate in isotropic algorithms is improved by using stratified or antithetic isotropy instead of naive isotropy. We show at the end of the pap er that b eyond the mathematical proof, the result holds on experiments. We conclude that one should use antithetic- isotropy or stratified-isotropy, and never standard-isotropy.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Paper)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:2429
Deposited By:Sylvain Gelly
Deposited On:22 November 2006