Continuous lunches are free plus the design of
optimal optimization algorithms
Anne Auger and Olivier Teytaud
This paper analyses extensions of No-Free-Lunch (NFL) theorems to
countably inﬁnite and uncountable inﬁnite domains and investigates the
design of optimal optimization algorithms.
The original NFL theorem due to Wolpert and Macready states that,
for ﬁnite search domains, all search heuristics have the same performance
when averaged over the uniform distribution over all possible functions.
For inﬁnite domains the extension of the concept of distribution over all
possible functions involves measurability issues and stochastic process the-
ory. For countably inﬁnite domains, we prove that the natural extension
of NFL theorems, for the current formalization of probability, does not
hold, but that a weaker form of NFL does hold, by stating the existence
of non-trivial distributions of ﬁtness leading to equal performances for all
search heuristics. Our main result is that for continuous domains, NFL
does not hold. This free-lunch theorem is based on the formalization of
the concept of random ﬁtness functions by means of random ﬁelds.
We also consider the design of optimal optimization algorithms for a
given random ﬁeld, in a black-box setting, namely, a complexity measure
based solely on the number of requests to the ﬁtness function. We derive
an optimal algorithm based on Bellman’s decomposition principle, for
a given number of iterates and a given distribution of ﬁtness functions.
We also approximate this algorithm thanks to a Monte-Carlo planning
algorithm close to the UCT (Upper Conﬁdence Trees) algorithm, and
provide experimental results.