Continuous lunches are free plus the design of
optimal optimization algorithms
## AbstractThis paper analyses extensions of No-Free-Lunch (NFL) theorems to countably inﬁnite and uncountable inﬁnite domains and investigates the design of optimal optimization algorithms. The original NFL theorem due to Wolpert and Macready states that, for ﬁnite search domains, all search heuristics have the same performance when averaged over the uniform distribution over all possible functions. For inﬁnite domains the extension of the concept of distribution over all possible functions involves measurability issues and stochastic process the- ory. For countably inﬁnite domains, we prove that the natural extension of NFL theorems, for the current formalization of probability, does not hold, but that a weaker form of NFL does hold, by stating the existence of non-trivial distributions of ﬁtness leading to equal performances for all search heuristics. Our main result is that for continuous domains, NFL does not hold. This free-lunch theorem is based on the formalization of the concept of random ﬁtness functions by means of random ﬁelds. We also consider the design of optimal optimization algorithms for a given random ﬁeld, in a black-box setting, namely, a complexity measure based solely on the number of requests to the ﬁtness function. We derive an optimal algorithm based on Bellman’s decomposition principle, for a given number of iterates and a given distribution of ﬁtness functions. We also approximate this algorithm thanks to a Monte-Carlo planning algorithm close to the UCT (Upper Conﬁdence Trees) algorithm, and provide experimental results.
[Edit] |