【1997 论文】优化问题中的没有免费午餐定理

No Free Lunch Theorems for Optimization

优化问题中的没有免费午餐定理

Abstract

摘要:

A framework is developed to explore the connection between effective optimization algorithms and the problems they are solving. A number of “no free lunch” (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over another class. These theorems result in a geometric interpretation of what it means for an algorithm to be well suited to an optimization problem. Applications of the NFL theorems to information-theoretic aspects of optimization and benchmark measures of performance are also presented. Other issues addressed include time-varying optimization problems and a priori “head-to-head” minimax distinctions between optimization algorithms, distinctions that result despite the NFL theorems’ enforcing of a type of uniformity over all algorithms.

开发了一个框架来探索有效的优化算法与其所解决的问题之间的联系。提出了许多“没有免费的午餐”(NFL)定理，这些定理表明，对于任何算法，在一类问题上的任何性能提升都会被在另一类问题上的性能抵消。这些定理从几何角度解释了算法是否适合于优化问题。还介绍了NFL定理在优化和性能基准度量的信息论方面的应用。解决的其他问题包括时变优化问题和优化算法之间先验的“头对头”极小极大区别，尽管NFL定理在所有算法上强制执行一种一致性，但这种区别仍然存在。

Index Terms— Evolutionary algorithms, information theory, optimization.

索引术语-进化算法，信息论，优化。

1. Introduction 

1.介绍

        THE past few decades have seen an increased interest in general-purpose “black-box” optimization algorithms in general-purpose “black-box” optimization algorithms that exploit limited knowledge concerning the optimization problem on which they are run. In large part these algorithms have drawn inspiration from optimization processes that occur in nature. In particular, the two most popular black-box optimization strategies, evolutionary algorithms [1]–[3] and simulated annealing [4], mimic processes in natural selection and statistical mechanics, respectively.

        在过去的几十年里，人们对通用的“黑箱”优化算法越来越感兴趣，这种算法利用了有限的关于优化问题的知识。在很大程度上，这些算法是从自然界中发生的优化过程中获得灵感的。特别是，两种最流行的黑盒优化策略，进化算法[1]-[3]和模拟退火[4]，分别模拟了自然选择和统计力学中的过程。

        In light of this interest in general-purpose optimization algorithms, it has become important to understand the relationship between how well an algorithm performs and the optimization problem on which it is run. In this paper we present a formal analysis that contributes toward such an understanding by addressing questions like the following: given the abundance of black-box optimization algorithms and of optimization problems, how can we best match algorithms to problems (i.e., how best can we relax the black-box nature of the algorithms and have them exploit some knowledge concerning the optimization problem)? In particular, while serious optimization practitioners almost always perform such matching, it is usually on a heuristic basis; can such matching be formally analyzed? More generally, what is the underlying mathematical “skeleton” of optimization theory before the “flesh” of the probability distributions of a particular context and set of optimization problems are imposed? What can information theory and Bayesian analysis contribute to an understanding of these issues? How a priori generalizable are the performance results of a certain algorithm on a certain class of problems to its performance on other classes of problems? How should we even measure such generalization? How should we assess the performance of algorithms on problems so that we may programmatically compare those algorithms?

        鉴于对通用优化算法的兴趣，理解算法的性能与运行算法的优化问题之间的关系变得非常重要。在本文中，我们提出了一个形式化的分析，通过解决以下问题有助于这样的理解:考虑到大量的黑箱优化算法和优化问题，我们如何才能最好地将算法与问题匹配(即，我们如何才能最好地放松算法的黑箱性质，并让它们利用一些关于优化问题的知识)?特别是，虽然认真的优化实践者几乎总是执行这样的匹配，但它通常是基于启发式的;这种匹配是否可以正式分析?更一般地说，在特定环境和一组优化问题的概率分布的“血肉”被强加之前，优化理论的潜在数学“骨架”是什么?信息论和贝叶斯分析对理解这些问题有什么帮助?某一算法在某一类问题上的性能结果如何先验地推广到它在其他类问题上的性能?我们该如何衡量这种泛化?我们应该如何评估算法在问题上的性能，以便我们可以通过编程来比较这些算法?

        Broadly speaking, we take two approaches to these questions. First, we investigate what a priori restrictions there are on the performance of one or more algorithms as one runs over the set of all optimization problems. Our second approach is to instead focus on a particular problem and consider the effects of running over all algorithms. In the current paper we present results from both types of analyses but concentrate largely on the first approach. The reader is referred to the companion paper [5] for more types of analysis involving the second approach.

        一般来说，我们对这些问题采取两种方法。首先，我们研究了当一个算法在所有优化问题集上运行时，对一个或多个算法的性能有什么先验限制。我们的第二种方法是专注于一个特定的问题，并考虑遍历所有算法的影响。在本文中，我们提出了两种分析的结果，但主要集中在第一种方法上。读者可参考配套论文[5]了解涉及第二种方法的更多类型的分析。

        We begin in Section II by introducing the necessary notation. Also discussed in this sec