Nelder–Mead method

is a commonly applied numerical method used to find the minimum or maximum of an objective function in a multidimensional space

applied to nonlinear optimization problems for which derivatives may not be known.

http://www.scholarpedia.org/article/Nelder-Mead_algorithm

Nelder-Mead algorithm

• Dr. Saša Singer, Department of Mathematics, University of Zagreb, Croatia

• Dr. John Nelder, Imperial College, London, UK

1. 1. The Nelder-Mead algorithm or simplex search algorithm, originally published in 1965 (Nelder and Mead, 1965), 

• is one of the best known algorithms for multidimensional unconstrained optimization without derivatives.
•  should not be confused with Dantzig's simplex method for linear programming, 

1.   which is completely different, 

1. as it solves a linearly constrained linear problem.

•  popular in many fields of science and technology, especially in chemistry and medicine.

1. The method does not require any derivative information, 

• • • which makes it suitable for problems with non-smooth functions. 
    • It is widely used to solve parameter estimation and similar statistical problems, 
      • where the function values are uncertain or subject to noise. 
      • It can also be used for problems with discontinuous functions, 
      • which occur frequently in statistics and experimental mathematics.

Basic description

The Nelder-Mead algorithm is designed to solve the classical unconstrained optimization problem of minimizing a given nonlinear function f:Rn→R . 

The method uses only function values at some points in Rn , and does not try to form an approximate gradient at any of these points.

Hence it belongs to the general class of direct search methods (see Wright, 1996; Powell, 1998; Kolda et al., 2003).

The method then performs 

• a sequence of transformations of the working simplex S , 
• aimed at decreasing the function values at its vertices. 
• At each step, the transformation is determined by computing one or more test points,
•  together with their function values, and by comparison of these function values with those at the vertices.

• This process is terminated when the working simplex S becomes sufficiently small in some sense, or 
• when the function values fj are close enough in some sense (provided fis continuous).
• The Nelder-Mead algorithm typically requires only one or two function evaluations at each step, while many other direct search methods use n or even more function evaluations.

The rest of the article is largely based on Wright (1996) and Lagarias et al. (1998), which are excellent starting references for the Nelder-Mead method.

Origins and history

Direct search methods first appeared in the 1950s and early 1960s with the growing use of computers to fit experimental data. The name “direct search” was introduced in 1961 by Hooke and Jeeves.

The first simplex-based direct search method was proposed by Spendley, Hext and Himsworth in 1962 (Spendley et al., 1962). It uses only two types of transformations to form a new simplex in each step:

• reflection away from the worst vertex (the one with the largest function value), or
• shrinkage towards the best vertex (the one with the smallest function value).

In these transformations, the angles between edges in every simplex remain constant throughout the iterations, so the working simplex can change in size, but not in shape.

In 1965, Nelder and Mead modified the original method of Spendley et al. by including two additional transformations—expansion and contraction, that allow the working simplex to change not only its size, but also its shape.

A figurative motivation is given in the introduction to their paper:

In the method to be described 

the simplex 

• • adapts itself to the local landscape, 

• elongating down long inclined planes, 
• changing direction on encountering a valley at an angle, and
•  contracting in the neighbourhood of a minimum.

The Nelder-Mead simplex method gained popularity very quickly. 

At that time, due to its simplicity and low storage requirements, it was

•  ideally suited for use on minicomputers, 
• especially in laboratories. 
• In the 1970s, the method became a standard member of several major software libraries. 
• Its popularity grew even more in the 1980s, when it first appeared as the “amoeba algorithm” in the widely used handbook Numerical Recipes (Press et al., second edition, 1992), and in Matlab software package, where it is now called “fminsearch” (Matlab, 2008).

• Despite its age and recent advances in direct search methods, the Nelder-Mead method is still among the most popular direct search methods in practice.

The Nelder-Mead simplex algorithm

Even though the method is quite simple, it is implemented in many different ways. 

• Apart from some minor computational details in the basic algorithm, the main difference between various implementations lies in the construction of the initial simplex, and 
• in the selection of convergence or termination tests used to end the iteration process. 
• The general algorithm is given below.

1. Construct the initial working simplex S .
2. Repeat the following steps until the termination test is satisfied:
   1.        calculate the termination test information;
   2.      if the termination test is not satisfied then transform the working simplex.

• Return the best vertex of the current simplex S and 
• the associated function value