Line Search Methods

重点

• Armijo condition的直观理解

背景: In gradient descent algorithms, step size may be too large or too small, as shown in the figures below.

Backtracking line search

 - Initialization: alpha (=1), tau (decay rate)
 - while f(x^t + alpha p^t) ">" f(x^t)
	 alpha = tau*alpha
	end
 - Update x^{t+1} = x^t

• $\alpha =\alpha \ast \tau$的作用是防止step length过小
• $f(x^t+\alpha p^t)“>”f(x^t):\text{ prevent steps that are too long relative to the decrease in }f\text{；通过Armijo condition实现}$

Wolfe conditions

Armijo condition

$$f(x^t+{\alpha }^t{p}^t)\le f(x^t)+{\alpha }^t{c}_1\cdot [g^t{]}^T{p}^k,$$
where $g^t$ denotes the first derivative of $f$; $p$ denotes the direction, $g^{T}p<0$ (remark: gradient descent $-g\iff p$).
In practice, $c_1$ is chosen quite small, say $c_1=10^{-4}$.
In the case that $p=g$, the Armijo condition in the 2nd step of pseudo-code step can be simplified as follows:
$$f(x-\alpha \nabla (f))>f(x)-c_1\alpha \Vert \nabla (f){\Vert }_2^2.$$
*B&V book, $c_1\in [0.01,0.03],\tau \in [0.1,0.8]$

$直观理解$

• require the reduction in $f$ to be at least a fixed fraction $\beta$ of the reduction promised by the first-order Taylor approximation of $f$ at $x^t$.
• aka significant decrease condition: require $\alpha$ to decrease the objective function by a significant amount.

Curvature condition

The curvature condition rules out small steps.
$$\nabla f(x^t+{\alpha }^t{p}^t{)}^T{p}^t\ge c_2\nabla f(x^t{)}^T{p}^t,$$
where $c_2\in (c_1,1)$.
$\text{The condition requires that the new slope is at least }{c}_2\text{ times the original gradient.}$

图片来源

• step sizes: https://people.maths.ox.ac.uk/hauser/hauser_lecture2.pdf
• Figs 3.3, 3.4:
  Numerical Optimization

参考文献：

1. https://people.maths.ox.ac.uk/hauser/hauser_lecture2.pdf
2. https://optimization.mccormick.northwestern.edu/index.php/Line_search_methods
3. Numerical Optimization