高级优化理论与方法（二）

上节回顾

Constrained

$$\begin{gathered} minf(x) \\ s.t.x\in \Omega \end{gathered}$$

Unconstrained

$$minf(x)$$

FONC

$x^{\ast }$ is optimal,$\forall d,\nabla f(x^{\ast }{)}^{T}d\ge 0$
(interior)$\nabla f(x^{\ast })=0$

SONC

$x^{\ast }$ local optimal,$\forall d,d^T\nabla F(x{)}^{T}d\ge 0$
(interior)$\nabla f(x^{\ast })=0,F(x^{\ast })\ge 0$

example

$minf(x_1,x_2)=x_1^2-x_2^2$

$x^{\ast }=[0,0{]}^T$

$\nabla f(x)=[2x_1,-2x_2{]}^T$

$\nabla f(x^{\ast })=[0,0{]}^T$

$$H(x)=\left[\begin{array}{cc}2& 0\\ 0& -2\end{array}\right]>0$$
$d_1=[1,0{]}^T$

$d_1^{T}F(x^{\ast })d_1=[2,0][1,0{]}^T=2>0$

$d_2=[0,1{]}^T$

$d_2^{T}F(x^{\ast })d_2=-2<0$

根据SONC，$[0,0{]}^T$ not local minimizer.

这节课的内容

SOSC

定理叙述

【Second-order Sufficient Condition]
Let $f\in C^2$ be defined on a region in which $x^{\ast }$ is an interior point.Suppose that:
①$\nabla f(x^{\ast })=0$
②$F(x^{\ast })>0$
Then, $x^{\ast }$ is a strict local minimizer of f.$\forall x\in N_{ϵ}(x^{\ast }):f(x^{\ast })<f(x)$
注：对于无约束优化问题，我们只能给出一些充分条件或者必要条件，充要条件是数学界的一个公开问题，目前还没有答案。

证明

证：
$f\in C^2\Rightarrow F(x^{\ast })=F(x^{\ast }{)}^T$
(由Clairaut’s Theorem and Schwarz’s Therem，$\forall i,j\in [1,n],\frac{{\partial }^{2}f(x^{\ast })}{\partial x_i\partial x_j}=\frac{{\partial }^{2}f(x^{\ast })}{\partial x_j\partial x_i}$)

Rayleigh’s Inequality:for a $P\in {\mathbb{R}}^{n\times n}$,symmetric, positive definite:
${\lambda }_{min}(P)||x|{|}^2\le x^{T}Px\le {\lambda }_{max}(P)||x|{|}^2$

where ${\lambda }_{min}(P)$ and ${\lambda }_{max}(P)$ are the minmal and maximal eigenvalue value of P, respectively.

a symmetric matrix is positive definite$\iff$all its eigenvalues are positive.

$\because d^{T}F(x^{\ast })d\ge {\lambda }_{min}(F(x^{\ast }))||d|{|}^2>0$

$\therefore f(x^{\ast }+d)-f(x^{\ast })=\frac{1}{2}{d}^{T}F(x^{\ast })d+o(||d|{|}^2)>0$

例子

$f(x)=x_1^2+x_2^2$
$\nabla f(x)=[2x_1,2x_2{]}^T$
$$H(x)=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right]>0$$
$x^{\ast }=[0,0{]}^T$

One-dimensional Search Methods

Iterative Method

Iterative Method意为迭代算法。此处算法用algorithm其实不太严谨，因为要设计到算法的复杂度证明、正确性证明、能否停止等等的算法严谨性问题，而method这个词则不用考虑这么多。迭代意为由某个初始点出发，找一些方向，往某些方向更新的过程。

Golden Section Search

Assume f: unimodular on $[a_0,b_0]$ (only one minimizer in $[a_0,b_0]$)
Basic Idea: “Narrow Down”
Binary Search does not work out.
Pick two instead of one points.

Method

input: $a_0,b_0,f,ϵ$
1.$i=0$

2.while $b_i-a_i\ge ϵ$ do

3.Pick $x<y$ from [a_i,b_i]

4.If $f(x)<f(y)$ then $a_{i+1}=a_i,b_{i+1}=y$;
else $b_{i+1}=b_i,a_{i+1}=x$

5.i++

6.END while

Issues

1.# while-loop
2.# computation of $f(\cdot )$

方法推理

W.O.L.G.(Without Loss of Generality)
Assume $b_0-a_0=1$
$a_1-a_0=b_1-b_0=\rho <\frac{1}{2}$

$\forall i:b_{i+1}-a_{i+1}=(1-\rho )(b_i-a_i)$

$b_1-a_1=1-2\rho$

$b_1-a_1=\rho (b_1-a_0)=\rho (1-\rho )\Rightarrow 1-2\rho =\rho -{\rho }^2\Rightarrow {\rho }^2-3\rho +1=0$

${\rho }_1=\frac{3+\sqrt{5}}{2}>\frac{1}{2}$（舍去）,${\rho }_2=\frac{3-\sqrt{5}}{2}<\frac{1}{2}$

算法描述

1.compile $b_1=a_0+(1-\rho )(b_0-a_0),a_1=a_0+\rho (b_0-a_0),f(a_1),f(b_1)$

2.i=0

3.while $b_i-a_i\ge ϵ$ do
if $f(a_{i+1})<f(b_{i+1})$ then
$b_{i+2}=a_{i+1},a_{i+2}=a_i+\rho (b_{i+1}-a_i),a_{i+1}=a_i$
else
$a_{i+2}=b_{i+1},b_{i+2}=b_i-\rho (b_i-a_{i+1}),b_{i+1}=b_i$

4.i++

5.END while

Time

1.While-Loop: time of $f(\cdot )$+O(1)
2.Loop: $(1-\rho {)}^N(b_0-a_0)<ϵ$
N=$argmin(lo{g}_{1-\rho }\frac{ϵ}{b_0-a_0})$

Example

$ϵ=0.3$
$f(x)=x^4-14x^3+60x^2-70x$
[0,2]
$(1-\rho {)}^N<\frac{0.3}{2}=0.15\Rightarrow N=4$

1.$a_1=a_0+\rho (b_0-a_0)=0.7633$
$b_1=a_0+(1-\rho )(b_0-a_0)=1.236$
$f(a_1)=-24.36$
$f(b_1)=-18.96$

2.[0,1.236]
$b_2=a_1=0.7639$
$a_1=a_0+\rho (1.236-0)=0.4721$
$f(b_2)=-24.36$
$f(a_2)=-21.10$

3.[0.4721,1.236]
$a_3=b_2=0.7639$
$b_3=a_2+(1-\rho )(1.236-0.4721)=0.9443$
$f(a_3)=-24.36$
$f(b_3)=-23.59$

4.[0.4721,0.9443]
$b_4=a_3=0.7639$
$a_4=0.4721+\rho (0.7443-0.4721)=0.6525$
$f(b_4)=-24.36$
$f(a_4)=-23.86$

5.[0.6525,09443]
$0.9443-0.6525<0.3=ϵ$
算法终止

Fibonacci Method

事实上，每一轮的$\rho$不一定要固定，也可以变化。假设$\rho$会变化，我们来推导一下每一轮之间$\rho$的关系。
${\rho }_1(1-{\rho }_0)=1-2{\rho }_0$
${\rho }_{k+1}(1-{\rho }_k)=1-2{\rho }_k$
${\rho }_{k+1}=1-\frac{{\rho }_k}{1-{\rho }_k}$

问题转化为
min $(1-{\rho }_0)(1-{\rho }_1)\cdots (1-{\rho }_k)$
s.t. ${\rho }_{k+1}=1-\frac{{\rho }_k}{1-{\rho }_k}$

结论为${\rho }_0=1-\frac{F_N}{F_{N+1}},{\rho }_{N-1}=1-\frac{F_1}{F_2}$
$F_k$为Fibonacci数列的第$k$项，$F_0=0,F_1=1,F_{k+2}=F_k+F_{k+1}$

注：用该方法来做比黄金分割法要快。

Bisection Method

Assume:f: unimodular on $[a_0,b_0]$, f continuously differentiable.

$f^{\prime }(c)<0:[c,b_0]$
$f^{\prime }(c)>0:[a_0,c]$
$f^{\prime }(c)=0:$return $c$

$(\frac{1}{2}{)}^N<ϵ$

Newton Method

Assume: $f\in C^2\Rightarrow x^{\ast }\in [a,b]:f^{\prime }(x^{\ast })=0$

$x_{k+1}=x_k-\frac{f(x_k)}{f^{\prime }(x_k)}$或$x_{k+1}=x_k-\frac{f^{\prime }(x_k)}{f^{\prime \prime }(x_k)}$

该方法只有在初始点选的比较好的时候才管用，若初始点选的不好，可能产生振荡不收敛的问题。

Example

$f(x)=\frac{1}{2}{x}^2-sinx$

$x_0=0.5$

$ϵ=10^{-5}$

$f^{\prime }(x)=x-cosx$

$f^{\prime \prime }(x)=1+sinx$

$x_1=0.5-\frac{0.5-cos0,5}{1+sin0.5}=0.7552$

$x_2=0.7391$

$x_3=0.7390$

$x_4=0.7390$

Secant Method

secant意为切线。

$f\in C^1$

$f^{\prime \prime }\approx \frac{f^{\prime }(x_{k+1})-f^{\prime }(x_k)}{x_{k+1}-x_k}$

$x_{k+1}=x_k-\frac{f^{\prime }(x_k)(x_k-x_{k-1})}{f^{\prime }(x_k)-f^{\prime }(x_{k-1})}$

Bracketing

Find the initial $a_0,b_0$
Suffice: $a_0,c,b_0\leftarrow f(a_0)>f(c),f(b_0)>f(c)$
该方法用于求得一个理想的区间，然后使用其它算法来做，但在实际应用中比较少见，且不太好用。

总结

本节课先回顾了FONC和SONC这两个找最值点的必要条件，然后给出了SOSC这个找最值点的充分条件。虽然看上去比较简单，但是关于无约束优化的定理目前也只发展到这种程度。目前数学界还没有找出一个充分必要条件。然后介绍了一维搜索方法中的迭代方法。重点介绍了黄金分割法，简略介绍了斐波那契法、二分法、牛顿法、割线法等方法。