高级优化理论与方法（三）

Gradient Method

方法推导

Fixed-Stepsize Gradient

$x^{k+1}=x^k+{\alpha }^k{d}^k$
$x^0\in {\mathbb{R}}^n$
${\alpha }^k\in \mathbb{R}$
$d^k\in {\mathbb{R}}^n$

$max\frac{\partial f(x)}{\partial d}=\nabla f(x{)}^T\cdot d$

$f(x^{k+1})=f(x^k-{\alpha }^k{d}^k)$

Cauchy-Schwartz inequality, $\nabla f(x{)}^T\cdot d=<\nabla f(x),d>\le ||\nabla f(x)||\cdot ||d||$

$<\nabla f(x),d>=||\nabla f(x)||\cdot ||d||\iff d=\nabla f(x)$

W.l.o.g, $||d||=1$

$d=\frac{\nabla f(x)}{||\nabla f(x)||}$

$<\nabla f(x),\frac{\nabla f(x)}{||\nabla f(x)||}>=||\nabla f(x)||$

Fixed-Stepsize Gradient: $x^{k+1}=x^k-\alpha \frac{\nabla f(x)}{||\nabla f(x)||}$

Steepest Descent Method

$x^{k+1}=x^k-{\alpha }^k\nabla f(x^k)$, where ${\alpha }^k=argminf(x^k-{\alpha }^k\nabla f(x^k))$

其实可以把这个求最值问题看成一个复合函数，从而变成一维向量空间的优化问题，用上一节课的内容解决（比如牛顿法）。转化后的复合函数如下：

$x^k(\alpha )=x^k-\alpha \nabla f(x^k)$

${\varphi }^k(\alpha )=f(x^k(\alpha ))$

Stop Criteria

$\nabla f(x^k)=0\Rightarrow$ FONC

$||x^{k+1}-x^k||<ϵ$

$\frac{||x^{k+1}-x^k||}{||x^k||}<ϵ$

$||f(x^{k+1})-f(x^k)||<ϵ$

$\frac{f(x^{k+1})-f(x^k)}{f(x^k)}<ϵ$

注：上述都是一些常见的停止原则，可根据实际需要选取。

方法的正确性证明

Good direction

Claim: $-\nabla f(x^k)$ is a “good” direction:$f(x^{k+1})<f(x^k)$

Pf: $f(x^{k+1})=f(x^k-\alpha \nabla f(x^k))=f(x^k)-\alpha ||\nabla f(x^k)|{|}^2+o(\alpha )$

If $\nabla f(x^k)\ne 0$, then $f(x^{k+1})-f(x^k)<0$ for small $\alpha$

注：此处的证明针对的是Fixed-Stepsize Gradient

zig-zag

Claim: $<x^{k+2}-x^{k+1},x^{k+1}-x^k>=0$

Pf: $<x^{k+2}-x^{k+1},x^{k+1}-x^k>=<{\alpha }^{k+1}\nabla f(x^{k+1}),{\alpha }^k\nabla f(x^k)>={\alpha }^{k+1}\cdot {\alpha }^k\cdot \nabla f(x^{k+1}{)}^T\cdot \nabla f(x^k)$

$\because {\alpha }^k$ minimizes, ${\varphi }^k(\alpha )=f(x^k(\alpha )),x^k(\alpha )=x^k-\alpha \nabla f(x^k)$

$\therefore$ FONC: $0={\varphi }^{k^{\prime }}({\alpha }^k)=\nabla f(x^k-{\alpha }^k\nabla f(x^k){)}^T(-\nabla f(x^k))=-\nabla f(x^{k+1}{)}^T\cdot \nabla f(x^k)$

Descent

Claim: If $\nabla f(x^k)\ne 0$, then $f(x^{k+1})<f(x^k)$

Pf:$\because {\alpha }^k$ minimizes, $f(x^k-{\alpha }^k\nabla f(x^k))={\varphi }^k(\alpha )$

$\therefore \forall \alpha \ge 0:{\varphi }^k({\alpha }^k)\le {\varphi }^k(\alpha )$

${\varphi }^{k^{\prime }}(0)=-||\nabla f(x^k)|{|}^2<0\Rightarrow \exists \overline{\alpha }>0:\forall \alpha \in (0,\overline{\alpha }],{\varphi }^k(\alpha )<{\varphi }^k(0)$

$f(x^{k+1})={\varphi }^k({\alpha }^k)\le {\varphi }^k(\alpha )<{\varphi }^k(0)=f(x^k)$

注：此处的证明针对的是Steepest Descent Method

quadratic function

$f(x)=\frac{1}{2}{x}^{T}Qx-bx$
$x\in {\mathbb{R}}^n,Q\in {\mathbb{R}}^{n\times n},b\in {\mathbb{R}}^n$
$Q$: symmetric, positive definite

$\nabla f(x)=\frac{1}{2}(x^{T}Q+Qx)-b=\frac{1}{2}(Qx+Qx)-b=Qx-b$

$F(x)=Q$

Lemma: ${\alpha }^k=\frac{{g^k}^T\cdot g^k}{{g^k}^{T}Q{g}^k}$, where $g^k=\nabla f(x^k)$

Pf: Assume $g^k\ne 0$ (If $g^k=0$, then $x^k$ strictly local min)

${\varphi }^k(\alpha )=f(x^k-\alpha g^k)$

${\varphi }^{k^{\prime }}(\alpha )=(Q(x^k-\alpha g^k)-b{)}^T\cdot (-g^k)=(x^k-\alpha g^k{)}^{T}Q(-g^k)+b^T{g}^k$

$\because {\alpha }^k>0$ minimizes

$\therefore {\varphi }^{k^{\prime }}({\alpha }^k)=0$

$\therefore {\alpha }^k{g^k}^{T}Q{g}^k=({x^k}^{T}Q-b^T)g^k$

$\because g^k=\nabla f(x^k)=Q{x}^k-b$

$\therefore {\alpha }^k=\frac{{g^k}^T\cdot g^k}{{g^k}^{T}Q{g}^k}$

注：这里给出的结论是最速下降法在二次函数时的特殊情况。

Example 1

$f(x)=x_1^2+x_2^2\Rightarrow f(x)=\frac{1}{2}{x}^{T}Qx-bx$
$$Q=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right]$$
$$b=\left[\begin{array}{c}0\\ 0\end{array}\right]$$
$$\nabla f(x)=\left[\begin{array}{c}2x_1\\ 2x_2\end{array}\right]$$
$$F(x)=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right]$$
$$x^0=\left[\begin{array}{c}1\\ 1\end{array}\right]$$
$x^{k+1}=x^k-{\alpha }^k{g}^k$
$x^1=x^0-{\alpha }^0{g}^0$
$$g^0=\nabla f(x^0)=\left[\begin{array}{c}2\\ 2\end{array}\right]$$
${\alpha }^0=\frac{{g^0}^T{g}^0}{{g^0}^{T}Q{g}^0}=\frac{8}{[4,4]\cdot [2,2{]}^T}=\frac{8}{16}=\frac{1}{2}$

$x^1=[1,1{]}^T-\frac{1}{2}[2,2{]}^T=[0,0{]}^T$

$$g^1=\nabla f(x^1)=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

Example 2

$f(x)=\frac{1}{5}{x}_1^2+x_2^2$
$$\nabla f(x)=\left[\begin{array}{c}\frac{2}{5}{x}_1\\ 2x_2\end{array}\right]$$
$$b=\left[\begin{array}{c}0\\ 0\end{array}\right]$$
$$x^0=\left[\begin{array}{c}1\\ 1\end{array}\right]$$
$$g^0=\left[\begin{array}{c}\frac{2}{5}\\ 2\end{array}\right]$$
${\alpha }^0=\frac{{g^0}^T{g}^0}{{g^0}^{T}Q{g}^0}=\frac{4.16}{8.064}=0.51587$

$x^1=[1,1{]}^T-0.51587\cdot [\frac{2}{5},2]=[0.79365,-0.031746{]}^T$

注：最速下降法在各个领域都有着非常广泛的应用，也是非常重要的一个优化方法。

Analysis

对于这个方法，我们需要考虑该方法能不能停下来（收敛性），因为什么停下来（正确性），什么时候停下来（算法的效率）。

由于该方法适用性太广，所以我们从二次函数入手，来做分析。

$f(x)=\frac{1}{2}{x}^{T}Qx-bx$

设$x^{\ast }$为最小值点，于是$\nabla f(x^{\ast })=0\Rightarrow Q{x}^{\ast }-b=0$

令$V(x)=f(x)+\frac{1}{2}{x^{\ast }}^{T}Q{x}^{\ast }$

$V(x)=\frac{1}{2}{x}^{T}Qx-bx+\frac{1}{2}{x^{\ast }}^{T}Q{x}^{\ast }=\frac{1}{2}{x}^{T}Qx-{x^{\ast }}^{T}Qx+\frac{1}{2}{x^{\ast }}^{T}Q{x}^{\ast }=\frac{1}{2}{x}^{T}Q(x-x^{\ast })+\frac{1}{2}{x^{\ast }}^{T}Q(x-x^{\ast })=\frac{1}{2}(x^{\ast }-x{)}^{T}Q(x^{\ast }-x)\Rightarrow V(x^{\ast })=0$

注：该等式利用了${x^{\ast }}^{T}Qx=x^{T}Q{x}^{\ast }$

$x^{k+1}=x^k-{\alpha }^k{g}^k,g^k=Q{x}^k-b$

Lemma

Lemma: $V(x^{k+1})=(1-r^k)V(x^{\ast })$, where $r^k=1$, if $g(k)=0$

If $g(k)\ne 0$, then $r^k={\alpha }^k\frac{{g^k}^{T}Q{g}^k}{{g^k}^T{Q}^{-1}{g}^k}(2\frac{{g^k}^T\cdot g^k}{{g^k}^{T}Q{g}^k}-{\alpha }^k)$

Pf: Let $y^k=x^k-x^{\ast }$

$V(x^k)=\frac{1}{2}{y}^{T}Q{y}^k$

$V(x^{k+1})=\frac{1}{2}(x^{k+1}-x^k{)}^{T}Q(x^{k+1}-x^{\ast })=\frac{1}{2}(x^k-x^{\ast }-{\alpha }^k{g}^k{)}^{T}Q(x^k-x^{\ast }-{\alpha }^k{g}^k)=\frac{1}{2}{y^k}^{T}Q{y}^k-{\alpha }^k{g^k}^{T}Q{y}^k+\frac{1}{2}{{\alpha }^k}^2{g^k}^{T}Q{g}^k$

$\frac{V(x^k)-V(x^{k+1})}{V(x^k)}=\frac{2{\alpha }^k{g^k}^{T}Q{y}^k-{{\alpha }^k}^2{g^k}^{T}Q{g}^k}{{y^k}^{T}Q{y}^k}$

$\because g^k=Q{x}^k-b=Q{x}^k-Q{x}^{\ast }=Q(x^k-x^{\ast })=Q{y}^k$

$\therefore {y^k}^{T}Q{y}^k=(Q^{-1}{g}^k{)}^{T}Q(Q^{-1}{g}^k)={g^k}^T{Q}^{-1}{g}^k,{g^k}^{T}Q{y}^k={g^k}^{T}Q{Q}^{-1}{g}^k={g^k}^T{g}^k$

$\frac{V(x^k)-V(x^{k+1})}{V(x^k)}=\frac{2{\alpha }^k{g^k}^{T}Q{y}^k-{{\alpha }^k}^2{g^k}^{T}Q{g}^k}{{y^k}^{T}Q{y}^k}=\frac{2{\alpha }^k{g^k}^{T}Q{y}^k-{{\alpha }^k}^2{g^k}^{T}Q{g}^k}{{g^k}^T{Q}^{-1}{g}^k}={\alpha }^k\frac{{g^k}^{T}Q{g}^k}{{g^k}^T{Q}^{-1}{g}^k}(2\frac{{g^k}^T{g}^k}{{g^k}^{T}Q{g}^k}-{\alpha }^k)=r^k$

Theorem 1

Thm: $x^k$ from a gradient method.

Suppose $r^k>0,\forall k$

Then, $x^k$ converges to $x^{\ast }$ for any $x^0\iff {\sum }_{k=0}^{\infty }{r}^k=\infty$

Pf: $r^k=1-\frac{V(x^{k+1})}{V(x^k)},\forall k:r^k\le 1$

$r^k\leftrightarrow V(x^{k+1})=0\leftrightarrow x^{k+1}=x^k$

$\forall k$: if $g^k\ne 0$, then $r^k<1$

$\forall k:r^k>0$

$g^k\ne 0\Rightarrow 0<r^k<1$

$\because V(x^{k+1})=(1-r^k)V(x^k)$

$\therefore x^k={\prod }_{i=0}^{k-1}(1-r^i)V(x^0)$

$x^k\to x^{\ast }\iff V(x^{\ast })\to 0\iff {\prod }_{i=0}^{\infty }(1-r^i)=0\iff {\sum }_{i=1}^{\infty }log(1-r^i)=\infty$

Rayleigh’s inequality: ${\lambda }_{min}(Q)||x|{|}^2\le x^{T}Qx\le {\lambda }_{max}(Q)||x|{|}^2$

$Q$ symmetric,$Q>0:{\lambda }_{min}(Q^{-1})=\frac{1}{{\lambda }_{max}(Q)},{\lambda }_{max}(Q^{-1})=\frac{1}{{\lambda }_{min}(Q)}$

Lemma: $\frac{{\lambda }_{min}(Q)}{{\lambda }_{max}(Q)}\le \frac{(x^{T}x{)}^2}{(x^{T}Qx)(x^T{Q}^{-1}x)}\le \frac{{\lambda }_{max}(Q)}{{\lambda }_{min}(Q)}$

Theorem 2

Thm: Steepest Decent: $x^k\to x^{\ast }$

Pf: $\forall k:g^k\ne 0$

${\alpha }^k=\frac{{g^k}^T{g}^k}{{g^k}^{T}Q{g}^k}\to r^k=\frac{({g^k}^T{g}^k{)}^2}{({g^k}^{T}Q{g}^k)({g^k}^T{Q}^{-1}{g}^k)}$

$r\ge \frac{{\lambda }_{min}(Q)}{{\lambda }_{max}(Q)}>0\Rightarrow {\sum }_{k=0}^{\infty }{r}^k=\infty$

总结

本节课介绍了梯度方法。先是推导出了固定步长的梯度法，然后在此基础上，考虑可变步长，于是推导出了最速下降法。并且给出了对于二次函数的最速下降法的具体做法和例子。最后做了一些理论性的分析，证明了梯度方法的收敛性。