高级优化理论与方法（四）

上节回顾

Fixed Stepsize

$x^{k+1}=x^k-\alpha \nabla f(x^k)$

Steepest Decent

$x^{k+1}=x^k-{\alpha }^k\nabla f(x^k)$,where ${\alpha }^k=argminf(x^k-{\alpha }^k\nabla f(x^k))$

Gradient Method

Analysis

Theorem 3

Thm: Fixed Stepsize:$x^k\to x^{\ast }$ for any $x^0\iff 0<\alpha <\frac{2}{{\lambda }_{max}(Q)}$

Pf: “$\Leftarrow$”: Rayleigh’s Inequality: ${\lambda }_{min}(Q){g^k}^T{g}^k\le {g^k}^{T}Q{g}^k\le {\lambda }_{max}(Q){g^k}^T{g}^k$

${g^k}^T{Q}^{-1}{g}^k\le \frac{1}{{\lambda }_{min}(Q)}{g^k}^T{g}^k$

$\Rightarrow r^k\ge \alpha \frac{{\lambda }_{min}(Q){g^k}^T{g}^k}{{\lambda }_{max}(Q^{-1}){g^k}^T{g}^k}(2\frac{{g^k}^T{g}^k}{{\lambda }_{max}(Q){g^k}^T{g}^k}-\alpha )=\alpha {\lambda }_{min}^2(Q)(\frac{2}{{\lambda }_{max}(Q)}-\alpha )\ge C>0$

${\sum }_{k=0}^{\infty }{r}^k=\infty$

$x^k\to x^{\ast }$

Pf: “$\Leftarrow$”: Assume $\alpha <0$ or $\alpha >\frac{2}{{\lambda }_{max}(Q)}$

Define $x^0$ with $x^0-x^{\ast }$ is an eigenvector of Q corresponding to ${\lambda }_{max}(Q)$

$x^{k+1}=x^k-\alpha (Q{x}^k-b)=x^k-\alpha (Q{x}^k-Q{x}^{\ast })$

$\to x^{k+1}-x^{\ast }=x^k-x^{\ast }-\alpha (Q{x}^k-Q{x}^{\ast })=(I_n-\alpha Q)(x^k-x^{\ast })=(I_n-\alpha Q{)}^{k+1}(x^0-x^{\ast })=(I_n-\alpha Q{)}^k(I_n-\alpha Q)(x^0-x^{\ast })=(I_n-\alpha Q{)}^k(x^0-x^{\ast }-\alpha Q(x^0-x^{\ast }))=(I_n-\alpha Q)(x^0-x^{\ast }-\alpha {\lambda }_{max}(Q)(x^0-x^{\ast }))=(1-\alpha {\lambda }_{max}(Q))(I_n-\alpha Q{)}^k(x^0-x^{\ast })=(1-\alpha {\lambda }_{max}(Q){)}^{k+1}(x^0-x^{\ast })$

$||x^{k+1}-x^{\ast }||=|1-\alpha {\lambda }_{max}(Q){|}^{k+1}||x^0-x^{\ast }||$

$\because \alpha >0$ or $\alpha >\frac{2}{{\lambda }_{max}(Q)}$

$\therefore |1-\alpha {\lambda }_{max}(Q)|>1$

$\therefore x^k$ not converge to $x^{\ast }$

Order of convergence

Def: Given $x^k\to x^{\ast }$, ${\lim }_{k\to \infty }||x^k-x^{\ast }||=0$. Order of convergence is $P\in \mathbb{R}$,if $0<{\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}=c<\infty$

order is $\infty$, if for all $p>0$: ${\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}=0$

sublinear: $p=1$,${\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}=1$

linear: $p=1$, ${\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}<1$

superlinear: $p=1$,${\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}>1$

注：对于二次函数， $p=2$。

Example 1

$x^k=\frac{1}{k}$

$x^k\to 0=x^{\ast }$

$\frac{\frac{1}{k+1}}{(\frac{1}{k}{)}^p}=\frac{k^p}{k+1}$

$p<1$: ${\lim }_{k\to \infty }\frac{k^p}{k+1}=0$

$p=1$: ${\lim }_{k\to \infty }\frac{k}{k+1}=1$

Example 2

$x^k=r^k$($0<r<1$)

$x^{\ast }=0$

$\frac{r^{k+1}}{(r^k{)}^p}=r^{k(1-p)+1}$

$p<1$: ${\lim }_{k\to \infty }{r}^{k(1-p)+1}=0$

$p=1$: ${\lim }_{k\to \infty }{r}^{k(1-p)+1}=r<1$

Example 3

$x^k=r^{q^k}$, $q>1$, $0<r<1$

$x^{\ast }=0$

$\frac{r^{q^{k+1}}}{(r^{q^k}{)}^p}=r^{q^{k+1}-p{q}^k}=r^{(q-p)q^k}$

$p<q$: ${\lim }_{k\to \infty }{r}^{(q-p)q^k}=0$

$p=q$: ${\lim }_{k\to \infty }{r}^{(q-p)q^k}=1$

Example 4

$x^k=1$

$x^k\to x^0=1$

$\frac{x^{k+1}-1}{(x^k-1{)}^p}=0$, $p=\infty$

Theorem

$||x^{k+1}-x^{\ast }||=O(||x^k-x^{\ast }|{|}^p)$

For large $k$: $\exists c\in \mathbb{R}$: $||x^{k+1}-x^{\ast }||\le c||x^k-x^{\ast }|{|}^p$

Thm: $x^k-x^{\ast }$ If $||x^{k+1}-x^{\ast }||=O(||x^k-x^{\ast }|{|}^p)$, then the order of convergence is at least $p$.

Pf: For large $k$, $\exists c$: $\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}\le c$

$\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^s}=\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^p}||x^k-x^{\ast }|{|}^{p-s}\le c||x^k-x^{\ast }|{|}^{p-s}$

If $s$ is the order of convergence, ${\lim }_{k\to \infty }\frac{||x^{k+1}-x^{\ast }||}{||x^k-x^{\ast }|{|}^s}>0$

$\Rightarrow c{\lim }_{k\to \infty }||x^k-x^{\ast }|{|}^{p-s}>0$

$\because {\lim }_{k\to \infty }||x^k-x^{\ast }||=0$

$\therefore$ if $p>s$, $c{\lim }_{k\to \infty }||x^k-x^{\ast }|{|}^{p-s}=0$

$\Rightarrow s\ge p$

Theorem

Thm: Stepest Decent: the order of convergence $\ge 1$

Pf: $Q$: ${\lambda }_{max}(Q)>{\lambda }_{min}(Q)>0$

Suffices to prove: $\exists c,||x^{k+1}-x^{\ast }||\ge c||x^k-x^{\ast }||$

$\because V(x^{k+1})=\frac{1}{2}(x^{k+1}-x^{\ast }{)}^{T}Q(x^{k+1}-x^{\ast })\le \frac{{\lambda }_{max}(Q)}{2}||x^{k+1}-x^{\ast }|{|}^2$

$V(x^{\ast })\ge \frac{{\lambda }_{min}(Q)}{2}||x^k-x^{\ast }|{|}^2$

$||x^{k+1}-x^{\ast }||\ge \sqrt{(1-r^k)\frac{{\lambda }_{min}(Q)}{{\lambda }_{max}(Q)}}||x^k-x^{\ast }||$

To prove : $r^k<1\Rightarrow g^k$ is not eigenvectir of $Q\iff r^k<1$

Newton Method

$f\in C^2$

$x^{\ast }$ FONC$\Rightarrow \nabla f(x^{\ast })=0$

$x^{k+1}=x^k-\frac{f^{\prime }(x^k)}{f^{\prime \prime }(x^k)}\Rightarrow x^{k+1}=x^k-[F(x^k){]}^{\prime }\nabla f(x^{\ast })$

优缺点

优点

Pro: simple, convergen order

缺点

Con: $F(x^k)<0$,

even if $F(x^k)>0$, not decent.

Compute $F^{-1}(x^k)$

Convergence Order

Example

$f(x)=\frac{1}{2}{x}^{T}Qx-b^{T}x$

$\nabla f(x)=Qx-b$

$F(x)=Q$

$x^1=x^0-Q^{-1}(Q{x}^0-b)=x^0-x^0+Q^{-1}b=Q^{-1}b=x^{\ast }$

Theorem

Thm: $f\in C^3$, $x^{\ast }$: $\nabla f(x^{\ast })=0$ and $F(x^{\ast })$ inventible. Then, for all $x^0$ sufficiently close to $x^{\ast }$, $x^{\ast }$ converges to $x^{\ast }$ with an order at least 2.

Pf: To prove: $||x^{k+1}-x^{\ast }||=O(||x^k-x^{\ast }|{|}^2)$

$||x^1-x^{\ast }||=||x^0-x^{\ast }-F^{-1}(x^0)\nabla f(x^{\ast })||=||F^{-1}(x^0)(F(x^0)(x^0-x^{\ast })-\nabla f(x^0))||\le ||F^{-1}(x^0)||\cdot ||(F(x^0)(x^0-x^{\ast })-\nabla f(x^0))||$

$\because F(x^{\ast })$ inventible, $f\in C^3$, $x^0$ sufficiently close to $x^{\ast }$, $||F^{-1}(x^{\ast })||$ constant $\Rightarrow ||F^{-1}(x^0)||<c_2$ for some $c_2\in \mathbb{R}$

Taylor expansion of $\nabla f(x)$: $\nabla f(x)-\nabla f(x^{\ast })=F(x^0)(x-x^0)+O(||x-x^0|{|}^2)$

$\forall x,||x-x^{\ast }||<ϵ:||\nabla f(x)-\nabla f(x^0)-F(x^0)(x-x^0)||\le c_1||x-x^0|{|}^2$ for some $c_1\in \mathbb{R}$

If x ∗ ∈ { x : ∣ ∣ x − x ∗ ∣ ∣ < ϵ } x^*\in \{x:||x-x^*||<\epsilon\} x∗∈{x:∣∣x−x∗∣∣<ϵ}: $||\nabla f(x^{\ast })-\nabla f(x^0)-F(x^0)(x^{\ast }-x^0)||\le c_1||x^{\ast }-x^0|{|}^2$

$||F(x^0)(x^0-x^{\ast })-\nabla f(x^0)||\le c_1||x^{\ast }-x^0|{|}^2$

$||x^1-x^{\ast }||\le c_1{c}_2||x^0-x^{\ast }|{|}^2$

$||x^{k+1}-x^{\ast }||\le c_1{c}_2||x^k-x^{\ast }|{|}^2$

Let $0<x<1$, choose $x^0$ satisfy $||x^0-x^{\ast }||\le \frac{\alpha }{c_1{c}_2}$

${\lim }_{k\to \infty }||x^k-x^{\ast }||\le {\alpha }^k||x^0-x^{\ast }||=0$

Theorem

Thm: $x^k$: Sequence generated by Newton’s Method. If $F(x^k)>0$ and $\nabla f(x^k)\ne 0$, then for $d^k=-F^{-1}(x^k)\nabla f(x^k)$, there exsits an $\overline{\alpha }>0$ s.t. $\forall \alpha \in (0,\overline{\alpha }):f(x^k+\alpha d^k)<f(x^k)$.

Pf: Let $\varphi (\alpha )=f(x^k+\alpha d^k)$

${\varphi }^{\prime }(\alpha )=\nabla f(x^k+\alpha d^k{)}^T{d}^k$

${\varphi }^{\prime }(0)=\nabla f(x^k{)}^T{d}^k=-\nabla f(x^k{)}^{T}F(x^k)\nabla f(x^k)$

$\because F(x^k)>0,\nabla f(x^k)\ne 0$

$\therefore {\varphi }^{\prime }(0)<0$

$\therefore \exists \overline{\alpha }>0$ s.t. $\forall \alpha \in (0,\overline{\alpha }):\varphi (\alpha )<\varphi (0)$

$\forall \alpha \in (0,\overline{\alpha }):f(x^k+\alpha d^k)<f(x^k)$

Modification

$x^{k+1}=x^k-{\alpha }^k{F}^{-1}(x^k)\nabla f(x^k)$, where ${\alpha }^k=argminf(x^k-\alpha F^{-1}(x^k)\nabla f(x^k)$

$F(x^k)$ not positive definite.

Let ${\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n$ be the eigenvalues of $F(x^k)$ corresponding to eigenvalues $v_1,v_2,\cdots ,v_n$

Consider $G=F(x^k)+\mu I_n,\mu >0$

$G{v}_i=(F(x^k)+\mu I_n)v_i=F(x^k)v_i+\mu v_i={\lambda }_i{v}_i+\mu v_i=({\lambda }_i+\mu )v_i\Rightarrow v_i$ eigrnvector of $G$ corresponding to ${\lambda }_i+\mu$

Choose $\mu$ large enough, s.t. ${\lambda }_i+\mu >0$ $\forall i\Rightarrow G$ positive definite

Modification: $x^{k+1}=x^k-{\alpha }^k(F(x^k)+\mu I_n{)}^{-1}\nabla f(x^k)$

${\alpha }^k=argminf(x^k-\alpha (F(x^k)+\mu I_n{)}^{-1}\nabla f(x^k))$

Conjugate Method

Def: $Q$: symmetric matrix from ${\mathbb{R}}^{n\times n}$. $d_0,\cdots ,d_m$ are Q-conjugate, if $\forall i\ne j$： $d_i^{T}Q{d}_j=0$

Orthogonal: $x^{T}y=0\stackrel{\Delta }{=}{x}^T{I}_{n}y$

Lemma

Lem: $Q$ symmetric, positive definite.$d_0,\cdots ,d_k$: non-zero, Q-conjugate. Then $d_0,\cdots ,d_k$ linearly independent.

$d_j^{T}Q(a_0{d}_0+\cdots +a_k{d}_k)=0$

$a_j{d}_j^{T}Q{d}_j=0$

$\because a>0,d_j\ne 0$

$\therefore a_j=0$

Conjugate Direction Algorithm

Input: $f(x)=\frac{1}{2}{x}^{T}Qx-b^{T}x,x_0,d_0,\cdots ,d_{n-1}$: Q-conjugate

$g^k=\nabla f(x^k)=Q{x}^k-b$

${\alpha }^k=-\frac{{g^k}^T{d}_k}{d_k^{T}Q{d}_k}$

$x^{k+1}=x^k+{\alpha }^k{d}_k$

Thm: For any $x_0$, CDA converges to $x^{\ast }$ in $n$ steps.

Pf: $\because d_0,\cdots ,d_{n-1}$ Q-conjugate

$\therefore d_0,\cdots ,d_{n-1}$ linearly independent

$\Rightarrow \exists {\beta }_0,\cdots ,{\beta }_{n-1}:x^{\ast }-x^0={\beta }_0{d}_0+\cdots +{\beta }_{n-1}{d}_{n-1}$

$\Rightarrow d_k^{T}Q(x^{\ast }-x^0)={\beta }_k{d}_k^{T}Q{d}_k$

$\Rightarrow {\beta }_k=\frac{d_k^{T}Q(x^{\ast }-x^0)}{d_k^{T}Q{d}_k}=-\frac{d_k^T{g}^k}{d_k^{T}Q{d}_k}={\alpha }^k$

$x^k=x^0+{\alpha }^0{d}_0+\cdots +{\alpha }^{k-1}{d}_{k-1}$

$d_k^{T}Q(x^{\ast }-x^0)=d_k^{T}Q(x^{\ast }-x^k+x^k-x^0)=d_k^{T}Q(x^{\ast }-x^k)=d_k^T(Q{x}^{\ast }-Q{x}^k)=d_k^T(b-Q{x}^k)=-d_k^T{g}^k$

Example

$$Q=\left[\begin{array}{ccc}3& 0& 1\\ 0& 4& 2\\ 1& 2& 3\end{array}\right]$$

Compute $d_0,d_1,d_2$
$$d_0=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$$

$$d_0^{T}Q{d}_1=[1,0,0]\left[\begin{array}{ccc}3& 0& 1\\ 0& 4& 2\\ 1& 2& 3\end{array}\right]\left[\begin{array}{c}{d}_1^1\\ d_1^2\\ d_1^3\end{array}\right]=3d_1^1+d_1^3=0$$
$$d_1=\left[\begin{array}{c}1\\ 0\\ -3\end{array}\right]$$
注：此处展示了如何获取Q-conjugate的向量$d_0$和$d_1$。先选取一个比较简单的$d_0$，然后代入$d_0^{T}Q{d}_1=0$，算出关于$d_1$的关系式$3d_1^1+d_1^3=0$，然后选定$d_1=[1,0,-3{]}^T$。
$$f(x)=\frac{1}{2}{x}^T\left[\begin{array}{cc}4& 2\\ 2& 2\end{array}\right]x-[-1,1]x$$
$$g(x)=\left[\begin{array}{cc}4& 2\\ 2& 2\end{array}\right]x-\left[\begin{array}{c}-1\\ 1\end{array}\right]$$
$$x^0=\left[\begin{array}{c}0\\ 0\end{array}\right]$$
$$d_0=\left[\begin{array}{c}1\\ 0\end{array}\right]$$
$$d_1=\left[\begin{array}{c}-\frac{3}{8}\\ \frac{3}{4}\end{array}\right]$$
$$g^0=\left[\begin{array}{c}1\\ -1\end{array}\right]$$
$${\alpha }^0=\frac{-{g^0}^T{d}_0}{d_0^{T}Q{d}_0}=\frac{-[1,-1]\left[\begin{array}{c}1\\ 0\end{array}\right]}{[1,0]\left[\begin{array}{cc}4& 2\\ 2& 2\end{array}\right]\left[\begin{array}{c}1\\ 0\end{array}\right]}=-\frac{1}{4}$$
$$x^1=x^0+{\alpha }^0{d}_0=\left[\begin{array}{c}-\frac{1}{4}\\ 0\end{array}\right]$$
$$g^1=\left[\begin{array}{c}0\\ -\frac{2}{3}\end{array}\right]$$
${\alpha }^1=2$
$$x^2=\left[\begin{array}{c}-1\\ \frac{3}{2}\end{array}\right]$$
$f(x^2)=0$
注：二次函数在$n$次迭代后必取到最值。

总结

本节课首先延续上节课的梯度方法，做了一些理论上的分析。然后提出了收敛速度的概念，从而可以进一步比较各个方法的收敛速度。最后介绍了牛顿法和共轭方向法。