高级优化理论与方法（五）

Conjugate Direction Algorithm

IN: $x_0,d_0,\cdots ,d_{n-1}$, Q-conjugate
Property: $x^{k+1}=x^k+{\alpha }^k{d}_k,{\alpha }^k=-\frac{{g^k}^T{d}_k}{{d_k}^{T}Q{d}_k}$

Lemma 1

In the conjugate direction algorithm, ${g^{k+1}}^T{d}_i=0,\forall 0\le k\le n-1,\forall 0\le i\le k$

Lemma 2

Lemma: $f(x^{k+1})=minf(x^0+{\sum }_{i=0}^k{\alpha }^i{d}_i)$

Pf: Let $D=[d_0,\cdots ,d_{k-1}]$ be an nxk-matrix,

$x(\alpha )=x^0+D^k\alpha ,\alpha \in {\mathbb{R}}^k$

${\varphi }^k(\alpha )=f(x(\alpha ))=f(x^0+D^k\alpha )$

$D{\varphi }^k(\alpha )=\nabla f(x^0+D^k\alpha {)}^T{D}^k$

Let $\overline{\alpha }=[{\alpha }^0,{\alpha }^1,\cdots ,{\alpha }^{k-1}],{\alpha }^i$ generated by algorithm.

$D{\varphi }^k(\overline{\alpha })=\nabla f(x^0+D^k\overline{\alpha }{)}^T{D}^k=\nabla f(x^k)D^k={g^k}^T{D}^k$

$\because$ Property①$\Rightarrow {g^k}^T{d}_i=0,\forall i\le k-1$

$\therefore D{\varphi }^k(\overline{\alpha })=0$, satisfies FONC

Conjugate Gradient Algorithm

IN: $f(x)=\frac{1}{2}{x}^{T}Qx-b^{T}x,x^0$

1. $k=0$
2. compute $g^0=\nabla f(x^0)$, If $g^0=0$, then stop; else $d_0=-g^0$
3. ${\alpha }^k=-\frac{g^k{d}_k}{{d_k}^{T}Q{d}_k}$
4. $x^{k+1}=x^k+{\alpha }^k{d}_k$
5. $g^{k+1}=\nabla f(x^{k+1})$, If $g^{k+1}=0$ then stop;
6. ${\beta }^k=\frac{{g^{k+1}}^{T}Q{d}_k}{{d_k}^{T}Q{d}_k}$
7. $d^{k+1}=-g^{k+1}+{\beta }^k{d}_k$
8. $k$++, goto 3

Theorem

Thm: $d_0,\cdots ,d_{n-1}$ computed in conjugate Gradient Algorithm are Q-conjugate.

Pf:
$n=2:$

${d_0}^{T}Q{d}_1=d_0^{T}Q(-g^1+{\beta }^0{d}_0)={d_0}^{T}Q(-g^1+\frac{{g^1}^{T}Q{d}_0}{{d_0}^{T}Q{d}_0}{d}_0)=0$

$n=k+1:$

To prove ${d_{k+1}}^{T}Q{d}_j=0,\forall j\le k$

(Known: ${d_k}^{T}Q{d}_j=0,\forall j\le k-1$)

${d_{k+1}}^{T}Q{d}_j=(-g^{k+1}+{\beta }^k{d}_k)Q{d}_j=-g^{k+1}Q{d}_j+{\beta }^k{d}_{k}Q{d}_j=-g^{k+1}Q{d}_j$

$\because g^{j+1}=Q{x}^{j+1}-b=Q{x}^j+{\alpha }^{j}Q{d}_j-b=g^j+{\alpha }^{j}Q{d}_j$

$\therefore {d_{k+1}}^{T}Q{d}_j=-g^{k+1}\frac{g^{j+1}-g^j}{{\alpha }^j},j\le k-1$

$\therefore \forall j\le k:d_j=-g^j+{\beta }^{j-1}{d}_{j-1}$

$\therefore 0={g^{k+1}}^T{d}_j=-{g^{k+1}}^T{g}^j+{\beta }^{j-1}{g^{k+1}}^T{d}_{j-1}=0$

$\therefore {g^{k+1}}^T{g}^j=0,\forall j\le k$

$\therefore {d_{k+1}}^{T}Q{d}_j=0,\forall j\le k-1$

${d_{k+1}}^{T}Q{d}_k=0$ (The same as n=2)

Example

$f(x)=\frac{3}{2}{x}_1^2+2x_2^2+\frac{3}{2}{x}_3^2+x_1{x}_2+2x_2{x}_3-3x_1-x_3$

$$Q=\left[\begin{array}{ccc}3& 0& 1\\ 0& 4& 2\\ 1& 2& 3\end{array}\right]$$
$$b=\left[\begin{array}{c}3\\ 0\\ 1\end{array}\right]$$
$$x^0=\left[\begin{array}{c}0\\ 0\\ 0\end{array}\right]$$
$$g^0=Q{x}^0-b=\left[\begin{array}{c}-3\\ 0\\ 1\end{array}\right]$$
$$d_0=\left[\begin{array}{c}3\\ 0\\ 1\end{array}\right]$$
$${\alpha }^0=-\frac{[-3,0,-1]\left[\begin{array}{c}3\\ 0\\ 1\end{array}\right]}{[3,0,1]Q\left[\begin{array}{c}3\\ 0\\ 1\end{array}\right]}=\frac{10}{36}=0.2778$$
$$x^1=x^0+{\alpha }^0{d}_0=\left[\begin{array}{c}0.8373\\ 0\\ 0.2778\end{array}\right]$$
$\cdots$
$$x^3=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$$

Non-Quadratic function f

Problems: Computation of ${\alpha }^k,{\beta }^k$
solutions: ${\alpha }^k=argmi{n}_{x>0}f(x^k+\alpha d_k)\to$ one-dimentional search

${\beta }^k$

Hestens-Stiefel-formula:

$x^{k+1}=x^k+{\alpha }^k{d}_k\Rightarrow Q{x}^{k+1}-b=Q{x}^k-b+{\alpha }^{k}Q{d}_k\Rightarrow g^{k+1}-g^k={\alpha }^{k}Q{d}_k\Rightarrow Q{d}_k=\frac{g^{k+1}-g^k}{{\alpha }^k}$

${\beta }^k=\frac{{g^k}^{T}Q{d}_k}{{d_k}^{T}Q{d}_k}=\frac{{g^k}^T(g^{k+1}-g^k)}{{d_k}^T(g^{k+1}-g^k)}$

Polak-Ribi$\grave{e}$re-formula:

$\because {g^k}^T{d}_k=-{g^k}^T{g}^k+{g_k}^T{\beta }^{k-1}{d}_{k-1}=-{g^k}^T{g}^k$

$\therefore {\beta }^k=\frac{{g^{k+1}}^T(g^{k+1}-g^k)}{{g^k}^T{g}^k}$

Fletcher-Reeves-Formula

$\because 0={g^{k+1}}^T{d}_k=-{g^{k+1}}^T{g}^k+{\beta }^k{g^{k+1}}^T{d}_{k-1}\Rightarrow {g^{k+1}}^T{g}^k=0$

$\therefore {\beta }^k=\frac{{g^{k+1}}^T{g}^{k+1}}{{g^k}^T{g}^k}$

Quasi-Newton Methods

牛顿法回顾

$x^{k+1}=x^k-{F(x^k)}^{-1}\nabla f(x^k),f\in C^3$

牛顿法优点：简单，适用性广，收敛速度快

Problems

not descent, even if ${F(x^k)}^{-1}>0\Rightarrow x^{k+1}=x^k-{\alpha }^k{F(x^k)}^{-1}\nabla f(x^k)$, where ${\alpha }^k=argmi{n}_{\alpha >0}f(x^k-{\alpha }^k{F(x^k)}^{-1}\nabla f(x^k))$

not positive definite: $\Rightarrow G=({F(x^k)}^{-1}+\mu I_n)$

computation of ${F(x^k)}^{-1}\nabla f(x^k)$

Solution:

Construct $H^k$(real-valued, positive definite, summetric)

$g^k=\nabla f(x^k)$

$d^k=-H^k{g}^k$

${\alpha }^k=argminf(x^k+\alpha d^k)$

$x^{k+1}=x^k+{\alpha }^k{d}^k$

Theorem

If $g^k\ne 0$ and $H^k$: nxn-metric(symmetric, positive definite), then $f(x^{k+1})<f(x^k)$

Theorem

Quadratic functions: $f(x)=\frac{1}{2}{x}^{T}Qx-b^{T}x$

Thm: Applying Quasi-Newton Method to a quadratic function with $Q=Q^T$ s.t.$H^{k+1}\Delta g_i=\Delta x^i,\forall 0\le i\le k$, where $\Delta g^i=g^{i+1}-g^i,\Delta x^i=x^{i+1}-x^i$, if ${\alpha }^i\ne 0$ for all $0\le i\le k$, then $d^0,\cdots ,d^{k+1}$ are Q-conjugate.

Corollary

Applying Quasi-Newton to quadratic functions, $n$ step converges.

computation of H

Rank-One-Correction

$H^{k+1}=H^k+a^k{z}^k{z^k}^T,a^k\in \mathbb{R},z^k\in \mathbb{R}$

$$z^k{z^k}^T=\left[\begin{array}{c}{z}_1^k\\ \cdots \\ z_n^k\end{array}\right]\times [z_1^k,\cdots ,z_n^k]=\left[\begin{array}{ccc}{z}_1^k{z}_1^k& \cdots & z_1^k{z}_n^k\\ \cdots & & \cdots \\ z_n^k{z}_1^k& \cdots & z_n^k{z}_n^k\end{array}\right]$$
$rank(z^k{z^k}^T)=1$

$H^{k+1}$ as a function of $H^k,\Delta g^k,\Delta x^k$

$H^{k+1}=H^k+\frac{(\Delta x^k-H^k\Delta g^k)(\Delta x^k-H^k\Delta g^k{)}^T}{\Delta {g^k}^T(\Delta x^k-H^k\Delta g^k)}$

Rank-One-Correction Algorithm

IN: $x^0,H^0$

1. $k:=0$
2. If $g^k=0$, then stop; else $d^k=-H^k{g}^k$
3. compute ${\alpha }^k=argminf(x^k+\alpha d^k),x^{k+1}=x^k+{\alpha }^k{d}^k$
4. compute $\Delta x^k={\alpha }^k{d}^k,\Delta g^k=g^{k+1}-g^k,H^{k+1}=H^k+\frac{(\Delta x^k-H^k\Delta g^k)(\Delta x^k-H^k\Delta g^k{)}^T}{\Delta {g^k}^T(\Delta x^k-H^k\Delta g^k)}$
5. $k$++,goto 2

Theorem

Apply Rank-One to a quadratic function, with $Q=Q^T$, we have $H^{k+1}\Delta g^i=\Delta x^i,\forall i\le k$

Problems

$H^k$ positive definite?
$\Delta {g^k}^T(\Delta {x^k}^T-H^k\Delta g^k)$ too small?

DFP algorithm

$H^{k+1}=H^k+\frac{\Delta x^k\Delta {x^k}^T}{\Delta {x^k}^T\Delta g^k}-\frac{(H^k\Delta g^k)(H^k\Delta g^k{)}^T}{\Delta {g^k}^T{H}^k\Delta g^k}$

Theorem

Apply to quadratic functions: $H^{k+1}\Delta g^i=\Delta x^i,\forall i\le k$

总结

为了使整体篇幅显得简洁，从这节课开始，逐渐删去一些繁琐的证明。这节课从上节课提到的共轭方向法讲起，又介绍了共轭梯度法。牛顿法存在一些缺陷，为了改进，提出了拟牛顿法。关于拟牛顿法中的$H$如何计算，又有一些不同的方法。这节课介绍了秩为1的修正方法，以及DFP算法。