高级优化理论与方法（六）

Quasi-Newton Methods

Rank-One Algorithm

核心公式

$d^k=-H^k{g}^k$
${\alpha }^k=argminf(x^k+\alpha d^k)$
$x^{k+1}=x^k+{\alpha }^k{d}^k$
$H^{k+1}=H^k+\frac{(\Delta x^k-H^k\Delta g^k)(\Delta x^k-H^k\Delta g^k{)}^T}{\Delta {g^k}^T(\Delta x^k-H^k\Delta g^k)}$

Theorem

Quadratic: $H^{k+1}\Delta g^i=\Delta x^i,\forall i\le k$

Example

$f(x)=x_1^2+\frac{1}{2}{x}_2^2+3=\frac{1}{2}{x}^T\left[\begin{array}{cc}2& 0\\ 0& 1\end{array}\right]x+3$

$g^k=\left[\begin{array}{cc}2& 0\\ 0& 1\end{array}\right]x^k$

$x^0=\left[\begin{array}{c}1\\ 2\end{array}\right]$

$H^0=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

$d^0=-H^0{g}^0=\left[\begin{array}{c}-2\\ -2\end{array}\right]$

${\alpha }^0=\frac{{g^0}^T{d}^0}{{d^0}^{T}Q{d}^0}=\frac{2}{3}$

$x^1=x^0+{\alpha }^0{d}^0=\left[\begin{array}{c}-\frac{1}{3}\\ \frac{2}{3}\end{array}\right]$

$H^1=\left[\begin{array}{cc}\frac{1}{2}& 0\\ 0& 1\end{array}\right]$

Problems

$H^k$ not positive definite$\Rightarrow$ not descent.

$\Delta {g^k}^T(\Delta x^k-H^k\Delta g^k)$ too small, close to zero.

DFP-Algorithm

算法步骤

IN: $x^0,H^0$（positive definite, symmetric）

1. $k:=0$
2. If $g^k=0$, then stop; else $d^k=-H^k{g}^k$
3. compute ${\alpha }^k=argminf(x^k+\alpha d^k),x^{k+1}=x^k+{\alpha }^k{d}^k$
4. compute $\Delta x^k={\alpha }^k{d}^k,\Delta g^k=g^{k+1}-g^k,H^{k+1}=H^k+\frac{\Delta x^k{\Delta x^k}^T}{{\Delta x^k}^T\Delta g^k}-\frac{(H^k\Delta g^k)(H^k\Delta g^k{)}^T}{{\Delta g^k}^T{H}^k\Delta g^k}$
5. $k$++,goto 2

Theorem

Applying DFP to quadratic functions: $H^{k+1}\Delta g^i=\Delta x^i,\forall i\le k$

Example

$f(x)=\frac{1}{2}{x}^T\left[\begin{array}{cc}4& 2\\ 2& 2\end{array}\right]x-x^T\left[\begin{array}{c}-1\\ 1\end{array}\right]$

$x^0=\left[\begin{array}{c}0\\ 0\end{array}\right]$

$H^0=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

$g^k=\left[\begin{array}{cc}4& 2\\ 2& 2\end{array}\right]x-\left[\begin{array}{c}-1\\ 1\end{array}\right]$

$g^0=\left[\begin{array}{c}1\\ -1\end{array}\right]$

$d^0=-H^0{g}^0=\left[\begin{array}{c}-1\\ 1\end{array}\right]$

${\alpha }^0=1$

$x^1=x^0+{\alpha }^0{d}^0=\left[\begin{array}{c}-1\\ 1\end{array}\right]$

$g^1=\left[\begin{array}{c}-1\\ 1\end{array}\right]$

$\Delta x^0=\left[\begin{array}{c}-1\\ 1\end{array}\right]$

$\Delta g^0=\left[\begin{array}{c}-2\\ 0\end{array}\right]$

$\Delta x^0{\Delta x^0}^T=\left[\begin{array}{c}-1\\ 1\end{array}\right][-1,1]=\left[\begin{array}{cc}1& -1\\ -1& 1\end{array}\right]$

${\Delta x^0}^T\Delta g^0=2$

$H^0\Delta g^0=\left[\begin{array}{c}-2\\ 0\end{array}\right]$

$(H^0\Delta g^0)(H^0\Delta g^0{)}^T=\left[\begin{array}{cc}4& 0\\ 0& 0\end{array}\right]$

${\Delta g^0}^T{H}^0\Delta g^0=4$

$H^1=H^0+\frac{\Delta x^0{\Delta x^0}^T}{{\Delta x^0}^T\Delta g^0}-\frac{(H^0\Delta g^0)(H^0\Delta g^0{)}^T}{{\Delta g^0}^T{H}^0\Delta g^0}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]+\frac{\left[\begin{array}{cc}1& -1\\ -1& 1\end{array}\right]}{2}-\frac{\left[\begin{array}{cc}4& 0\\ 0& 0\end{array}\right]}{4}=\left[\begin{array}{cc}\frac{1}{2}& -\frac{1}{2}\\ -\frac{1}{2}& \frac{3}{2}\end{array}\right]$

$d^1=-H^1{g}^1=\left[\begin{array}{c}0\\ 1\end{array}\right]$

${\alpha }^1=\frac{1}{2}$

$x^2=x^1+{\alpha }^1{d}^1=\left[\begin{array}{c}-1\\ \frac{3}{2}\end{array}\right]=x^{\ast }$

Theorem

If $g^k\ne 0,H^k$ being positive definite implies $H^{k+1}$ being positive definite.

BFGS

前两个算法的核心是用$H^k$来表示$F(x^k{)}^{-1}$，即二阶导的逆。两种算法的不同之处仅仅在于用不同的公式来计算$H^k$。BFGS是用$G^{k+1}$来表示$F(x^k)$，从而推导出$H^k$。BFGS的收敛速度在某些情况下较其它两种方法快，但是在非二次函数的情形下难以证明。这里只给出$H^k$的公式，不做过多展开。
$H^{k+1}=H^k+(1+\frac{{\Delta g^k}^T{H}^k\Delta g^k}{{\Delta g^k}^T\Delta x^k})\frac{\Delta x^k{\Delta x^k}^T}{\Delta x^k\Delta g^k}-\frac{H^k\Delta g^k{\Delta x^k}^T+(H^k\Delta g^k{\Delta x^k}^T{)}^T}{{\Delta g^k}^T\Delta x^k}$

Solving Linear Equations

$Ax=b,A\in {\mathbb{R}}^{m\times n},x\in {\mathbb{R}}^n,b\in {\mathbb{R}}^m$

$\{\begin{array}{l}{a}_{11}{x}_1+a_{12}{x}_2+\cdots +a_{1n}{x}_n=b_1\\ \cdots \\ a_{m1}{x}_1+a_{m2}{x}_2+\cdots +a_{mn}{x}_n=b_n\end{array}$

$n$代表变量个数，$m$代表方程个数。默认这$m$个方程线性无关。

Case 1

$m\ge n,rankA=n$

可以把问题看成用$m$个数据点来拟合有$n$个变量的函数，此时可能不存在严格的解。于是我们不要求严格地解这个方程，而是把解方程问题转化为数据拟合问题（Data fitting）。

在数据拟合问题中，我们一般采取的是Least Square fitting，即求$min||Ax-b|{|}^2$

Lemma

$rankA=n\iff rank{A}^{T}A=n$

Theorem

The unique vector $x^{\ast }$ minimizing $||Ax-b|{|}^2$ is given by $x^{\ast }=(A^{T}A{)}^{-1}{A}^{T}b$

Proposition

$A=[a_1,a_2,\cdots ,a_n]$

$span[a_1,a_2,\cdots ,a_n]$表示由这些向量张成的空间。

设$b$在该空间上的投影为$v$，则$v=A{x}^{\ast }=A(A^{T}A{)}^{-1}{A}^{T}b$

注：该命题是通过几何方法给出了 $x^{\ast }=(A^{T}A{)}^{-1}{A}^{T}b$的一个证明。

Example

$i$	1	2	3
$t_i$	2	3	4
$y_i$	3	4	15

$y=mt+c$

$A=\left[\begin{array}{cc}2& 1\\ 3& 1\\ 4& 1\end{array}\right]$

$b=\left[\begin{array}{c}3\\ 4\\ 15\end{array}\right]$

$x^{\ast }=\left[\begin{array}{c}6\\ -\frac{32}{3}\end{array}\right]$

注：此时，$x=\left[\begin{array}{c}m\\ c\end{array}\right]$

Recursive Least-Square Algorithm

根据原始数据，已经计算出了$x^{\ast }$的值，那么此时如果给出一些新的数据，如何在借助已知值的基础上快速算出新的$x^{\ast }$，就是这个算法要解决的问题。
$min||A_{0}x-b_0|{|}^2\Leftarrow x_0=G_0^{-1}{A}_0^T{b}_0$
$G_0=A_0^{-1}{A}_0$

$min{\left|\left|\left[\begin{array}{c}{A}_0\\ A_1\end{array}\right]x-\left[\begin{array}{c}{b}_0\\ b_1\end{array}\right]\right|\right|}^2$

$x_1=G_1^{-1}{\left[\begin{array}{c}{A}_0\\ A_1\end{array}\right]}^T\left[\begin{array}{c}{b}_0\\ b_1\end{array}\right]$

$G_1={\left[\begin{array}{c}{A}_0\\ A_1\end{array}\right]}^T\left[\begin{array}{c}{A}_0\\ A_1\end{array}\right]=A_0^T{A}_0+A_1^T{A}_1$

Goal: express $x_1$ as a function of $x_0,A_1,b_1,G_0$

$x_1=x_0+G_1^{-1}{A}_1^T(b_1-A_1{x}_0)$

$k\to k+1:\{\begin{array}{l}{G}_{k+1}=G_k+A_{k+1}^T{A}_{k+1}\\ x_{k+1}=x_k+G_{k+1}^{-1}{A}_{k+1}^T(b_{k+1}-A_{k+1}{x}_k)\end{array}$

Let $G_{k+1}^{-1}=P_{k+1},G_k^{-1}=P_k$

$P_{k+1}=P_k-P_k{A}_{k+1}^T(I+A_{k+1}{P}_k{A}_{k+1}^T{)}^{-1}{A}_{k+1}{P}_k$

$x_{k+1}=x_k+P_k{A}_{k+1}^T(b_{k+1}-A_{k+1}{x}_k)$

Lemma

$X$ invertible matrix, $U$ and $V$ satisfy $I+V{X}^{-1}U$ being invertible, then $(X+UV{)}^{-1}=X^{-1}-(X^{-1}U)(I+V{X}^{-1}U{)}^{-1}(V{X}^{-1})$

总结

本节课先就上节课没讲完的拟牛顿法接着讲，回顾了秩为1的修正方法。由于该方法存在一些问题，为了解决这些问题，又介绍了DFP算法。接着又简要介绍了收敛速度较快的BFGS算法。接着开始了一个新的主题——解线性方程组。由于问题的复杂性，我们讲问题分成两种情况来分类讨论，即$m\ge n$和$m<n$两种情况。目前只讲了第一种情况。第一种情况本质上是一个数据拟合问题，根据数据，求得$min||Ax-b|{|}^2$。然后，介绍了直接的求解公式。最后，对于如何在充分利用已知数据的基础上，对新数据进行拟合，又提出了递归最小二乘法。