无约束优化的基础

1、梯度与黑塞矩阵

定义1：设$n$元函数$f(x)$对自变量$x=(x_1,x_2,\dots ,x_n{)}^T$各自分量$x_i$的一阶偏导数为
$$\frac{\partial f(x)}{\partial x_i},i=1,2,\dots ,n$$
那么称向量
$$\nabla f(x)=(\frac{\partial f(x)}{\partial x_1},\frac{\partial f(x)}{\partial x_2},\dots ,\frac{\partial f(x)}{\partial x_n}{)}^T$$
为函数$f(x)$在$x$处的一阶导数或梯度

定义2：设$n$元函数$f(x)$对自变量$x=(x_1,x_2,\dots ,x_n{)}^T$各自分量$x_i$的二阶偏导数为
$$\frac{{\partial }^{2}f(x)}{\partial x_i{x}_j},i,j=1,2,\dots ,n$$
那么称矩阵
$${\nabla }^{2}f(x)=\left(\begin{array}{cccc}\frac{{\partial }^{2}f(x)}{\partial x_1^2}& \frac{{\partial }^{2}f(x)}{\partial x_1{x}_2}& \dots & \frac{{\partial }^{2}f(x)}{\partial x_i{x}_n}\\ \frac{{\partial }^{2}f(x)}{\partial x_2{x}_1}& \frac{{\partial }^{2}f(x)}{\partial x_2^2}& \dots & \frac{{\partial }^{2}f(x)}{\partial x_2{x}_n}\\ ⋮& ⋮& & ⋮\\ \frac{{\partial }^{2}f(x)}{\partial x_n{x}_1}& \frac{{\partial }^{2}f(x)}{\partial x_n{x}_2}& \cdots & \frac{{\partial }^{2}f(x)}{\partial x_n^2}\end{array}\right)$$
为函数$f(x)$在$x$处的二阶导数矩阵或$Hessen$矩阵

定义3：如果$f(x)$梯度的所有分量函数在$x$都连续，则称$f(x)$在$x$连续可微；如果$f(x)$的$Hessen$矩阵的各个分量函数都连续，则$f(x)$在$x$二阶连续可微。

定义4：如果$f$在开集$D$上每一点都连续可微，则称$f$在$D$上一阶连续可微；如果如果$f$在开集$D$上每一点上二阶连续可微，则称$f$在$D$上二阶连续可微

注：(1)、定义4中之所以选择开集$D$，而不是闭集，是因为闭集的边界不可微
(2)、如果$f(x)$在$x$二阶连续可微，则
$$\frac{{\partial }^{2}f(x)}{\partial x_i{x}_j}=\frac{{\partial }^{2}f(x)}{\partial x_j{x}_x}$$
即表明${\nabla }^{2}f(x)$是一个对称矩阵

例1：设$A\in {\mathbb{R}}^{nxn}$,$b\in {\mathbb{R}}^n$,求二次函数
$$f(x)=\frac{1}{2}{x}^{T}Ax+b^{T}x$$
在$x$的梯度和$Hesse$矩阵
解：由于$$\begin{array}{rl}f(x)& =\frac{1}{2}\sum _{i=1}^{i=n}\sum _{j=1}^{j=n}{a}_{ij}{x}_i{x}_j+\sum _{i=1}^{i=n}{b}_i{x}_i\end{array}$$
则$k=1,2,\cdots ,n$时
$$\begin{array}{rl}\frac{\partial f(x)}{\partial x_k}& =\frac{1}{2}\sum _{j=1,j\ne k}^{j=n}{a}_{kj}{x}_j+\frac{1}{2}\sum _{i=1,i\ne k}^{i=n}{a}_{ik}{x}_i+2a_{kk}{x}_k+b_k\\ & =\frac{1}{2}\sum _{j=1}^{j=n}{a}_{kj}{x}_j+\frac{1}{2}\sum _{i=1}^{i=n}{a}_{ik}{x}_i+b_k\end{array}$$
故$\frac{\partial f(x)}{\partial x}=\frac{1}{2}(A+A^T)x+b$
和上面的分析类似，我们可以证明${\nabla }^{2}f(x)=\frac{1}{2}(A+A^T)$

2、方向导数

定义5：设$f:{\mathbb{R}}^n\to \mathbb{R}$在开集$D$上连续可微，对于$x\in {\mathbb{R}}^n,d\in {\mathbb{R}}^n$,则$f$在点$x$关于方向$d$的方向导数定义为
$$\frac{\partial f}{\partial d}(x)=\underset{\theta \to 0}{\lim }\frac{f(x+\theta d)-f(x)}{\theta }$$
上述定义的方向导数等于$\nabla f(x{)}^{T}d$,其中$\nabla f(x)$表示$f$在$x$处的梯度，$d$为方向.

注：(1)、显然方向导数是偏导数的推广，偏导数刻画的函数沿着特定方向的微商,而方向导数是任意方向的微商
(2)、就是关于这里方向导数的定义，采用的我后面参考的几本书上其中的定义，不过我当时一看觉得有问题，我当时认为方向导数应该这样定义
$$\frac{\partial f}{\partial d}(x)=\underset{\theta \to 0}{\lim }\frac{f(x+\theta d)-f(x)}{\theta \Vert d\Vert }$$
上面的范数我们就取欧式范数，或者原始的定义方向选取的是单位方向。后来在维基百科发现方向导数的定义，它认为两者都可以，仔细一想，才是我狭隘了。如果有人留意此贴，希望大家思考一下。

3、多元函数的泰勒公式

定义6：若$f(x)$在$D$上一阶连续可微，对任何$x,x+d\in D$则有
$$f(x+d)=f(x)+\nabla f(x{)}^{T}d+o(\Vert d\Vert )麦克劳林余项$$
$$f(x+d)=f(x)+\nabla f(x+td{)}^{T}d,t\in (0,1)柯西余项$$
$$f(x+d)=f(x)+{\int }_0^1\nabla f(x+td{)}^{T}ddt积分余项$$
定义7：若$f(x)$在$D$上二阶连续可微，对任何$x,x+d\in D$则有
$$f(x+d)=f(x)+\nabla f(x{)}^{T}d+\frac{1}{2}{d}^T{\nabla }^{2}f(x)d+o(\Vert d{\Vert }^2)麦克劳林余项$$
$$f(x+d)=f(x)+\nabla f(x{)}^{T}d+\frac{1}{2}{d}^T{\nabla }^{2}f(x+td)dt\in (0,1)柯西余项$$
$$f(x+d)=f(x)+\nabla f(x{)}^{T}d+{\int }_0^1(1-t)[d^T{\nabla }^{2}f(x+td)d]dt积分余项$$
证明：因为这个不是很显然
我们利用一元函数的泰勒展开证明，令$\varphi (t)=f(x+td)$
则${\varphi }^{\prime }(t)=\nabla f(x+td{)}^{T}d$,${\varphi }^{\prime \prime }(t)=d^T{\nabla }^{2}f(x+td)d$，由$\varphi (1)-\varphi (0)={\int }_0^1{\varphi }^{\prime }(t)dt$知
$$\begin{array}{rl}f(x+d)-f(x)& ={\int }_0^1[\nabla f(x+td{)}^{T}d]dt=-{\int }_0^1[\nabla f(x+td{)}^{T}d]d(1-t)\\ & =(t-1)\nabla f(x+td{)}^{T}d{|}_0^1+{\int }_0^1(1-t)d[\nabla f(x+td{)}^{T}d]\\ & =\nabla f(x{)}^{T}d+{\int }_0^1(1-t)[d^T\nabla f(x+td{)}^{T}d]dt\end{array}$$

4、两个普通公式的证明

此处是我临时起意加上的，肯定很多书上也找不到，主要的是
定义8：若$f(x)$在$开集D\in {\mathbb{R}}^n$上二阶连续可微，对任何$x,x+td\in D$则有
$$\frac{df(x+td)}{dt}=\nabla f(x+td{)}^{T}d$$
$$\frac{d^{2}f(x+td)}{d{t}^2}=d^T{\nabla }^{2}f(x+td)d$$
这个公式我们在上面的证明中用到，但是看起来却不是那么显然，我来证明一下：
证明：$$\begin{array}{rl}\frac{df(x+td)}{dt}& =\frac{df(x_1+t{d}_1,x_2+t{d}_2,\cdots ,x_n+t{d}_n)}{dt}\\ & =\frac{\partial f(x+td)}{\partial (x_1+t{d}_1)}{d}_1+\frac{\partial f(x+td)}{\partial (x_2+t{d}_2)}{d}_2+\cdots +\frac{\partial f(x+td)}{\partial (x_n+t{d}_n)}{d}_n\\ & =(\frac{\partial f(x+td)}{\partial (x_1+t{d}_1)},\frac{\partial f(x+td)}{\partial (x_2+t{d}_2)},\cdots ,\frac{\partial f(x+td)}{\partial (x_n+t{d}_n)})\left(\begin{array}{c}{d}_1\\ d_2\\ ⋮\\ d_n\end{array}\right)\\ & =(\frac{\partial f(x+td)}{\partial x_1},\frac{\partial f(x+td)}{\partial x_2},\cdots ,\frac{\partial f(x+td)}{\partial x_n})\left(\begin{array}{c}{d}_1\\ d_2\\ ⋮\\ d_n\end{array}\right)\\ & =\nabla f(x+td{)}^{T}d\end{array}$$
$$\begin{array}{rl}\frac{d^{2}f(x+td)}{d{t}^2}& =\frac{d^{2}f(x_1+t{d}_1,x_2+t{d}_2,\cdots ,x_n+t{d}_n)}{d{t}^2}\\ & =\frac{{\partial }^{2}f(x+td)}{{\partial }^2(x_1+t{d}_1)}{d}_1^2+\frac{{\partial }^{2}f(x+td)}{\partial (x_1+t{d}_1)\partial (x_2+t{d}_2)}{d}_1{d}_2+\cdots +\frac{{\partial }^{2}f(x+td)}{\partial (x_1+t{d}_1)\partial (x_n+t{d}_n)}{d}_1{d}_n\\ & +\frac{{\partial }^{2}f(x+td)}{\partial (x_2+t{d}_2)\partial (x_1+t{d}_1)}{d}_2{d}_1+\frac{{\partial }^{2}f(x+td)}{{\partial }^2(x_2+t{d}_2)}{d}_2^2+\cdots +\frac{{\partial }^{2}f(x+td)}{\partial (x_2+t{d}_2)\partial (x_n+t{d}_n)}{d}_2{d}_n\\ & ⋮\\ & +\frac{{\partial }^{2}f(x+td)}{\partial (x_n+t{d}_n)\partial (x_1+t{d}_1)}{d}_n{d}_1+\frac{{\partial }^{2}f(x+td)}{\partial (x_n+t{d}_n)\partial (x_2+t{d}_2)}{d}_n{d}_2+\cdots +\frac{{\partial }^{2}f(x+td)}{{\partial }^2(x_n+t{d}_n)}{d}_2{d}_n\\ & =\left(\begin{array}{cccc}{d}_1& d_2& \cdots & d_n\end{array}\right)\left(\begin{array}{cccc}\frac{{\partial }^{2}f(x+td)}{{\partial }^2{x}_1}& \frac{{\partial }^{2}f(x+td)}{\partial x_1\partial x_2}\cdots & \frac{{\partial }^{2}f(x+td)}{\partial x_1\partial x_n}& \\ \frac{{\partial }^{2}f(x+td)}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f(x+td)}{{\partial }^2{x}_2}\cdots & \frac{{\partial }^{2}f(x+td)}{\partial x_2\partial x_n}& \\ ⋮& ⋮& ⋮& \\ \frac{{\partial }^{2}f(x+td)}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f(x+td)}{\partial x_n\partial x_2}\cdots & \frac{{\partial }^{2}f(x+td)}{{\partial }^2{x}_n}& \end{array}\right)\left(\begin{array}{c}{d}_1\\ d_2\\ ⋮\\ d_n\end{array}\right)\\ & =d^T{\nabla }^{2}f(x+td)d\end{array}$$


此次内容参考书籍：
[1]、倪勤：最优化方法与程序设计
[2]、袁亚湘,孙文瑜：最优化理论与方法