Hessian矩阵及其辨识 实变函数 极值 无约束梯度分析
本文主要介绍Hessian矩阵的定义以及与实变函数无约束极值的关系,主要内容来自张贤达《矩阵分析与优化(第二版)》第三章和第四章的相关内容。
Hessian矩阵的定义
Hessian矩阵可以理解为矩阵的二阶偏导。
实值标量函数
f
(
x
)
f(x)
f(x)在列向量
x
∈
R
m
×
1
x\in\mathbb{R}^{m\times 1}
x∈Rm×1处的Hessian矩阵为:
H
[
f
(
x
)
]
=
∂
f
2
(
x
)
∂
x
∂
x
T
=
∂
∂
x
[
∂
f
(
x
)
∂
x
T
]
∈
R
m
×
m
H[f(x)]= \frac{\partial f^2(x)}{\partial x\partial x^T} = \frac{\partial}{\partial x} [ \frac{\partial f(x)}{\partial x^T}]\in\mathbb{R}^{m\times m}
H[f(x)]=∂x∂xT∂f2(x)=∂x∂[∂xT∂f(x)]∈Rm×m
其第
(
i
,
j
)
(i,j)
(i,j)个元素定义为:
H
[
f
(
x
)
]
i
,
j
=
[
∂
f
2
(
x
)
∂
x
∂
x
T
]
i
,
j
=
∂
∂
x
i
[
∂
f
(
x
)
∂
x
j
]
H[f(x)]_{i,j}=[\frac{\partial f^2(x)}{\partial x\partial x^T}]_{i,j}= \frac{\partial}{\partial x_i}[ \frac{\partial f(x)}{\partial x_j}]
H[f(x)]i,j=[∂x∂xT∂f2(x)]i,j=∂xi∂[∂xj∂f(x)]
展开的形式为:
H
[
f
(
x
)
]
=
∂
f
2
(
x
)
∂
x
∂
x
T
=
[
∂
f
2
(
x
)
∂
x
1
∂
x
1
⋯
∂
f
2
(
x
)
∂
x
1
∂
x
m
⋮
⋱
⋮
∂
f
2
(
x
)
∂
x
m
∂
x
1
⋯
∂
f
2
(
x
)
∂
x
m
∂
x
m
]
∈
R
m
×
m
H[f(x)]=\frac{\partial f^2(x)}{\partial x\partial x^T}=\left[ \begin{matrix} \frac{\partial f^2(x)}{\partial x_1\partial x_1} & \cdots & \frac{\partial f^2(x)}{\partial x_1\partial x_m} \\ \vdots & \ddots & \vdots \\ \frac{\partial f^2(x)}{\partial x_m\partial x_1} & \cdots & \frac{\partial f^2(x)}{\partial x_m\partial x_m} \\ \end{matrix} \right]\in \mathbb{R}^{m\times m}
H[f(x)]=∂x∂xT∂f2(x)=⎣⎢⎢⎡∂x1∂x1∂f2(x)⋮∂xm∂x1∂f2(x)⋯⋱⋯∂x1∂xm∂f2(x)⋮∂xm∂xm∂f2(x)⎦⎥⎥⎤∈Rm×m
同样,实值标量函数
f
(
X
)
f(X)
f(X)对矩阵变元
X
∈
R
m
×
n
X\in \mathbb{R}^{m\times n}
X∈Rm×n的Hessian矩阵定义为:
H
[
f
(
X
)
]
=
[
∂
2
f
(
X
)
∂
X
11
∂
X
11
⋯
∂
2
f
(
X
)
∂
X
11
∂
X
m
1
⋯
∂
f
2
(
X
)
∂
X
11
∂
X
m
n
⋮
⋱
⋮
⋱
⋮
∂
2
f
(
X
)
∂
X
m
1
∂
X
11
⋯
∂
2
f
(
X
)
∂
X
m
1
∂
X
m
1
⋯
∂
2
f
(
X
)
∂
X
m
1
∂
X
m
n
⋮
⋱
⋮
⋱
⋮
∂
2
f
(
X
)
∂
X
m
n
∂
X
11
⋯
∂
2
f
(
X
)
∂
X
m
n
∂
X
m
1
⋯
∂
2
f
(
X
)
∂
X
m
n
∂
X
m
n
]
∈
R
m
n
×
m
n
H[f(X)] =\left[ \begin{matrix} \frac{\partial^2 f(X)}{\partial X_{11}\partial X_{11}} & \cdots & \frac{\partial^2 f(X)}{\partial X_{11} \partial X_{m1}} & \cdots & \frac{\partial f^2(X)}{\partial X_{11}\partial X_{mn}} \\ \vdots & \ddots & \vdots &\ddots & \vdots\\ \frac{\partial^2 f(X)}{\partial X_{m1}\partial X_{11}} & \cdots & \frac{\partial^2 f(X)}{\partial X_{m1}\partial X_{m1}} & \cdots & \frac{\partial^2 f(X)}{\partial X_{m1}\partial X_{mn}} \\ \vdots & \ddots & \vdots &\ddots & \vdots\\ \frac{\partial^2 f(X)}{\partial X_{mn}\partial X_{11}} & \cdots & \frac{\partial^2 f(X)}{\partial X_{mn}\partial X_{m1}} &\cdots & \frac{\partial^2 f(X)}{\partial X_{mn}\partial X_{mn}} \\ \end{matrix} \right]\in \mathbb{R}^{mn\times mn}
H[f(X)]=⎣⎢⎢⎢⎢⎢⎢⎢⎢⎡∂X11∂X11∂2f(X)⋮∂Xm1∂X11∂2f(X)⋮∂Xmn∂X11∂2f(X)⋯⋱⋯⋱⋯∂X11∂Xm1∂2f(X)⋮∂Xm1∂Xm1∂2f(X)⋮∂Xmn∂Xm1∂2f(X)⋯⋱⋯⋱⋯∂X11∂Xmn∂f2(X)⋮∂Xm1∂Xmn∂2f(X)⋮∂Xmn∂Xmn∂2f(X)⎦⎥⎥⎥⎥⎥⎥⎥⎥⎤∈Rmn×mn
可以看出,实标量函数的Hessian矩阵是对称矩阵。
Hessian矩阵的辨识
同Jacobian矩阵类似,Hessian矩阵与矩阵微分的关系称为Hessian矩阵的辨识,提供了除定义外求Hessian矩阵的另一种方法。
实标量函数对向量变元
x
∈
R
m
×
1
x\in \mathbb{R}^{m\times 1}
x∈Rm×1的Hessian矩阵的辨识为:
d
2
f
(
x
)
=
(
d
x
)
T
H
[
f
(
x
)
]
d
x
d^2f(x)=(dx)^TH[f(x)]dx
d2f(x)=(dx)TH[f(x)]dx
在运算中,如果矩阵
B
B
B满足:
d
2
f
(
x
)
=
(
d
x
)
T
B
d
x
d^2f(x)=(dx)^TBdx
d2f(x)=(dx)TBdx
则可得:
H
[
f
(
x
)
]
=
1
2
(
B
T
+
B
)
H[f(x)]=\frac{1}{2}(B^T+B)
H[f(x)]=21(BT+B)
这是为了保证所得Hessian矩阵的对称性。
实值标量函数对矩阵变元
X
∈
R
m
×
n
X\in \mathbb{R}^{m\times n}
X∈Rm×n的Hessian矩阵的辨识为:
d
2
f
(
X
)
=
(
d
(
v
e
c
X
)
)
T
H
[
f
(
X
)
]
d
(
v
e
c
X
)
d^2f(X)=(d(vecX))^TH[f(X)]d(vecX)
d2f(X)=(d(vecX))TH[f(X)]d(vecX)
这里
v
e
c
vec
vec表示矩阵向量化运算,
v
e
c
X
vecX
vecX定义为:
v
e
c
X
=
[
X
11
,
⋯
,
X
m
1
,
⋯
,
X
1
n
,
⋯
,
X
m
n
]
T
vecX=[X_{11},\cdots,X_{m1},\cdots,X_{1n},\cdots,X_{mn}]^T
vecX=[X11,⋯,Xm1,⋯,X1n,⋯,Xmn]T
在运算中,如果矩阵
B
B
B满足:
d
2
f
(
X
)
=
(
d
(
v
e
c
X
)
)
T
B
d
(
v
e
c
X
)
d^2f(X)=(d(vecX))^TBd(vecX)
d2f(X)=(d(vecX))TBd(vecX)
则可得:
H
[
f
(
x
)
]
=
1
2
(
B
T
+
B
)
H[f(x)]=\frac{1}{2}(B^T+B)
H[f(x)]=21(BT+B)
这同样是为了保证所得Hessian矩阵的对称性。
实变函数平稳点与极值点的条件
上一篇Jacobian矩阵,梯度矩阵和这一篇Hessian矩阵都是为了下面这个表。
解释一下,
R
→
R
\mathbb{R}→\mathbb{R}
R→R即表示实数到实数的映射函数,
R
n
→
R
\mathbb{R^n}→\mathbb{R}
Rn→R表示向量到实数的映射函数,
R
m
×
n
→
R
\mathbb{R^{m\times n}}→\mathbb{R}
Rm×n→R表示矩阵到实数的映射函数。
实数到实数我们很熟悉了,一阶导为0则为极点,二阶导大于0则为极小点,二阶导小于0则为极大点。矩阵和向量到实数的函数,一阶导为0或0矩阵则为极点,二阶导正定则为极小点,二阶导负定则为极大点。半正定的情况都不是严格的极大点和极小点。
张贤达书上第四章还谈了复变函数的Hessian矩阵及其平稳点和极点条件,由于我暂时不关注,也未做总结。