4.3 基于梯度的优化方法
1.向量微积分
(31条消息) 向量微积分基础_文剑木然的专栏-CSDN博客_向量微积分
常用求导公式（基于分母布局，结果转置即为分子布局）
∂
A
x
∂
x
=
A
(1)
\frac{\partial \mathbf{A} \mathbf{x}}{\partial \mathbf{x}}=\mathbf{A}\tag{1}
∂x∂Ax=A(1)
∂ x ⊤ A ∂ x = A ⊤ (2) \frac{\partial \mathbf{x}^{\top} \mathbf{A}}{\partial \mathbf{x}}=\mathbf{A}^{\top}\tag{2} ∂x∂x⊤A=A⊤(2)
∂ x ⊤ x ∂ x = 2 x ⊤ (3) \frac{\partial \mathbf{x}^{\top} \mathbf{x}}{\partial \mathbf{x}}=2 \mathbf{x}^{\top}\tag{3} ∂x∂x⊤x=2x⊤(3)
∂ x ⊤ A x ∂ x = x ⊤ ( A + A ⊤ ) (4) \frac{\partial \mathbf{x}^{\top} \mathbf{A} \mathbf{x}}{\partial \mathbf{x}}=\mathbf{x}^{\top}\left(\mathbf{A}+\mathbf{A}^{\top}\right)\tag{4} ∂x∂x⊤Ax=x⊤(A+A⊤)(4)
∂ ( u + v ) ∂ x = ∂ u ∂ x + ∂ v ∂ x (5) \frac{\partial(\mathbf{u}+\mathbf{v})}{\partial \mathbf{x}}=\frac{\partial \mathbf{u}}{\partial \mathbf{x}}+\frac{\partial \mathbf{v}}{\partial \mathbf{x}}\tag{5} ∂x∂(u+v)=∂x∂u+∂x∂v(5)
∂ ( u ⋅ v ) ∂ x = ∂ u ⊤ v ∂ x = u ⊤ ∂ v ∂ x + v ⊤ ∂ u ∂ x (6) \frac{\partial(\mathbf{u} \cdot \mathbf{v})}{\partial \mathbf{x}}=\frac{\partial \mathbf{u}^{\top} \mathbf{v}}{\partial \mathbf{x}}=\mathbf{u}^{\top} \frac{\partial \mathbf{v}}{\partial \mathbf{x}}+\mathbf{v}^{\top} \frac{\partial \mathbf{u}}{\partial \mathbf{x}}\tag{6} ∂x∂(u⋅v)=∂x∂u⊤v=u⊤∂x∂v+v⊤∂x∂u(6)
∂ f ( u ) ∂ x = ∂ f ( u ) ∂ u ∂ u ∂ x (7) \frac{\partial \mathbf{f}(\mathbf{u})}{\partial \mathbf{x}}=\frac{\partial \mathbf{f}(\mathbf{u})}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial \mathbf{x}}\tag{7} ∂x∂f(u)=∂u∂f(u)∂x∂u(7)
2.方向导数
数学篇-方向导数（讲的很通俗易懂） - 知乎 (zhihu.com)
如果函数
f
(
x
,
y
)
f(x,y)
f(x,y) 在点
P
0
(
x
0
,
y
0
)
P_0(x_0,y_0)
P0(x0,y0)可微分,那么函数在该点沿任一方向
l
l
l 的方向导数存在,且有
∂
f
∂
l
∣
(
x
0
,
y
0
)
=
f
x
(
x
0
,
y
0
)
cos
α
+
f
y
(
x
0
,
y
0
)
cos
β
(8)
\left.\frac{\partial f}{\partial l}\right|_{\left(x_{0}, y_{0}\right)}=f_{x}\left(x_{0}, y_{0}\right) \cos \alpha+f_{y}\left(x_{0}, y_{0}\right) \cos \beta\tag{8}
∂l∂f∣∣∣∣(x0,y0)=fx(x0,y0)cosα+fy(x0,y0)cosβ(8)
其中，
c
o
s
α
cos\alpha
cosα 和
c
o
s
β
cos\beta
cosβ的方向余弦.
证明: 由假设
f
(
x
,
y
)
f(x,y)
f(x,y) 在点
(
x
0
,
y
0
)
(x_0,y_0)
(x0,y0)可微分,故有
f
(
x
0
+
Δ
x
,
y
0
+
Δ
y
)
−
f
(
x
0
,
y
0
)
=
f
x
(
x
0
,
y
0
)
Δ
x
+
f
y
(
x
0
,
y
0
)
Δ
y
+
o
(
(
Δ
x
)
2
+
(
Δ
y
)
2
)
(9)
\begin{array}{c} f\left(x_{0}+\Delta x, y_{0}+\Delta y\right)-f\left(x_{0}, y_{0}\right) \\=f_{x}\left(x_{0}, y_{0}\right) \Delta x+f_{y}\left(x_{0}, y_{0}\right) \Delta y+o\left(\sqrt{(\Delta x)^{2}+(\Delta y)^{2}}\right) \end{array}\tag{9}
f(x0+Δx,y0+Δy)−f(x0,y0)=fx(x0,y0)Δx+fy(x0,y0)Δy+o((Δx)2+(Δy)2)(9)
但点
(
x
0
+
Δ
x
,
y
0
+
Δ
y
)
(x_0+\Delta x,y_0+\Delta y)
(x0+Δx,y0+Δy) 在以
(
x
0
,
y
0
)
(x_0,y_0)
(x0,y0) 为始点的射线
l
l
l 上时,应有
Δ x = t cos α , Δ y = t cos β ( Δ x ) 2 + ( Δ y ) 2 = t (10) \begin{array}{c} \Delta x=t \cos \alpha, \Delta y=t \cos \beta \\ \sqrt{(\Delta x)^{2}+(\Delta y)^{2}}=t \end{array}\tag{10} Δx=tcosα,Δy=tcosβ(Δx)2+(Δy)2=t(10)
所以
lim
t
→
0
+
f
(
x
0
+
t
cos
α
,
y
0
+
t
cos
β
)
−
f
(
x
0
,
y
0
)
t
=
f
x
(
x
0
,
y
0
)
cos
α
+
f
y
(
x
0
,
y
0
)
cos
β
(11)
\lim _{t \rightarrow 0^{+}} \frac{f\left(x_{0}+t \cos \alpha, y_{0}+t \cos \beta\right)-f\left(x_{0}, y_{0}\right)}{t} \\ =f_{x}\left(x_{0}, y_{0}\right) \cos \alpha+f_{y}\left(x_{0}, y_{0}\right) \cos \beta \tag{11}
t→0+limtf(x0+tcosα,y0+tcosβ)−f(x0,y0)=fx(x0,y0)cosα+fy(x0,y0)cosβ(11)
这就证明了方向导数存在,且其值为
∂
f
∂
l
∣
(
x
0
,
y
0
)
=
f
x
(
x
0
,
y
0
)
cos
α
+
f
y
(
x
0
,
y
0
)
cos
β
(12)
\left.\frac{\partial f}{\partial l}\right|_{\left(x_{0}, y_{0}\right)}=f_{x}\left(x_{0}, y_{0}\right) \cos \alpha+f_{y}\left(x_{0}, y_{0}\right) \cos \beta\tag{12}
∂l∂f∣∣∣∣(x0,y0)=fx(x0,y0)cosα+fy(x0,y0)cosβ(12)
用x表示多维向量，用u表示方向，用a表示t，即可得到
∂
∂
α
f
(
x
+
α
u
)
=
u
T
∇
x
f
(
x
)
=
f
x
(
x
0
,
y
0
)
c
o
s
α
+
f
y
(
x
0
,
y
0
)
c
o
s
β
(13)
\frac{\partial}{\partial \alpha}f(x+\alpha u) = u^T \nabla_xf(x) = f_x(x0,y0) cos\alpha+f_y(x0,y0) cos\beta\tag{13}
∂α∂f(x+αu)=uT∇xf(x)=fx(x0,y0)cosα+fy(x0,y0)cosβ(13)
(7)式第一个等号是花书上给出的，目前仍有疑惑，我的推导如下
令
t
=
x
+
α
u
(14)
令 t = x+\alpha u\tag{14}
令t=x+αu(14)
∂
∂
α
f
(
x
+
α
u
)
=
∂
f
(
t
)
∂
α
=
∂
f
(
t
)
∂
t
⋅
∂
t
∂
α
(15)
\frac{\partial}{\partial \alpha} f(x+\alpha u)=\frac{\partial f(t)}{\partial \alpha}=\frac{\partial f(t)}{\partial t} \cdot \frac{\partial t}{\partial \alpha}\tag{15}
∂α∂f(x+αu)=∂α∂f(t)=∂t∂f(t)⋅∂α∂t(15)
=
∂
f
(
t
)
∂
t
⋅
∂
x
+
α
u
∂
α
(16)
=\frac{\partial f(t)}{\partial t} \cdot \frac{\partial x+\alpha u}{\partial \alpha}\tag{16}
=∂t∂f(t)⋅∂α∂x+αu(16)
=
∂
f
(
t
)
∂
t
⋅
u
(17)
=\frac{\partial f(t)}{\partial t} \cdot u\tag{17}
=∂t∂f(t)⋅u(17)
=
∇
x
f
(
x
)
T
⋅
u
(
取
α
=
0
)
(18)
=\nabla_{x} f(x)^T \cdot u (取\alpha=0)\tag{18}
=∇xf(x)T⋅u(取α=0)(18)
希望有明白的人指出我的问题
补充：问题解决了，由于我的公式是基于分母布局（横向），所以在(17)-(18)的时候得到的式子应该是 ∇ x f ( x ) T ⋅ u \nabla_{x} f(x)^T \cdot u ∇xf(x)T⋅u（怕误导大家，上面已修改，但是之前是没加转置的），又因为花书中的推导都是基于分子布局的，所以最终结果与我的结果会刚好差一个转置。另外，书中所有的向量都是列向量，尤其是梯度向量。
3.梯度下降
一阶优化方法
略
4.牛顿法
二阶优化方法
参考链接：数值优化（Numerical Optimization）(3)-牛顿法 - 知乎 (zhihu.com)
f
(
x
)
≈
f
(
x
k
)
+
(
x
−
x
k
)
T
∇
f
(
x
k
)
+
1
2
(
x
−
x
k
)
T
H
(
x
−
x
k
)
(19)
f(\boldsymbol x) \approx f(\boldsymbol{x_{k}})+\left(\boldsymbol x-\boldsymbol{x_{k}}\right)^T\nabla f\left(\boldsymbol{x_{k}}\right)+\frac{1}{2}\left(\boldsymbol x-\boldsymbol{x_{k}}\right)^{T} H\left(\boldsymbol x-\boldsymbol{x_{k}}\right)\tag{19}
f(x)≈f(xk)+(x−xk)T∇f(xk)+21(x−xk)TH(x−xk)(19)
要找到
f
(
x
)
f(x)
f(x)的最小点，对
f
f
f求导，得
f
′
(
x
)
=
∇
f
(
x
k
)
T
+
1
2
(
x
−
x
k
)
T
(
H
+
H
T
)
=
∇
f
(
x
k
)
T
+
(
x
−
x
k
)
T
H
(20)
f'(\boldsymbol x) =\nabla f(\boldsymbol{x_k})^T +\frac{1}{2}(\boldsymbol{x-x_k})^T(H+H^T) \\= \nabla f(\boldsymbol{x_k})^T +(\boldsymbol x-\boldsymbol{x_{k}})^TH\tag{20}
f′(x)=∇f(xk)T+21(x−xk)T(H+HT)=∇f(xk)T+(x−xk)TH(20)
令
f
′
(
x
)
=
0
f'(\boldsymbol x)=\boldsymbol0
f′(x)=0，又
H
H
H为对称矩阵，即
H
=
H
T
H = H^T
H=HT
x
−
x
k
=
−
(
H
−
1
∗
∇
f
(
x
k
)
T
)
T
=
−
H
−
1
∗
∇
f
(
x
k
)
(21)
\boldsymbol x-\boldsymbol{x_{k}} =- (H^{-1}*\nabla f(\boldsymbol{x_k})^T)^T =- H^{-1}*\nabla f(\boldsymbol{x_k})\tag{21}
x−xk=−(H−1∗∇f(xk)T)T=−H−1∗∇f(xk)(21)
x
k
+
1
=
x
k
−
H
−
1
∇
f
(
x
k
)
(22)
\boldsymbol{x_{k+1}} = \boldsymbol{x_k}-H^{-1}\nabla f(\boldsymbol{x_k})\tag{22}
xk+1=xk−H−1∇f(xk)(22)
当 f f f是一个正定二次函数时，牛顿法只要应用一次(22)就能跳到函数最小点
如果 f f f不是真正二次，但能在局部近似为正定二次，牛顿法则需要多次迭代
当附近的临界点是最小点牛顿法才适用，在鞍点附近是有害的
上面这些话我还需要好好琢磨琢磨
扫一扫
4341
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
