现有目标函数:
f
(
x
)
=
1
2
∥
A
x
−
b
∥
2
2
f(x)=\frac{1}{2} \parallel Ax-b \parallel_2^2
f(x)=21∥Ax−b∥22
其中
A
∈
R
N
×
m
A\in \mathbb{R}^{N \times m}
A∈RN×m,
x
∈
R
m
x \in \mathbb{R}^m
x∈Rm,
b
∈
R
n
b\in \mathbb{R}^n
b∈Rn
则求
f
′
(
x
)
f^{'}(x)
f′(x)的推导如下:
f
(
x
)
=
1
2
∥
A
x
−
b
∥
2
2
=
1
2
(
A
x
−
b
)
T
(
A
x
−
b
)
=
1
2
(
x
T
A
T
−
b
T
)
(
A
x
−
b
)
=
1
2
(
x
T
A
T
A
x
−
x
T
A
T
b
−
b
T
A
x
+
b
T
b
)
\begin {aligned} f(x)&=\frac{1}{2} \parallel Ax-b \parallel_2^2 \\ &=\frac{1}{2}(Ax-b)^T(Ax-b)\\ &=\frac{1}{2}(x^TA^T-b^T)(Ax-b)\\ &=\frac{1}{2}(x^TA^TAx-x^TA^Tb-b^TAx+b^Tb) \end {aligned}
f(x)=21∥Ax−b∥22=21(Ax−b)T(Ax−b)=21(xTAT−bT)(Ax−b)=21(xTATAx−xTATb−bTAx+bTb)
利用如下性质:
∂
x
T
D
x
∂
x
=
(
D
+
D
T
)
x
∂
D
T
x
∂
x
=
D
∂
x
T
D
∂
x
=
D
\begin {aligned} \frac{\partial \ x^TDx}{\partial x}&=(D+D^T)x \\ \frac{\partial D^Tx}{\partial x}&=D\\ \frac{\partial x^TD}{\partial x}&=D \end {aligned}
∂x∂ xTDx∂x∂DTx∂x∂xTD=(D+DT)x=D=D
则有
∂
f
(
x
)
∂
x
=
1
2
{
[
A
T
A
+
(
A
T
A
)
T
]
x
−
A
T
b
−
A
T
b
}
=
1
2
{
2
A
T
A
x
−
2
A
T
b
}
=
A
T
A
x
−
2
A
T
b
=
A
T
(
A
x
−
b
)
\begin {aligned} \frac{\partial f(x)}{\partial x}&=\frac{1}{2} \{[A^TA+(A^TA)^T]x-A^Tb-A^Tb \} \\ &=\frac{1}{2} \{ 2A^TAx-2A^Tb \}\\ &=A^TAx-2A^Tb\\ &=A^T(Ax-b) \end {aligned}
∂x∂f(x)=21{[ATA+(ATA)T]x−ATb−ATb}=21{2ATAx−2ATb}=ATAx−2ATb=AT(Ax−b)
故
∂
f
(
x
)
∂
x
=
A
T
(
A
x
−
b
)
\frac{\partial f(x)}{\partial x}=A^T(Ax-b)
∂x∂f(x)=AT(Ax−b)