正规方程之矩阵求导(Matrix derivatives)

1、背景

最近，开始学习机器学习之旅。我的学习方式是：跟着斯坦福公开课Andrew Ng的讲义和视屏开始学习。

下面，简单介绍一下线性回归下面的普通方程之矩阵求导。
对应英文章节：Linear Regression-->The normal equations-->Matrix derivatives

---

2、矩阵求导公式

$\color{red}{\nabla_AtrAB = B^T}$                                        $(1)$

$\color{red}{\nabla_{A^T}f(A) = (\nabla_Af(A))^T}$                         $(2)$

$\color{red}{\nabla_AtrABA^TC = CAB + C^TAB^T}$        $(3)$

$\color{red}{\nabla_A|A| = |A|(A^{-1})^T}$                              $(4)$

---

3、公式推导

3.1 公式(1)的推导

$知识须知:$
$1、tr为矩阵的迹，即对角线元素之和。tr既然是对角线元素之和，那么矩阵为正方形矩阵。$
$2、\nabla符号为求梯度，实际上就是求导。\nabla_A就是对A矩阵求导。$

$设 \phantom{1}A = \begin{pmatrix} a_{11} &a_{12} \\ a_{21} & a_{22}\end{pmatrix}$ $,B = \begin{pmatrix} b_{11} &b_{12} \\ b_{21} & b_{22}\end{pmatrix}$

$根据矩阵的相乘，那么有:$

AB=(a11b11+a12b21a21b11+a22b21a11b12+a12b22a21b12+a22b22) A B = ( a 11 b 11 + a 12 b 21 a 11 b 12 + a 12 b 22 a 21 b 11 + a 22 b 21 a 21 b 12 + a 22 b 22 )

$$AB= \begin{pmatrix} a_{11}b_{11} +a_{12}b_{21} & a_{11}b_{12}+a_{12}b_{22} \\a_{21}b_{11}+a_{22}b_{21} &a_{21}b_{12}+a_{22}b_{22}\end{pmatrix}$$

$设 \phantom{1} F = trAB$
$而\phantom{1} trAB = a_{11}b_{11}+a_{12}b_{21}+a_{21}b_{12}+a_{22}b_{22}$

$则有\phantom{1}\nabla_AtrAB = \begin{pmatrix} \frac{\partial F}{\partial a_{11}} & \frac{\partial F}{\partial a_{12}} \\ \frac{\partial F}{\partial a_{21}} & \frac{\partial F}{\partial a_{22}} \end{pmatrix} = \begin{pmatrix} b_{11} & b_{21} \\ b_{12} & b_{22} \end{pmatrix} = B^T$

$得证$

3.2 公式(2)的推导

$设\phantom{1} A = \begin{pmatrix} a_{11} &a_{12} \\ a_{21} & a_{22}\end{pmatrix}$

$设\phantom{1} f(A) = 2a_{11}+3a_{12}+4a_{21}+5a_{22}$

$则\phantom{1} \nabla_Af(A)=\begin{pmatrix} 2 & 3\\4 & 5 \end{pmatrix}$

$转置后为 \phantom{1} (\nabla_Af(A))^T=\begin{pmatrix} 2 & 4\\3 & 5 \end{pmatrix}$

$而 \phantom{1} \nabla_{A^T}f(A) = \begin{pmatrix} 2 & 4\\3 & 5 \end{pmatrix} = (\nabla_Af(A))^T \phantom{1}得证$

3.3 公式(3)的推导

$知识须知:$
$1、tr(A) = tr(A^T)$
$2、\nabla_Atr(AB) = B^T$
$3、(AB)^T = B^TA^T$

$令\phantom{1}X = A$

$\nabla_A tr(ABA^TC) = \nabla_Xtr(XBA^TC)+\nabla_Xtr(ABX^TC)$
$\phantom{10000000000} =(BA^TC)^T+\nabla_Xtr(CABX^T)$
$\phantom{10000000000} =B^TAC^T+\nabla_Xtr(X(CAB)^T)$
$\phantom{10000000000} =B^TAC^T+((CAB)^T)^T$
$\phantom{10000000000} =B^TAC^T+\nabla_Xtr(X(CAB)^T)$
$\phantom{10000000000} =C^TAB^T+CAB \phantom{1}得证$

3.4 公式(4)的推导

$知识须知:$
$1、AA^{-1} = E = 1 \phantom{1}(单位矩阵)$
$2、tra = a \phantom{1} (a为是实数)$
$3、traA = atrA$
$4、\nabla_AtrAB = B^T$
$5、|A|为行列式，既然是行列式就能求出一个实数，则|A|为一个实数$

$\nabla_A|A| = |A|\nabla_Atr1$
$=|A|\nabla_AtrE$
$=|A|\nabla_AtrAA^{-1}$
$=|A|(A^{-1})^T$

---

4、相关资料

视屏网站：斯坦福公开课视频-机器学习
讲义网站：斯坦福公开课讲义-机器学习