多维正态分布的极大似然估计

二维正态分布

(X,Y)服从参数为μ1, μ2, σ1, σ2, ρ的二维正态分布，记作(X, Y)~N(μ1, μ2, σ1, σ2, ρ)，它的密度函数：
$$\begin{array}{l}f\left(x,y\right)\\ =\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-{\rho }^2}}\exp \left(-\frac{1}{2\left(1-{\rho }^2\right)}\left[\frac{{\left(x-{\mu }_1\right)}^2}{{\sigma }_1^2}-2\rho \frac{\left(x-{\mu }_1\right)\left(y-{\mu }_2\right)}{2{\sigma }_1{\sigma }_2}+\frac{{\left(y-{\mu }_2\right)}^2}{{\sigma }_2^2}\right]\right)\\ =\frac{1}{(\sqrt{2\pi }{)}^2{\sigma }_1{\sigma }_2\sqrt{1-{\rho }^2}}\exp \left(-\frac{1}{2\left(1-{\rho }^2\right)}\left[\frac{{\left(x-{\mu }_1\right)}^2}{{\sigma }_1^2}-2\rho \frac{\left(x-{\mu }_1\right)\left(y-{\mu }_2\right)}{2{\sigma }_1{\sigma }_2}+\frac{{\left(y-{\mu }_2\right)}^2}{{\sigma }_2^2}\right]\right)\end{array}$$
其中μ1是第1维度的均值，σ1是第1维度的方差，ρ是将两个维度的相关性规范到-1到+1之间的统计量，称为样本的相关系数，定义为：
$$\rho =\frac{\mathrm{COV}(X,Y)}{{\sigma }_1{\sigma }_2},|\rho |<1$$
对于二维正态随机变量(X,Y)，X和Y相互独立的充要条件是二者的协方差为0，也就是参数ρ=0。由于一维随机变量没有是否独立一说，ρ一定是0，因此没有在一维随机变量的正态分布中体现ρ。

多维正态分布

假设n维随机变量$x={\left[x_1,x_2,\cdots ,x_n\right]}^{\mathrm{T}}$的各个维度之间互不相关，且服从正态分布(维度不相关多元正态分布)，各个维度的均值为$E(x)={\left[{\mu }_1,{\mu }_2,\cdots ,{\mu }_n\right]}^{\mathrm{T}}$,各个维度的方差为$\sigma (x)={\left[{\sigma }_1,{\sigma }_2,\cdots ,{\sigma }_n\right]}^{\mathrm{T}}$
用列向量的形式表示随机变量和参数，对于n维随机变量有：
$$x=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right],\mu =\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_n\end{array}\right],\sigma =\left[\begin{array}{c}{\sigma }_1\\ {\sigma }_2\\ ⋮\\ {\sigma }_n\end{array}\right]$$
根据联合概率密度公式：
$$\begin{array}{l}f(x)=p\left(x_1,x_2\dots x_n\right)=p\left(x_1\right)p\left(x_2\right)\dots p\left(x_n\right)\\ =\frac{1}{\sqrt{2\pi }{\sigma }_1}\exp \left(-\frac{1}{2}{\left(\frac{x_1-{\mu }_1}{{\sigma }_1}\right)}^2\right)\frac{1}{\sqrt{2\pi }{\sigma }_2}\exp \left(-\frac{1}{2}{\left(\frac{x_2-{\mu }_2}{{\sigma }_2}\right)}^2\right)\cdots \frac{1}{\sqrt{2\pi }{\sigma }_n}\exp \left(-\frac{1}{2}{\left(\frac{x_n-{\mu }_n}{{\sigma }_n}\right)}^2\right)\\ =\frac{1}{(\sqrt{2\pi }{)}^n{\sigma }_1{\sigma }_2\cdots {\sigma }_n}\exp \left(-\frac{1}{2}\left[{\left(\frac{x_1-{\mu }_1}{{\sigma }_1}\right)}^2+{\left(\frac{x_2-{\mu }_2}{{\sigma }_2}\right)}^2+\cdots +{\left(\frac{x_n-{\mu }_n}{{\sigma }_n}\right)}^2\right]\right)\end{array}$$

令$z^2=\frac{{\left(x_1-{\mu }_1\right)}^2}{{\sigma }_1^2}+\frac{{\left(x_2-{\mu }_2\right)}^2}{{\sigma }_2^2}\cdots +\frac{{\left(x_n-{\mu }_n\right)}^2}{{\sigma }_n^2},{\sigma }_z={\sigma }_1{\sigma }_2\cdots {\sigma }_n$

则$f(x)$可以化为:
$$f(z)=\frac{1}{(\sqrt{2\pi }{)}^n{\sigma }_z}{e}^{-\frac{z^2}{2}}①$$

因为多元正态分布有着很强的几何思想，单纯从代数的角度看待z很难看出z的概率分布规律，这里需要转换成矩阵形式：
$$\begin{array}{l}{z}^2=z^{\mathrm{T}}z\\ =\left[\begin{array}{llll}{x}_1-{\mu }_1& x_2-{\mu }_2& \cdots & x_n-{\mu }_n\end{array}\right]\left[\begin{array}{cccc}\frac{1}{{\sigma }_1^2}& 0& \cdots & 0\\ 0& \frac{1}{{\sigma }_2^2}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & \frac{1}{{\sigma }_n^2}\end{array}\right]\left[\begin{array}{c}{x}_1-{\mu }_1\\ x_2-{\mu }_2\\ ⋮\\ x_n-{\mu }_n\end{array}\right]\end{array}②$$
上面的等式比较长，这里做一下变量替换，记

$x-\mu ={\left[x_1-{\mu }_1,x_2-{\mu }_2,\cdots ,x_n-{\mu }_n\right]}^{\mathrm{T}}$

定义一个符号 \quad $\Sigma =\left[\begin{array}{cccc}{\sigma }_1^2& 0& \cdots & 0\\ 0& {\sigma }_2^2& \cdots & 0\\ ⋮& \cdots & \cdots & ⋮\\ 0& 0& \cdots & {\sigma }_n^2\end{array}\right]$

$\Sigma$ 表示变量 $x$ 的协方差矩阵， $i$ 行 $j$ 列的元素值表示 $x_i$ 与 $x_j$ 的协方差。
因为现在变量之间是相互独立的，所以只有对角线上$(i=j)$存在元素，其他地方都等于0，且 $x_i$ 与它本身的协方差就等于方差。
$\Sigma$ 为一个对角矩阵，根据对角矩阵的性质，$\Sigma$ 的逆矩阵为：
$${\left(\Sigma \right)}^{-1}=\left[\begin{array}{cccc}\frac{1}{{\sigma }_1^2}& 0& \cdots & 0\\ 0& \frac{1}{{\sigma }_2^2}& \cdots & 0\\ ⋮& \cdots & \cdots & ⋮\\ 0& 0& \cdots & \frac{1}{{\sigma }_n^2}\end{array}\right]$$
因为对角矩阵的行列式 = 对角元素的乘积

$|\Sigma |={\sigma }_1^2{\sigma }_2^2\cdots {\sigma }_n^2$

${\sigma }_z={\left|\Sigma \right|}^{\frac{1}{2}}={\sigma }_1{\sigma }_2\dots {\sigma }_n$

带入②中可得：
$z^{\mathrm{T}}z={\left(x-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x-\mu \right)③$
带入①中可得：
$f(z)=\frac{1}{(\sqrt{2\pi }{)}^n{\sigma }_z}{e}^{-\frac{z^2}{2}}=\frac{1}{(\sqrt{2\pi }{)}^n{|\Sigma |}^{\frac{1}{2}}}{e}^{-\frac{{\left(x-\mu \right)}^{\mathrm{T}}{\left(\Sigma \right)}^{-1}\left(x-\mu \right)}{2}}$

所以得到：
$$\begin{array}{rl}f(x)& =\frac{1}{(\sqrt{2\pi }{)}^n{\sigma }_1{\sigma }_2\cdots {\sigma }_n}\exp \left(-\frac{1}{2}\left[{\left(\frac{x_1-{\mu }_1}{{\sigma }_1}\right)}^2+{\left(\frac{x_2-{\mu }_2}{{\sigma }_2}\right)}^2+\cdots +{\left(\frac{x_n-{\mu }_n}{{\sigma }_n}\right)}^2\right]\right)\\ & =\frac{1}{(\sqrt{2\pi }{)}^n\sqrt{|\Sigma |}}\exp \left(-\frac{1}{2}(x-\mu {)}^{\mathrm{T}}{\Sigma }^{-1}(x-\mu )\right)\\ & =(2\pi {)}^{-\frac{n}{2}}|\Sigma {|}^{-\frac{1}{2}}\exp \left(-\frac{1}{2}(x-\mu {)}^{\mathrm{T}}{\Sigma }^{-1}(x-\mu )\right)\\ & =f(x;\mu ,\Sigma )\end{array}$$

最大似然估计量

n维相互独立的随机变量 $x$ 服从正态分布：
$x\sim N\left(\mu ,{\sigma }^2\right),{\sigma }_i\ge 0$
多维正态分布的最终形式为：
$f(x)=f(x;\mu ,\Sigma )$
假设有m个可观察样本，那么最大似然函数是：

$$\begin{array}{rl}L(\mu ,\Sigma )& =\prod _{i=1}^{m}f\left(x^{(i)};\mu ,\Sigma \right)\\ & =\prod _{i=1}^m(2\pi {)}^{-\frac{n}{2}}|\Sigma {|}^{-\frac{1}{2}}\exp \left(-\frac{1}{2}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =(2\pi {)}^{-\frac{mn}{2}}|\Sigma {|}^{-\frac{m}{2}}\exp \left(-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\end{array}$$
其对数似然函数是：
$\begin{array}{rl}\ln L(\mu ,\Sigma )& =\ln (2\pi {)}^{-\frac{mn}{2}}+\ln |\Sigma {|}^{-\frac{m}{2}}+\ln \exp \left(-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =-\frac{mn}{2}\ln 2\pi -\frac{m}{2}\ln |\Sigma |-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\\ & =C-\frac{m}{2}\ln |\Sigma |-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\end{array}$
其中m和n是已知的，m为可观察样本的个数，n为单个样本的特征维数，C 是一个常数, $C=-\frac{mn}{2}\ln 2\pi$。

求极值需要对μ和∑求偏导：

$\{\begin{array}{l}\frac{\partial \ln L}{\partial \mu }=0\\ \frac{\partial \ln L}{\partial \Sigma }=0\end{array}$

μ和∑是矩阵，涉及到矩阵的求导法则。先看对μ的求导，$\mathrm{lnL}$由3个因子组成，只有一个因子含有μ，因此：

$$\frac{\partial \ln L}{\partial \mu }=\frac{\partial }{\partial \mu }\left(-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)$$

其中：
$\begin{array}{rl}(x-\mu {)}^{\mathrm{T}}{\Sigma }^{-1}(x-\mu )& =\left(x^{\mathrm{T}}-{\mu }^{\mathrm{T}}\right){\Sigma }^{-1}(x-\mu )\\ & =\left(x^{\mathrm{T}}{\Sigma }^{-1}-{\mu }^{\mathrm{T}}{\Sigma }^{-1}\right)(x-\mu )\\ & =x^{\mathrm{T}}{\Sigma }^{-1}x-x^{\mathrm{T}}{\Sigma }^{-1}\mu -{\mu }^{\mathrm{T}}{\Sigma }^{-1}x+{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \end{array}$

上式中：
$x^{\mathrm{T}}{\Sigma }^{-1}\mu =\left[\begin{array}{llll}{x}_1& x_2& \cdots & x_n\end{array}\right]{\Sigma }^{-1}\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_n\end{array}\right]=\left[\begin{array}{llll}{\mu }_1& {\mu }_2& \cdots & {\mu }_n\end{array}\right]{\Sigma }^{-1}\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]={\mu }^{\mathrm{T}}{\Sigma }^{-1}x$

因此：
$\begin{array}{rl}(x-\mu {)}^{\mathrm{T}}{\Sigma }^{-1}(x-\mu )& =x^{\mathrm{T}}{\Sigma }^{-1}x-x^{\mathrm{T}}{\Sigma }^{-1}\mu -{\mu }^{\mathrm{T}}{\Sigma }^{-1}x+{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \\ & =x^{\mathrm{T}}{\Sigma }^{-1}x-2x^{\mathrm{T}}{\Sigma }^{-1}\mu +{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \\ \sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& =\sum _{i=1}^m\left(x^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}{x}^{(i)}-2x^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\mu +{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \right)\\ & =\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}{x}^{(i)}-2\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\mu +m{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \end{array}$

将此结论代入$\frac{\partial \ln L}{\partial \mu }$中：
$\begin{array}{rl}\frac{\partial \ln L}{\partial \mu }& =\frac{\partial }{\partial \mu }\left(-\frac{1}{2}\left(\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}{x}^{(i)}-2\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\mu +m{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \right)\right)\\ & =\frac{\partial }{\partial \mu }\left(-\frac{1}{2}\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}{x}^{(i)}\right)+\frac{\partial }{\partial \mu }\left(\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\mu \right)-\frac{1}{2}\frac{\partial }{\partial \mu }m{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu \\ & =\underset{a_1}{\underbrace{\frac{\partial }{\partial \mu }\left(\sum _{i=1}^m{x}^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\mu \right)}}-\underset{a_2}{\underbrace{\frac{1}{2}\frac{\partial }{\partial \mu }m{\mu }^{\mathrm{T}}{\Sigma }^{-1}\mu }}\end{array}$
μ和∑是矩阵，根据矩阵的求导法则：

$$iff(\mathbf{X})={\mathbf{A}}^{\mathrm{T}}\mathbf{X},then\frac{\mathrm{d}f}{\mathrm{d}\mathbf{X}}=\mathbf{A}$$

$$\Rightarrow a_1=\sum _{i=1}^m{\left(x^{(i{)}^{\mathrm{T}}}{\Sigma }^{-1}\right)}^{\mathrm{T}}=\sum _{i=1}^m{\sum }^{-{}^{1^{\mathrm{T}}}}{x}^{(i)}$$
因为${\sum }^{-1}$是一个对称矩阵，所以：
$${\Sigma }^{-1^T}={\Sigma }^{-1},a1=\sum _{i=1}^m{\Sigma }^{-1^T}{x}^{(i)}=\sum _{i=1}^m{\Sigma }^{-1}{x}^{(i)}={\Sigma }^{-1}\sum _{i=1}^m{x}^{(i)}$$
根据矩阵的求导法则：
$iff(X)=X^{T}AX,then\frac{df}{dX}=AX+A^{T}X$
$whenA=A^T,then\frac{df}{dX}=AX+A^{T}X=2AX$
$\Rightarrow a_2=\frac{1}{2}\frac{\partial (m{\mu }^T{\Sigma }^{-1})}{\partial \mu }=m{\Sigma }^{-1}\mu$

将$a_1,a_2$代入$\frac{\partial \ln L}{\partial \mu }$中：

$$\begin{array}{c}\frac{\partial \ln L}{\partial \mu }=a_1+a_2={\sum }_{i=1}^m{\sum }^{-1}{x}^{(i)}-m{\sum }^{-1}\mu =0\\ \hat{\mu }=\frac{{\sum }^{-1}{\sum }_{i=1}^m{x}^{(i)}}{m{\sum }^{-1}}=\frac{1}{m}{\sum }_{i=1}^m{x}^{(i)}=\bar{x}\end{array}$$

再看对 $\Sigma$ 求偏导：

$\begin{array}{rl}\frac{\partial \ln L}{\partial \Sigma }& =\frac{\partial }{\partial \Sigma }\left(C-\frac{m}{2}\ln |\Sigma |-\frac{1}{2}\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =-\frac{m}{2}\underset{b_1}{\underbrace{\frac{\partial }{\partial \Sigma }\ln |\Sigma |}}-\frac{1}{2}\underset{b_2}{\underbrace{\frac{\partial }{\partial \sum }\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)}}\end{array}$

$\Sigma$ 和 ${\Sigma }^{-1}$都是实对称矩阵，根据矩阵的求导法则，当A是实对称矩阵是：
$\frac{\partial \ln A}{\partial A}=A^{-1}\Rightarrow b_1=\frac{\partial \ln |\Sigma |}{\partial \Sigma }={\Sigma }^{-1}$

再看$b_2$。设$\omega ,p,q$ 是$\Sigma$第 $p$ 行第$q$列的元素,$E_{pq}$是一个第 $p$ 行第$q$ 列元素为1,其他元素全为0的矩阵， $E$与 ${\Sigma }^{-1}$同阶。根据矩阵的求导公式：
$$\begin{array}{rl}\frac{\partial {\mathbf{X}}^{-1}}{\partial x}=-{\mathbf{X}}^{-1}\frac{\partial \mathbf{X}}{\partial x}{\mathbf{X}}^{-1}& \\ \Rightarrow \frac{\partial {\Sigma }^{-1}}{\partial {\omega }_{pq}}=-{\Sigma }^{-1}\frac{\partial \Sigma }{\partial {\omega }_{pq}}{\Sigma }^{-1}=& -{\Sigma }^{-1}{E}_{pq}{\Sigma }^{-1}\\ \Rightarrow \frac{\partial {\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)}{\partial {\omega }_{pq}}& ={\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\frac{\partial {\Sigma }^{-1}}{\partial {\omega }_{pq}}\left(x^{(i)}-\mu \right)\\ & ={\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\left(-{\Sigma }^{-1}{E}_{pq}{\Sigma }^{-1}\right)\left(x^{(i)}-\mu \right)\\ & =-{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\left({\Sigma }^{-1}{E}_{pq}{\Sigma }^{-1}\right)\left(x^{(i)}-\mu \right)\end{array}$$

已经知道了${\Sigma }^{-1}$是一个对称矩阵，矩阵乘法满足结合律，在不改变矩阵顺序的条件下可以任意加括号：
$\begin{array}{rl}\frac{\partial {\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)}{\partial {\omega }_{pq}}& =-{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\left({\Sigma }^{-1}{E}_{pq}{\Sigma }^{-1}\right)\left(x^{(i)}-\mu \right)\\ & =-\left({\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)E_{pq}\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =-\left({\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\left({\Sigma }^{-1}\right)}^{\mathrm{T}}\right)E_{pq}\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =-\underset{A^{\mathrm{T}}{B}^{\mathrm{T}}=(AB{)}^{\mathrm{T}}}{\underbrace{{\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}^{\mathrm{T}}}}{E}_{pq}\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)\\ & =-{\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_p^{\mathrm{T}}{\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_q\end{array}$
其中$({\Sigma }^{-1}(x^{(i)}-\mu ){)}^T$是一个1 * n的矩阵，$({\Sigma }^{-1}(x^{(i)}-\mu ){)}_p^T$表示矩阵中的第p个元素；${\Sigma }^{-1}(x^{(i)}-\mu )$是一个n*1的矩阵，$({\Sigma }^{-1}(x^{(i)}-\mu ){)}_q$表示矩阵中的第q个元素。将该结论推广到矩阵对矩阵的的求导，根据矩阵对矩阵的求导公式：

$\begin{array}{l}\left[\begin{array}{cccc}\frac{\partial \mathbf{F}}{\partial x_{11}}& \frac{\partial \mathbf{F}}{\partial x_{12}}& \cdots & \frac{\partial \mathbf{F}}{\partial x_{1s}}\\ \frac{\partial \mathbf{F}}{\partial x_{21}}& \frac{\partial \mathbf{F}}{\partial x_{22}}& \cdots & \frac{\partial \mathbf{F}}{\partial x_{2s}}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial \mathbf{F}}{\partial x_{r1}}& \frac{\partial \mathbf{F}}{\partial x_{r2}}& \cdots & \frac{\partial \mathbf{F}}{\partial x_{rs}}\end{array}\right]\\ \\ \\ \frac{\partial }{\partial \Sigma }{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\\ =-\left[\begin{array}{cccc}\frac{\partial }{\partial {\omega }_{11}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \frac{\partial }{\partial {\omega }_{12}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \cdots & \frac{\partial }{\partial {\omega }_{1n}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\\ \frac{\partial }{\partial {\omega }_{21}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \frac{\partial }{\partial {\omega }_{22}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \cdots & \frac{\partial }{\partial {\omega }_{2n}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial }{\partial {\omega }_{n1}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \frac{\partial }{\partial {\omega }_{n1}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)& \cdots & \frac{\partial }{\partial {\omega }_{nn}}{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)\end{array}\right]\end{array}$

其中：
$A_2=\left[\begin{array}{llll}{\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_1& {\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_2& \cdots & {\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_n\end{array}\right]={\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}^{\mathrm{T}}$

在$A_1$中，$({\Sigma }^{-1}(x^{(i)}-\mu ){)}^T$是一个1 * n的矩阵，$({\Sigma }^{-1}(x^{(i)}-\mu ){)}_i^T$表示矩阵中的第i个元素，是一个标量；${\Sigma }^{-1}(x^{(i)}-\mu )$是一个n*1的矩阵，$({\Sigma }^{-1}(x^{(i)}-\mu ){)}_i$表示矩阵中的第i个元素，也是一个标量，因此：

$\begin{array}{l}{\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_i^{\mathrm{T}}={\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}_i\\ \frac{\partial }{\partial \Sigma }{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)=-A_1{A}_2\\ =-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right)\right)}^{\mathrm{T}}\\ =-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\left({\Sigma }^{-1}\right)}^{\mathrm{T}}\\ =-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\end{array}$

终于可以求得$b_2$了：
$$b_2=\frac{\partial }{\partial \Sigma }\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)=\sum _{i=1}^m\left(-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)$$

现在可以看看最终的似然函数：

$\begin{array}{rl}\frac{\partial \ln L}{\partial \Sigma }& =-\frac{m}{2}\underset{b_1}{\underbrace{\frac{\partial }{\partial \Sigma }\ln \left|\Sigma \right|}}-\frac{1}{2}\underset{b_2}{\underbrace{\frac{\partial }{\partial \Sigma }\sum _{i=1}^m{\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\left(x^{(i)}-\mu \right)}}\\ & =-\frac{m}{2}{\Sigma }^{-1}-\frac{1}{2}\sum _{i=1}^m\left(-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)\end{array}$

$I$是单位矩阵，${\Sigma }^{-1}.I={\Sigma }^{-1}$
$\begin{array}{rl}\frac{\partial \ln L}{\partial \Sigma }& =-\frac{m}{2}{\Sigma }^{-1}I-\frac{1}{2}\sum _{i=1}^m\left(-{\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)\\ & =-\frac{m}{2}{\Sigma }^{-1}I+\frac{1}{2}\sum _{i=1}^m\left({\Sigma }^{-1}\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)\\ & =-\frac{1}{2}{\Sigma }^{-1}\left(m\mathbf{I}-\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)\\ & =-\frac{1}{2}{\Sigma }^{-1}\left(m\Sigma {\Sigma }^{-1}-\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}{\Sigma }^{-1}\right)\\ & =-\frac{1}{2}{\Sigma }^{-1}\left(m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\right){\Sigma }^{-1}\\ & =0\end{array}$
等号两侧同时左乘 $\Sigma$ :
$$\begin{array}{c}\begin{array}{rlr}\Sigma \left(-\frac{1}{2}{\Sigma }^{-1}\right)& \left(m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\right){\Sigma }^{-1}& =\Sigma 0\\ -\frac{1}{2}I\left(m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\right){\Sigma }^{-1}& =0& \\ \left(m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\right){\Sigma }^{-1}& =0& \end{array}\end{array}$$

两侧同时右乘 $\Sigma$ :
$\begin{array}{rl}\left(m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\right){\Sigma }^{-1}\Sigma & =0\Sigma \\ m\Sigma -\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}& =0\end{array}$

最终解得：
$\begin{array}{r}\Sigma =\frac{1}{m}\sum _{i=1}^m\left(x^{(i)}-\mu \right){\left(x^{(i)}-\mu \right)}^{\mathrm{T}}\end{array}$

综上所述，多维正态分布的最大似然估计量是：
$$\hat{\mu }=\frac{1}{m}\sum _{i=1}^m{x}^{(i)}=\bar{x}$$
$$\Sigma =\frac{1}{m}\sum _{i=1}^m(x^{(i)}-\mu )(x^{(i)}-\mu {)}^T$$

参考博客，讲的非常好！！！

参考博客