机器学习-白板推导系列笔记（二）-数学基础

此文章主要是结合哔站shuhuai008大佬的白板推导视频：数学基础_150min

全部笔记的汇总贴：机器学习-白板推导系列笔记

一、概述

假设有以下数据：

$$X=(x_1,x_1,\cdots ,x_N{)}^T={\left(\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right)}_{N\times p}$$
其中$x_i\in {\mathbb{R}}^p$且$x_i\stackrel{iid}{\sim }N(\mu ,\Sigma )$
则参数$\theta =(\mu ,\Sigma )$

二、通过极大似然估计高斯分布的均值和方差

（一）极大似然

$${\theta }_{MLE}=\underset{\theta }{\mathrm{argmax}}P(X|\theta )$$

（二）高斯分布

一维高斯分布：$p(x)=\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(x-\mu {)}^2}{2{\sigma }^2})$
多维高斯分布：$p(x)=\frac{1}{(2\pi {)}^{D/2}|\Sigma {|}^{1/2}}exp(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))$

（三）一维高斯分布下的估计

1.关于$\theta$的似然函数

$$\begin{gathered} logP(X|\theta )=log\prod _{i=1}^{N}p(x_i|\theta ) \\ =\sum _{i=1}^{N}log\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}) \\ =\sum _{i=1}^N[log\frac{1}{\sqrt{2\pi }}+log\frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}] \end{gathered}$$

2.通过极大似然估计法求解${\mu }_{MLE}$

$$\begin{gathered} {\mu }_{MLE}=\underset{\mu }{\mathrm{argmax}}logP(X|\theta ) \\ =\underset{\mu }{\mathrm{argmax}}\sum _{i=1}^N-\frac{(x_i-\mu {)}^2}{2{\sigma }^2} \\ =\underset{\mu }{\mathrm{argmin}}\sum _{i=1}^N(x_i-\mu {)}^2 \end{gathered}$$
对$\mu$求导
$$\begin{gathered} \frac{\partial {\sum }_{i=1}^N(x_i-\mu {)}^2}{\partial \mu }=\sum _{i=1}^{N}2(x_i-\mu )(-1)=0 \\ \iff \sum _{i=1}^N(x_i-\mu )=0 \\ \iff \sum _{i=1}^N{x}_i-\underset{N\mu }{\underbrace{\sum _{i=1}^N\mu }}=0 \end{gathered}$$
解得$${\mu }_{MLE}=\frac{1}{N}\sum _{i=1}^N{x}_i$$

3.证明${\mu }_{MLE}$是无偏估计

$$E[{\mu }_{MLE}]=\frac{1}{N}\sum _{i=1}^{N}E[x_i]=\frac{1}{N}\sum _{i=1}^N\mu =\frac{1}{N}N\mu =\mu$$

4.通过极大似然估计法求解${\sigma }_{MLE}$

$$\begin{gathered} {\sigma }_{MLE}^2=\underset{\sigma }{\mathrm{argmax}}P(X|\theta ) \\ =\underset{\sigma }{\mathrm{argmax}}\underset{L}{\underbrace{\sum _{i=1}^N(-log\sigma -\frac{(x_i-\mu {)}^2}{2{\sigma }^2})}} \\ \frac{\partial L}{\partial \sigma }=\sum _{i=1}^N[-\frac{1}{\sigma }+(x_i-\mu {)}^2{\sigma }^{-3}] \\ \iff \sum _{i=1}^N[-{\sigma }^2+(x_i-\mu {)}^2]=0 \\ \iff -\sum _{i=1}^N{\sigma }^2+\sum _{i=1}^N(x_i-\mu {)}^2=0 \\ {\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2 \end{gathered}$$
$\mu$取${\mu }_{MLE}$时，$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2$$

5.证明${\sigma }_{MLE}^2$是有偏估计

要证明${\sigma }_{MLE}^2$是有偏估计就需要判断$E[{\sigma }_{MLE}^2]\stackrel{?}{=}{\sigma }^2$，证明如下：

V a r [ μ M L E ] = V a r [ 1 N ∑ i = 1 N x i ] = 1 N 2 ∑ i = 1 N V a r [ x i ] = 1 N 2 ∑ i =