MLE(最大似然), LMS(均方误差),KL(散度), H(交叉熵).之间的等效性证明

UTF8gbsn

我们在机器学习中常用的损失函数, 比如MLE(极大似然估计), KL(散度),
H(交叉熵), 均方误差等之间是什么关系? 本文就简单的讲解一下它们之间的关系.

均方误差 vs MLE

如果你的数据分布是来自指数型数据分布, 那么他们是等价的.
具体证明这里就不详细去证明了. 详情参看文章.

Charnes, A., Frome, E. L., Yu, P. L. (1976). 
The equivalence of generalized least squares and
 maximum likelihood estimates in the exponential family. 
Journal of the American Statistical Association, 71(353), 169–171. 
https://doi.org/10.1080/01621459.1976.10481508

为什么不讲, 因为我也没看这篇文章.

KL vs 交叉熵

$$\begin{array}{rl}KL({\theta }^t,\theta )& =\int P(x|{\theta }^t)log\frac{P(x|{\theta }^t)}{P(x|\theta )}\\ H({\theta }^t,\theta )& =-\int P(x|{\theta }^t)logP(x|\theta )\\ H({\theta }^t)& =-\int P(x|{\theta }^t)logP(x|{\theta }^t)\end{array}$$

$$H({\theta }^t,\theta )=KL({\theta }^t,\theta )+H({\theta }^t)$$

其中$\theta$, 是真实概率模型的参数.
而${\theta }^t$是我们估计的概率模型的参数.
由此可见KL散度和交叉熵实际上是一回事.

$$\arg \underset{\theta }{\min }H({\theta }^t,\theta )\equiv \arg \underset{\theta }{\min }KL({\theta }^t,\theta )$$

KL vs MLE

那么, 我们再来看看极大似然估计和KL散度之间的关系.

$$\begin{array}{rl}L(\theta )& =\arg \underset{\theta }{\max }\sum _i^{N}logP(x_i|\theta )\\ & =\arg \underset{\theta }{\max }\sum _i^{N}logP(x_i|\theta )-\sum _i^{N}logP(x_i|{\theta }^t)\\ & =\arg \underset{\theta }{\max }\sum _i^{N}log\frac{P(x_i|\theta )}{P(x_i|{\theta }^t)}\\ & =\arg \underset{\theta }{\min }\sum _i^{N}log\frac{P(x_i|{\theta }^t)}{P(x_i|\theta )}\\ & =\arg \underset{\theta }{\min }\frac{1}{N}\arg \underset{\theta }{\min }\sum _i^{N}log\frac{P(x_i|{\theta }^t)}{P(x_i|\theta )}\\ & =\arg \underset{\theta }{\min }E(log\frac{P(x|{\theta }^t)}{P(x|\theta )})\end{array}$$

如果当$N\to +\infty$,
那么可得上式最后一项等于$log\frac{P(x|{\theta }^t)}{P(x|\theta )}$的期望.
$$\begin{array}{rl}\arg \underset{\theta }{\min }E(log\frac{P(x|{\theta }^t)}{P(x|\theta )})& =\arg \underset{\theta }{\min }\int P(x|{\theta }^t)log\frac{P(x_i|{\theta }^t)}{P(x_i|\theta )}\\ & =\arg \underset{\theta }{\min }KL({\theta }^t,\theta )\end{array}$$

可见MLE和KL散度也是等价的.

总结

由此可见, 在一般意义下MLE, KL, H交叉熵, 均方误差都是等价.
加粗样式