KL散度（相对熵）

1.定义

先给出公式：
$$KL(p(x)||q(x))=\int p(x)\log \frac{p(x)}{q(x)}dx=\int p(x)\log p(x)dx-\int p(x)\log q(x)dx\text{\hspace{1em}(1)}$$
(1)中第二项 $-\int p(x)\log q(x)dx$就是所谓的交叉熵。
(1)恒$\ge 0$，其中第一项是常数，因此把交叉熵作为代价函数，其最小时$KL=0$，$q(x)=p(x)$，学到的模型最优。

3.交叉熵损失函数避免梯度消失

给定$X_{N\times p}$，如果采用平方损失函数,我们有如下模型：
$$\begin{gathered} S_{N\times 1}=X_{N\times p}{W}_{p\times 1} \\ {Y}_{N\times 1}=\sigma (S_{N\times 1}) \\ Loss=\frac{1}{2}||Y-\hat{Y}|{|}_2^2 \end{gathered}$$根据链式求导法则：$$\begin{gathered} {\nabla }_{w}L=\frac{\partial L}{\partial Y}\frac{\partial Y}{\partial S}\frac{\partial S}{\partial W} \\ =X^T\cdot \sigma (S)\odot \sigma (S)\odot (1-\sigma (S)) \end{gathered}$$因为$$\underset{x->\infty }{\mathrm{limits}}\sigma (x)=0$$，因此$\sigma$函数会导致梯度消失现象。
如果采用交叉熵损失函数，我们有如下模型：
$$\begin{gathered} S_{N\times 1}=X_{N\times p}{W}_{p\times 1} \\ {Y}_{N\times 1}=\sigma (S_{N\times 1}) \\ Loss=\sum _{i=1}^N\widehat{y_i}\log y_i+(1-\widehat{y_i})\log (1-y_i) \end{gathered}$$根据链式求导法则：
$$\begin{gathered} \frac{\partial L}{\partial Y}=\left[\begin{array}{c}\cdots \\ \frac{\widehat{y_i}}{y_i}-\frac{1-{\hat{y}}_i}{1-y_i}\\ \cdots \end{array}\right]=\left[\begin{array}{c}\cdots \\ \frac{\widehat{y_i}-y_i}{y_i(1-y_i)}\\ \cdots \end{array}\right] \\ \frac{\partial Y}{\partial S}=\left[\begin{array}{c}\cdots \\ y_i(1-y_i)\\ \cdots \end{array}\right] \end{gathered}$$因此$${\nabla }_{w}L=X^T\cdot \left[\begin{array}{c}\cdots \\ \widehat{y_i}-y_i\\ \cdots \end{array}\right]=X^T\cdot (\hat{Y}-Y)$$消去了$\sigma (S)\odot \sigma (S)$，解决了梯度消失问题。

2.非负性证明

已知 $$\begin{gathered} \ln x\le x-1 \\ forx\in (0,+\infty )\text{\hspace{1em}(2)} \end{gathered}$$因此$$\begin{array}{rl}& -KL(p(x)||q(x))=\int p(x)\log \frac{q(x)}{p(x)}dx\\ & \le \int p(x)(\frac{q(x)}{p(x)}-1)dx\\ & =\int (q(x)-p(x))dx\\ & =\int q(x)dx-\int p(x)dx\\ & =1-1=0\end{array}\text{\hspace{1em}(3)}$$因此$$\begin{gathered} KL(p(x)||q(x))\ge 0 \\ KL(p(x)||q(x))=0iff.p(x)=q(x)\text{\hspace{1em}(4)} \end{gathered}$$

3.参考

https://blog.csdn.net/weixinhum/article/details/85064685