Logistics Regression公式推导

以前一直以为逻辑回归的公式(sigmoid函数)是人为臆造的，学了机器学习课程才知道，它的背后是有贝叶斯定理这样的数学理论支撑的。

为什么是Sigmoid函数？

回顾贝叶斯分类器，贝叶斯分类器是一种生成式学习方。为了获取$P(Y|X)$，我们将它转化为$P(Y)$和P(X|Y)，然后从数据集中估计这两个参数。那么问题来了，我们是否可以直接估计$P(Y|X)$呢？

在逻辑回归模型中，我们做如下假设：

• 设$X$是一个实数值矩阵，表示$n$个特征，$<X_1,X_2,\dots ,X_n>$
• 设$Y$是一个布尔值矩阵
• 假设在给定$Y$后所有的$X_i$之间都是条件独立的，即
  $$\begin{gathered} P(X|Y) \\ =P(X_1,X_2,\dots ,X_n|Y) \\ =P(X_1|Y)P(X_2|Y)\dots P(X_n|Y) \end{gathered}$$
• 假设在给定$Y=y_k$后，每一个$X_i$都服从高斯分布$N({\mu }_{ik},{\sigma }_i)$，即
  $(X_i|Y=y_k)\sim N({\mu }_{ik},{\sigma }_i)$
  $P\left(X_i|Y=0\right)=\frac{1}{\sqrt{2\pi }{\sigma }_i}\exp \left\{-\frac{{\left(X_i-{\mu }_{i0}\right)}^2}{2{\sigma }_i^2}\right\}$
• 假设类别$Y$的先验服从伯努利分布，即
  $P(Y=1)=\pi ,P(Y=0)=1-\pi$

然后我们就可以开始推公式了。

P ( Y = 1 ∣ X ) = P ( X ∣ Y = 1 ) P ( Y = 1 ) P ( X ) = P ( X ∣ Y = 1 ) P ( Y = 1 ) P ( X ∣ Y = 1 ) P ( Y = 1 ) + P ( X ∣ Y = 0 ) P ( Y = 0 ) = 1 1 + P ( X ∣ Y = 0 ) P ( Y = 0 ) P ( X ∣ Y = 1 ) P ( Y = 1 ) = 1 1 + 1 − π π ⋅ P ( X ∣ Y = 0 ) P ( X ∣ Y = 1 ) = 1 1 + 1 − π π ⋅ ∏ i P ( X i ∣ Y = 0 ) ∏ i P ( X i ∣ Y = 1 ) = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ⋅ ∏ i P ( X i ∣ Y = 0 ) ∏ i P ( X i ∣ Y = 1 ) ) } = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ) + ln ⁡ ( ∏ i P ( X i ∣ Y = 0 ) ∏ i P ( X i ∣ Y = 1 ) ) } = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ) + ∑ i ( ln ⁡ ( P ( X i ∣ Y = 0 ) ) − ln ⁡ ( P ( X i ∣ Y = 1 ) ) ) } P\left( Y=1|X \right) \\ =\frac{P\left( X|Y=1 \right) P\left( Y=1 \right)}{P\left( X \right)} \\ =\frac{P\left( X|Y=1 \right) P\left( Y=1 \right)}{P\left( X|Y=1 \right) P\left( Y=1 \right) +P\left( X|Y=0 \right) P\left( Y=0 \right)} \\ =\frac{1}{1+\frac{P\left( X|Y=0 \right) P\left( Y=0 \right)}{P\left( X|Y=1 \right) P\left( Y=1 \right)}} \\ =\frac{1}{1+\frac{1-\pi}{\pi}\cdot \frac{P\left( X|Y=0 \right)}{P\left( X|Y=1 \right)}} \\ =\frac{1}{1+\frac{1-\pi}{\pi}\cdot \frac{\prod_i{P\left( X_i|Y=0 \right)}}{\prod_i{P\left( X_i|Y=1 \right)}}} \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi}\cdot \frac{\prod_i{P\left( X_i|Y=0 \right)}}{\prod_i{P\left( X_i|Y=1 \right)}} \right) \right\}} \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi} \right) +\ln \left( \frac{\prod_i{P\left( X_i|Y=0 \right)}}{\prod_i{P\left( X_i|Y=1 \right)}} \right) \right\}} \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi} \right) +\sum_i{\left( \ln \left( P\left( X_i|Y=0 \right) \right) -\ln \left( P\left( X_i|Y=1 \right) \right) \right)} \right\}} P(Y=1∣X)=P(X)P(X∣Y=1)P(Y=1)​=P(X∣Y=1)P(Y=1)+P(X∣Y=0)P(Y=0)P(X∣Y=1)P(Y=1)​=1+P(X∣Y=1)P(Y=1)P(X∣Y=0)P(Y=0)​1​=1+π1−π​⋅P(X∣Y=1)P(X∣Y=0)​1​=1+π1−π​⋅∏i​P(Xi​∣Y=1)∏i​P(Xi​∣Y=0)​1​=1+exp{ln(π1−π​⋅∏i​P(Xi​∣Y=1)∏i​P(Xi​∣Y=0)​)}1​=1+exp{ln(π1−π​)+ln(∏i​P(Xi​∣Y=1)∏i​P(Xi​∣Y=0)​)}1​=1+exp{ln(π1−π​)+∑i​(ln(P(Xi​∣Y=0))−ln(P(Xi​∣Y=1)))}1​

我们观察分母的最后一项中的
$\ln \left(P\left(X_i|Y=0\right)\right)-\ln \left(P\left(X_i|Y=1\right)\right)$

可以发现由于我们有概率密度函数
$$\begin{gathered} P\left(X_i|Y=0\right)=\frac{1}{\sqrt{2\pi }{\sigma }_i}\exp \left\{-\frac{{\left(X_i-{\mu }_{i0}\right)}^2}{2{\sigma }_i^2}\right\} \\ P\left(X_i|Y=1\right)=\frac{1}{\sqrt{2\pi }{\sigma }_i}\exp \left\{-\frac{{\left(X_i-{\mu }_{i1}\right)}^2}{2{\sigma }_i^2}\right\} \end{gathered}$$

所以有
$$\begin{gathered} \ln \left(P\left(X_i|Y=0\right)\right)-\ln \left(P\left(X_i|Y=1\right)\right) \\ =-\frac{{\left(X_i-{\mu }_{i0}\right)}^2}{2{\sigma }_i^2}+\frac{{\left(X_i-{\mu }_{i1}\right)}^2}{2{\sigma }_i^2} \\ =\frac{-{\left(X_i-{\mu }_{i0}\right)}^2+{\left(X_i-{\mu }_{i1}\right)}^2}{2{\sigma }_i^2} \\ =\frac{-X_{\begin{array}{c}i\end{array}}^2+2X_i{\mu }_{i0}-{\mu }_{i0}^2+X_{\begin{array}{c}i\end{array}}^2-2X_i{\mu }_{i1}+{\mu }_{i1}^2}{2{\sigma }_i^2} \\ =\frac{2\left({\mu }_{i0}-{\mu }_{i1}\right)X_i-{\mu }_{i0}^2+{\mu }_{i1}^2}{2{\sigma }_i^2} \\ =\frac{{\mu }_{i0}-{\mu }_{i1}}{{\sigma }_i^2}{X}_i+\frac{-{\mu }_{i0}^2+{\mu }_{i1}^2}{2{\sigma }_i^2} \end{gathered}$$

因此，

P ( Y = 1 ∣ X ) = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ) + ∑ i ( ln ⁡ ( P ( X i ∣ Y = 0 ) ) − ln ⁡ ( P ( X i ∣ Y = 1 ) ) ) } = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ) + ∑ i ( μ i 0 − μ i 1 σ i 2 X i + − μ i 0 2 + μ i 1 2 2 σ i 2 ) } = 1 1 + exp ⁡ { ln ⁡ ( 1 − π π ) + ∑ i ( − μ i 0 2 + μ i 1 2 2 σ i 2 ) + ∑ i ( μ i 0 − μ i 1 σ i 2 X i ) } P\left( Y=1|X \right) \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi} \right) +\sum_i{\left( \ln \left( P\left( X_i|Y=0 \right) \right) -\ln \left( P\left( X_i|Y=1 \right) \right) \right)} \right\}} \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi} \right) +\sum_i{\left( \frac{\mu _{i0}-\mu _{i1}}{\sigma _{i}^{2}}X_i+\frac{-\mu _{i0}^{2}+\mu _{i1}^{2}}{2\sigma _{i}^{2}} \right)} \right\}} \\ =\frac{1}{1+\exp \left\{ \ln \left( \frac{1-\pi}{\pi} \right) +\sum_i{\left( \frac{-\mu _{i0}^{2}+\mu _{i1}^{2}}{2\sigma _{i}^{2}} \right)}+\sum_i{\left( \frac{\mu _{i0}-\mu _{i1}}{\sigma _{i}^{2}}X_i \right)} \right\}} P(Y=1∣X)=1+exp{ln(π1−π​)+∑i​(ln(P(Xi​∣Y=0))−ln(P(Xi​∣Y=1)))}1​=1+exp{ln(π1−π​)+∑i​(σi2​μi0​−μi1​​Xi​+2σi2​−μi02​+μi12​​)}1​=1+exp{ln(π1−π​)+∑i​(2σi2​−μi02​+μi12​​)+∑i​(σi2​μi0​−μi1​​Xi​)}1​

即
$$P\left(Y=1|X\right)=\frac{1}{1+\exp \left\{\ln \left(\frac{1-\pi }{\pi }\right)+{\sum }_i\left(\frac{-{\mu }_{i0}^2+{\mu }_{i1}^2}{2{\sigma }_i^2}\right)+{\sum }_i\left(\frac{{\mu }_{i0}-{\mu }_{i1}}{{\sigma }_i^2}{X}_i\right)\right\}}$$

令
$w_0=\ln \left(\frac{1-\pi }{\pi }\right)+{\sum }_i\left(\frac{-{\mu }_{i0}^2+{\mu }_{i1}^2}{2{\sigma }_i^2}\right)$
$w_i=\frac{{\mu }_{i0}-{\mu }_{i1}}{{\sigma }_i^2}$
则
P ( Y = 1 ∣ X ) = 1 1 + exp ⁡ { w 0 + ∑ i ( w i X i ) } P\left( Y=1|X \right)=\frac{1}{1+\exp \left\{ w_0+\sum_i{\left( w_iX_i \right)} \right\}} P(Y=1∣X)=1+exp{w0​+∑i​(wi​Xi​)}1​
即
$$P\left(Y=1|X\right)=\frac{1}{1+e^{wX+b}}$$
也就是我们常说的sigmoid函数。

损失函数推导

我们采用极大似然估计(MLE)，但是$P(<X_i,y_i>|w)$很难求，数据集中很难找到这样的数据，因此我们采用更弱一些的条件极大似然(MCLE)，求$P(Y=y_i|X_i,w)$。
在这里我们需要一个真实的场景。