LR推导

二分类逻辑回归：

假设样本x，预测值$y\in\{1,0\}$。

$$p(Y=1|x)=\frac{exp(wx+b)}{1+exp(wx+b)} = \pi(x)$$
$$p(Y=0|x)=\frac{1}{1+exp(wx+b)}=1-\pi(x)$$

损失函数：

$$\begin{align} L&=\prod_{i=1}^N p(Y_i=1|x)^{y_i}p(Y_i=0|x)^{1-y_i} \\ &=\prod_{i=1}^{N}\pi(x)^{y_i}(1-\pi(x)^{1-y_i} \end{align}$$
对数似然函数：
$$\begin{align} \ell&=logL\\ &=\sum_{i=1}^N y_i log\pi(x) +(1-y_i)log(1-\pi(x))\\ &=\sum_{i=1}^N y_i(wx+b-log(1+exp(wx+b)))+(1-y_i)(-log(1+exp(wx+b)))\\ &=\sum_{i=1}^N y_i(wx+b)+(y_i+1-y_i)(-log(1+exp(wx+b)))\\ &=\sum_{i=1}^N y_i(wx+b)-log(1+exp(wx+b)) \end{align}$$
对对数似然函数关于w求导
$$\begin{align} \frac{\partial \ell}{\partial w^{k}}&=\sum_{i=1}^N y_ix_i^k-\frac{exp(wx+b)x_i^k}{1+exp(wx+b)}\\ &=\sum_{i=1}^N(y_i-(1-\frac{1}{1+exp(wx+b)} ) ) x_i^k\\ &=\sum_{i=}^N(y_i-\pi(x_i))x_i^k \end{align}$$
令上式为0，得到
$$\sum_{i=1}^N(y_i-1+\frac{1}{1+exp(wx+b)})x_i^k=0$$
即
$$\sum_{i=1}^N(1-y_i)x_i^k=\sum_{i=1}^N(\frac{1}{1+exp(wx+b)})x_i^k$$
到这里，我们应该就可以看到逻辑斯蒂回归是没有解析解的。

参考:

1.http://blog.csdn.net/zouxy09/article/details/20319673
2.http://bigbear2017.github.io/logistic-regression/

Written with StackEdit.