logistic回归参数求解推导

记录一下逻辑回归的参数求解推导过程：

损失函数

线性回归的表达式为：$f(x)=wx+b$，为了消除后面的$b$，令$\theta =[wb],x=[x1{]}^T$，则$f(x)=\theta x$

将其转换为逻辑回归模型：$y=\sigma (f(x))=\sigma \left(\theta x\right)=\frac{1}{1+e^{-\theta x}}$

我们把单个样本看作一个事件，那么这个时间发生的概率为：
$$P(y\mid x)=\{\begin{array}{r}p,y=1\\ 1-p,y=0\end{array}$$
它等价于：$P\left(y_i\mid {\mathbf{x}}_i\right)=p^{y_i}(1-p{)}^{1-y_i}$

如果我们采集到了一组数据一共N个， { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ( x 3 , y 3 ) … ( x N , y N ) } , \left\{\left({x}_{1}, y_{1}\right),\left({x}_{2}, y_{2}\right),\left({x}_{3}, y_{3}\right) \ldots\left({x}_{N}, y_{N}\right)\right\}, {(x1​,y1​),(x2​,y2​),(x3​,y3​)…(xN​,yN​)}, 这个合成在一起的合事件发生的总概率如下：
$$\begin{array}{rl}{P}_{total}& =P(y_1|x_1)P(y_2|x_2)P(y_3|x_3)\dots P(y_N|x_N)\\ & =\prod _{i=1}^N{p}^{y_i}(1-p{)}^{1-y_i}\\ F(\theta )& =ln(P_{total})=\sum _{i=1}^{N}ln(p^{y_i}(1-p{)}^{1-y_i})\\ & =\sum _{i=1}^N{y}_{i}lnp+(1-y_i)ln(1-p)\\ 其中p& =\frac{1}{1+e^{-\theta x}}\end{array}$$
为了符合损失函数的含义，将其定义为为：
$$L(\theta )=-F(\theta )$$

推导

$$\frac{\partial L}{\partial \theta }=\frac{\partial L}{\partial p}\times \frac{\partial p}{\partial \theta }$$

先求$\frac{\partial p}{\partial \theta }$ :
$$\begin{array}{rl}{p}^{\prime }& =(\frac{1}{1+e^{-\theta x}}{)}^{\prime }\\ & =\frac{-1}{(1+e^{-\theta x}{)}^2}\cdot e^{-\theta x}\cdot -x\\ & =\frac{1}{1+e^{-\theta x}}\cdot \frac{e^{-\theta x}}{1+e^{-\theta x}}\cdot x\\ & =p(1-p)x\end{array}$$

求$\frac{\partial F}{\partial \theta }$ :
$$\begin{array}{rl}\nabla F(\theta )& =\nabla (\sum _{i=1}^N{y}_{i}lnp+(1-y_i)ln(1-p))\\ & =\frac{\partial F}{\partial p}\times \frac{\partial p}{\partial \theta }\\ & =(\sum _{i=1}^N{y}_i\frac{1}{p}+(1-y_i)\frac{-1}{1-p})p^{\prime }\\ & =\sum _{i=1}^N{y}_i(1-p)x_i-(1-y_i)p{x}_i\\ & =\sum _{i=1}^N(y_i-p)x_i\end{array}$$
因此 $\frac{\partial L}{\partial \theta }={\sum }_{i=1}^N(p-y_i)x_i$

梯度更新

通过反向传播，$\theta$ 的更新过程如下：
$$\theta :=\theta -\alpha \sum _{i=1}^N(\frac{1}{1+e^{-\theta x_i}})x_i$$

---

参考自：逻辑回归 logistics regression 公式推导