逻辑回归公式详细推导（LR推导）

逻辑回归公式详细推导（LR推导）

0 逻辑回归介绍及准备知识

逻辑回归本质上是线性回归，只是在特征映射到结果的过程中加入了$\sigma$(z)函数，其中$\sigma (z)=\frac{1}{1+e^{-z}}$.

逻辑回归线性边界的形式如下：$f(x)={\theta }_0+{\theta }_1{x}_1+...+{\theta }_n{x}_n={\sum }_{i=1}^n{\theta }_i{x}_i={\theta }^{T}X$

构造假设函数（预测函数）：$h_{\theta }(x)=\sigma (f(x))=\sigma ({\theta }^{T}X)=\frac{1}{1+e^{-{\theta }^{T}X}}$

对于$\sigma (z)$函数求导（后面会用到）：

${\sigma }^{\prime }(z)=\frac{d}{dz}\frac{1}{1+e^{-z}}$
$=\frac{-1}{(\frac{1}{1+e^{-z}}{)}^2}\cdot (1+e^{-z}{)}^{\prime }=\frac{e^{-z}}{(1+e^{-z}{)}^2}$
$=\frac{e^{-z}}{1+e^{-z}}\cdot \frac{1}{1+e^{-z}}=\frac{1}{1+e^{-z}}\cdot (1-\frac{1}{1+e^{-z}})=\sigma (z)\cdot (1-\sigma (z))$

1 预测函数

因为逻辑回归用于二分类问题，故满足二重伯努利公式：

$p(y=1|x;\theta )=h_{\theta }(x)$        \;\;\;\quad\quad 给定$\theta 和x$，对于正样本$y=1$，希望所得概率接近1

$p(y=0|x;\theta )=1-h_{\theta }(x)$ \quad 给定$\theta 和x$，对于负样本$y=0$，希望所得概率接近0

$⟹$ $p(y|x;\theta )=[h_{\theta }(x){]}^y[1-h_{\theta }(x){]}^{1-y}$

2 损失函数

损失函数可以根据最大似然函数得到：

对于m个样本 { ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) , . . . , ( x ( m ) , y ( m ) ) } \{(x^{(1)},y^{(1)}), (x^{(2)},y^{(2)}), ..., (x^{(m)},y^{(m)})\} {(x(1),y(1)),(x(2),y(2)),...,(x(m),y(m))}

似然函数为:$$L(\theta )=\prod _{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )=\prod _{i=1}^m[h_{\theta }(x^{(i)}){]}^{y^{(i)}}[1-h_{\theta }(x^{(i)}){]}^{1-y^{(i)}}$$

对数似然函数为：

$l(\theta )=logL(\theta )=log[{\prod }_{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )]={\sum }_{i=1}^{m}log[p(y^{(i)}|x^{(i)};\theta )]$

= ∑ i = 1 m l o g { [ h θ ( x ( i ) ) ] y ( i ) [ 1 − h θ ( x ( i ) ) ] 1 − y ( i ) } =\sum_{i=1}^mlog{\{[h_\theta(x^{(i)})]^{y^{(i)}}[1-h_\theta(x^{(i)})]^{1-y^{(i)}}\}} =∑i=1m​log{[hθ​(x(i))]y(i)[1−hθ​(x(i))]1−y(i)}

$={\sum }_{i=1}^m[y^{(i)}log(h\theta (x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))]$

因为最大似然函数求最大，而损失函数要求最小，故将$l(\theta )$乘以$-1$，同时为了样本规模对损失函数造成影响，在乘以$\frac{1}{m}$，即损失函数为：
$J(\theta )=-\frac{1}{m}l(\theta )=-\frac{1}{m}{\sum }_{i=1}^m[y^{(i)}log(h\theta (x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))]$

3 更新参数

使用梯度下降法更新参数$\theta$，即${\theta }_j={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )(j=0,1,...,n)$，其中n为特征数量，n+1个参数指n个特征加上一个偏置项（吴恩达老师将这部分的n个特征的参数更新与偏置项更新分别开了）。

计算$\frac{\partial }{\partial {\theta }_j}J(\theta )$部分：

$\frac{\partial }{\partial {\theta }_j}J(\theta )=-\frac{1}{m}\frac{\partial l(\theta )}{\partial {\theta }_j}$

则(用到了链式求导法则)：
$\frac{\partial l(\theta )}{\partial {\theta }_j}=\frac{\partial l(\theta )}{\partial h_{\theta }(x^{(i)})}\frac{\partial h_{\theta }(x^{(i)})}{\partial {\theta }_j}$

$=\frac{\partial l(\theta )}{\partial h_{\theta }(x^{(i)})}\cdot \frac{\partial h_{\theta }(x^{(i)})}{\partial ({\theta }^T{x}^{(i)})}\cdot \frac{\partial ({\theta }^T{x}^{(i)})}{\partial {\theta }_j}$

$=\frac{\partial l(\theta )}{\partial h_{\theta }(x^{(i)})}\cdot \frac{\partial \sigma ({\theta }^T{x}^{(i)})}{\partial ({\theta }^T{x}^{(i)})}\cdot \frac{\partial {\theta }^T{x}^{(i)}}{\partial {\theta }_j}$

= ∑ i = 1 m { [ y ( i ) 1 h θ ( x ( i ) ) − ( 1 − y ( i ) ) 1 1 − h θ ( x ( i ) ) ] ⋅ σ ( θ T x ( i ) ) [ 1 − σ ( θ T x ( i ) ) ] ⋅ x j ( i ) } =\sum_{i=1}^m\{[y^{(i)}\frac{1}{h_\theta(x^{(i)})}-(1-y^{(i)})\frac{1}{1-h_\theta(x^{(i)})}]\cdot\sigma(\theta^Tx^{(i)})[1-\sigma(\theta^Tx^{(i)})]\cdot x_j^{(i)}\} =∑i=1m​{[y(i)hθ​(x(i))1​−(1−y(i))1−hθ​(x(i))1​]⋅σ(θTx(i))[1−σ(θTx(i))]⋅xj(i)​}

= ∑ i = 1 m { [ y ( i ) 1 h θ ( x ( i ) ) − ( 1 − y ( i ) ) 1 1 − h θ ( x ( i ) ) ] ⋅ h θ ( x ( i ) ) [ 1 − h θ ( x ( i ) ) ] ⋅ x j ( i ) } =\sum_{i=1}^m\{[y^{(i)}\frac{1}{h_\theta(x^{(i)})}-(1-y^{(i)})\frac{1}{1-h_\theta(x^{(i)})}]\cdot h_\theta(x^{(i)})[1-h_\theta(x^{(i)})]\cdot x_j^{(i)}\} =∑i=1m​{[y(i)hθ​(x(i))1​−(1−y(i))1−hθ​(x(i))1​]⋅hθ​(x(i))[1−hθ​(x(i))]⋅xj(i)​}

= ∑ i = 1 m { [ y ( i ) ( 1 − h θ ( x ( i ) ) ) − ( 1 − y ( i ) ) h θ ( x ( i ) ) ] ⋅ x j ( i ) } =\sum_{i=1}^m\{[y^{(i)}(1-h_\theta(x^{(i)}))-(1-y^{(i)})h_\theta(x^{(i)})]\cdot x_j^{(i)}\} =∑i=1m​{[y(i)(1−hθ​(x(i)))−(1−y(i))hθ​(x(i))]⋅xj(i)​}

= ∑ i = 1 m { [ y ( i ) − h θ ( x ( i ) ) ] ⋅ x j ( i ) } =\sum_{i=1}^m\{[y^{(i)}-h_\theta(x^{(i)})]\cdot x_j^{(i)}\} =∑i=1m​{[y(i)−hθ​(x(i))]⋅xj(i)​}

所以， ∂ ∂ θ j J ( θ ) = − 1 m ∑ i = 1 m { [ y ( i ) − h θ ( x ( i ) ) ] ⋅ x j ( i ) } \frac{\partial}{\partial\theta_j}J(\theta)=-\frac{1}{m}\sum_{i=1}^m\{[y^{(i)}-h_\theta(x^{(i)})]\cdot x_j^{(i)}\} ∂θj​∂​J(θ)=−m1​∑i=1m​{[y(i)−hθ​(x(i))]⋅xj(i)​}

故参数更新为

${\theta }_j={\theta }_j-\alpha \cdot (-\frac{1}{m}{\sum }_{i=1}^m[y^{(i)}-h_{\theta }(x^{(i)})]\cdot x_j^{(i)})$

$={\theta }_j-\alpha \cdot \frac{1}{m}{\sum }_{i=1}^m[h_{\theta }(x^{(i)})-y^{(i)}]\cdot x_j^{(i)}(j=0,1,...,n)$

遍历训练数据中所有的样本进行计算，将参数进行更新，这种算法叫做批梯度下降(Batch Gradient Descent)，上式便是批梯度下降。但是，如果样本规模非常大，则计算量也将十分巨大。因此，比较实用的算法是随机梯度下降(Stochastic Gradient Descent)。在SGD算法中，每次更新的迭代，只计算一个样本。这样对于一个具有数百万样本的训练数据，完成一次遍历就会对更新数百万次，效率大大提升。

随机梯度下降公式如下：

${\theta }_j={\theta }_j-\alpha \cdot \frac{1}{m}[h_{\theta }(x^{(i)})-y^{(i)}]\cdot x_j^{(i)}(j=0,1,...,n)$

以上就是逻辑回归的详细推导，用到了大量的概率论知识，如果感到不熟悉的读者，可以去看一下概率论与统计学相关的知识。