逻辑回归的梯度下降公式详细推导过程

逻辑回归的梯度下降公式

逻辑回归的代价函数公式如下：

$$J(\theta )=-\frac{1}{m}\left[\sum _{i=1}^m{y}^{(i)}\log h_{\theta }\left(x^{(i)}\right)+\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)\right]$$

其梯度下降公式如下：

$${\theta }_j:={\theta }_j-\alpha \sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)x_j^{(i)}$$

详细推导过程

公式回顾

Sigmoid公式及其求导（详细推导过程）：

$$g(x)=\frac{1}{1+e^{-x}}$$

$$g^{\prime }(x)=g(x)(1-g(x))$$

推导过程如下：

$$\begin{array}{rl}\frac{\partial J(\theta )}{\partial {\theta }_J}& =-\frac{1}{m}\sum _{i=1}^m\left(y^{(i)}\frac{1}{h_{\theta }\left(x^{(i)}\right)}\frac{\partial h_{\theta }\left(x^i\right)}{\partial {\theta }_j}-\left(1-y^{(i)}\right)\frac{1}{1-h_{\theta }\left(x^{(i)}\right)}\frac{\partial h_{\theta }\left(x^i\right)}{\partial {\theta }_j}\right)\\ & \\ & =-\frac{1}{m}\sum _{i=1}^m\left(y^{(i)}\frac{1}{g\left({\theta }^T{x}^{(i)}\right)}-\left(1-y^{(i)}\right)\frac{1}{1-g\left({\theta }^T{x}^{(i)}\right)}\right)\cdot \frac{\partial g\left({\theta }^T{x}^{(i)}\right)}{\partial {\theta }_j}\\ & \\ & =-\frac{1}{m}\sum _{i=1}^m\left(y^{(i)}\frac{1}{g\left({\theta }^T{x}^{(i)}\right)}-\left(1-y^{(i)}\right)\frac{1}{1-g\left({\theta }^T{x}^{(i)}\right)}\right)\cdot g\left({\theta }^T{x}^{(i)}\right)\left(1-g\left({\theta }^T{x}^{(i)}\right)\right)x_j^{(i)}\\ & \\ & =-\frac{1}{m}\sum _{i=1}^m(y^{(i)}\left(1-g\left({\theta }^T{x}^{(i)}\right)-\left(1-y^{(i)}\right)g\left({\theta }^T{x}^{(i)}\right)\right)\cdot x_j^{(i)}\\ & \\ & =-\frac{1}{m}\sum _{i=1}^m\left(y^{(i)}-g\left({\theta }^T{x}^{(i)}\right)\right)\cdot x_j^{(i)}\\ & \\ & =\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)\cdot x_j^{(i)}\end{array}$$

参考资料

考研必备数学公式大全: https://blog.csdn.net/zhaohongfei_358/article/details/106039576

Sigmoid函数求导过程：https://blog.csdn.net/zhaohongfei_358/article/details/119274445