逻辑回归梯度下降公式详细推导

标签（空格分隔）： 机器学习

---

　　在一次面试中被问道了逻辑回归公式的推导，虽然知道原理，但是从来未从头到尾推导过，面试时估计有点困，又不完全熟悉，又略有紧张（借口找的齐全了…）。面试官还算仁慈，没有太严格。但是自己还是应该会熟练掌握才对的…写此博客，以备复习。

　　假设$h_{\theta}(x) = \frac{1}{1+exp(-\theta^T x)}$，且$(x^{(i)},y^{(i)})$为样本对，共有m个样本。则极大似然函数有：

$$\begin{align} l(\theta)&=log(\prod\limits_{i=1}^m h_{\theta}(x^{i})^{y^{(i)}}(1 - h_{\theta}(x^{(i)})^{1-y^{(i)}}) \\ &=\sum\limits_{i=1}^m [y^{(i)}log(h_{\theta}(x^{i})) + (1-y^{(i)})(1 - h_{\theta}(x^{(i)})] \end{align}$$

$$\begin{align} \frac{\partial l(\theta)}{\partial \theta} &= \sum\limits_{i=1}^m [\frac{y^{(i)}}{h_{\theta}(x^{i})}\frac{\partial h_{\theta}(x^{i})}{\partial \theta} + \frac{1-y^{(i)}}{1-h_{\theta}(x^{i})}\frac{\partial h_{\theta}(x^{i})}{\partial \theta}(-1)] \\ &= \sum\limits_{i=1}^m [\frac{\partial h_{\theta}(x^{i})}{\partial \theta} (\frac{y^{(i)}}{h_{\theta}(x^{i})} - \frac{1-y^{(i)}}{1-h_{\theta}(x^{i})})] \\ &= \sum\limits_{i=1}^m [\frac{\partial h_{\theta}(x^{i})}{\partial \theta}(\frac{y^{(i)}(1-h_{\theta}(x^{i}))+(y^{(i)}-1)h_{\theta}(x^{i})}{h_{\theta}(x^{i})(1-h_{\theta}(x^{i}))})] \\ &=\sum\limits_{i=1}^m [\frac{\partial h_{\theta}(x^{i})}{\partial \theta}(\frac{y^{(i)}-h_{\theta}(x^{i})}{h_{\theta}(x^{i})(1-h_{\theta}(x^{i}))})] \end{align}$$

其中，

$$\frac{\partial h_{\theta}(x^{i})}{\partial \theta}=h_{\theta}(x^{i})(1-h_{\theta}(x^{i}))x^{(j)}$$

化简得：

$$\begin{align} \nabla_\theta l(\theta) = \frac{\partial l(\theta)}{\partial \theta} = \sum\limits_{i=1}^m （y^{(i)}-h_{\theta}(x^{i}))x^{(j)} \end{align}$$