【吴恩达 机器学习】逻辑回归的损失函数偏导

1) 逻辑回归（Logistic Regression, Logistic Function, Sigmoid Function）的损失函数为：

$$J(\theta) =- \frac{1}{m}\sum_{i=1}^{m}[y^{(i)}logh_{\theta}(x^{(i)}) + (1-y^{(i)})log(1-h_{\theta}(x^{(i)}))]$$
其中 $h_\theta(x^{(i)}) = \frac{1}{1+exp(-\theta^Tx^{(i)})}$， $\theta^Tx^{(i)} = \theta_0x_0 + \theta_1x_1 + \theta_2x_2 + ...$

2) 对其中一个参数，比如$\theta_1$对其求偏导：

$$\frac{\partial J(\theta)}{\partial \theta_1} = - \frac{1}{m} \sum_{i=1}^{m}[\frac {y^{(i)}}{h_\theta(x^{(i)})lne} \frac{\partial h_\theta(x^{(i)})}{ \partial \theta_1} + \frac{(1-y^{(i)})}{(1-h_\theta(x^{(i)}))lne} \frac{-\partial h_\theta(x^{(i)})}{\partial \theta_1}]$$

3) 提取公因式$\frac{\partial h_\theta(x^{(i)})}{\partial \theta_1}$：

$$=- \frac{1}{m} \sum_{i=1}^{m}[\frac{\partial h_\theta(x^{(i)})}{\partial \theta_1} (\frac {y^{(i)}}{h_\theta(x^{(i)})} + \frac{(1-y^{(i)})}{(1-h_\theta(x^{(i)}))})]$$

$$=- \frac{1}{m} \sum_{i=1}^{m}[ \frac{\partial h_\theta(x^{(i)})}{\partial \theta_1} \frac{y^{(i)} - h_\theta(x^{(i)})}{h_\theta(x^{(i)})(1-h_\theta(x^{(i)}))}]$$

4) 其中假设函数$h_\theta(x^{(i)})$对$\theta_1$求偏导：

$$\frac{\partial h_\theta(x^{(i)})}{\partial \theta_1} = \frac{-1}{(1+e^{-\theta^Tx^{(i)}})^2} e^{-\theta^Tx^{(i)} } (-1) x_1^{(i)} = h_\theta(x^{(i)}) (1 - h_\theta(x^{(i)})) x_1^{(i)}$$

5) 所以$\theta_1$偏导：

$$=- \frac{1}{m} \sum_{i=1}^{m} h_\theta(x) (1 - h_\theta(x)) x_1^{(i)} \frac{y^{(i)} - h_\theta(x^{(i)})}{h_\theta(x^{(i)})(1-h_\theta(x^{(i)}))} = - \frac{1}{m} \sum_{i=1}^{m} [( y^{(i)} - h_\theta(x^{(i)})) x_1^{(i)}]$$

6) 所以对任意的参数$\theta_j$求偏导：

$$\frac{\partial J(\theta)}{\partial \theta_j} = - \frac{1}{m} \sum_{i=1}^{m} [( y^{(i)} - h_\theta(x^{(i)})) x_j^{(i)}]$$