【机器学习笔记】Logistics Regression中损失函数求偏导的具体步骤

根据吴恩达老师机器学习课程中在 Logistics Regression 中定义的损失函数：
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\log \left(h_{\theta }\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)\right]$$

对其中每个参数 ${\theta }_j$ 求偏导数，过程如下：

$$\begin{array}{rl}\frac{\partial }{\partial {\theta }_j}J(\theta )& =\frac{\partial }{\partial {\theta }_j}\frac{-1}{m}\sum _{i=1}^m\left[y^{(i)}\log \left(h_{\theta }\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\frac{\partial }{\partial {\theta }_j}\log \left(h_{\theta }\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[\frac{y^{(i)}\frac{\partial }{\partial {\theta }_j}{h}_{\theta }\left(x^{(i)}\right)}{h_{\theta }\left(x^{(i)}\right)}+\frac{\left(1-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\left(1-h_{\theta }\left(x^{(i)}\right)\right)}{1-h_{\theta }\left(x^{(i)}\right)}\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[\frac{y^{(i)}\frac{\partial }{\partial {\theta }_j}\sigma \left({\theta }^T{x}^{(i)}\right)}{h_{\theta }\left(x^{(i)}\right)}+\frac{\left(1-y^{(i)}\right)\frac{\partial }{\partial {\theta }_j}\left(1-\sigma \left({\theta }^T{x}^{(i)}\right)\right)}{1-h_{\theta }\left(x^{(i)}\right)}\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[\frac{y^{(i)}\sigma \left({\theta }^T{x}^{(i)}\right)\left(1-\sigma \left({\theta }^T{x}^{(i)}\right)\right)\frac{\partial }{\partial {\theta }_j}{\theta }^T{x}^{(i)}}{h_{\theta }\left(x^{(i)}\right)}+\frac{-\left(1-y^{(i)}\right)\sigma \left({\theta }^T{x}^{(i)}\right)\left(1-\sigma \left({\theta }^T{x}^{(i)}\right)\right)\frac{\partial }{\partial {\theta }_j}{\theta }^T{x}^{(i)}}{1-h_{\theta }\left(x^{(i)}\right)}\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[\frac{y^{(i)}{h}_{\theta }\left(x^{(i)}\right)\left(1-h_{\theta }\left(x^{(i)}\right)\right)\frac{\partial }{\partial {\theta }_j}{\theta }^T{x}^{(i)}}{h_{\theta }\left(x^{(i)}\right)}-\frac{\left(1-y^{(i)}\right)h_{\theta }\left(x^{(i)}\right)\left(1-h_{\theta }\left(x^{(i)}\right)\right)\frac{\partial }{\partial {\theta }_j}{\theta }^T{x}^{(i)}}{1-h_{\theta }\left(x^{(i)}\right)}\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\left(1-h_{\theta }\left(x^{(i)}\right)\right)x_j^{(i)}-\left(1-y^{(i)}\right)h_{\theta }\left(x^{(i)}\right)x_j^{(i)}\right]\\ & =-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\left(1-h_{\theta }\left(x^{(i)}\right)\right)-\left(1-y^{(i)}\right)h_{\theta }\left(x^{(i)}\right)\right]x_j^{(i)}\\ & =-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}-y^{(i)}{h}_{\theta }\left(x^{(i)}\right)-h_{\theta }\left(x^{(i)}\right)+y^{(i)}{h}_{\theta }\left(x^{(i)}\right)\right]x_j^{(i)}\\ & =-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}-h_{\theta }\left(x^{(i)}\right)\right]x_j^{(i)}\\ & =\frac{1}{m}\sum _{i=1}^m\left[h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right]x_j^{(i)}\end{array}$$

可以发现，该偏导数和线性回归中损失函数对参数 $\theta$ 的偏导数形式是一致的，线性回归的损失函数定义为：

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)}^2$$
其偏导数为：
$$\begin{array}{rl}\frac{\partial J(\theta )}{\partial {\theta }_j}& =\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }\left(x^{(i)}\right)-y^{(i)}\right)\cdot x_j^{(i)}\end{array}$$

将其进行向量化：
$$\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{1}{m}{\overrightarrow{x_j}}^T(X\theta -\vec{y})$$
进一步得到损失函数的梯度：
$$\nabla J(\theta )=\frac{1}{m}{X}^T(X\theta -\vec{y})$$

然后通过该梯度进行参数更新：
$$\theta :=\theta -\frac{\alpha }{m}{X}^T(X\theta -\vec{y})$$

其他内容可参考：吴恩达机器学习