Derivative of cost function of logical regression逻辑回归代价函数的一些数学推导

Recently I have some misconceptions of Logistic Regression so I reviewed it and tried to do some mathematical derivations manually.

Generally Speaking

In general, suppose we have $m$ examples and $n$ parameters in each example. The Cost Function of Logistic Regression is
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^{m}log(h_{\theta }(x^{(i)})+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))$$
in which $x^{(i)}$represents the features of $i^{th}$ example, $y^{(i)}$ denotes the label of $i^{th}$ example,
$$\log (h_{\theta }(x^{(i)}))=\log (\frac{1}{1+e^{-{\theta }^T{x}^{(i)}}})=-\log (1+e^{-{\theta }^T{x}^{(i)}})\text{\hspace{1em}(1)}$$
and
KaTeX parse error: No such environment: align* at position 8: \begin{̲a̲l̲i̲g̲n̲*̲}̲ &\log(1-h_\the…

In formula$(1)(2)$, we use Sigmoid Function as activate function, so we use updated $\log (h_{\theta }(x^{(i)}))$ and $\log (1-h_{\theta }(x^{(i)})$ by formula $(1)(2)$ as:
$$\begin{array}{r}J(\theta )=-\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log (1+e^{-{\theta }^T{x}^{(i)}})+(1-y^{(i)})(-{\theta }^T{x}^{(i)}-\log (1+e^{-{\theta }^T{x}^{(i)}})\left)\right]\end{array}\text{\hspace{1em}(3)}$$

Mathematic Deduction of $J(\theta )$

we could observe and further do some mathematical derivation on formula$(3)$:

$$\begin{array}{rl}J(\theta )& =-\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log (1+e^{-{\theta }^T{x}^{(i)}})-{\theta }^T{x}^{(i)}+y^{(i)}{\theta }^T{x}^{(i)}-\log (1+e^{-{\theta }^T{x}^{(i)}})+y^{(i)}\log (1+e^{-{\theta }^T{x}^{(i)}})\\ & =-\frac{1}{m}\sum _{i=1}^m[-{\theta }^T{x}^{(i)}+y^{(i)}{\theta }^T{x}^{(i)}-\log (1+e^{-{\theta }^T{x}^{(i)}})\end{array}\text{\hspace{1em}(4)}$$

Here note that ${\theta }^T{x}^{(i)}=\log (e^{{\theta }^{T}x(i)})$, we continue doing some derivations using formula$(4)$:
$$\begin{array}{rl}J(\theta )& =-\frac{1}{m}\sum _{i=1}^m[-\log (e^{{\theta }^{T}x(i)})+y^{(i)}{\theta }^T{x}^{(i)}-\log (1+e^{-{\theta }^T{x}^{(i)}})\\ & =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{\theta }^T{x}^{(i)}-\log ((1+e^{-{\theta }^T{x}^{(i)}})(e^{{\theta }^{T}x(i)})\left)\right]\\ & =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{\theta }^T{x}^{(i)}-\log (1+e^{{\theta }^T{x}^{(i)}})]\end{array}$$

So the Cost Function of Logistic Regression can be written as:
$$\begin{array}{rl}J(\theta )& =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{\theta }^T{x}^{(i)}-\log (1+e^{{\theta }^T{x}^{(i)}})]\end{array}\text{\hspace{1em}(5)}$$

Mathematic Deduction of $\frac{\partial J}{\partial {\theta }_j}$

From above process, we can calculate $\frac{\partial J}{\partial {\theta }_i}$ more easily using formula$(5)$:
$$\begin{array}{rl}\frac{\partial J}{\partial {\theta }_j}& =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{x}_j^{(i)}-\frac{x_j^{(i)}{e}^{{\theta }^T{x}^{(i)}}}{(1+e^{{\theta }^T{x}^{(i)}})}]\\ & =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{x}_j^{(i)}-\frac{x_j^{(i)}}{(1+e^{-{\theta }^T{x}^{(i)}})}]\end{array}\text{\hspace{1em}(6)}$$
Notice that:
$$h_{\theta }(x^{(i)})=\frac{1}{1+e^{-{\theta }^T{x}^{(i)}}}$$
So the formula$(6)$ can be written as:

$$\begin{array}{rl}\frac{\partial J}{\partial {\theta }_j}& =-\frac{1}{m}\sum _{i=1}^m[y^{(i)}{x}_j^{(i)}-h_{\theta }(x^{(i)})x_j^{(i)}]\\ & =\frac{1}{m}\sum _{i=1}^m\left[x_j^{(i)}\right(h_{\theta }(x^{(i)})-y^{(i)}\left)\right]\end{array}\text{\hspace{1em}(7)}$$
Until now, we get the $\frac{\partial J}{\partial {\theta }_j}$. If we want to use MATLAB or Python to program, we could use some Liner Algebra knowledge to deduct it more deeply:
$$\begin{array}{rl}\left[\begin{array}{c}\frac{\partial J}{\partial {\theta }_0}\\ \frac{\partial J}{\partial {\theta }_1}\\ \frac{\partial J}{\partial {\theta }_2}\\ ⋮\\ \frac{\partial J}{\partial {\theta }_n}\end{array}\right]& =\frac{1}{m}\left[\begin{array}{c}{\sum }_{i=1}^m\left(x_0^{(i)}\right(h_{\theta }(x^{(i)})-y^{(i)}\left)\right)\\ {\sum }_{i=1}^m\left(x_1^{(i)}\right(h_{\theta }(x^{(i)})-y^{(i)}\left)\right)\\ {\sum }_{i=1}^m\left(x_2^{(i)}\right(h_{\theta }(x^{(i)})-y^{(i)}\left)\right)\\ ⋮\\ {\sum }_{i=1}^m\left(x_n^{(i)}\right(h_{\theta }(x^{(i)})-y^{(i)}\left)\right)\end{array}\right]\end{array}\text{\hspace{1em}(8)}$$
In formula $(8)$, we found that the $(h_{\theta }(x^{(i)})-y^{(i)})$ is a scalar(single number), and the $x_j^{(i)}$denotes the $j^{th}$ parameter in the $i^{th}$ example, which also is constant, so we can conclude that:
$$\begin{array}{rl}\left[\begin{array}{c}\frac{\partial J}{\partial {\theta }_0}\\ \frac{\partial J}{\partial {\theta }_1}\\ \frac{\partial J}{\partial {\theta }_2}\\ ⋮\\ \frac{\partial J}{\partial {\theta }_n}\end{array}\right]& =\frac{1}{m}\sum _{i=1}^m\left(\right(h_{\theta }(x^{(i)})-y^{(i)}\left)x^{(i)}\right)\end{array}\text{\hspace{1em}(9)}$$
in which
$$x^{(i)}=\left[\begin{array}{c}{x}_0^{(i)}\\ x_1^{(i)}\\ x_2^{(i)}\\ ⋮\\ x_n^{(i)}\end{array}\right]$$
Finally using the above formula we have:
$$\begin{array}{r}\left[\begin{array}{c}\frac{\partial J}{\partial {\theta }_0}\\ \frac{\partial J}{\partial {\theta }_1}\\ \frac{\partial J}{\partial {\theta }_2}\\ ⋮\\ \frac{\partial J}{\partial {\theta }_n}\end{array}\right]=\frac{1}{m}{X}^T(h_{\theta }(x)-y)\end{array}\text{\hspace{1em}(10)}$$
where
$$\begin{array}{rl}& X=\left[\begin{array}{cccccc}{x}_0^{(1)}& x_1^{(1)}& x_2^{(1)}& x_3^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& x_2^{(2)}& x_3^{(2)}& \cdots & x_n^{(2)}\\ x_0^{(3)}& x_1^{(3)}& x_2^{(3)}& x_3^{(3)}& \cdots & x_n^{(3)}\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮\\ x_0^{(m)}& x_1^{(m)}& x_2^{(m)}& x_3^{(m)}& \cdots & x_n^{(m)}\end{array}\right]\\ & \\ & \\ & h_{\theta }(x)-y=\left[\begin{array}{c}{h}_{\theta }(x^{(1)})-y^{(1)}\\ h_{\theta }(x^{(2)})-y^{(2)}\\ h_{\theta }(x^{(3)})-y^{(3)}\\ ⋮\\ h_{\theta }(x^{(m)})-y^{(m)}\end{array}\right]\end{array}$$
We could easily program Logistic Regression Derivatives using formula $(10)$.

An important detail

Note that $x_0^{(i)}$denotes the Bias in every example, which constantly equals to $1$. Nothing considerable unless if we add regularization methods for avoiding overfitting problem.

Examples in MATLAB

In this part, I use MATLAB to program the Liner Regression Cost Function. In order to avoid overfitting, I do some regularization operations (L2 regularization) to improve the quality of parameters.

function [J, grad] = lrCostFunction(theta, X, y, lambda)
%LRCOSTFUNCTION Compute cost and gradient for logistic regression with 
%regularization
%   J = LRCOSTFUNCTION(theta, X, y, lambda) computes the cost of using
%   theta as the parameter for regularized logistic regression and the
%   gradient of the cost w.r.t. to the parameters. 

% Initialize some useful values
m = length(y); % number of training examples

% You need to return the following variables correctly 
J = 0;
grad = zeros(size(theta));

% =============================================================
h_x = sigmoid(X*theta);

grad = 1/m .*(X' * (h_x - y));
temp = theta; 
temp(1) = 0;   % because we don't add anything for j = 0  
J = 1/m*(sum(-y.*log(h_x) - (1-y) .* log(1-h_x))) + lambda/(2*m) .* sum(temp.^2);
grad = grad + lambda ./ m .* temp;

% =============================================================

grad = grad(:);

end

function g = sigmoid(z)
%SIGMOID Compute sigmoid functoon
%   J = SIGMOID(z) computes the sigmoid of z.

g = 1.0 ./ (1.0 + exp(-z));
end

reference

Andrew Ng, Machine Learning Course in Coursera: https://www.coursera.org/learn/machine-learning
Neural Networks and Deep Learning: https://nndl.github.io/