Logistic Regression 逻辑回归 + Newton's Method

Logistic Regression 逻辑回归

内容来自于CS229听课笔记以及百度百科，blog补充
Logistic regression is a classification model, but you can use it to solve regression problems if you want to.
WARNING: do not use linear regression to solve claasification problems.

Logistic regression

sigmoid function: $g(x)=\frac{1}{1+e^{-x}}$
define $h_{\theta }(x)=g({\theta }^{T}X)=\frac{1}{1+e^{-{\theta }^{T}X}},P(y=1|x;\theta )=h_{\theta }(x),P(y=0|x;\theta )=1-h_{\theta }(x)$
combine these two equations : $P(y|x;\theta )=(h_{\theta }(x){)}^y(1-h_{\theta }(x){)}^{1-y}$
use maximum likelihood estimation(MLE):
likelihood $L(\theta )=P(\vec{y}|x;\theta )=\prod _{i=1}^m(h_{\theta }(x^{(i)}){)}^{y^{(i)}}(1-h_{\theta }(x^{(i)}){)}^{1-y^{(i)}}$
we like sum rather than product $l(\theta )=\sum _{i=1}^m[y^{(i)}log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))]$
This function is also called cross entropy function for binary classification. We will talk about it later.
use gradient descent to find the optima:
$$\begin{gathered} {\theta }_j:={\theta }_j+\alpha \frac{\partial }{\partial {\theta }_j}l(\theta ) \\ {\theta }_j:={\theta }_j+\alpha \sum _{i=1}^m(y^{(i)}-h_{\theta }(x^{(i)}))x_j^{(i)} \end{gathered}$$
$l(\theta )$ is a convex function and it doesn’t have local optima.

Softmax Regression

You can regard softmax regression as multiclass logistic regression.
define $K$ – #class
$y$ – label, $y$ is a one hot vector
$\theta$ – parameter, $\theta =\left[\begin{array}{c}-{\theta }_1^T-\\ -{\theta }_2^T-\\ ...\\ -{\theta }_K^T-\end{array}\right]$, ${\theta }_k$ is the parameter of $k^{th}$ class
$h_{\theta }(x)$ – hypothesis, $h_{\theta }(x)=\left[\begin{array}{c}P(y^{(1)}=1|x^{(i)};\theta )\\ ...\\ P(y^{(K)}=1|x^{(i)};\theta )\end{array}\right]=\frac{1}{\sum _{j=1}^{K}exp({\theta }_j^T{X}^{(i)})}\left[\begin{array}{c}exp({\theta }_1^T{X}^{(i)})\\ ...\\ exp({\theta }_K^T{X}^{(i)})\end{array}\right]$
Each value of dimension in hypothesis represents the probability of corresponding dimension.
Cross entropy error function:
L o s s = − ∑ i 1 { a } l n p i Loss = - \sum\limits_i1\{a\}lnp_i Loss=−i∑​1{a}lnpi​
1 { a } 1\{a\} 1{a} is indicate function, if a is true 1 { a } = 1 1\{a\} = 1 1{a}=1, else 1 { a } = 0 1\{a\} = 0 1{a}=0
log or ln are both correct
J ( θ ) = − 1 m [ ∑ i = 1 m ∑ j = 1 K 1 { y j ( i ) = 1 } l n e x p ( θ j T X ( i ) ) ∑ j = 1 K e x p ( θ j T X ( i ) ) ] J(\theta) = -\frac1m[\sum\limits_{i=1}^m\sum\limits_{j=1}^K1\{y^{(i)}_j=1\}ln\frac{exp(\theta^T_jX^{(i)})}{\sum\limits_{j=1}^Kexp(\theta_j^TX^{(i)})}] J(θ)=−m1​[i=1∑m​j=1∑K​1{yj(i)​=1}lnj=1∑K​exp(θjT​X(i))exp(θjT​X(i))​]
usually we would add a weight decay term in case of over parameterization(any parameter is too large):
J ( θ ) = − 1 m [ ∑ i = 1 m ∑ j = 1 K 1 { y j ( i ) = 1 } l n e x p ( θ j T X ( i ) ) ∑ j = 1 K e x p ( θ j T X ( i ) ) ] + λ 2 ∑ i = 1 K ∑ j = 1 n θ i j 2 J(\theta) = -\frac1m[\sum\limits_{i=1}^m\sum\limits_{j=1}^K1\{y^{(i)}_j=1\}ln\frac{exp(\theta^T_jX^{(i)})}{\sum\limits_{j=1}^Kexp(\theta_j^TX^{(i)})}]+\frac\lambda2\sum\limits_{i=1}^K\sum\limits_{j=1}^n\theta_{ij}^2 J(θ)=−m1​[i=1∑m​j=1∑K​1{yj(i)​=1}lnj=1∑K​exp(θjT​X(i))exp(θjT​X(i))​]+2λ​i=1∑K​j=1∑n​θij2​
use gradient descent to solve:
${\theta }_j:={\theta }_j-\alpha {\nabla }_{{\theta }_j}J(\theta )$

θ j : = θ j + α 1 m ∑ i = 1 m [ x ( i ) ( 1 { y j ( i ) = 1 } − P ( y ( j ) = 1 ∣ x ( i ) ; θ ) ) ] + λ θ j \theta_j := \theta_j +\alpha \frac1m\sum\limits_{i=1}^m[x^{(i)}(1\{y^{(i)}_j=1\}- P(y^{(j)}=1|x^{(i)};\theta))]+\lambda\theta_j θj​:=θj​+αm1​i=1∑m​[x(i)(1{yj(i)​=1}−P(y(j)=1∣x(i);θ))]+λθj​

Softmax VS Logistic(one vs all)

If the examples are mutually exclusive, use Softmax Regression(it’s faster).
If the examples are not mutually exclusive, use Logistic Regression with one versus all strategy.

Newton’s Method

$l^{\prime }=f$
${\theta }^{(t+1)}:={\theta }^{(t)}-\frac{f({\theta }^{(i)})}{f^{\prime }({\theta }^{(t)})}$
${\theta }^{(t+1)}:={\theta }^{(t)}-\frac{l^{\prime }({\theta }^{(t)})}{l^{\prime \prime }({\theta }^{(t)})}$
${\theta }^{(t+1)}:={\theta }^{(t)}-H^{-1}{\nabla }_{\theta }l$
$H$ is Hessian matrix.
Use Newton’s method when the number of parameters is small.

Common loss functions

Classification Error $J(\theta )=\frac{erroritems}{allitems}$

Mean Squared Error(MSE) $J(\theta )=\frac{1}{n}\sum _i^n({\hat{y}}_i-y_i{)}^2$

Cross Entropy Error Function L o s s = − 1 N ∑ i 1 { a } l n p i Loss = - \frac1N\sum\limits_i1\{a\}lnp_i Loss=−N1​i∑​1{a}lnpi​