Part Ⅳ

Part Ⅳ

4.1 Words list

• logistic regression logistic 回归
• exponential family 指数族
• Newton’s method 牛顿方法
• generalized linear models 广义线性模型

4.2 The Newton’s method

The Newton’s method is a method for solving the root of an equation. Here’s a picture of the Newton’s method in action.

Newton’s method performs the following update:

$\theta :=\theta -\alpha \frac{f(\theta )}{f^{{}^{\prime }}(\theta )}$.

Newton’s method gives a way of getting $f(\theta )=0$.What if we want to use it to maximize some functions $\ell$.The maximize of $\ell$ correspond to points where its first derivative $\ell \prime (\theta )$ is zero.And we obtain update rule:

$\theta :=\theta -\alpha \frac{{\ell }^{{}^{\prime }}(\theta )}{{\ell }^{{}^{\prime \prime }}(\theta )}$.

If $\theta$ is a vector-value.The update rule is given by

$\theta :=\theta -H^{-1}{\nabla }_{\theta }\ell (\theta )$.

4.3 The exponential family

The exponential family be written in the from

$p(y;\eta )=b(y)exp({\eta }^{T}T(y)-a(\eta ))$.

Here, $\eta$ is called the natural parameter (also called the canonical parameter) of the distribution; $T(y)$ is the sufficient statistic (for the distributions we consider, it will often be the case that $T(y)=y$); and $a(\eta )$ is the log partition function. The quantity $e^{-a(\eta )}$） essentially plays the role of a normalization constant, that makes sure the distribution $p(y;\eta )$ sums/integrates over y to 1.

Gaussian and Bernoulli are is familiar to us.They are proved to belong to the exponential family.

In the Bernoulli distribution:

$T(y)=y$.

$b(y)=1$.

$a(\eta )=-log(1-\varphi )=log(1+e^{-\eta })$.

In the Gaussian distribution:

$\eta =\mu$.

$T(y)=y$.

$a(\eta )=-{\mu }^2/2=-{\eta }^2/2$.

$b(y)=(1/\sqrt{2\pi })exp((-y^2/2))$.

4.4 The generalized linear models

The generalized linear models is use to the prediction function of independent variable as the estimated value of dependent variable.

The generalized linear models are based on three assumptions:

1. $y|x;\theta$ ∼ Exponential Family$(\eta )$. I.e., given $x$ and $\theta$, the distribution of $y$ follows some exponential family distribution, with parameter $\eta$.
2. Given $x$, our goal is to predict the expected value of $T(y)$ given $x$.In most of our examples, we will have $T(y)=y$, so this means we would like the prediction $h(x)$ output by our learned hypothesis $h$ to satisfy $h(x)=E[y|x]$. (Note that this assumption is satisfied in the choices for $h_{\theta }(x)$ for both logistic regression and linear regression. For instance, in logistic regression, we had $h_{\theta }(x)=p(y=1|x;\theta )=0\cdot p(y=0|x;\theta )+1\cdot p(y=1|x;\theta )=E[y|x;\theta ].)$.
3. The natural parameter $\eta$ and the inputs $x$ are related linearly: $\eta ={\theta }^{T}x$.(Or, if $\eta$ is vector-valued, then ${\eta }_i={\theta }_i^{T}x$).

There are three examples of the generalized linear models:

1. Ordinary Least Squares.

According to assumption 2,we get:

$h_{\theta }(x)=E[y|x]$.

Because of the Ordinary least Squares is satisfied Gaussian distribution so $E[y|x]=u$ because of $h_{\theta }(x)$ belong to the exponential family,so $h_{\theta }(x)=\mu =\eta$,according to assumption 3 we get $h_{\theta }(x)=\mu =\eta ={\theta }^{T}x$.

2.Logistic Regression.

We get the equation according to above three assumptions:

$h_{\theta }(x)=E[y|x;\theta ]=\varphi =1/(1+e^{-\eta })=1/(1+e^{-{\theta }^{T}x})$.

3.Softmax Regression(For solving the multiple classification).

Softmax Regression is for solving the multiple classification. So the first of all,we need to prove it belong to exponential family.Then we can get:

$\eta =\left[\begin{array}{c}log({\varphi }_1/{\varphi }_k)\\ log({\varphi }_2/{\varphi }_k)\\ ⋮\\ log({\varphi }_{k-1/}{\varphi }_k)\end{array}\right]$.

$a(\eta )=-log(k)$. $b(y)=1$.

because ${\eta }_i=log\frac{{\varphi }_i}{{\varphi }_k}$,so

$e^{{\eta }_i}={\varphi }_i/{\varphi }_k$.

${\varphi }_i=e^{{\eta }_i}{\varphi }_k$.

${\varphi }_k{\sum }_{i=1}^k{e}^{{\eta }_i}={\sum }_{i=1}^k{\varphi }_i=1$.

so the

${\varphi }_i=e^{{\eta }_i}/{\sum }_{j=1}^k{e}^{{\eta }_j}$.

we come to the conclusion

$p(y=i|x;\theta )={\varphi }_i$.

$=\frac{e^{{\eta }_i}}{{\sum }_{j=1}^k{e}^{{\eta }_j}}$.

$=\frac{e^{{\theta }_i^{T}x}}{{\sum }_{j=1}^k{e}^{{\theta }_j^{T}x}}$.

So our hypothesis will output:

$h_{\theta }(x)=E[T(y)|x;\theta ]=\left[\begin{array}{c}{\varphi }_1\\ {\varphi }_2\\ ⋮\\ {\varphi }_{k-1}\end{array}\right]$.

Hence,we can use the log-likelihood to solve the parameters of this model.

$\ell (\theta )={\sum }_{i=1}^{m}logp(y^{(i)}|x^{(i)};\theta )$.
= ∑ i = 1 m l o g ∏ l = 1 k ( e θ l T x i ∑ j = 1 k e θ j T x ( i ) ) 1 { y ( i ) = l } = \sum_{i=1}^m log \prod_{l=1}^k(\frac{e^{\theta_l^Tx^{i}}}{\sum_{j=1}^ke^{\theta_j^T x^{(i)}}})^{1\{y^{(i)}=l\}} =i=1∑m​logl=1∏k​(∑j=1k​eθjT​x(i)eθlT​xi​)1{y(i)=l}.

Using a method such as gradient ascent or Newton’s method.