机器学习笔记1-Supervised learning

1.1 Generalized Linear Models
both of these methods mentioned before are special cases of a broader family of models, called Generalized Linear Models (GLMs). We will also show how other models in the GLM family can be derived and applied to other classification and regression problem.
The exponential family
We say that a class of distributions is in the exponential family if it can be written in the form :

$$p(y:{\eta})=b(y)exp({{\eta}^T}T(y)-a({\eta}))$$
${\eta}$ is called the natural parameter(also called the canonical parameter);
$T(y)$ is the sufficient statistic ;
$a({\eta})$ is the log partition function;
The quantity $e^{-a({\eta})}$ essentially plays the role of a normalization constant, that makes sure the distribution $p(y;{\eta})$ sums/integrates over y to 1.
A fixed choice of T, a and b defines a family (or set) of distributions that is parameterized by ${\eta}$; as we vary ${\eta}$, we then get different distributions within this family.
We now show that the Bernoulli and the Gaussian distributions are examples of exponential family distribution:
$Bernoulli({\phi})$:
the Bernoulli distribution with mean ${\phi}$;
$p(y = 1;{\phi}) = {\phi}; p(y = 0;{\phi}) = 1−{\phi}$
We write the Bernoulli distribution as:
$$p(y;{\phi})={{\phi}^y{(1-{\phi})}^{(1-y)}}$$
$$=exp(ylog{\phi} + (1-y)log(1-{\phi}))$$
$$=exp((log({\frac{\phi}{1-{\phi}}}))y+log(1-{\phi}))$$
$${\eta}=(log({\frac{\phi}{1-{\phi}}}))=log(1+e^{\eta})$$
$$b(y)=1$$
$$T(y)=y$$
$$a({\eta})=-log(1-{\phi})$$
Gaussian distribution
Recall that, when deriving linear regression, the value of σ2 had no effect on our final choice of θ and hθ(x). Thus, we can choose an arbitrary value for σ2 without changing anything. To simplify the derivation below, lets set σ2 = 1 We then have:
$$p(y;{\mu})={\frac{1}{\sqrt{(2{\pi})}}}exp(-{\frac{1}{2}}{(y-{\mu})}^2)$$
$$={\frac{1}{\sqrt{(2{\pi})}}}exp(-{\frac{1}{2}}{y}^2)exp({\mu}y-{\frac{1}{2}}{\mu}^2)$$
Thus, we see that the Gaussian is in the exponential family, with:
$${\eta}={\mu}$$
$$T(y)=y$$
$$a({\eta})={\frac{1}{2}}{\mu}^2={\frac{1}{2}}{\eta}^2$$
$$b(y)={\frac{1}{\sqrt{(2{\pi})}}}exp(-{\frac{1}{2}}{y}^2)$$
There’re many other distributions that are members of the exponential family: The multinomial (which we’ll see later), the Poisson (for modelling count-data; also see the problem set); the gamma and the exponential (for modelling continuous, non-negative random variables, such as timeintervals); the beta and the Dirichlet (for distributions over probabilities); and many more. In the next section, we will describe a general “recipe” for constructing models in which y (given x and θ) comes from any of these distributions.
Constructing GLMs
More generally, consider a classification or regression problem where we would like to predict the value of some random variable y as a function of x. To derive a GLM for this problem, we will make the following three assumptions about the conditional distribution of y given x and about our model:
1. $y | x;{\theta}$ ∼ ExponentialFamily( ${\eta}$). I.e., given x and θ, the distribution of y follows some exponential family distribution, with parameter ${\eta}$.
2. Given x, our goal is to predict the expected value of T(y) given x. In most of our examples, we will have T(y) = y, so this means we would like the prediction h(x) output by our learned hypothesis h to satisfy $h(x) = E[y|x]$. (Note that this assumption is satisfied in the choices for $h_θ(x)$ for both logistic regression and linear regression. For instance, in logistic regression, we had $h_θ(x) = p(y = 1|x;θ) = 0·p(y = 0|x;θ) + 1·p(y = 1|x;θ) = E[y|x;θ].)$
3. The natural parameter η and the inputs x are related linearly: η = $θ_Tx$. (Or, if η is vector-valued, then $η_i = {θ_i}^Tx$.)
Ordinary Least Squares
To show that ordinary least squares is a special case of the GLM family of models, consider the setting where the target variable y (also called the response variable in GLM terminology) is continuous, and we model the conditional distribution of y given x as as a Gaussian N(µ,σ2). (Here, µ may depend x.) So, we let the ExponentialFamily(η) distribution above be the Gaussian distribution. As we saw previously, in the formulation of the Gaussian as an exponential family distribution, we had µ = η. So, we have:
$$hθ(x) = E[y|x;θ]$$
$$= µ$$
$$= η$$
$$= θ^Tx$$
Logistic Regression
We now consider logistic regression. Here we are interested in binary classification, so y ∈{0,1}. Given that y is binary-valued, it therefore seems natural to choose the Bernoulli family of distributions to model the conditional distribution of y given x. In our formulation of the Bernoulli distribution as an exponential family distribution, we had ${\phi} = 1/(1 + e^{−η})$. Furthermore, note that if y|x;θ ∼ Bernoulli( ${\phi}$), then E[y|x;θ] = ${\phi}$. So, following a similar derivation as the one for ordinary least squares, we get:
$$h_{\theta}(x) = E[y|x;{\theta}]$$
$$= {\phi}$$
$$= 1/(1 + e^{−η})$$
$$= 1/(1 + e^{−θ^Tx})$$
So, this gives us hypothesis functions of the form $h_θ(x) = 1/(1 + e^{−θ^Tx})$. If you are previously wondering how we came up with the form of the logistic function $1/(1 + e^{−z})$, this gives one answer: Once we assume that y conditioned on x is Bernoulli, it arises as a consequence of the definition of GLMs and exponential family distributions. To introduce a little more terminology, the function g giving the distribution’s mean as a function of the natural parameter (g(η) = E[T(y);η]) is called the canonical response function. Its inverse, g−1, is called the canonical link function. Thus, the canonical response function for the Gaussian family is just the identify function; and the canonical response function for the Bernoulli is the logistic function