chapter 5 Logistic Regression

chapter 5 Logistic Regression

Speech and Language Processing ed3 读书笔记

Generative and Discriminative Classifiers: The most important difference between naive Bayes and logistic regression is that logistic regression is a discriminative classifier while naive Bayes is a generative classifier.

A generative model like naive Bayes makes use of likelihood term, which expresses how to generate the features of a document if we knew it was of class c. By contrast a discriminative model in this text categorization scenario attempts to directly compute $P(c|d)$.

A probability machine learning system for classification has four components:

1. A feature representation of the input. For each input observation $x^{(i)}$, this will be a vector of features $[x_1,x_2,\dots ,x_n]$. We will generally refer to feature $i$ for input $x^{(j)}$ as $x_i^{(j)}$, sometimes simplified as $x_i$, but we will also see the notation $f_i,f_i(x)$, or, for multiclass classification, $f_i(c,x)$.
2. A classification function that computes $\hat{y}$, the estimated class, via $p(y|x)$. In the next section we will introduce the sigmoid and softmax tools for classification.
3. An objective function for learning, usually involving minimizing error on training examples. We will introduce the cross-entropy loss function.
4. An algorithm for optimizing the objective function. We introduce the stochastic gradient descent algorithm.

Logistic regression has two phases:

training: we train the system (specifically the weights $w$ and $b$) using stochastic gradient descent and the cross-entropy loss.

test: Given a test example $x$ we compute $p(y|x)$ and return the higher probability label $y=1$ or $y=0$.

5.1 Classification: the sigmoid

Logistic regression learns, from a training set, a vector of weights $w$ and a bias term $b$.

To make a decision on a test instance $x$,
$$z=w\cdot x+b$$
Since weights are real-valued, the output might even be negative; $z$ ranges from $-\infty$ to $\infty$. To create a probability $P(+|x)$ that $x$ is classified as positive, we’ll pass $z$ through the sigmoid function, $\sigma (z)$. During the process of training, we will use a objective function to characterize the error between the estimated value $\hat{y}$ and the golden labels $y$. In logistic regression, $\sigma (z)$ also serves as the estimated value. As $\hat{y}$ is a real number between 0 and 1, we maps golden labels + and - into real numbers 1 and 0.
$$P(+|x)=P(Y=1|x)=\hat{y}=\sigma (z)=\frac{1}{1+e^{-z}}$$

$$P(-|x)=P(Y=0|x)=1-\hat{y}=1-\sigma (z)=1-\sigma (w\cdot x+b)$$

5.1.1 Example: sentiment classification

Designing features: Features are generally designed by examining the training set with an eye to linguistic intuitions and the linguistic literature on the domain. A careful error analysis on the training or dev set. of an early version of a system often provides insights into features.

• combination features or feature interactions
• feature templates
• representation learning: ways to learn features automatically in an unsupervised way from the input.

Choosing a classifier: Logistic regression has a number of advantages over naive Bayes. Naive Bayes has overly strong conditional independence assumptions. Consider two features which are strongly correlated; in fact, imagine that we just add the same feature $f_1$ twice.

When there are many correlated features, logistic regression will assign a more accurate probability than naive Bayes. So logistic regression generally works better on larger documents or datasets and is a common default. Despite the less accurate probabilities, naive Bayes still often makes the correct classification decision. Furthermore, naive Bayes works extremely well (even better than logistic regression) on very small datasets (Ng and Jordan, 2002) or short documents (Wang and Manning, 2012). Furthermore, naive Bayes is easy to implement and very fast to train (there’s no optimization step). So it’s still a reasonable approach to use in some situations.

5.2 Learning in Logistic Regression

loss function or the cost function: cross-entropy loss

gradient descent: stochastic gradient descent

5.3 The cross-entropy loss function

$$L(\hat{y},y)=\text{How much }\hat{y}\text{ differs from the true }y$$

$$L_{\text{MSE}}(\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2$$

It turns out that this MSE loss, which is very useful for some algorithms like linear regression, becomes harder to optimize (technically, non-convex), when it’s applied to probabilistic classification.

Instead, we use a loss function that prefers the correct class labels of the training example to be more likely. This is called conditional maximum likelihood estimation: we choose the parameters $w,b$ that maximize the log probability of the true $y$ labels in the training data given the observations $x$. The resulting loss function is the negative log likelihood loss, generally called the cross entropy loss.

Let’s derive this loss function, applied to a single observation $x$. We’d like to learn weights that maximize the probability of the correct label $p(y|x)$. Since there are only two discrete outcomes (1 or 0), this is a Bernoulli distribution, and we can express the probability density function $p(y|x)$ that our classifier produces for one observation by combining Eq.2 and Eq.3:
$$p(y|x)={\hat{y}}^y(1-\hat{y}{)}^{1-y}$$
To facilitate differential,
$$\log p(y|x)=\log [{\hat{y}}^y(1-\hat{y}{)}^{1-y}]=y\log \hat{y}+(1-y)\log (1-\hat{y})$$
In order to turn this into loss function (something that we need to minimize), we’ll just flip the sign. The result is the cross-entropy loss $L_{CE}$:
$$\begin{gathered} L_{CE}=-\log p(y|x)=-[y\log \hat{y}+(1-y)\log (1-\hat{y})] \\ =-[y\log \sigma (w\cdot x+b)+(1-y)\log (1-\sigma (w\cdot x+b))] \end{gathered}$$
Why does minimizing this negative log probability do what we want? A perfect classifier would assign probability 1 to the correct outcome (y=1 or y=0) and probability 0 to the incorrect outcome. That means the higher $\hat{y}$ (the closer it is to 1), the better the classifier; the lower $\hat{y}$ is (the closer it is to 0), the worse the classifier. The negative log of this probability is a convenient loss metric since it goes from 0 (negative log of 1, no loss) to infinity (negative log of 0, infinite loss). This loss function also insures that as probability of the correct answer is maximized, the probability of the incorrect answer is minimized; since the two sum to one, any increase in the probability of the correct answer is coming at the expense of the incorrect answer. It’s called the cross-entropy loss, because it is also the formula for the cross-entropy between the true probability distribution $y$ and our estimated distribution $\hat{y}$.

Let’s now extend it from one example to the whole training set: we’ll continue to use the notation that $x^{(i)}$ and $y^{(i)}$ mean the ith training features and training label, respectively. We make the assumption that the training examples are independent:
$$\log p(\text{training labels})=\log \prod _{i=1}^{m}p(y^{(i)}|x^{(i)})=\sum _{i=1}^m\log p(y^{(i)}|x^{(i)})=-\sum _{i=1}^m{L}_{CE}({\hat{y}}^{(i)},y^{(i)})$$
We’ll define the cost function for the whole dataset as the average loss for each example:
$$\begin{gathered} Cost(w,b)=\frac{1}{m}\sum _{i=1}^m{L}_{CE}({\hat{y}}^{(i)},y^{(i)}) \\ =-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}\log \sigma (w\cdot x^{(i)}+b)+(1-y^{(i)})\log (1-\sigma (w\cdot x^{(i)}+b)) \end{gathered}$$

5.4 Gradient Descent

Our goal with gradient descent is to find the optimal weights: minimize the loss function we’ve defined for the model. We’ll explicitly represent the fact that the loss function L is parameterized by the weights, which we’ll refer to in machine learning in general as $\theta$ (in the case of logistic regression $\theta =w,b$):
$$\hat{\theta }={\mathrm{argmin}}_{\theta }\frac{1}{m}\sum _{i=1}^m{L}_{CE}(y^{(i)},x^{(i)};\theta )$$
In each dimension $w_i$ (plus the bias $b$), we express the slope as a partial derivative $\frac{\partial }{\partial w_i}$ of the loss function. The gradient is then defined as a vector of these partials. We’ll represent $\hat{y}$ as $f(x;\theta )$ to make the dependence on $\theta$ more obvious:
$${\nabla }_{\theta }L(f(x;\theta ),y)=\left[\begin{array}{c}\frac{\partial }{\partial w_1}L(f(x;\theta ),y)\\ \frac{\partial }{\partial w_2}L(f(x;\theta ),y)\\ ⋮\\ \frac{\partial }{\partial w_n}L(f(x;\theta ),y)\end{array}\right]$$
The final equation for updating q based on the gradient is thus
$${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }(f(x;\theta ),y)$$

5.4.1 The Gradient for Logistic Regression

$$\frac{\partial L_{CE}(w,b)}{\partial w_j}=[\sigma (w\cdot x+b)-y]x_j$$

This makes use of the function 1fg which evaluates to 1{} if the condition in the brackets is true and to 0 otherwise.
$$\frac{\partial Cost(w,b)}{\partial w_j}=\sum _{i=1}^m[\sigma (w\cdot x^{(i)}+b)-y^{(i)}]x_j^{(i)}$$

5.4.2 The Stochastic Gradient Descent Algorithm

import numpy as np
n_epochs=50
t0,t1=5,50
n_features=10 

def learning_schedule(t):
    return t0/(t+t1)

theta=np.random.rand(n_features,1)

for epoch in range(n_epochs):
    for i in range(m):
        random_index = np.random.randint(m)
        xi=X_train[random_index:random_index+1]
        #Note that using slicing here is to ensure the shape of xi is (n_features,1), and the shape of transpose of xi is (1,n_features)
        yi=y[random_index:random_index+1]
        gradients=2*xi.T.dot(xi.dot(theta)-yi)
        eta=learning_schedule(epoch*m +i)
        theta=theta-eta*gradients

Stochastic gradient descent is called stochastic because it chooses a single random example at a time, moving the weights so as to improve performance on that single example. That can result in very choppy movements, so it’s also common to do minibatch gradient descent, which computes the gradient over batches of training instances rather than a single instance.

The learning rate $\eta$ is a parameter that must be adjusted. If it’s too high, the learner will take steps that are too large, overshooting the minimum of the loss function. If it’s too low, the learner will take steps that are too small, and take too long to get to the minimum. It is most common to begin the learning rate at a higher value, and then slowly decrease it, so that it is a function of the iteration $k$ of training; you will sometimes see the notation ${\eta }_k$ to mean the value of the learning rate at iteration $k$.

5.4.3 Working through an example

$\frac{\partial L_{CE}(w,b)}{\partial b}=\sigma (w\cdot x+b)-y$

5.5 Regularization

To avoid overfitting, a regularization term is added to the objective function, resulting in the following objective:
$$\hat{w}={\mathrm{argmax}}_w\left[\sum _{i=1}^m\log p(y^{(i)}|x^{(i)})-\alpha R(w)\right]$$
The new component, $R(w)$ is called a regularization term, and is used to penalize large weights. Thus a setting of the weights that matches the training data perfectly, but uses many weights with high values to do so, will be penalized more than a setting that matches the data a little less well, but does so using smaller weights.

There are two common regularization terms $R(w)$. L2 regularization is a quadratic function of the weight values, named because it uses the (square of the) L2 norm of the weight values. The L2 norm, $||W|{|}_2$, is the same as the Euclidean distance:
$$R(w)=||w|{|}_2^2=\sum _{j=1}^N{w}_j^2$$
The L2 regularized objective function becomes:
$$\hat{w}={\mathrm{argmax}}_w\left[\sum _{i=1}^m\log P(y^{(i)}|x^{(i)})-\alpha \sum _{j=1}^N{w}_j^2\right]$$
L1 regularization is a linear function of the weight values, named after the L1 norm $||W|{|}_1$, the sum of the absolute values of the weights, or Manhattan distance (the Manhattan distance is the distance you’d have to walk between two points in a city with a street grid like New York):
$$R(w)=||W|{|}_1=\sum _{j=1}^N|w_j|$$
The L1 regularized objective function becomes:
$$\hat{w}={\mathrm{argmax}}_w\left[\sum _{i=1}^m\log p(y^{(i)}|x^{(i)})-\alpha \sum _{j=1}^N|w_j|\right]$$
These kinds of regularization come from statistics, where L1 regularization is called the ‘lasso’ or lasso regression (Tibshirani, 1996) and L2 regression is called ridge regression, and both are commonly used in language processing. L2 regularization is easier to optimize because of its simple derivative (the derivative of $w^2$ is just $2w$), while L1 regularization is more complex (the derivative of |w| is noncontinuous at zero). But where L2 prefers weight vectors with many small weights, L1 prefers sparse solutions with some larger weights but many more weights set to zero. Thus L1 regularization leads to much sparser weight vectors, that is, far fewer features.

Both L1 and L2 regularization have Bayesian interpretations as constraints on the prior of how weights should look. L1 regularization can be viewed as a Laplace prior on the weights. L2 regularization corresponds to assuming that weights are distributed according to a gaussian distribution with mean $\mu =0$. In a gaussian or normal distribution, the further away a value is from the mean, the lower its probability (scaled by the variance $\sigma$). By using a gaussian prior on the weights, we are saying that weights prefer to have the value 0. A gaussian for a weight $w_j$ is
$$\frac{1}{\sqrt{2\pi {\sigma }_j^2}}\exp \left(-\frac{(w_j-{\mu }_j{)}^2}{2{\sigma }_j^2}\right)$$
If we multiply each weight by a gaussian prior on the weight, we are thus maximizing the following constraint:
$$\hat{w}={\mathrm{argmax}}_w\left[\prod _{i=1}^{m}p(y^{(i)}|x^{(i)})\times \prod _{j=1}^N\frac{1}{\sqrt{2\pi {\sigma }_j^2}}\exp \left(-\frac{(w_j-{\mu }_j{)}^2}{2{\sigma }_j^2}\right)\right]$$
which in log space, with $\mu =0$, and assuming $2{\sigma }^2=1$, corresponds to
$$\hat{w}={\mathrm{argmax}}_w\left[\sum _{i=1}^m\log p(y^{(i)}|x^{(i)})-\alpha \sum _{j=1}^N{w}_j^2\right]$$

5.6 Multinomial logistic regression

Multinomial logistic regression, also called softmax regression (or, historically, the maxent classifier). In multinomial logistic regression the target $y$ is a variable that ranges over more than two classes; we want to know the probability of $y$ being in each potential class $c\in C$, $p(y=c|x)$.

The multinomial logistic classifier uses a generalization of the sigmoid, called the softmax function, to compute the probability $p(y=c|x)$. The softmax function takes a vector $z=[z_1,z_2,\dots z_k]$ of $k$ arbitrary values and maps them to a probability distribution, with each value in the range (0,1], and all the values summing to 1. Like the sigmoid, it is an exponential function.
$$\text{softmax}(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^k{e}^{z_j}},1\le i\le k$$
The softmax of an input vector $z=[z_1,z_2,\dots z_k]$ is thus a vector itself:
$$\text{softmax}(z)=\left[\frac{e^{z_1}}{{\sum }_{j=1}^k{e}^{z_j}},\frac{e^{z_2}}{{\sum }_{j=1}^k{e}^{z_j}},\dots ,\frac{e^{z_k}}{{\sum }_{j=1}^k{e}^{z_j}}\right]$$
The probability for each class $c$ is
$$p(y=c|x)=\frac{e^{w_c\cdot x+b_c}}{{\sum }_{j=1}^k{e}^{w_j\cdot x+b_j}}$$

5.6.1 Features in Multinomial Logistic Regression

For multiclass classification the input features need to be a function of both the observation $x$ and the candidate output class $c$. we will use the notation $f_i(c,x)$, meaning feature $i$ for a particular class $c$ for a given observation $x$.

5.6.2 Learning in Multinomial Logistic Regression

The loss function for a single example $x$ is the sum of the logs of the $K$ output classes:
L C E ( y ^ , y ) = − ∑ k = 1 K 1 { y = k } log ⁡ p ( y = k ∣ x ) = − ∑ k = 1 K 1 { y = k } log ⁡ e w k ⋅ x + b k ∑ j = 1 K e w j ⋅ x + b j L_{CE}(\hat y,y)=-\sum_{k=1}^K 1\{y=k\}\log p(y=k|x)=-\sum_{k=1}^K 1\{y=k\}\log\frac{e^{w_k\cdot x+b_k}}{\sum_{j=1}^K e^{w_j\cdot x+b_j}} LCE​(y^​,y)=−k=1∑K​1{y=k}logp(y=k∣x)=−k=1∑K​1{y=k}log∑j=1K​ewj​⋅x+bj​ewk​⋅x+bk​​

∂ L C E ∂ w k = ( 1 { y = k } − p ( y = k ∣ x ) ) x k = ( 1 { y = k } − e w k ⋅ x + b k ∑ j = 1 K e w j ⋅ x + b j ) x k \frac{\partial L_{CE}}{\partial w_k}=(1\{y=k\}- p(y=k|x))x_k=\left(1\{y=k\}-\frac{e^{w_k\cdot x+b_k}}{\sum_{j=1}^K e^{w_j\cdot x+b_j}}\right)x_k ∂wk​∂LCE​​=(1{y=k}−p(y=k∣x))xk​=(1{y=k}−∑j=1K​ewj​⋅x+bj​ewk​⋅x+bk​​)xk​

5.7 Interpreting models

We want our decision to be interpretable. Because the features to logistic regression are often human-designed, one way to understand a classifier’s decision is to understand the role each feature it plays in the decision. Logistic regression can be combined with statistical tests (the likelihood ratio test, or the Wald test); investigating whether a particular feature is significant by one of these tests, or inspecting its magnitude (how large is the weight $w$ associated with the feature?) can help us interpret why the classifier made the decision it makes. This is enormously important for building transparent models.

Furthermore, in addition to its use as a classifier, logistic regression in NLP and many other fields is widely used as an analytic tool for testing hypotheses about the effect of various explanatory variables (features). In such cases, logistic regression allows us to test whether some feature is associated with some outcome above and beyond the effect of other features.

5.8 Advanced: Deriving the Gradient Equation

5.9 Summary

This chapter introduced the logistic regression model of classification.

• Logistic regression is a supervised machine learning classifier that extracts real-valued features from the input, multiplies each by a weight, sums them, and passes the sum through a sigmoid function to generate a probability. A threshold is used to make a decision.
• Logistic regression can be used with two classes (e.g., positive and negative sentiment) or with multiple classes (multinomial logistic regression, for example for n-ary text classification, part-of-speech labeling, etc.).
• Multinomial logistic regression uses the softmax function to compute probabilities. The weights (vector $w$ and bias $b$) are learned from a labeled training set via a loss function, such as the cross-entropy loss, that must be minimized.
• Minimizing this loss function is a convex optimization problem, and iterative algorithms like gradient descent are used to find the optimal weights.
• Regularization is used to avoid overfitting.
• Logistic regression is also one of the most useful analytic tools, because of its ability to transparently study the importance of individual features.