Coursera Machine Learning Week 6 - Advice for Applying Machine Learning

Advice for Applying Machine Learning

1.

You train a learning algorithm, and find that it has unacceptably high error on the test set. You plot the learning curve, and obtain the figure below. Is the algorithm suffering from high bias, high variance, or neither?

Neither
(CHECKED) High bias
High variance

2.

Suppose you have implemented regularized logistic regression to classify what object is in an image (i.e., to do object recognition). However, when you test your hypothesis on a new set of images, you find that it makes unacceptably large errors with its predictions on the new images. However, your hypothesis performs well (has low error) on the training set. Which of the following are promising steps to take? Check all that apply.

Try evaluating the hypothesis on a cross validation set rather than the test set.
(CHECKED) Try increasing the regularization parameter λ.
Try decreasing the regularization parameter λ.
(CHECKED) Try using a smaller set of features.

3.

Suppose you have implemented regularized logistic regression to predict what items customers will purchase on a web shopping site. However, when you test your hypothesis on a new set of customers, you find that it makes unacceptably large errors in its predictions. Furthermore, the hypothesis performs poorly on the training set. Which of the following might be promising steps to take? Check all that apply.

(CHECKED) Try adding polynomial features.
Try using a smaller set of features.
(CHECKED) Try to obtain and use additional features.
Try increasing the regularization parameter λ.

4.

Which of the following statements are true? Check all that apply.

Suppose you are training a regularized linear regression model.The recommended way to choose what value of regularization parameter λ to use is to choose the value of λ which gives the lowest training set error.

(CHECKED) The performance of a learning algorithm on the training set will typically be better than its performance on the test set.

Suppose you are training a regularized linear regression model. The recommended way to choose what value of regularization parameter λ to use is to choose the value of λ which gives the lowest test set error.

(CHECKED) Suppose you are training a regularized linear regression model. The recommended way to choose what value of regularization parameter λ to use is to choose the value of λ which gives the lowest cross validation error.

5.

Which of the following statements are true? Check all that apply.

(CHECKED) A model with more parameters is more prone to overfitting and typically has higher variance.

(CHECKED) If a learning algorithm is suffering from high bias, only adding more training examples may not improve the test error significantly.

If a neural network has much lower training error than test error, then adding more layers will help bring the test error down because we can fit the test set better.

(CHECKED) When debugging learning algorithms, it is useful to plot a learning curve to understand if there is a high bias or high variance problem.

Regularized Linear Regression and Bias/Variance

linearRegCostFunction.m

h = X*theta;
J = sum (((h-y).^2))/(2*m)+ lambda * (sum(theta.^2)-theta(1,1)^2)/(2*m);
g = X'*(h-y)/m;
thetaT = theta;
thetaT(1,1)=0;
grad = g + (lambda*thetaT)/m;

learningCurve.m

for i=1:m
    xt = X(1:i,:);
    yt = y(1:i,:);
    theta = trainLinearReg(xt,yt,lambda);
    error_train(i) = linearRegCostFunction(xt,yt,theta,0); 
    error_val(i) = linearRegCostFunction(Xval,yval,theta,0);

polyFeatures.m

for i=1:p
    X_poly(:,i) = X.^i;
end

validationCurve.m

for i = 1:length(lambda_vec)
    lambda = lambda_vec(i);
    theta = trainLinearReg(X,y,lambda);
    error_train(i) = linearRegCostFunction(X,y,theta,0); 
    error_val(i) = linearRegCostFunction(Xval,yval,theta,0);
end

Machine Learning System Design

1.

You are working on a spam classification system using regularized logistic regression. “Spam” is a positive class (y = 1) and “not spam” is the negative class (y = 0). You have trained your classifier and there are m = 1000 examples in the cross-validation set. The chart of predicted class vs. actual class is:

                    Actual Class: 1 Actual Class: 0
Predicted Class: 1  85              890
Predicted Class: 0  15              10

For reference:
Accuracy = (true positives + true negatives) / (total examples)
Precision = (true positives) / (true positives + false positives)
Recall = (true positives) / (true positives + false negatives)
F1 score = (2 * precision * recall) / (precision + recall)

What is the classifier’s F1 score (as a value from 0 to 1)?

Enter your answer in the box below. If necessary, provide at least two values after the decimal point.

tp 85, fp 890
fn 15, tn 10 


Accuracy = (85+10)/1000 = 0.095
Precision = 85/(85+890) = 0.0871795
Recall = 85/(85+15) = 0.85
F1 score = (2*0.0871795*0.85)/(0.0871795+0.85) = 0.15814 = 0.16

2.

Suppose a massive dataset is available for training a learning algorithm. Training on a lot of data is likely to give good performance when two of the following conditions hold true.

Which are the two?

(CHECKED) Our learning algorithm is able to represent fairly complex functions (for example, if we train a neural network or other model with a large number of parameters).

(CHECKED) A human expert on the application domain can confidently predict y when given only the features x
(or more generally, if we have some way to be confident that x contains sufficient information to predict y
accurately).

When we are willing to include high order polynomial features of x (such as x21, x22, x1x2, etc.).

The classes are not too skewed.

3.

Suppose you have trained a logistic regression classifier which is outputing hθ(x).

Currently, you predict 1 if hθ(x)≥threshold, and predict 0 if hθ(x)

(CHECKED) The classifier is likely to now have higher precision.

The classifier is likely to now have higher recall.

The classifier is likely to have unchanged precision and recall, but higher accuracy.

The classifier is likely to have unchanged precision and recall, and thus the same F1 score.

4.

Suppose you are working on a spam classifier, where spam emails are positive examples (y=1) and non-spam emails are negative examples (y=0). You have a training set of emails in which 99% of the emails are non-spam and the other 1% is spam. Which of the following statements are true? Check all that apply.

(CHECKED) If you always predict non-spam (output y=0), your classifier will have an accuracy of 99%.
   a1  a0
p1 tp  fp  0   0
p0 fn  tn  1   99
acc=99%
prec=0/0?
recall=0%

(CHECKED) If you always predict spam (output y=1), your classifier will have a recall of 100% and precision of 1%.
   a1  a0
p1 tp  fp  1   99
p0 fn  tn  0   0
acc=1%
recall=100%
prec=1%

If you always predict spam (output y=1), your classifier will have a recall of 0% and precision of 99%.

(CHECKED) If you always predict non-spam (output y=0), your classifier will have a recall of 0%.

5.

Which of the following statements are true? Check all that apply.

It is a good idea to spend a lot of time collecting a large amount of data before building your first version of a learning algorithm.

After training a logistic regression classifier, you must use 0.5 as your threshold for predicting whether an example is positive or negative.

If your model is underfitting the training set, then obtaining more data is likely to help.

(CHECKED) Using a very large training set makes it unlikely for model to overfit the training data.

(CHECKED) On skewed datasets (e.g., when there are more positive examples than negative examples), accuracy is not a good measure of performance and you should instead use F1 score based on the precision and recall.

-eof-