Coursera 上学习吴恩达的机器学习课程笔记 Week6-Evaluating a Learning Algorithm

Evaluating a Learning Algorithm

coursera machine learning Week 6

Evaluating a learning Algorithm

Deciding What to Try next

• Debugging a learning algorithm

  • Get more training examples
  • Try smaller sets of features
  • Try getting additional features
  • Try adding polynomial features
  • Increasing or decreasing λ

• Machine learning diagnostic

  • Diagnostic:A test that you can run to gain insight what is/isn’t working with a learning algorithm, and gain guidance as to how best to improve its performance.
  • Diagnostics can take time to implement, but doing so can be a very good use of your time.

Evaluating a Hypothesis

• split all the data into a training set and test set

• Training/testing procedure for linear regression

  • Learn parameter theta from training data( minimizing training error J(theta))
  • Compute test set error
  • Misclassification error (0/1 misclassification error)

Model Selection and Train/Validation/Test Sets

• Just because a learning algorithm fits a training set well, that does not mean it is a good hypothesis. It could over fit and as a result your predictions on the test set would be poor. The error of your hypothesis as measured on the data set with which you trained the parameters will be lower than the error on any other data set.

• Given many models with different polynomial degrees, we can use a systematic approach to identify the ‘best’ function. In order to choose the model of your hypothesis, you can test each degree of polynomial and look at the error result.

• One way to break down our dataset into the three sets

  • Training set: 60%
  • Cross validation set: 20%
  • Test set: 20%

• We can now calculate three separate error values for the three different sets using the following method:

  • 1.Optimize the parameters in Θ using the training set for each polynomial degree.
  • 2.Find the polynomial degree d with the least error using the cross validation set.
  • 3.Estimate the generalization error using the test set with Jtest(Θ(d)), (d = theta from polynomial with lower error);
  • This way, the degree of the polynomial d has not been trained using the test set.