机器学习工程实践方法论

Andrew Ng 斯坦福机器学习公开课听课笔记（11）

Key ideas:

1. Diagnostics for debugging learning algorithms
2. Error analyses and ablative analysis
3 . How to get started on a machine learning problem
- premature (statistical) optimization. 过早统计优化

Debugging Learning Algorithms
Motivating example:
. Anti-spam You carefully choose a small set of 100 words to use as features.
. Bayesian logistic regression, implemented with gradient descent, gets 20% test error, which is unacceptably high.

. What to do next?
. Bayesian logistic regression:

. Common approach: Try improving the algorithm in different ways.
- Try getting more training examples
- Try a smaller set of features
- Try a larger set of features
- Try changing the features: Email header vs. email body features
- Run gradient descent for more iterations
- Try Newton’s method
- Use a different value for lambda
- Try using an SVM

Diagnostic for bias vs. variance
Better approach:
- Run diagnostics to figure out what the problem is.
- Fix whatever the problem is

Suppose you suspect the problem is either”
- Overfiting (high bias, 高偏差-过拟合)
- Too few features to classify spam(high variance，高方差-欠拟合).

Diagnostic:
- Variance: Training error will the much lower than test error.
- Bias: Training error will also be high.

方法：检查训练误差与测试误差的差异，判断是高偏差还是高方差。（图形或数值）
Diagnostics tell you what to try next
Bayesian logistic regression, implemented with gradient descent

Fixes to try:
- Try getting more training examples.       Fixes high variance.
- Try a smaller set of features                     Fixes high variance
- Try a larger set of features                        Fixes high bias
- Try changing the features: Try Email header vs. email body features    Fixes high bias
- Run gradient descent for more iterations             Fixes optimization algorithm
- Try Newton’s method                                                Fixes optimization algorithm
- Use a different value for lambda                            Fixes optimization objective
- Try using an SVM                                                      Fixes optimization objective

Optimization algorithm diagnostics
. Bias vs. variance is one common diagnostic

. For other problems. it’s usually up to your own ingenuity to construct your own diagnostics to figure out what’s wrong.

. Another example:
- Bayesian logistic regression gets 2% error on spam, and 2% spam error on non-spam. (Unacceptably high error on non-spam.)
- SVM using a linear kernel gets 10% error on spam, and 0.01% error on non-spam. (Acceptable performance)
- But you want to use logistic regression because of computation efficiency. etc

. What to do next?

More diagnostics
. Other common questions:
- Is the algorithm(gradient descent for logistic regression) converging?
- Are you optimizing the right function?
- I.e. what you care about (weights w higher or non-spam than for spam, 准确率的加权值)
- Bayesian logistic regression? Correct value for lambda?
- SVM? Correct value for C?