机器学习笔记 ---- Evaluations and Diagnostics on Algorithms

Improvements and Diagnostics on Algorithms

1. How to Evaluate A Hypothesis

Split training set into 2 parts: training set + test set
If $J_{test}(\theta)$ high, $J(\theta)$ low, then overfitting occurs.


Linear Regression Test Error:
Same as $J(\theta)$

Logistic Regression Test Error:

err(hΘ(x),y)={1,0,if hΘ(x)≥0.5 and y=0 or hΘ(x)<0.5 and y=1otherwise e r r ( h Θ ( x ) , y ) = { 1 , if  h Θ ( x ) ≥ 0.5   a n d   y = 0   o r   h Θ ( x ) < 0.5   a n d   y = 1 0 , o t h e r w i s e

$$err(h_\Theta(x),y) = \begin{cases} 1,& \mbox{if } h_\Theta(x) \geq 0.5\ and\ y =0\ or\ h_\Theta(x) < 0.5\ and\ y = 1\newline 0,& \mbox otherwise \end{cases}$$
then
$$Test Error = \frac{1}{M_{test}} \sum err(h_\Theta(x),y)$$

2. Model Selection

Split training set into 3 parts: training set + cross validation set (CV) + test set
1) Optimize the parameters in Θ using the training set for each polynomial degree.
2) Find the polynomial degree d with the least error using the cross validation set.
3) Estimate the generalization error using the test set with $J_{test}(\Theta^{(d)})$, ($d$ = theta from polynomial with lower error);
In reality, CV set and test set should be randomly picked!

3. Diagnosing Bias & Variance

Training error decreases with $d$ increases.
Validation error first decreases, then increases as $d$ becomes bigger.

High Bias:
$J_{CV}(\theta) \approx J_{train}(\theta)$ is high
High Variance:
$J_{CV}(\theta)$ high, $J_{train}(\theta)$ low

4. Choosing $\lambda$ When Doing Regularization

Try $\lambda: = \lambda *2$, Pick the one wth least $J_{CV}(\theta)$ and see its test error
High Bias:
$J_{CV}(\theta) \approx J_{train}(\theta)$ is high, $\lambda$ is big
High Variance:
$J_{CV}(\theta)$ high, $J_{train}(\theta)$ low, $\lambda$ is small

5. Learning Curves

x-axis is m, y-axis is error

High Bias:

If bias is high, adding more training data won’t help.

High Variance:

If variance is high, adding more training data may help.

6. Solutions for Bias & Variance

High Bias:
-more features;
-more polynomials;
-decreasing $\lambda$

High Variance:
-more examples;
-less features;
-increasing $\lambda$

7.Bias & Variance for Neural Network

Small Network: High Bias
Big Network: High Variance, using $\lambda$ doing regularization

8. Error Metrics: Precision & Recall

Put y=1 in presence of rare classes.
- Precision: Of all y=1 predictions, how many are correctly detected?
- Recall: Of all the rare cases, how many are correctly detected?

How to compare precision and recall? Using F score.
F score = $2 \frac{PR}{P+R}$