What is early stopping?

NN practitioners often use nets with many times as many parameters as training cases. E.g., Nelson and Illingworth (1991, p. 165) discuss training a network with 16,219 parameters with only 50 training cases! The method used is called "early stopping" or "stopped training" and proceeds as follows:

1. Divide the available data into training and validation sets.

2. Use a large number of hidden units.

3. Use very small random initial values.

4. Use a slow learning rate.

5. Compute the validation error rate periodically during training.

6. Stop training when the validation error rate "starts to go up".

It is crucial to realize that the validation error is not a good estimate of the generalization error. One method for getting an unbiased estimate of the generalization error is to run the net on a third set of data, the test set, which is not used at all during the training process. For other methods, see "How can generalization error be estimated?"

Early stopping has several advantages:

1. It is fast.

2. It can be applied successfully to networks in which the number of weights far exceeds the sample size.

3. It requires only one major decision by the user: what proportion of validation cases to use.

l        But there are several unresolved practical issues in early stopping:

l        How many cases do you assign to the training and validation sets?

l        Do you split the data into training and validation sets randomly or by some systematic algorithm? How do you tell when the validation error rate "starts to go up"?

 It may go up and down numerous times during training. The safest approach is to train to convergence, then go back and see which iteration had the lowest validation error. For more elaborate algorithms, see Prechelt (1994, 1998). Statisticians tend to be skeptical of stopped training because it appears to be statistically inefficient due to the use of the split-sample technique; i.e., neither training nor validation makes use of the entire sample, and because the usual statistical theory does not apply. However, there has been recent progress addressing both of the above concerns (Wang 1994). Early stopping is closely related to ridge regression. If the learning rate is sufficiently small, the sequence of weight vectors on each iteration will approximate the path of continuous steepest descent down the error surface. Early stopping chooses a point along this path that optimizes an estimate of the generalization error computed from the validation set. Ridge regression also defines a path of weight vectors by varying the ridge value. The ridge value is often chosen by optimizing an estimate of the generalization error computed by cross-validation, generalized cross-validation, or bootstrapping (see "What are cross-validation and bootstrapping?") There always exists a positive ridge value that will improve the expected generalization error in a linear model. A similar result has been obtained for early stopping in linear models (Wang, Venkatesh, and Judd 1994). In linear models, the ridge path lies close to, but does not coincide with, the path of continuous steepest descent; in nonlinear models, the two paths can diverge widely.

reference:

http://www.faqs.org/faqs/ai-faq/neural-nets/part3/