Estimation with Bootstrap

The bootstrap method is a resampling technique used to estimate statistics on a population by sampling a dataset with replacement. It can be used to estimate summary statistics such as the mean or standard deviation.It is used in applied machine learning to estimate the skill of machine learning models when making predictions on data not included in the training data.

• The bootstrap method involves iteratively resampling a dataset with replacement.
• That when using the bootstrap you must choose the size of the sample and the number of repeats.
• The scikit-learn provides a function that you can use to resample a dataset for the bootstrap method.

1.1 Bootstrap Method

The bootstrap method is a statistical technique for estimating quantities about a population by averaging estimates from multiple small data samples.Importantly, samples are constructed by drawing observations from a large data sample one at a time and returning them to the data sample after they have been chosen. This allows a given observation to be included in a given small sample more than once. This approach to sampling is called sampling with replacement. The process for building one sample can be summarized as follows:

1. Choose the size of the sample.

2. While the size of the sample is less than the chosen size

2.1 Randomly select an observation from the dataset

2.2 Add it to the sample

 The bootstrap method can be used to estimate a quantity of a population. This is done by repeatedly taking small samples, calculating the statistic, and taking the average of the calculated statistics. We can summarize this procedure as follows:

1. Choose a number of bootstrap samples to perform

2. Choose a sample size

3. For each bootstrap sample

3.1 Draw a sample with replacement with the chosen size

3.2 Calculate the statistic on the sample

4. Calculate the mean of the calculated sample statistics.

 This is done by training the model on the sample and evaluating the skill of the model on those samples not included in the sample. These samples not included in a given sample are called the out-of-bag samples, or OOB for short. This procedure of using the bootstrap method to estimate the skill of the model can be summarized as follows:

1. Choose a number of bootstrap samples to perform

2. Choose a sample size

3. For each bootstrap sample

3.1 Draw a sample with replacement with the chosen size

3.2 Fit a model on the data sample

3.3 Estimate the skill of the model on the out-of-bag sample.

4. Calculate the mean of the sample of model skill estimates.

Importantly, any data preparation prior to fitting the model or tuning of the hyperparameter of the model must occur within the for-loop on the data sample. This is to avoid data leakage where knowledge of the test dataset is used to improve the model. This, in turn, can result in an optimistic estimate of the model skill. A useful feature of the bootstrap method is that the resulting sample of estimations often forms a Gaussian distribution. In additional to summarizing this distribution with a central tendency, measures of variance can be given, such as standard deviation and standard error. Further, a confidence interval can be calculated and used to bound the presented estimate. This is useful when presenting the estimated skill of a machine learning model.

1.2  Configuration of the Bootstrap

There are two parameters that must be chosen when performing the bootstrap: the size of the sample and the number of repetitions of the procedure to perform.

1.2.1 Sample Size

In machine learning, it is common to use a sample size that is the same as the original dataset.

The bootstrap sample is the same size as the original dataset. As a result, some samples will be represented multiple times in the bootstrap sample while others will not be selected at all.

If the dataset is enormous and computational efficiency is an issue, smaller samples can be used, such as 50% or 80% of the size of the dataset.

1.2.2 Repetitions

The number of repetitions must be large enough to ensure that meaningful statistics, such as the mean, standard deviation, and standard error can be calculated on the sample. A minimum might be 20 or 30 repetitions. A smaller number of repeats can be used, which will further add variance to the estimated statistic. Ideally, the sample of estimates would be as large as possible given the time resources, with hundreds or thousands of repeats.

1.3 Worked Example

We can make the bootstrap procedure concrete with a small worked example .

[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]

The first step is to choose the size of the sample. Here, we will use 4. Next, we must randomly choose the first observation from the dataset. Let’s choose 0.2

sample = [0.2]

This observation is returned to the dataset and we repeat this step 3 more times.

sample = [0.2, 0.1, 0.2, 0.6]

We now have our data sample. The example purposefully demonstrates that the same value can appear zero, one or more times in the sample. Here the observation 0.2 appears twice. An estimate can then be calculated on the drawn sample.

statistic = calculation([0.2, 0.1, 0.2, 0.6])

Those observations not chosen for the sample may be used as out of sample observations.

oob = [0.3, 0.4, 0.5]

In the case of evaluating a machine learning model, the model is fit on the drawn sample and evaluated on the out-of-bag sample.

train = [0.2, 0.1, 0.2, 0.6]
test = [0.3, 0.4, 0.5]
model = fit(train)
statistic = evaluate(model, test)

1.4 Bootstrap in Python

We do not have to implement the bootstrap method manually. The scikit-learn library provides an implementation that will create a single bootstrap sample of a dataset. The resample() scikit-learn function can be used. It takes as arguments the data array, whether or not to sample with replacement, the size of the sample, and the seed for the pseudorandom number generator used prior to the sampling. For example, we can create a bootstrap that creates a sample with replacement with 4 observations and uses a value of 1 for the pseudorandom number generator.

# resample with replacement
boot = resample(data, replace=True, n_samples=4, random_state=1)

Unfortunately, the API does not include any mechanism to easily gather the out-of-bag observations that could be used as a test set to evaluate a fit model. At least in the univariate case we can gather the out-of-bag observations using a simple Python list comprehension

 

# out of bag observations
oob = [x for x in data if x not in boot]

we can tie all of this together with our small dataset used in the worked example of the prior section.

# scikit-learn bootstrap
from sklearn.utils import resample
# data sample
data = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]
# prepare bootstrap sample
boot = resample(data,replace=True, n_samples=4, random_state=1)
print('Bootstrap Sample: %s' % boot)
# out of bag observations
oob = [x for x in data if x not in boot]
print('OOB Sample: %s' % oob)

Running the example prints the observations in the bootstrap sample and those observations in the out-of-bag sample.