How to Remove Outliers for Machine Learning

After completing this tutorial, you will know:

• That an outlier is an unlikely observation in a dataset and may have one of many causes.
• How to use simple univariate statistics like standard deviation and interquartile range to identify and remove outliers from a data sample.
• How to use an outlier detection model to identify and remove rows from a training dataset in order to lift predictive modeling performance.

目录：

1. What are outliers?
2. Test Datasets
3. Standard Deviation Method

1. What are outliers?
There is no precise way to define and identify outliers in general because of the specifics of each dataset.

2. Test Datasets
We will generate a population 10000 random numbers drawn from a Gaussian distribution with a mean of 50 and a standard of 5.

from numpy.random import seed
from numpy.random import randn
from numpy import mean
from numpy import std

seed(1)
data = 5 * randn(10000) + 50
print(mean(data), std(data))

3. Standard Deviation Method

Univariable case

data_mean = mean(data)
data_std = std(data)
cut_off = data_std * 3
lower, upper = data_mean - cut_off, data_mean + cut_off
outlier = [x for x in data if x<lower or x>upper]
print(len(outlier))

Multiivariable case

from numpy.random import seed
from numpy.random import randn, standard_t
from numpy import mean, std
import numpy as np

seed(1)
#data = 5 * randn(2, 10000) + 50
data = 5 * standard_t(4, [2,10000]) + 50
#print(data[:,0])

noise = 1 * randn(2, 100) + 20
x1 = np.hstack([data[0,:], noise[0,:]])
x2 = np.hstack([data[1,:], noise[1,:]])

cut_off_0, cut_off_1 = std(x1)*5, std(x2)*5
lower_0 = mean(x1) - cut_off_0
upper_0 = mean(x1) + cut_off_0
lower_1 = mean(x2) - cut_off_1
upper_1 = mean(x2) + cut_off_1
print(lower_0, upper_0, ">>>>", lower_1, upper_1)



outlier = []
for i in range(len(x1)):
    if x1[i] < lower_0 or x1[i] > upper_0 or x2[i] < lower_1 or x2[i] > upper_1:
        outlier.append(0)
    else:
        outlier.append(2)

import matplotlib.pyplot as plt
plt.scatter(x1, x2, c = outlier)
plt.show()