python异常值删除,Python从数据中删除异常值

I have a data frame as following:

ID Value

A 70

A 80

B 75

C 10

B 50

A 1000

C 60

B 2000

.. ..

I would like to group this data by ID, remove the outliers from the grouped data (the ones we see from the boxplot) and then calculate mean.

So far

grouped = df.groupby('ID')

statBefore = pd.DataFrame({'mean': grouped['Value'].mean(), 'median': grouped['Value'].median(), 'std' : grouped['Value'].std()})

How can I find outliers, remove them and get the statistics.

解决方案

I believe the method you're referring to is to remove values > 1.5 * the interquartile range away from the median. So first, calculate your initial statistics:

statBefore = pd.DataFrame({'q1': grouped['Value'].quantile(.25), \

'median': grouped['Value'].median(), 'q3' : grouped['Value'].quantile(.75)})

And then determine whether values in the original DF are outliers:

def is_outlier(row):

iq_range = statBefore.loc[row.ID]['q3'] - statBefore.loc[row.ID]['q1']

median = statBefore.loc[row.ID]['median']

if row.Value > (median + (1.5* iq_range)) or row.Value < (median - (1.5* iq_range)):

return True

else:

return False

#apply the function to the original df:

df.loc[:, 'outlier'] = df.apply(is_outlier, axis = 1)

#filter to only non-outliers:

df_no_outliers = df[~(df.outlier)]