机器学习之不平衡数据集的处理方法

1，不平衡数据集

1.1 定义

不平衡数据集指的是数据集各个类别的样本数目相差巨大。以二分类问题为例，假设正类的样本数量远大于负类的样本数量，这种情况下的数据称为不平衡数据

1.2 举例

在二分类问题中，训练集中class 1的样本数比上class 2的样本数的比值为60:1。使用逻辑回归进行分类，最后结果是其忽略了class 2，将所有的训练样本都分类为class 1

在三分类问题中，三个类别分别为A，B，C，训练集中A类的样本占70%，B类的样本占25%，C类的样本占5%。最后我的分类器对类A的样本过拟合了，而对其它两个类别的样本欠拟合

1.3 实例

训练数据不均衡是常见并且合理的情况，比如：

• 在欺诈交易识别中，绝大部分交易是正常的，只有极少部分的交易属于欺诈交易
• 在客户流失问题中，绝大部分的客户是会继续享受其服务的（非流失对象），只有极少数部分的客户不会再继续享受其服务（流失对象

1.4 导致的问题

如果训练集的90%的样本是属于同一个类的，而我们的分类器将所有的样本都分类为该类，在这种情况下，该分类器是无效的，尽管最后的分类准确度为90%。所以在数据不均衡时，准确度（Accuracy）这个评价指标参考意义就不大了.

实际上，如果不均衡比例超过4:1，分类器就会偏向于大的类别

2. 不平衡数据集常用的处理方法

2.1 扩充数据集

首先想到能否获得更多数据，尤其是小类（该类样本数据极少）的数据

2.2 对数据集进行重采样

a）过采样（over-sampling）：对小类的数据样本进行过采样来增加小类的数据样本个数

import numpy as np
from imblearn.over_sampling import RandomOverSampler
from collections import Counter

x = ['the first record in group 1', 'the second record in group 1' , "the third record in group 1",
     "the first record in group 2", "the second record in group 2"]

y = ['group1','group1','group1','group2','group2']

x = np.asarray(x).reshape(-1, 1)
print(x.shape)
print(x)
print(Counter(y))

ros=RandomOverSampler(random_state=0) #采用随机过采样（上采样）
x_resample,y_resample=ros.fit_sample(x,y)

print(x_resample.shape)
print(x_resample)
print(Counter(y_resample))

(5, 1)
[['the first record in group 1']
 ['the second record in group 1']
 ['the third record in group 1']
 ['the first record in group 2']
 ['the second record in group 2']]
Counter({'group1': 3, 'group2': 2})
(6, 1)
[['the first record in group 1']
 ['the second record in group 1']
 ['the third record in group 1']
 ['the first record in group 2']
 ['the second record in group 2']
 ['the first record in group 2']]
Counter({'group1': 3, 'group2': 3})

b）欠采样（under-sampling）：对大类的数据样本进行欠采样来减少大类的数据样本个数

import numpy as np
from imblearn.under_sampling import RandomUnderSampler
from collections import Counter

x = ['the first record in group 1', 'the second record in group 1' , "the third record in group 1",
     "the first record in group 2", "the second record in group 2"]

y = ['group1','group1','group1','group2','group2']

x = np.asarray(x).reshape(-1, 1)
print(x.shape)
print(x)
print(Counter(y))

rus=RandomUnderSampler(random_state=0,replacement=True) #采用随机欠采样（下采样）
x_resample,y_resample=rus.fit_sample(x,y)

print(x_resample.shape)
print(x_resample)
print(Counter(y_resample))

(5, 1)
[['the first record in group 1']
 ['the second record in group 1']
 ['the third record in group 1']
 ['the first record in group 2']
 ['the second record in group 2']]
Counter({'group1': 3, 'group2': 2})
(4, 1)
[['the first record in group 1']
 ['the second record in group 1']
 ['the first record in group 2']
 ['the second record in group 2']]
Counter({'group1': 2, 'group2': 2})

参考资料
https://blog.csdn.net/asialee_bird/article/details/83714612#%EF%BC%883%EF%BC%89%E4%BA%BA%E9%80%A0%E6%95%B0%E6%8D%AE
https://www.cnblogs.com/always-fight/p/10285904.html