最近接手的一个项目,最后被要求给出查准率-查全率曲线(Precision-Recall Curve)以及曲线对应的查准率、查全率数值序列。查准率-查全率曲线以前使用不多,所以临时coding,算交了差。
概念Precision-Recall Curve和ROC Curve很相似,同样从混淆矩阵说起,二分类问题的混合矩阵如下图:
根据上面混淆矩阵,定义查准率和查全率:
(1)查准率(Precision),又叫准确率,Precision=TP/(TP+FP);
(2)查全率(Recall),又叫召回率,Recall=TP/(TP+FN),可以看出查全率和真正率是同一个值,只是应用场景不同而已。
依次设置不同的决策边界,分别计算出查准率、查全率,然后以查准率为纵坐标、以查全率为横坐标画出的曲线即为Precision-Recall Curve.
将各类别分别定义为正,在同一坐标系中画出得到各类别的Precision-Recall Curve,可以比较模型对各类别的区分效果。
查准率和查全率是一对矛盾体,是此消彼长的关系。要想抓取更多的目标类别即提高查全率,必须扩大决策边界,那么查准率必然会降低;同样,要想提高查准率,必须缩小决策边界,那么查全率必然会降低,所以通常会结合业务需要在两者之间寻找平衡。比较不同Precision-Recall Curve的效果,可以从图中直接观察,外面的曲线比里面的曲线效果要好;但大多情况下,两个曲线会相交,所以更一般的比较指标是曲线下方的面积,面积越大越好。
实现Python实现如下,其中preds是预测结果,可以为预测概率或评分;labels是目标变量,取值0或1;n表示在计算Precison和Recall时切分的数量;asc表示数据是否升序排列,当preds为评分时升序,当preds为概率时降序;返回的数据框PRdf包含TP、FP、FN、P(Precison)、R(Recall)、Area等值。
import numpy as np
from pandas import DataFrame, Series
import matplotlib.pyplot as plt
def PlotPR(preds, labels, n, asc):
# preds is score: asc=1
# preds is prob: asc=0
pred = preds # 预测值
bad = labels # 取1为bad, 0为good
acds = DataFrame({'pred': pred, 'bad': bad}, columns=['pred', 'bad'])
acds['good'] = 1 - acds.bad
if asc==1:
acds1 = acds.sort_values(by='pred', ascending=True)
acds0 = acds.sort_values(by='pred', ascending=False)
elif asc==0:
acds1 = acds.sort_values(by='pred', ascending=False)
acds0 = acds.sort_values(by='pred', ascending=True)
acds1.reset_index(drop=True, inplace=True)
acds0.reset_index(drop=True, inplace=True)
qe = list(np.arange(0, 1, 1.0/n))
qe.append(1)
qe = qe[1:]
ac_index = Series(acds1.index)
ac_index = ac_index.quantile(q = qe)
ac_index = np.ceil(ac_index).astype(int)
ac_index = list(ac_index)
# calculate Precison-Recall data
TP1 = [0]
FP1 = [0]
FN1 = [0]
TP0 = [0]
FP0 = [0]
FN0 = [0]
P1 = [1]
R1 = [0]
Area1 = [0]
P0 = [1]
R0 = [0]
Area0 = [0]
for i in ac_index:
# calculate Precison-Recall data of class 1
tempdf1 = acds1.loc[acds1.index <= i]
TP1 = TP1 + [sum(tempdf1.bad)]
FP1 = FP1 + [len(tempdf1)-sum(tempdf1.bad)]
FN1 = FN1 + [sum(acds1.bad)-sum(tempdf1.bad)]
P1 = P1 + [TP1[-1]/(TP1[-1]+FP1[-1])]
R1 = R1 + [TP1[-1]/(TP1[-1]+FN1[-1])]
Area1 = Area1 + [(P1[-1] + P1[-2]) * (R1[-1] - R1[-2])/2]
# calculate Precison-Recall data of class 0
tempdf0 = acds0.loc[acds0.index <= i]
TP0 = TP0 + [sum(tempdf0.good)]
FP0 = FP0 + [len(tempdf0)-sum(tempdf0.good)]
FN0 = FN0 + [sum(acds.good)-sum(tempdf0.good)]
P0 = P0 + [TP0[-1]/(TP0[-1]+FP0[-1])]
R0 = R0 + [TP0[-1]/(TP0[-1]+FN0[-1])]
Area0 = Area0 + [(P0[-1] + P0[-2]) * (R0[-1] - R0[-2])/2]
PRdf = DataFrame({'TP1': TP1, 'FP1': FP1, 'FN1': FN1,
'TP0': TP0, 'FP0': FP0, 'FN0': FN0,
'P1': P1, 'R1': R1, 'Area1': Area1,
'P0': P0, 'R0': R0, 'Area0': Area0})
# calculate the area
area1 = round(sum(PRdf.Area1), 4)
area0 = round(sum(PRdf.Area0), 4)
# chart
plt.plot(PRdf.R0, PRdf.P0, color='black', linestyle='-', linewidth=2, label='Precision-Recall Curve of class 0 (area=%s)'%area0)
plt.plot(PRdf.R1, PRdf.P1, color='green', linestyle='-', linewidth=2, label='Precision-Recall Curve of class 1 (area=%s)'%area1)
plt.xlim([0, 1])
plt.ylim([0, 1.05])
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precison-Recall Curve', fontsize=15)
plt.legend(loc='best')
return PRdf
下面以网上的breast-cancer数据集为例进行展示(真实数据不便展示),简单的处理后,用随机森林进行训练和预测。
import pandas as pd
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data',header=None)
Xvar = ['X'+str(i) for i in list(range(1, 31))]
var= ['id', 'y'] + Xvar
df.columns = var
df['y'] = df['y'].map({'M': 1, 'B': 0})
X = df.loc[:, Xvar].values
y = df['y'].values
# 抽样
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=1)
# 标准化
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# 训练
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=300, max_depth=3, random_state=1234)
rf = rf.fit(X_train, y_train)
# 预测
y_pred = rf.predict(X_test)
y_prob = rf.predict_proba(X_test)
y_prob_1 = y_prob[:, 1]
(1)设置n=100,n越大Precision-Recall Curve越精准:
prdf = PlotPR(y_prob_1, y_test, 100, 0)
Precision-Recall Curve如下:
(2)设置n=10:
prdf = PlotPR(y_prob_1, y_test, 10, 0)
Precision-Recall Curve如下:
查看prdf数据框:
(3)python中的scikitplot模块已有相关评估曲线的实现,之前的文章中也整理过,现用该模块画出上述预测的曲线:
import scikitplot as skplt
skplt.metrics.plot_precision_recall_curve(y_test, y_prob)
Precision-Recall Curve如下:
可以看出,类别1和类别0的曲线和area与上述n=100的运行结果基本一致。此外,该模块实现的结果中还输出了micro-average曲线,就是微平均曲线。微平均是指,先将各混淆矩阵对应的TP、FP、FN的值进行平均,再用平均值计算Precison和Recall,得到micro-Precison和micro-Recall;相应的,宏平均是指,先计算各混淆矩阵的Precison和Recall,再对Precison和Recall进行平均,得到macro-Precison和macro-Recall.有兴趣的可以在上述代码中实现试试。