pyspark MulticlassClassificationEvaluator的一些使用总结

数据挖掘过程当中，模型评估必须可少。
最近做一个pyspark的项目，评估模型过程中使用了MulticlassClassificationEvaluator进行模型评估，踩了不少坑，所以在此做个记录，分享给大家。
官方参考文档：

MulticlassClassificationEvaluator

• Init signature: MulticlassClassificationEvaluator(predictionCol=‘prediction’, labelCol=‘label’, metricName=‘f1’)
• Docstring:
  … note:: Experimental
  Evaluator for Multiclass Classification, which expects two input
  columns: prediction and label.

从帮助文档我们可以看出，方法MulticlassClassificationEvaluator里面有三个参数，分别为predictionCol, labelCol, metricName，重点解释一下metricName。

metricName
查阅文档可知，metricName配置f1、weightedPrecision、weightedRecall、accuracy

使用示例如下：

为了方便理解，我们将MulticlassClassificationEvaluator同sklean.metrics进行对比。

from pyspark.sql import SparkSession
import pandas as pd

from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from sklearn import metrics

spark= SparkSession.builder.master('local').appName('test').getOrCreate()
# 创建一个数据集
data =  [(0.0, 0.0), 
         (0.0, 1.0), 
         (0.0, 0.0),
         (1.0, 0.0), 
         (1.0, 1.0), 
         (1.0, 1.0), 
         (1.0, 1.0), 
         (2.0, 2.0), 
         (2.0, 0.0)]
data_pd_df = pd.DataFrame(data, columns=["prediction", "label"])
data_spark_df = spark.createDataFrame(data, ["prediction", "label"])

evaluator_acc = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="accuracy")
evaluator_f1 = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="f1")
evaluator_pre = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="weightedPrecision")
evaluator_recall = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="weightedRecall")

sklern_acc = metrics.accuracy_score(data_pd_df['label'], data_pd_df['prediction'])
sklern_f1 = metrics.f1_score(data_pd_df['label'], data_pd_df['prediction'], average='weighted')
sklern_pre = metrics.precision_score(data_pd_df['label'], data_pd_df['prediction'], average='weighted')
sklern_recall = metrics.recall_score(data_pd_df['label'], data_pd_df['prediction'], average='weighted')

print('pyspark accuracy: %.6f' %evaluator_acc.evaluate(data_spark_df))
print('pyspark f1-score: %.6f' %evaluator_f1.evaluate(data_spark_df))
print('pyspark precision: %.6f' %evaluator_pre.evaluate(data_spark_df))
print('pyspark recall: %.6f' %evaluator_recall.evaluate(data_spark_df))
print('-----------------------')
print('sklearn accuracy: %.6f' %sklern_acc)
print('sklearn f1-score: %.6f' %sklern_f1)
print('sklearn precision: %.6f' %sklern_pre)
print('sklearn recall: %.6f' %sklern_recall)

pyspark accuracy: 0.666667
pyspark f1-score: 0.661376
pyspark precision: 0.685185
pyspark recall: 0.666667
-----------------------
sklearn accuracy: 0.666667
sklearn f1-score: 0.661376
sklearn precision: 0.685185
sklearn recall: 0.666667

从上代码可以看出，MulticlassClassificationEvaluator和metrics运行结果完全一致，但细心的同学应该发现，在metrics模式下，配置了一个及其重要的参数average=weight才使得两种结果保持一致。
metrics模式下，参数average解释如下：

Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance.
计算每个标签的度量，并找到它们的平均权重按支持（每个标签的真实实例数）。这个更改“宏”以解释标签不平衡。

产生的一个坑

MulticlassClassificationEvaluator对于评估二分类结果不太友好，在sklearn模式下，参数average默认值为binary，而MulticlassClassificationEvaluator当中均为weight模式。这时，两种模型将会形成完全不一样的结果。具体示例如下：

# 产生一个二分类的数据
data =  [(0.0, 0.0), 
         (0.0, 0.0), 
         (0.0, 0.0),
         (1.0, 0.0), 
         (1.0, 1.0), 
         (1.0, 0.0), 
         (1.0, 1.0)]

data_pd_df = pd.DataFrame(data, columns=["prediction", "label"])
data_spark_df = spark.createDataFrame(data, ["prediction", "label"])

evaluator_acc = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="accuracy")
evaluator_f1 = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="f1")
evaluator_pre = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="weightedPrecision")
evaluator_recall = MulticlassClassificationEvaluator(predictionCol="prediction", metricName="weightedRecall")

sklern_acc = metrics.accuracy_score(data_pd_df['label'], data_pd_df['prediction'])
sklern_f1 = metrics.f1_score(data_pd_df['label'], data_pd_df['prediction'])
sklern_pre = metrics.precision_score(data_pd_df['label'], data_pd_df['prediction'])
sklern_recall = metrics.recall_score(data_pd_df['label'], data_pd_df['prediction'])

print('pyspark accuracy: %.6f' %evaluator_acc.evaluate(data_spark_df))
print('pyspark f1-score: %.6f' %evaluator_f1.evaluate(data_spark_df))
print('pyspark precision: %.6f' %evaluator_pre.evaluate(data_spark_df))
print('pyspark recall: %.6f' %evaluator_recall.evaluate(data_spark_df))
print('-----------------------')
print('sklearn accuracy: %.6f' %sklern_acc)
print('sklearn f1-score: %.6f' %sklern_f1)
print('sklearn precision: %.6f' %sklern_pre)
print('sklearn recall: %.6f' %sklern_recall)

pyspark accuracy: 0.714286
pyspark f1-score: 0.726190
pyspark precision: 0.857143
pyspark recall: 0.714286
-----------------------
sklearn accuracy: 0.714286
sklearn f1-score: 0.666667
sklearn precision: 0.500000
sklearn recall: 1.000000

由上面的例子可以看出来，在二分类情况下，尽量不要使用MulticlassClassificationEvaluator，它与我们二分类通常的评估标准有很大差别。

如果必须要使用？

2020-5-21，继续
如果必须要使用的话，我自己写了个函数，供大家大家参考。

def precise_recall_f1(pred_label,test_label):
    p_label = pred_label.withColumnRenamed('prediction','p_label')
    d_label = test_label.withColumnRenamed('label','d_label')
    # p_label=p_label.withColumn('id',monotonically_increasing_id())
    # d_label=d_label.withColumn('id',monotonically_increasing_id())
    result=p_label.join(d_label, on=USER_NO, how='left')
    tp = result[(result.d_label == 1) & (result.p_label == 1)].count()
    tn = result[(result.d_label == 0) & (result.p_label == 0)].count()
    fp = result[(result.d_label == 0) & (result.p_label == 1)].count()
    fn = result[(result.d_label == 1) & (result.p_label == 0)].count()
    try:
        p = float(tp)/(tp + fp)
    except:
        p = 0
    try:
        r = float(tp)/(tp + fn)
    except:
        r = 0
    try:
        f1 = 2*p*r/(p+r)
    except:
        f1 = 0
    return p,r,f1