最全【Spark ML】（二）Spark ML 分类算法_spark分类算法

最新推荐文章于 2024-08-16 21:51:12 发布

2401_84159813

最新推荐文章于 2024-08-16 21:51:12 发布

阅读量854

点赞数 10

分类专栏：程序员文章标签： spark-ml 分类 spark

本文链接：https://blog.csdn.net/2401_84159813/article/details/138443563

版权

# 获得每次迭代的优化目标(损失 + 惩罚项)
objectiveHistory = trainingSummary.objectiveHistory
print("objectiveHistory:")
for objective in objectiveHistory:
    print(objective)

# Obtain the receiver-operating characteristic as a dataframe and areaUnderROC.
trainingSummary.roc.show()
print("areaUnderROC: " + str(trainingSummary.areaUnderROC))

# 设置模型阈值，使得最大化F度量值
fMeasure = trainingSummary.fMeasureByThreshold
maxFMeasure = fMeasure.groupBy().max('F-Measure').select('max(F-Measure)').head()
bestThreshold = fMeasure.where(fMeasure['F-Measure'] == maxFMeasure['max(F-Measure)']) \
    .select('threshold').head()['threshold']
lr.setThreshold(bestThreshold)

spark.stop()


日志信息：

objectiveHistory:
0.683314913574
…
±–±-------------------+
|FPR| TPR|
±–±-------------------+
|0.0| 0.0|
|0.0|0.017543859649122806|
|0.0| 0.03508771929824561|
|0.0| 0.05263157894736842|
|0.0| 0.07017543859649122|
|0.0| 0.08771929824561403|
|0.0| 0.10526315789473684|
|0.0| 0.12280701754385964|
|0.0| 0.14035087719298245|
|0.0| 0.15789473684210525|
|0.0| 0.17543859649122806|
|0.0| 0.19298245614035087|
|0.0| 0.21052631578947367|
|0.0| 0.22807017543859648|
|0.0| 0.24561403508771928|
|0.0| 0.2631578947368421|
|0.0| 0.2807017543859649|
|0.0| 0.2982456140350877|
|0.0| 0.3157894736842105|
|0.0| 0.3333333333333333|
±–±-------------------+
only showing top 20 rows

areaUnderROC: 1.0


#### 1.2 多分类LR


示例代码如下：

-- coding: utf-8 --

from future import print_function
from pyspark.ml.classification import LogisticRegression
from pyspark.sql import SparkSession

if name == “__main__”:
spark = SparkSession
.builder
.appName(“MulticlassLogisticRegressionWithElasticNet”)
.getOrCreate()

training = spark \
    .read \
    .format("libsvm") \
    .load("../data/mllib/sample\_multiclass\_classification\_data.txt")

lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
lrModel = lr.fit(training)

# 系数和截距项
print("Coefficients: \n" + str(lrModel.coefficientMatrix))
print("Intercept: " + str(lrModel.interceptVector))

trainingSummary = lrModel.summary

# 获得每次迭代的优化目标(损失 + 惩罚项)
objectiveHistory = trainingSummary.objectiveHistory
print("objectiveHistory:")
for objective in objectiveHistory:
    print(objective)

# 可以查看每个类的FPR & TPR
print("False positive rate by label:")
for i, rate in enumerate(trainingSummary.falsePositiveRateByLabel):
    print("label %d: %s" % (i, rate))

print("True positive rate by label:")
for i, rate in enumerate(trainingSummary.truePositiveRateByLabel):
    print("label %d: %s" % (i, rate))

print("Precision by label:")
for i, prec in enumerate(trainingSummary.precisionByLabel):
    print