pyspark ml 报错: need struct type but got struct type:tinyint,size:int,indices:array

pyspark model transform 之后出现列probability,若要提取预测为1的概率:
在这里插入图片描述
直接对原列处理的报错:

`argument 1 requires string type, however, 'probability' is of struct<type:tinyint,size:int,indices:array<int>,values:array<double>> type.`

先转化该列数据格式:

prob_df1=prob_df0.withColumn("probability",prob_df0["probability"].cast("String"))

对string数据格式提取:

prob_df = prob_df1.withColumn('probabilityre',split(regexp_replace("probability", "^\[|\]", ""), ",")[1].cast(DoubleType())).select('label', 'probabilityre', 'prediction').withColumnRenamed("probabilityre","probability")
发布了29 篇原创文章 · 获赞 20 · 访问量 4万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览