pyspark.sql.functions.udf中使用numpy，出现Py4JJavaError错误

背景：pyspark，通过udf定义函数，以辅助添加新列

出错原因：udf不能返回numpy类型

举例：

df.head()

Row(artist=‘Martha Tilston’, auth=‘Logged In’, firstName=‘Colin’, gender=‘M’, userId=‘30’, hour=8)

# 切割时间，每6个小时为一组
get_6hour = udf(lambda x: np.floor(x/6),IntegerType())
df.withColumn('6hour',get_6hour(df.hour)).head()

Py4JJavaError: An error occurred while calling o2099.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 207.0 failed 1 times, most recent failure: Lost task 0.0 in stage 207.0 (TID 9384, localhost, executor driver): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)……

正解：(将udf中的返回类型更改为int)

get_6hour = udf(lambda x: int(np.floor(x/6)),IntegerType())
df.withColumn('6hour',get_6hour(df.hour)).head()

Row(artist=‘Martha Tilston’, auth=‘Logged In’, firstName=‘Colin’, gender=‘M’, userId=‘30’, hour=8, 6hour=1)

如果你的问题解决了，欢迎点赞+关注哦~