Spark Machine Learning Error analysis and solution(Error with RDD[Vector];Vector takes type paramet)

最新推荐文章于 2022-12-13 14:34:26 发布

原创最新推荐文章于 2022-12-13 14:34:26 发布 · 1.2k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#Spark Machine Learning #RDD[Vector] #spark.ml.linalg.Vector #spark.mllib.linalg

spark 专栏收录该内容

3 篇文章

订阅专栏

本文介绍了一个常见的Spark机器学习应用错误，即在同一个项目中混合使用了Spark ML与MLlib的不同版本，导致类型不匹配的问题，并提供了解决方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

While we follow spark example

case class model_instance (features: Vector)

//and

val df = rawData.map(line =>
     | { model_instance( Vectors.dense(line.split(",").filter(p => p.matches("\\d*(\\.?)\\d*"))
     | .map(_.toDouble)) )}).toDF()

Errors may occur like below:

Column features must be of type org.apache.spark.ml.linalg.VectorUDT

Error with RDD[Vector] in function parameter

type Vector takes type parameters

for the error we should realize that the ml and mllib are in different version of Spark Machine Learning, you may not mix them together, just use this:

import org.apache.spark.ml.linalg.{Vector, Vectors}

//or

import org.apache.spark.mllib.linalg.{Vector, Vectors}

//remember not mix above import together, just use one case