Spark ml报错：Error:scalac: missing or invalid dependency detected while loading class file ‘SQLImplici

最新推荐文章于 2022-02-18 15:51:20 发布

RayBreslin

最新推荐文章于 2022-02-18 15:51:20 发布

阅读量3k

点赞数

分类专栏： Spark ML Spark 机器学习文章标签： spark ml 2.3.0 推荐

本文链接：https://blog.csdn.net/u010886217/article/details/90515060

版权

Spark 同时被 3 个专栏收录

68 篇文章 0 订阅

订阅专栏

Spark ML

7 篇文章 2 订阅

订阅专栏

机器学习

7 篇文章 0 订阅

订阅专栏

一、问题描述

采用spark2.3.0的 ml库进行推荐算法建模，但是，运行代码到如下推荐个数的时候

...
model.recommendForAllUsers(7).show(false)
...

报错如下图：

具体报错如下：


Information:Module "sparktest" was fully rebuilt due to project configuration/dependencies changes
Information:2019/5/20 8:53 - Build completed with 23 errors and 0 warnings in 4 s 71 ms
Error:scalac: missing or invalid dependency detected while loading class file 'SQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.
Error:scalac: missing or invalid dependency detected while loading class file 'LowPrioritySQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LowPrioritySQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.
Error:scalac: missing or invalid dependency detected while loading class file 'package.class'.
Could not access type Row in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.
Error:scalac: missing or invalid dependency detected while loading class file 'Dataset.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Dataset.class' was compiled against an incompatible version of org.apache.spark.sql.
Error:scalac: missing or invalid dependency detected while loading class file 'SparkSession.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SparkSession.class' was compiled against an incompatible version of org.apache.spark.sql.

二、问题原因

百思不得其解，因为pom依赖也已经引入，而且代码也不报错，纠结了一周

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-mllib_2.11</artifactId>
      <version>2.3.0</version>
<!--      <scope>runtime</scope>-->
    </dependency>

网上诸多查找，然后发现，十分可能是下载依赖的时候出现问题。查看依赖位置，发现依赖jar包后面有lastUpdated字样，证明是依赖下载失败。