Spark3.0.3版本报““main“ java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream“

最新推荐文章于 2024-10-09 10:53:13 发布

终极量化数据

最新推荐文章于 2024-10-09 10:53:13 发布

阅读量1.5k

点赞数 1

分类专栏： Spark 大数据文章标签： spark java big data

本文链接：https://blog.csdn.net/suwei825/article/details/120449015

版权

大数据同时被 2 个专栏收录

32 篇文章 7 订阅

订阅专栏

Spark

12 篇文章 0 订阅

订阅专栏

大数据开发环境就是这样，你刚填完一个坑，就掉入了另外一个坑，运行一个spark远程调用示例

  SparkConf sparkConf = new SparkConf()
                .setMaster("spark://ss3:7077")
                .setAppName("JavaSparkPi");

        SparkSession spark = SparkSession
                .builder()
                .config(sparkConf)
                .getOrCreate();

结果在创建SparkConf对象的时候，报如下错误

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
	at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)
	at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)
	at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
	at JavaWordCount.main(JavaWordCount.java:54)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 4 more

查找资料后了解到，从spark1.4以后，所有spark的编译都是没有将hadoop的classpath编译进去的，所以必须在spark-env.sh中指定hadoop中的所有jar包。

既然原因找到了，那我们就去spark修改spark-env.sh, 加上以下配置，前提是hadoop命令已经可以使用

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

同时还需要检查pom配置，下面贴出正确的pom依赖配置

       <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
<!--            <scope>provided</scope>-->
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
<!--            <scope>provided</scope>-->
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>${spark.version}</version>
<!--            <scope>provided</scope>-->
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.12</artifactId>
            <version>${spark.version}</version>
            <!--<scope>provided</scope>-->
        </dependency>
         <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.12</artifactId>
            <version>${spark.version}</version>
            <!--<scope>provided</scope>-->
        </dependency>

        <dependency>
            <!-- scala语言核心包 -->
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <exclusions>
                <exclusion>
                    <groupId>com.google.code.gson</groupId>
                    <artifactId>gson</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty-all</artifactId>
                </exclusion>
            </exclusions>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>${hive.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.8.1</version>
        </dependency>
    </dependencies>