spark 问题笔记

最新推荐文章于 2022-07-05 09:30:53 发布

新手路上的程序员

最新推荐文章于 2022-07-05 09:30:53 发布

阅读量394

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/yidan7063/article/details/109219246

版权

spark 专栏收录该内容

15 篇文章 1 订阅

订阅专栏

1.在idea中运行报错:scala.Product.$init$(Lscala/Product;)V
检查pom文件中scala的版本和idea中scala的版本是否一致 idea中scala版本为2.11
解决办法:<artifactId>spark-core_2.12</artifactId> -> <artifactId>spark-core_2.11</artifactId>

2.idea中 java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
在程序main()方法下加入System.setProperty("hadoop.home.dir", "D:\\hadoop-2.9.2")
D:\\hadoop-2.9.2 window环境中Hadoop文件的地址

3.Error:(17, 34) Unable to find encoder for type String. An implicit Encoder[String] is needed to store String instances in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. val value = df1.createDataset(strings)
这里缺失了隐式转换 implicit val encoder=org.apache.spark.sql.Encoders.STRING

4.http调用livy时报错java.lang.NullPointerException
http中的参数的前后不能有空格,检查是否有空格

5.livy session 返回值中文乱码解决:

livy：
'conf': {'spark.driver.extraJavaOptions': '-Dfile.encoding=utf-8',
'spark.executor.extraJavaOptions': '-Dfile.encoding=utf-8'}
spark:
 –conf spark.driver.extraJavaOptions=” -Dfile.encoding=utf-8 “ \
 –conf spark.executor.extraJavaOptions=” -Dfile.encoding=utf-8 “ \

6.livy session模式指定大jar包(使用的是batch模式的大jar包)导致jar包冲突解决:
去掉batch模式大jar包中的和scala相关的依赖

        <dependency>
            <groupId>org.json4s</groupId>
            <artifactId>json4s-jackson_2.11</artifactId>
            <version>3.2.11</version>
            <exclusions>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-reflect</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.scala-lang.modules</groupId>
                    <artifactId>scala-xml_2.11</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-compiler</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.scala-lang.modules</groupId>
                    <artifactId>scala-parser-combinators_2.11</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scalap</artifactId>
                </exclusion>
            </exclusions>
        </dependency>