实践数据湖iceberg 第十七课 hadoop2.7,spark3 on yarn运行iceberg配置

*星星之火*

已于 2022-02-22 20:15:52 修改

阅读量1.3k

点赞数 1

分类专栏： iceberg spark 文章标签：数据湖 iceberg yarn spark

于 2022-02-15 16:46:23 首次发布

本文链接：https://blog.csdn.net/spark_dev/article/details/122947494

版权

iceberg 同时被 2 个专栏收录

44 篇文章 114 订阅

订阅专栏

spark

8 篇文章 1 订阅

订阅专栏

系列文章目录

文章目录

系列文章目录
前言
1. hadoop2.7 上安装 spark3.2 报错java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
2. 问题分析
3. 解决方法
4.spark3 on yarn + iceberg0.13启动
总结

前言

spark版本： spark-3.2.0-bin-hadoop2.7
hadoop版本： hadoop2.7.2

1. hadoop2.7 上安装 spark3.2 报错java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig

直接解压，跑spark-shell --master yarn
说明：已经配置了HADOOP_HOME,HADOOP_CONF_DIR, 解压后，spark-shell会自动找HADOOP_HOME

[root@hadoop101 spark]# spark-shell --master yarn
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/02/14 21:00:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
  at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:175)
  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
  at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
  at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
  ... 55 elided
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 68 more
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.0
      /_/
         
Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information

2. 问题分析

spark3的jersey版本：

[root@hadoop103 spark-3.2.0-bin-hadoop2.7]# ls jars/jersey-*
jars/jersey-client-2.34.jar  jars/jersey-common-2.34.jar  jars/jersey-container-servlet-2.34.jar  jars/jersey-container-servlet-core-2.34.jar  jars/jersey-hk2-2.34.jar  jars/jersey-server-2.34.jar
[root@hadoop103 spark-3.2.0-bin-hadoop2.7]#

hadoop的jersey版本：

[root@hadoop103 spark-3.2.0-bin-hadoop2.7]# ls /opt/module/hadoop/share/hadoop/yarn/lib/jersey-*
/opt/module/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar  /opt/module/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar  /opt/module/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar
/opt/module/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar    /opt/module/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar
[root@hadoop103 spark-3.2.0-bin-hadoop2.7]#

网上很多建议，把 jersey-core-1.9.jar ，jersey-client-1.9.jar ，jersey-guice-1.9.jar 放到$SPARK_HOME/jars 下面。
照做，发现不可行 (spark-shell --master yarn, spark-sql --master yarn都试试，有时有一个行，奇怪)。

1.9和2.3的包，里面的类是不会冲突。
在这里插入图片描述

3. 解决方法

yarn-site.xml中，加上如下配置，重启yarn （不重启，不生效）。

<property>
            <name>yarn.timeline-service.enabled</name>
                <value>false</value>
</property>

4.spark3 on yarn + iceberg0.13启动

[root@hadoop101 spark]#   bin/spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.0    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions     --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog     --conf spark.sql.catalog.spark_catalog.type=hive     --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog     --conf spark.sql.catalog.local.type=hadoop     --conf spark.sql.catalog.local.warehouse=/tmp/iceberg/warehouse --master yarn

总结

至此，终于spark3 cluster模式运行iceberg环境准备完成

缺点：把yarn.timeline-service.enabled关闭了，先玩起来再说

*星星之火*

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
实践数据湖iceberg 第十七课 hadoop2.7,spark3 on yarn运行iceberg配置

系列文章目录实践数据湖iceberg 第一课入门实践数据湖iceberg 第二课 iceberg基于hadoop的底层数据格式实践数据湖iceberg 第三课在sqlclient中，以sql方式从kafka读数据到iceberg实践数据湖iceberg 第四课在sqlclient中，以sql方式从kafka读数据到iceberg（升级版本到flink1.12.7）实践数据湖iceberg 第五课 hive catalog特点实践数据湖iceberg 第六课从kafka写入到iceber
复制链接

扫一扫