TiSpark安装相关bug

已于 2022-08-01 18:31:45 修改
阅读量437 收藏
点赞数
文章标签： spark bug scala
于 2022-08-01 18:31:25 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_40505721/article/details/126076362
版权

Bug 1 ------ hostname

22/07/30 18:48:06 WARN Utils: Your hostname, spider-cluster-03 resolves to a loopback using 10.245.146.66 instead (on interface eno3)
22/07/30 18:48:06 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another addres
22/07/30 18:48:07 WARN NativeCodeLoader: Unable to load native-hadoop library for youruiltin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newL
22/07/30 18:48:14 ERROR Main: Failed to initialize Spark session.
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstrua:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1(SparkSession.
	at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1$adapted(Spark
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$applyion.scala:1219)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:96
	at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
	at $line3.$read$$iw$$iw.<init>(<console>:15)
	at $line3.$read$$iw.<init>(<console>:42)
	at $line3.$read.<init>(<console>:44)
	at $line3.$read$.<init>(<console>:48)
	at $line3.$read$.<clinit>(<console>)
	at $line3.$eval$.$print$lzycompute(<console>:7)
	at $line3.$eval$.$print(<console>:6)
	at $line3.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:6
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.sca
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.sc
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileC
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
	at scala.tools.nsc.interpreter.IMain.$anonfun$quietRun$1(IMain.scala:216)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
	at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:216)
	at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$2(SparkILoop.scal
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$1(SparkILoop.scal
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:97)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:83)
	at org.apache.spark.repl.SparkILoop.$anonfun$process$4(SparkILoop.scala:165)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.interpreter.ILoop.$anonfun$mumly$1(ILoop.scala:166)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
	at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:163)
	at org.apache.spark.repl.SparkILoop.loopPostInit$1(SparkILoop.scala:153)
	at org.apache.spark.repl.SparkILoop.$anonfun$process$10(SparkILoop.scala:221)
	at org.apache.spark.repl.SparkILoop.withSuppressedSettings$1(SparkILoop.scala:
	at org.apache.spark.repl.SparkILoop.startup$1(SparkILoop.scala:201)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:236)
	at org.apache.spark.repl.Main$.doMain(Main.scala:78)
	at org.apache.spark.repl.Main$.main(Main.scala:58)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:6
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$ruala:958)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: scala/Function1$class
	at org.apache.spark.sql.TiExtensions.<init>(TiExtensions.scala:23)
	... 67 more
Caused by: java.lang.ClassNotFoundException: scala.Function1$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 68 more

解决方案：

修改配置文件，配置 SPARK_LOCAL_IP 变量即可：

sudo vim conf/spark-env.sh
export SPARK_LOCAL_IP=10.xxx.xxx.66

Bug 2 ------ InvocationTargetException

22/07/30 19:08:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/07/30 19:08:29 ERROR Main: Failed to initialize Spark session.
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1(SparkSession.scala:1222)
	at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1$adapted(SparkSession.scala:1219)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$applyExtensions(SparkSession.scala:1219)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:960)
	at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
	at $line3.$read$$iw$$iw.<init>(<console>:15)
	at $line3.$read$$iw.<init>(<console>:42)
	at $line3.$read.<init>(<console>:44)
	at $line3.$read$.<init>(<console>:48)
	at $line3.$read$.<clinit>(<console>)
	at $line3.$eval$.$print$lzycompute(<console>:7)
	at $line3.$eval$.$print(<console>:6)
	at $line3.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
	at scala.tools.nsc.interpreter.IMain.$anonfun$quietRun$1(IMain.scala:216)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
	at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:216)
	at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$2(SparkILoop.scala:83)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.repl.SparkILoop.$anonfun$initializeSpark$1(SparkILoop.scala:83)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:97)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:83)
	at org.apache.spark.repl.SparkILoop.$anonfun$process$4(SparkILoop.scala:165)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.interpreter.ILoop.$anonfun$mumly$1(ILoop.scala:166)
	at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:206)
	at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:163)
	at org.apache.spark.repl.SparkILoop.loopPostInit$1(SparkILoop.scala:153)
	at org.apache.spark.repl.SparkILoop.$anonfun$process$10(SparkILoop.scala:221)
	at org.apache.spark.repl.SparkILoop.withSuppressedSettings$1(SparkILoop.scala:189)
	at org.apache.spark.repl.SparkILoop.startup$1(SparkILoop.scala:201)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:236)
	at org.apache.spark.repl.Main$.doMain(Main.scala:78)
	at org.apache.spark.repl.Main$.main(Main.scala:58)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: scala/Function1$class
	at org.apache.spark.sql.TiExtensions.<init>(TiExtensions.scala:23)
	... 67 more
Caused by: java.lang.ClassNotFoundException: scala.Function1$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 68 more

解决方案

版本号的问题

TiSpark 版本TiDB、TiKV、PD 版本Spark 版本Scala 版本
2.4.x-scala_2.115.x，4.x2.3.x，2.4.x2.11
2.4.x-scala_2.125.x，4.x2.4.x2.12
2.5.x5.x，4.x3.0.x，3.1.x2.12
3.0.x5.x，4.x3.0.x，3.1.x，3.2.x2.12

改成3.X就好了，不过spark-defaults.conf文件里需要加上

spark.sql.catalog.tidb_catalog org.apache.spark.sql.catalyst.catalog.TiCatalog
spark.sql.catalog.tidb_catalog.pd.addresses $pd_host:$pd_port
陆溪云
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
ERROR spark.SparkContext:Error initializing SparkContext.Call From to failed on connection exception
zhengzaifeidelushang的博客
11-11 1249
Spark集群搭建完，节点启动Spark-shell报错 报错如下： bigdata1节点启动Spark-shell报错： ERROR repl.Main: Failed to initialize Spark session.java.net.ConnectException: Call From bigdata1/192.168.239.131 to bigdata1:8020 failed on connection exception: java.net.ConnectException: 拒绝连接
bugfree安装配置
06-14
Bugfree 安装配置详解 Bugfree 安装配置是一个复杂的过程，需要了解 XAMPP 软件包的安装和配置，以及 Bugfree 的安装和配置过程。下面将详细介绍 Bugfree 安装配置的每个步骤，并解释每个步骤的目的和作用。 一、...
Spark-遇到的坑-启动spark报错
热门推荐
a1032818891的博客
07-26 1万+
1.解决措施： Please instead use:  - ./spark-submit with --num-executors to specify the number of executors  - Or set SPARK_EXECUTOR_INSTANCES  - spark.executor.instances to configure the number of instanc...
Spark启动报错
zhikanjiani的博客
07-30 4059
1、spark-shell --master local[2]启动出现如下报错： [hadoop@hadoop002 spark-2.4.2-bin-2.6.0-cdh5.7.0]$ spark-shell --master local[2] 19/07/23 00:52:14 WARN util.NativeCodeLoader: Unable to load native-hadoop lib...
Spark启动过程中遇到的错误 ./spark-shell --master spark://node001:7077
qq_34072169的博客
02-20 1900
20/02/20 19:52:17 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using t...
我的Bugfree安装过程
11-08
【Bugfree安装过程详解】 Bugfree是一款开源的缺陷跟踪管理系统，用于帮助软件开发团队管理和跟踪软件中的错误和问题。在本文中，我们将详细介绍如何在Windows系统上安装Bugfree，包括必要的依赖软件XAMPP的安装。 ...
Linux系统下安装bugfree操作方法
01-09
BugFree是借鉴微软的研发流程和Bug管理理念，使用PHP+MySQL独立写出的一个Bug管理 系统。简单实用、免费并且开放源代码(遵循GNU GPL)。... 　具体安装操作步骤： 　步骤一：下载XAMpp安装包 　此安装包中有Ap
bugfree安装文件
04-14
安装BugFree时，通常需要遵循以下步骤： 1. 下载并解压缩安装包。 2. 配置服务器环境，确保支持PHP和MySQL。 3. 创建数据库，并导入BugFree提供的数据库脚本。 4. 修改配置文件（如config.inc.php），设置数据库...
安装bugfree工具.7z
01-20
在本教程中，我们将探讨如何在低版本的XAMPP环境中安装和配置BugFree，这对于初学者来说是非常实用的步骤。 首先，你需要下载并安装XAMPP。确保选择与你的操作系统兼容的版本，并且这个版本需要能够支持BugFree的...
运行spark程序时报错org.apache.hadoop.security.AccessControlException: Permission denied:
buside的博客
05-10 666
1、问题：运行spark程序时报一下错误 2、解决方案 关闭权限检测，在cloudera Manager中去掉dfs.permissions并重启HDFS即可
Spark StandAlone 模式下 ./pyspark 连接被拒
CSDN1csdn1的博客
04-01 725
执行 ./pyspark --master spark://Master:7077 后报错如下： ERROR SparkContext: Error initializing SparkContext. java.net.ConnectException: Call From Master/192.168.3.23 to Master:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more deta
TiSpark 3.0.0 新特性实践
TiDB 社区干货传送门
06-21 341
作者： 数据小黑 原文来源： https://tidb.net/blog/a84d0ac7 ...
ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memo
qq_42103298的博客
03-01 787
做推荐系统实战的时候，执行FeatureEngineering（scala）文件出现以下错误： ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200 解决办法： 1.点击Edit Configurations 2.如果你的设置界面有下面红框，直接粘贴以下这串文字 -Xms128m -
Win安装Spark问题：ERROR SparkContext: Error initializing SparkContext
Circle-C的博客
11-28 5838
问题背景: Java版本：1.8.0.291 Spark版本：spark-3.2.0-bin-hadoop3.2 Windows操作系统64位 Spark初次安装 问题描述： 配置完Java环境和Spark路径之后，在cmd中执行Spark-shell时，出现如下错误 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN".
Spark启动报错问题
理科男同学
06-04 1552
今天准备把Spark和Hive进行集成一下，当启动Spark的时候，报了如下错误： [rzf@hadoop100 spark]$ bin/spark-shell 21/06/04 11:52:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j pro
Spark-shell单节点启动报错
zhikanjiani的博客
07-10 1269
单节点部署的hdfs和yarn，在这基础上自定义部署了spark-2.4.2-bin-2.6.0-cdh5.7.0，输入spark-shell启动报错： [hadoop@hadoop002 conf]$ spark-shell 19/06/20 04:44:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for...
TiSpark (Beta) 用户指南
最新发布
weixin_42241611的博客
04-09 195
TiSpark 是将 Spark SQL 直接运行在 TiDB 存储引擎 TiKV 上的 OLAP 解决方案。TiSpark 架构图如下：TiSpark 架构图TiSpark 深度整合了 Spark Catalyst 引擎, 可以对计算提供精确的控制，使 Spark 能够高效的读取 TiKV 中的数据，提供索引支持以实现高速的点查；通过多种计算下推减少 Spark SQL 需要处理的数据大小，以加速查询；利用 TiDB 的内建的统计信息选择更优的查询计划。
Bugfree安装与使用指南
"bugfree安装文档及基本使用手册" 本文档主要介绍了Bugfree的安装过程以及基本使用方法，适合初学者和学习者参考。Bugfree是一个开源的缺陷跟踪系统，用于帮助软件开发团队管理项目的错误和问题。 1. Bugfree安装 ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值