解决centos7上初始化spark意外

最新推荐文章于 2022-03-14 11:13:41 发布

rui_1993

最新推荐文章于 2022-03-14 11:13:41 发布

阅读量923

点赞数 4

分类专栏： spark学习文章标签： spark Linux

本文链接：https://blog.csdn.net/rui_1993/article/details/88076275

版权

spark学习专栏收录该内容

2 篇文章 0 订阅

订阅专栏

在Linux（CentOS7）上初始化spark遇到“org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@VM_0_9_centos:34068”问题，具体如下：

ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@VM_0_9_centos:34068
 at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
 at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
 at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
 at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
 at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
 at org.apache.spark.executor.Executor.<init>(Executor.scala:178)
 at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
 at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:127)
 at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:178)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
 at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
 at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
 at py4j.Gateway.invoke(Gateway.java:238)
 at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
 at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
 at py4j.GatewayConnection.run(GatewayConnection.java:238)
 at java.lang.Thread.run(Thread.java:748)

解决方法：在spark-env.shell中设置SPARK_LOCAL_IP=127.0.0.1后运行：
export SPARK_LOCAL_HOSTNAME=localhost