执行./pyspark,./spark-shell报错拒绝链接，Error initializing SparkContext. java.net.ConnectException:

最新推荐文章于 2024-03-23 23:48:36 发布
最新推荐文章于 2024-03-23 23:48:36 发布
阅读量3.2k 收藏 10
点赞数 7
分类专栏： spark 大数据 文章标签： spark java .net
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_50641264/article/details/122457952
版权

1.错误描述,执行./pyspark,./spark-shell报错拒绝链接

22/01/12 03:49:27 ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From 6274master/192.168.47.10 to 6274master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

(base) [root@6274master bin]# ./pyspark
Python 3.8.12 (default, Oct 12 2021, 13:49:34) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
22/01/12 03:49:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/01/12 03:49:27 ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From 6274master/192.168.47.10 to 6274master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:913)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:828)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1577)
	at org.apache.hadoop.ipc.Client.call(Client.java:1519)
	at org.apache.hadoop.ipc.Client.call(Client.java:1416)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
	at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
	at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
	at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:610)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:701)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1647)
	at org.apache.hadoop.ipc.Client.call(Client.java:1463)
	... 37 more
22/01/12 03:49:27 WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext should be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
py4j.Gateway.invoke(Gateway.java:238)
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
py4j.ClientServerConnection.run(ClientServerConnection.java:106)
java.lang.Thread.run(Thread.java:748)
22/01/12 03:49:27 ERROR SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From 6274master/192.168.47.10 to 6274master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:913)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:828)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1577)
	at org.apache.hadoop.ipc.Client.call(Client.java:1519)
	at org.apache.hadoop.ipc.Client.call(Client.java:1416)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
	at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
	at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
	at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:610)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:701)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1647)
	at org.apache.hadoop.ipc.Client.call(Client.java:1463)
	... 37 more
/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/shell.py:42: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/shell.py", line 38, in <module>
    spark = SparkSession._create_shell_session()  # type: ignore
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/sql/session.py", line 553, in _create_shell_session
    return SparkSession.builder.getOrCreate()
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/sql/session.py", line 228, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/context.py", line 392, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/context.py", line 146, in __init__
    self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/context.py", line 209, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/pyspark/context.py", line 329, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in __call__
    return_value = get_return_value(
  File "/usr/local/spark-3.2.0-bin-hadoop3.2/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.net.ConnectException: Call From 6274master/192.168.47.10 to 6274master:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:913)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:828)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1577)
	at org.apache.hadoop.ipc.Client.call(Client.java:1519)
	at org.apache.hadoop.ipc.Client.call(Client.java:1416)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
	at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
	at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
	at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:610)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:701)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1647)
	at org.apache.hadoop.ipc.Client.call(Client.java:1463)
	... 37 more

2.错误原因

今天我将spark的history-server打开了，测试的时候用的好好的，但是一会发现启动不了spark作业提交不了。
通过分析日志并查看HDFS的Web界面，发现应该是我的spark连接不到HDFS的ActiveNN，而spark启动就需要连接HDFS的服务只有写入job日志这一项，所以我查看了指定sparkJob日志写入路径的spark-defaults.conf文件，果然路径指定的是standByNN

spark.eventLog.dir hdfs://hadoop002:8020/g6_direcory

3.解决办法

  • 将spark-defaults.conf和spark-env.sh 里面日志目录文件路径从单一NN改为命名空间的路径就好
    我的命名空间是

hadoop配置文件，查看hadoop core-site.xml

<property>
      <name>fs.defaultFS</name>
      <value>hdfs://ruozeclusterg6</value>
</property>

修改spark-defaults.conf

spark.eventLog.enabled           true
spark.eventLog.dir              hdfs://ruozeclusterg6:8020/g6_direcory

修改spark-env.s

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://ruozeclusterg6:8020/g6_direcory"

这三处修改保持一致重启环境

4.检查测试

[hadoop@hadoop002 spark]$ spark-shell 
19/05/14 06:00:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://hadoop002:4040
Spark context available as 'sc' (master = local[*], app id = local-1557828013138).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.2
      /_/
 
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.

重启成功，各位刷到的铁铁留个免费的双击加关注

大白菜程序猿
关注
  • 7
    点赞
  • 踩
  • 10
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 4
    评论
专栏目录
Spark 报错解决--Error initializing SparkContext
weixin_34049948的博客
05-14 1万+
在提交spark作业的时候，spark出现报错 ./spark-shell 19/05/14 05:37:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting defau...
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException
技术探求
07-18 5353
spark Shell启动的时候报错：ERROR spark.SparkContext: Error initializing SparkContext.ConnectException: Call From hadoop202/192.168.1.202 to hadoop202:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: spark连接HDFS拒绝连接，其实是犯了一
SparkContext
最新发布
03-23 2万+
SparkContext是应用启动时创建的Spark上下文对象，是进行Spark应用开发的主要接口，是Spark上层应用与底层实现的中转站。我们俗称上下文，上接机群环境，下接Spark 内核，就是给我们提供的一个入口。
Spark-hadoop集群中8020：Connection refused
weixin_51744807的博客
07-29 2048
前提条件，使用standalone模式下的Spark，使用spark-shell运行example中的相关测试案例spark-examples_2.12-3.0.0.jar，来。通过netstat查看8020端口，发现该端口不存在，是属于hadoop集群中一个内部通讯的端口。经过查看端口8020的使用情况发现，8020端口一般作为hadoop2.x系列的内部通讯端口使用。通过查看namenode端口的使用发现，hadoop3.x默认使用9820端口，因此，发现问题所在是在。，但是8020端口拒绝连接。
spark本地运行问题解决
G_Dylan1的博客
09-16 956
1.直接运行本地模式，显示未知服务，那么很可能就是/etc/hosts文件中的主机名和IP地址没配或者配置错误 2.如果本地模式出现ERROR SparkContext: Error initializing SparkContext. java.net.BindException: 无法指定被请求的地址: Service ‘sparkDriver’ failed after 16 retries (starting from 0)! Consider explicitly setting the appr
解决ERROR spark.SparkContext: Error initializing SparkContext
mo
11-25 7900
今天在开启spark-shell时报了如下错误: 从报错的日志上面来看可以知道它说SparkContext初始化异常，并且和hadoop的端口通信的时候显示连接失败。刚开始的时候我检查了启动的spark进程发现也没问题。 [root@hadoop102 logs]# showjps.sh ===================== root@hadoop102 ======================= 7315 Worker 7395 Jps 7237 Master ===============
spark on yarn java8_spark on yarn 错误
weixin_30954817的博客
02-26 243
hduser@xxx:/data1/hadoop/spark/bin$ ./spark-shell --master yarn --deploy-mode client2020-04-13 10:04:25 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using buil...
Spark启动过程中遇到的错误 ./spark-shell –master spark://node001:7077
01-20
20/02/20 19:52:17 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the...
An internal error occurred during: java.lang.NullPointerException
07-27
实用开发工具Myeclipse，在启动tomcat时，不间断的会弹出 An internal error occurred during: "Calculating completion proposals...". java.lang.NullPointerException的错误对话框！
Java中Thread.join()的使用方法
08-18
Java 中 Thread.join() 的使用方法 Thread.join() 是 Java 中的一个重要方法，该方法允许一个线程等待另一个线程的执行完成。下面我们将详细介绍 Thread.join() 的使用方法、源码分析和示例代码。 1. join() 的...
解决java执行cmd命令调用ffmpeg报错Concat error - No such filter '[0,0]'问题
08-24
主要介绍了java执行cmd命令，调用ffmpeg报错Concat error - No such filter '[0,0]'解决方法,本文通过截图实例代码说明给大家介绍的非常详细，对大家的工作或学习有一定的参考借鉴价值，需要的朋友可以参考下
honeybadger-ruby：Ruby gem向honeybadger.io报告错误
02-01
Ruby的Honeybadger 这是将应用程序与 :high_voltage: 。 当发生未捕获的异常时，Honeybadger会将相关数据过帐到您环境中指定的Honeybadger服务器。 文档和支持 有关全面的文档和支持，。 变更日志 ...
启动spark-shell遇到的一些问题
记录
04-12 5291
执行：spark-shell 1、报When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in th environmwnt 原因是没有配置之前在/etc/profile下配置了hadoop的环境变量，但是没有配置HADOOP_CONF_DIR 解决办法： vim /U...
spark-shell启动spark报错
zhang__rong的博客
11-05 1763
spark-shell启动spark报错 前言 　　离线安装好CDH、Coudera Manager之后，通过Coudera Manager安装所有自带的应用，包括hdfs、hive、yarn、spark、hbase等应用，过程很是波折，此处就不抱怨了，直接进入主题。 描述 　　在安装有spark的节点上，通过spark-shell启动spark，满怀期待的启动spark，but，来了个晴...
Spark中 java.net.ConnectException: 拒绝连接问题
haibei153的博客
08-12 1万+
在使用Spark的过程中报错java.net.ConnectException: 拒绝连接 在逐步排查后最终锁定为JDK配置的问题，可能自己在之前动过JDK相关的配置，导致在使用Spark中不能生效出现报错 总结问题原因：在修改完/etc/profile.d/my_env.sh中环境变量后没有进行source /etc/profile,直接向服务器进行分发，导致出错。 解决办法：1.在完成系统环境变量的修改后，进行source /etc/profile，（注意此过程为root操作，如果不想使用roo.
ERROR spark.SparkContext:Error initializing SparkContext.Call From to failed on connection exception
zhengzaifeidelushang的博客
11-11 1221
Spark集群搭建完，节点启动Spark-shell报错 报错如下： bigdata1节点启动Spark-shell报错： ERROR repl.Main: Failed to initialize Spark session.java.net.ConnectException: Call From bigdata1/192.168.239.131 to bigdata1:8020 failed on connection exception: java.net.ConnectException: 拒绝连接
Call From master/192.168.170.128 to master:8020 failed on connection exception: java.net.ConnectExce
qiweiqian的博客
06-11 1万+
在执行一个Hadoop的文件读写任务是发生这个错误：Call From master/192.168.170.128 to master:8020 failed on connection exception: java.net.ConnectException: Connection。 意思大致是无法连接到master的8020端口。该问题造成的主要原因是和Hadoop以及Linux系
解决通过spark-shell 连接集群产生的不能连接master问题
lwnowhy的专栏
05-07 7630
spark集群启动后通过MASTER=spark://spark01:7077 ./spark-shell 连接 报以下错误
出现 java.net.ConnectException: Connection refused 异常的原因及解决方法
热门推荐
CG国斌的博客
04-17 33万+
1 异常描述在启动 Tomcat 服务器的时候，控制台一直输出异常信息，然后停止服务器，报出如下异常：2 异常原因通过观察上图中被标记出来的异常信息，咱们可以知道 java.net.ConnectException: Connection refused 此异常，为：连接被拒绝异常。之前也在网上搜索过该异常的原因，大多数人给出的答案是端口号被占用，或者在启动本次 Tomcat 服务器的之前“关闭
ERROR spark.SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File does not exist: hdfs://m0:9000/directory
10-26
这个错误提示表明SparkContext初始化失败，因为指定的文件不存在。具体来说，这个错误是由于Spark无法找到指定的HDFS文件系统中的目录/directory所致。可能的原因包括：文件路径错误、文件被删除或移动、HDFS文件系统不可用等。为了解决这个问题，你可以尝试以下几个方案： 1.检查文件路径是否正确，确保文件存在于指定的HDFS目录中。 2.检查HDFS文件系统是否可用，确保HDFS服务已经启动并且可以正常访问。 3.检查文件权限，确保当前用户有访问指定文件的权限。 4.如果文件确实被删除或移动了，你需要重新上传或者移动文件到正确的位置。 5.如果以上方法都无法解决问题，你可以尝试重新安装或者升级Spark，或者调整Spark的配置参数，例如增加内存大小等。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

大白菜程序猿

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值