单节点部署的hdfs和yarn,在这基础上自定义部署了spark-2.4.2-bin-2.6.0-cdh5.7.0,输入spark-shell启动报错:
[hadoop@hadoop002 conf]$ spark-shell
19/06/20 04:44:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/06/20 04:44:56 ERROR spark.SparkContext: Error initializing SparkContext.
java.net.ConnectException: Call From hadoop002/10.0.0.132 to hadoop002:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Caused by: java.net.ConnectException: Connection refused
19/06/20 04:44:56 ERROR repl.Main: Failed to initialize Spark session.
java.net.ConnectException: Call From hadoop002/10.0.0.132 to hadoop002:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
第一个报错解决方法:start-all.sh,进程都启动后再次使用命令spark-shell尝试启动,继续进行报错,日志如下:
[hadoop@hadoop002 conf]$ spark-shell
19/06/20 04:52:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/06/20 04:53:08 ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File does not exist: hdfs://hadoop002:9000/g6_directory
19/06/20 04:53:08 ERROR repl.Main: Failed to initialize Spark session.
java.io.FileNotFoundException: File does not exist: hdfs://hadoop002:9000/g6_directory
第二个报错提示找不到这个g6_directory这个目录,于是我用hdfs dfs -mkdir g6_directory,新建了一个目录哈哈。
再次启动可以了