项目场景:
在本地测试使用spark链接hdfs的时候出现链接不上hdfs的信息,代码如下:
object WorldCountHdfs {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local").setAppName("hdfsWordCount")
//获取spark上下文
val sc = new SparkContext(conf)
val rdd = sc.textFile("hdfs://node03:8020/sparkworldcount/worldcount.txt")
rdd.flatMap(_.split(" ")).map((_,1)).reduceByKey((_+_)).collect().foreach(println)
sc.stop()
}
}
问题描述
出现以下错误信息:
WARN DFSClient: Failed to connect to /192.168.1.125:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information
java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelIm