IDEA开发Spark提示：Failed to locate the winutils binary in the hadoop binary path

今天在运行Sparkstreaming时，出现了spark提示错误：

19/11/19 13:14:36 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:390)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:610)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:277)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:265)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:810)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:780)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:653)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2468)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2468)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2468)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:292)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:838)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
	at cn.edu.zut.bigdata.sparkstreaming.SparkStreamingTelnet$.delayedEndpoint$cn$edu$zut$bigdata$sparkstreaming$SparkStreamingTelnet$1(SparkStreamingTelnet.scala:10)
	at cn.edu.zut.bigdata.sparkstreaming.SparkStreamingTelnet$delayedInit$body.apply(SparkStreamingTelnet.scala:6)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at cn.edu.zut.bigdata.sparkstreaming.SparkStreamingTelnet$.main(SparkStreamingTelnet.scala:6)
	at cn.edu.zut.bigdata.sparkstreaming.SparkStreamingTelnet.main(SparkStreamingTelnet.scala)

程序代码如下：

package cn.edu.zut.bigdata.sparkstreaming

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

object SparkStreamingTelnet extends App {
            //配置对象：这里先使用本地模式
     val conf = new SparkConf().setAppName("stream").setMaster("local[3]")
            //创建StreamingContext对象，默认10秒处理一次数据
     val ssc = new StreamingContext(conf,Seconds(10))
            //从socket接收数据
     val lineDStream = ssc.socketTextStream("node01",6666)
            //统计词频
     val result = lineDStream.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
       result.print()
  ssc.start()
  ssc.awaitTermination()
}

错误原因分析：

这是主要是因为：hadoop只能运行在linux环境下，如果我们在windows下用idea开发spark的时候底层比方说文件系统这些方面调用hadoop的时候是没法调用的，这也就是为什么会提示这样的错误。

但是如果我们需要使用Hadoop上的hdfs等程序时，而我们的Windows环境上并没有Hadoop，我们应该怎么解决？

解决办法：

1.下载winutils的windows版本。打开GitHub地址：https://github.com/srccodes/hadoop-common-2.2.0-bin, 直接下载此项目的zip包，下载后是文件名是hadoop-common-2.2.0-bin-master.zip，随便解压到一个目录；
2.设置环境变量，在系统变量中添加HADOOP_HOME（图一），指向你解压的文件路径。然后再path中添加%HADOOP_HOME%bin（图二）；


3.重启开发工具idea，再次运行程序，正常执行。