小记--bug解决:Idea本地运行Spark作业,缺失winutils.exe hadoop.dll文件

1. 问题发生场景:

• window 环境,使用idea 开发Spark作业,并 运行job作业,报错

{"time":"2020-01-19 11:24:41","logtype":"WARN","loginfo":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"}
{"time":"2020-01-19 11:24:41","logtype":"ERROR","loginfo":"Failed to locate the winutils binary in the hadoop binary path"}
java.io.IOException: Could not locate executable D:\hadoop\hadoop-2.6.0-cdh5.15.1\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:407)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:422)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:415)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
	at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:168)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:132)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:100)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:435)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:341)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:308)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:895)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:861)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:728)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:293)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
	at main.scala.com.xiaolin.huawei.ads.Companys$.main(Companys.scala:22)
	at main.scala.com.xiaolin.huawei.ads.Companys.main(Companys.scala)

2. 解决问题:

• 产生问题原因: window环境问题 不兼容原因,缺失 winutils.exe hadoop.dll文件
• 下载路径: https://github.com/steveloughran/winutils
• 将 下载的文件放置到 hadoop/bin(注:已经配置系统环境变量) 目录下,并且将 hadoop.dll 复制到 window/system32/目录下
• 重启idea 或电脑