IDEA环境安装及运行

最新推荐文章于 2024-07-29 19:50:16 发布

段智华

最新推荐文章于 2024-07-29 19:50:16 发布

阅读量1.2k

点赞数

分类专栏：生产实战spark

热烈祝贺Gavin大咖2024年北京航空航天大学两本新书《Transformer&ChatGPT解密：原理、源码及案例》、《Transformer& Rasa 解密: 原理、源码及案例》出版发行

本文链接：https://blog.csdn.net/duan_zhihua/article/details/79062215

版权

生产实战spark 专栏收录该内容

13 篇文章 2 订阅

订阅专栏

IDEA环境安装及运行

（1）windows 10 + idea 运行spark 提示 Winutils异常：

解决办法：

1，建立目录 C:\hadoop\bin，将网上下载的winutils.exe文件放在此目录下。（网上提供的下载地址https://github.com/srccodes/hadoop-common-2.2.0-bin）

2，环境变量配置：HADOOP_HOME 配置为 C:\hadoop

3，注销或重启windows 10，Winutils问题解决。

18/01/15 10:04:17 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
	at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2373)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:295)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
	at com.dt.spark.cores.DatasetOps$.main(DatasetOps.scala:27)
	at com.dt.spark.cores.DatasetOps.main(DatasetOps.scala)

（2） idea +git

在IDEA 中使用Git 可参考 https://www.cnblogs.com/zbw911/p/6206689.html

git下载提示

15:31	Couldn't save uncommitted changes.
		Tried to save uncommitted changes in stash before Update, but failed with an error.
		stash file://D:/WorkSpace/test1.0: unable to auto-detect email address (got 'lenovo@duanzhihua.(none)')
		Cannot save the current index state

解决：

1，进入git的目录

C:\Users\lenovo>cd C:\git\bin

2，设置用户名密码即可

C:\Users\lenovo>
C:\Users\lenovo>cd C:\git\bin

C:\git\bin>git config --global user.email "duanzhihua@189.cn"

C:\git\bin>git config --global user.name "duanzhihua"

C:\git\bin>