小记--bug:org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(Ljava/lang/String；)V

最新推荐文章于 2023-10-15 19:23:59 发布

xiaolin_xinji

最新推荐文章于 2023-10-15 19:23:59 发布

阅读量1.3k

点赞数 1

分类专栏： Spark BUG 文章标签： spark hadoop

本文链接：https://blog.csdn.net/weixin_44131414/article/details/116749786

版权

Spark 同时被 2 个专栏收录

25 篇文章 0 订阅

订阅专栏

BUG

14 篇文章 0 订阅

订阅专栏

在使用新版本的hadoop3.2.2时,启动spark sql 发现以下报错

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(Ljava/lang/String;)V
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:84)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:319)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:304)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1860)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:718)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:668)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2464)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:292)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
	at com.xiaolin.ML.LogisticRegression.code10.LinearRegression01$.main(LinearRegression01.scala:15)
	at com.xiaolin.ML.LogisticRegression.code10.LinearRegression01.main(LinearRegression01.scala)

Process finished with exit code 1

找到相关类发现

新增代码块 ,报红在2.6版本中无

   String ruleMechanism = conf.get(HADOOP_SECURITY_AUTH_TO_LOCAL_MECHANISM,  DEFAULT_MECHANISM);
    setRuleMechanism(ruleMechanism);

解决方案:


    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>${hadoop.version}</version>
    </dependency>

    <!-- 引入该jar 即可-->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-auth</artifactId>
      <version>${hadoop.version}</version>
    </dependency>