记一次Spark Kerberos的故障解决

有同事反馈，Livy Server启动的所有Spark AM失败。Livy启动的Spark AM默认会enableHiveSupport，且使用$LIVY_HOME/conf/livy.conf的如下配置作为spark.yarn.keytab和spark.yarn.kerberos。

livy.server.launch.kerberos.keytab

livy.server.launch.kerberos.principal

由于不知道报错信息，查看Spark AM log，有所发现：

Attempting to login to Kerberos using principal: ...

...

GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

说明spark.yarn.principal和spark.yarn.keytab已经设置，但是校验Kerberos ticket失败了。根据错误信息，对照Spark-Hive的代码HiveClientImpl.scala.

 

// Set up kerberos credentials for UserGroupInformation.loginUser within
// current class loader
if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) {
  val principalName = sparkConf.get("spark.yarn.principal")
  val keytabFileName =