华为FusionInsight sparkstreaming消费kafka kerberos异常解决

背景

使用FusionInsight Manager上的sparkstreaming消费kafka数据到hive

编写

修改FusionInsight client的spark2中的demo进行流程测试
修改maven中的依赖，将所有的dependency增加<scope>provided</scope>，使用maven打包

	<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-source-plugin</artifactId>
                <version>2.1</version>
                <configuration>
                    <attach>true</attach>
                </configuration>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.1.6</version>
                <configuration>
                    <scalaCompatVersion>2.11</scalaCompatVersion>
                    <scalaVersion>2.11.12</scalaVersion>
                    <encoding>UTF-8</encoding>
                </configuration>
                <executions>
                    <execution>
                        <id>compile-scala</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>test-compile-scala</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
            </plugin>
        </plugins>
    </build>

调试

将jar上传到安装有FusionInsight client的服务器上

加载环境变量

source /opt/client/bigdata_env

编写spark-job.sh文件，方便提交任务

#! /bin/bash

base_path=/home/ooxx/spark-job
taskName=kafkaToHive
kafkaBrokers=192.168.52.101:21007,192.168.52.102:21007,192.168.52.103:21007
kafkaTopics=kafkaToHive
#spark流计算时间间隔，单位秒
sparkStreamingBatchTime=10
hiveTable=kafka_to_hive
hiveDataBase=default

loginUser=ooxx
#client模式，配置本地路径；cluster模式，配置成user_container.keytab
keytabPath=user_container.keytab
#client模式，配置本地路径；cluster模式，配置成krb5.conf
krb5Path=krb5.conf

spark-submit --master yarn \
--deploy-mode cluster \
--keytab user.keytab \
--principal ooxx \
--files ./jaas.conf,./user_container.keytab,./krb5.conf \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf -Djava.security.krb5.conf=./krb5.conf" \
--conf "spark.yarn.cluster.driver.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--jars $SPARK_HOME/jars/streamingClient010/kafka-clients-1.1.0.jar,$SPARK_HOME/jars/streamingClient010/kafka_2.11-1.1.0.jar,$SPARK_HOME/jars/streamingClient010/spark-streaming-kafka-0-10_2.11-2.3.2.jar \
--class com.ooxx.kafka.KafkaToHive \
--name $taskName \
--driver-memory 2g --executor-cores 2 --executor-memory 2g --num-executors 4  \
${base_path}/huawei_kafka_to_hive-1.0-SNAPSHOT.jar $taskName $kafkaBrokers $kafkaTopics $sparkStreamingBatchTime $hiveTable $hiveDataBase $loginUser $keytabPath $krb5Path > ${base_path}/logs/KafkaToHive.log 2>&1 &

jaas.conf文件内容

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./user_container.keytab"
principal="ooxx"
useTicketCache=false
storeKey=true
debug=true;
};

遇到的问题

问题1

Could not find a 'KafkaClient' entry in the JAAS configuration. 
System property 'java.security.auth.login.config' is ./__spark_conf__/__hadoop_conf__/jaas-zk.conf

百度和官网都有提示信息

client模式使用：--driver-java-options "-Djava.security.auth.login.config=./jaas.conf"和--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"
cluster模式使用：--conf "spark.yarn.cluster.driver.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"和--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"
这两者都需要--files上传本地文件

问题2

Exception in thread "main" java.lang.IllegalArgumentException: Can't get Kerberos realm
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:318)
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:364)
	at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:53)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance$lzycompute(SparkHadoopUtil.scala:409)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance(SparkHadoopUtil.scala:409)
	at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:430)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:192)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:285)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:110)
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
	... 9 more
Caused by: KrbException: Cannot locate default realm
	at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
	... 15 more

这个问题卡了我好几天，不过也让我对spark执行任务的流程更清晰了些。每次执行applicationmaster都能正常启动，executor一直报这个错，然后退出，重试几次后整体失败。百度了半天，发现报这个错的解决办法都有涉及到 java.security.krb5.conf。然后就假设是不是executor角色的container找不到krb5.conf文件导致的，索性将java.security.krb5.conf也配置到spark.executor.extraJavaOptions中，并且上传krb5.conf，所以就是

--files ./jaas.conf,./user_container.keytab,./krb5.conf \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf -Djava.security.krb5.conf=./krb5.conf" \

延伸

在解决问题的过程中，也翻到一些文章，其中有提到sparkstreaming长时间运行会存在票据过期问题，所以配置了如下配置

--keytab user.keytab \
--principal ooxx \

然后文章又提到此user.keytab需要和–files中的user.keytab不同名称但是内容相同，所以将–files中的user.keytab改成user_container.keytab，并且修改对应的jaas.conf中keytab文件名称和在driver端执行的代码中的keytab名称。目前还没对票据过期这个问题进行测试认证。