HBase Master启动认证kerberos失败&问题解决

1. 问题复原

在kerberos环境下启动HBase时，包如下错误：

2018-03-29 11:59:12,828 INFO  [main-SendThread(hadoop2497.jd.163.org:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop2497.jd.163.org/10.196.67.44:2181, initiating session
2018-03-29 11:59:12,837 INFO  [main-SendThread(hadoop2497.jd.163.org:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop2497.jd.163.org/10.196.67.44:2181, sessionid = 0x2621ecd5e1a0165, negotiated timeout = 40000
2018-03-29 11:59:12,861 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster. 
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2512)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:231)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2522)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: master:16000-0x2621ecd5e1a0165, quorum=hadoop2496.jd.163.org:2181,hadoop2497.jd.163.org:2181,hadoop2498.jd.163.org:2181, baseZNode=/hbase-secure Unexpected KeeperException creating base node
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:206)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:187)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:572)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:412)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2505)
        ... 5 more
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hbase-secure
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:565)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:544)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1204)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1182)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:194)
        ... 13 more

查看了下日志，使用keytab也登录成功了：

2018-03-29 11:59:12,180 INFO  [main] security.UserGroupInformation: Login successful for user hbase/atlas1.jd.163.org@HADOOP2.HZ.NETEASE.COM using keytab file /home/hadoop/yarn/conf/hbase.service.keytab

不清楚为什么在HBase启动的时候为何不能自动初始化/hbase-secure节点？

解决方案

手动创建带acl权限的/hbase-secure节点。

命令如下：

create /hbase-secure "" sasl:hbase:cdrwa

参考：
* https://community.hortonworks.com/articles/29900/zookeeper-using-superdigest-to-gain-full-access-to.html
* https://community.hortonworks.com/articles/90705/hive-llap-fails-with-invalidacl-for-llap-sasluser.html