KeeperErrorCode = NoAuth for /hbase/tokenauth/keys

kerberos配置hbase出現問題

---

環境如下：

问题描述

想要在hadoop ha的場景上，基於kerberos配置hbase ha，出現了如下的bug

org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,035 ERROR [main-EventThread] zookeeper.ZKWatcher: regionserver:16020-0x3029dc0d4ec0021, quorum=hadoop102:2181,hadoop103:2181,hadoop104:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,038 ERROR [main-EventThread] regionserver.HRegionServer: ***** ABORTING region server hadoop102,16020,1687508213386: Unexpected exception handling nodeCreated event *****
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,041 ERROR [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2023-06-23 16:19:56,060 INFO  [main-EventThread] regionserver.HRegionServer:
    "exceptions.ScannerResetException" : 0,

---

原因分析：

這個問題昨天卡了一天，我發現就是在只有在hadoop102幾點上啟動了master和regionserver，我通過hadop102:16010 web頁面訪問，發現是regionserver是dead，所以後面就一直卡死在這裡了

通過bug日誌看，可以知道是kerberos權限認證的問題，看了一下當下的配置文件

hadoop102 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop102@EXAMPLE.COM";
};

hadoop103 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop103@EXAMPLE.COM";
};

hadoop104 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop104@EXAMPLE.COM";
};

我發現只有hadoop102上的master和regionserver啟動起來了，hadoop103和hadoop104沒有啟動，所以下意識就把hadoop103和hadoop104的hbase-jaas.conf配置文件改成了和hadoop102一樣的，重啟hbase，發現所有的服務是可以啟動的，但是無法執行hbase的插入語句

base(main):002:0> create 'student','info'

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2946)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1942)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:603)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:

Create a table with namespace=ns1 and table qualifier=t1
  hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1
  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
  hbase> # The above in shorthand would be the following:
  hbase> create 't1', 'f1', 'f2', 'f3'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
  hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}}
  hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'weekly'}

Table configuration options can be put at the end.
Examples:

  hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' }
  hbase> # Optionally pre-split the table into NUMREGIONS, using
  hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDem
  hbase> create 't1', {NAME => 'f1', DFS_REPLICATION => 1}

You can also keep around a reference to the created table:

  hbase> t1 = create 't1', 'f1'

Which gives you a reference to the table named 't1', on which you can then
call methods.

Took 8.8778 seconds                                                                                                                                                 
hbase(main):003:0>  put 'student','1001','info:sex','male'

ERROR: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on hadoop102,16020,1687510685378
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2429)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Put a cell 'value' at specified table/row/column and optionally
timestamp coordinates.  To put a cell value into table 'ns1:t1' or 't1'
at row 'r1' under column 'c1' marked with the time 'ts1', do:

  hbase> put 'ns1:t1', 'r1', 'c1', 'value'
  hbase> put 't1', 'r1', 'c1', 'value'
  hbase> put 't1', 'r1', 'c1', 'value', ts1
  hbase> put 't1', 'r1', 'c1', 'value', {ATTRIBUTES=>{'mykey'=>'myvalue'}}
  hbase> put 't1', 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
  hbase> put 't1', 'r1', 'c1', 'value', ts1, {VISIBILITY=>'PRIVATE|SECRET'}

The same commands also can be run on a table reference. Suppose you had a reference
t to table 't1', the corresponding command would be:

  hbase> t.put 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}

解决方案：

提示：这里填写该问题的具体解决方案：

看到這裡我發現所有節點的regionserver都沒有正常啟動，全是dead狀態，所以我就猜測是zookeeper中的hbase數據損壞導致的，所以就想把zookeeper中的hbase信息刪除

[zk: hadoop102:2181(CONNECTED) 0] ls
ls [-s] [-w] [-R] path
[zk: hadoop102:2181(CONNECTED) 1] ls /
[dolphinscheduler, hadoop-ha, hbase, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 2] deleteall /hbase
Authentication is not valid : /hbase/replication
[zk: hadoop102:2181(CONNECTED) 3] getAcl /hbase
'sasl,'hbase/hadoop102@EXAMPLE.COM
: cdrwa

發現刪除失敗，一直再報Authentication is not valid : /hbase/replication這個bug，這個是由於zookeeper开启了ACL導致的，最後的解決方案是在zookeeper的配置文件zoo.cfg中加入一行skipACL=yes

#kerberos认证配置
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
sessionRequireClientSASLAuth=true
skipACL=yes

分發zoo.cfg到zk所有節點，重啟zookeeper，再刪除/hbase節點數據

[zk: hadoop102:2181(CONNECTED) 0] ls /
[dolphinscheduler, hadoop-ha, hbase, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 1] deleteall /hbase
[zk: hadoop102:2181(CONNECTED) 2] ls /
[dolphinscheduler, hadoop-ha, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 3] quit;
ZooKeeper -server host:port cmd args

成功刪除！！！
到這裡的時候基本上就已經解決成功了
為了保險起見，我把hdfs上的hbase所有文件也刪除了

hadoop fs -rm -r -f /hbase/*

刪除zoo.cfg中的skipACL=yes，然後重啟zk，重啟hbase，訪問hadoop102:16010 web網頁：

可以看到已經沒有dead regionserver了
再執行hbase 插入語句

hbase(main):001:0> create 'student','info'
Created table student
Took 2.6728 seconds                                                                                                                                                
=> Hbase::Table - student
hbase(main):002:0> put 'student','1001','info:sex','male'
Took 0.1907 seconds                                                                                                                                                
hbase(main):003:0> put 'student','1001','info:age','18'
Took 0.0055 seconds                                                                                                                                                
hbase(main):004:0>  scan 'student'
ROW                                       COLUMN+CELL                                                                                                              
 1001                                     column=info:age, timestamp=1687568561569, value=18                                                                       
 1001                                     column=info:sex, timestamp=1687568556688, value=male                                                                     
1 row(s)
Took 0.0611 seconds                                                                                                                                                
hbase(main):005:0> scan 'student',{STARTROW => '1001', STOPROW  => '1001'}
ROW                                       COLUMN+CELL                                                                                                              
 1001                                     column=info:age, timestamp=1687568561569, value=18                                                                       
 1001                                     column=info:sex, timestamp=1687568556688, value=male                                                                     
1 row(s)
Took 0.0131 seconds                                                                                                                                                
hbase(main):006:0> describe 'student'
Table student is ENABLED                                                                                                                                           
student                                                                                                                                                            
COLUMN FAMILIES DESCRIPTION                                                                                                                                        
{NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
 DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY =
> 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}                
1 row(s)
Took 0.0590 seconds                                                                                                                                                
hbase(main):007:0> quit

至此，bug已經解決完

總結：

解決的bug一共有這麼幾個地方：

1.更改所有節點的 hbase-jaas.conf，保持和hadoop102一致

hadoop102 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop102@EXAMPLE.COM";
};

2.刪除zookeeper中的/hbase數據

在zoo.cfg中加入skipACL=yes，再重啟zk，然後刪除/hbase

3.刪除hdfs上hbase舊數據

hdfs fs -rm -r -f /hbase/*

4.重啟hbase，執行建表和數據插入語句

參考：https://zhuanlan.zhihu.com/p/396007109