必须先吐槽一下 Cloudera 6.x 和 Hbase 2.0 太坑了!
不久前生产上的一套Hbase集群出现著名的RIT(Regions in Transition)问题。
查看hbase web ui
于是通过hbck命令查看一下集群状态,果然好多inconsistency
...
ERROR: Region { meta => XXX,XXX:,1573019231000.ff2aecaf28917792395c341d01e0b8cc., hdfs => hdfs://nameservice1/hbase/data/default/XXX/ff2aecaf28917792395c341d01e0b8cc, deployed => , replicaId => 0 } not deployed on any region server.
...
ERROR: Found inconsistency in table XXX
...
9 inconsistencies detected.
Status: INCONSISTENT
看到错误提示问题明显了,这个Region在hdfs中有数据文件但没有依赖任何Region Server,原因可能region被原来的Region Server unassigned了,但是还没有被assigned到一个新的Region Server上。
那么尝试用hbase hbck -repair
和hbase hbck -fixMeta -fixAssignments
来修复吧,于是就有了下面的提示,hbase2.0+以后hbck的所有修复功能全都不支持...
-----------------------------------------------------------------------
NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
In general, all Read-Only options are supported and can be be used
safely. Most -fix/ -repair options are NOT supported. Please see usage
below for details on which options are not supported.
-----------------------------------------------------------------------
NOTE: Following options are NOT supported as of HBase version 2.0+.
UNSUPPORTED Metadata Repair options: (expert features, use with caution!)
-fix Try to fix region assignments. This is for backwards compatiblity
-fixAssignments Try to fix region assignments. Replaces the old -fix
-fixMeta Try to fix meta problems. This assumes HDFS region info is good.