Hbase故障修复过程

Hbase故障修复过程

• 故障原因

可能是虚拟主机突然断电，hbase的所有节点都在这个虚拟机上，因此全部停机。启动起来后，发现很多region丢失：is not online。

• 检测HDFS文件是否损坏

使用hadoop命令: hadoop fsck / , 可以看到/ 目录下fs是否是Healthy。

• 查看-ROOT-、.META.表的状态

通过可视化界面查看-ROOT-、.META.表的状态是否正常。
http://hmaster:60010/master-status

看到只有-ROOT-表，而没有了.META.表，说明meta表损坏，而数据并未丢失。

通过scan ‘.META.’ 确定META表在Region Server2上，而Region Server 2在hbase启动后，过一段时间后，IPC端口就不通了，master无法与region server通讯，无法stop，也无法执行修复命令。
所以需要在hbase服务启动后，在正常状态时，迅速执行修复命令。

zookeeper上存储hbase 基本信息，可以删除后，重新启动，会重新自动创建。

[hbase@hmaster hb]$ hbase-0.94.27/bin/hbase zkcli
Connecting to hslave1,hmaster,hslave2:2181
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:host.name=hmaster
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_51
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_51/jre
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hbase/hb/hbase-0.94.27/bin/../conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/hbase/hb/hbase-0.94.27/bin/..:/home/hbase/hb/hbase-0.94.27/bin/../hbase-0.94.27.jar:/home/hbase/hb/hbase-0.94.27/bin/../hbase-0.94.27-tests.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/activation-1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/asm-3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/avro-1.5.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/avro-ipc-1.5.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-beanutils-1.7.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-cli-1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-codec-1.4.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-collections-3.2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-configuration-1.6.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-digester-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-el-1.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-httpclient-3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-io-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-lang-2.5.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-logging-1.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-math-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-net-1.4.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/core-3.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/guava-11.0.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/hadoop-core-1.0.4.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/hamcrest-core-1.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/high-scale-lib-1.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/httpclient-4.1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/httpcore-4.1.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-core-asl-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-xc-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jamon-runtime-2.3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jasper-compiler-5.5.23.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jasper-runtime-5.5.23.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jaxb-api-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-core-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-json-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-server-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jettison-1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jetty-6.1.26.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jetty-util-6.1.26.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jruby-complete-1.6.5.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsp-2.1-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsr305-1.3.9.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/junit-4.11.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/libthrift-0.8.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/log4j-1.2.16.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/metrics-core-2.1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/netty-3.2.4.Final.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/protobuf-java-2.4.0a.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/slf4j-api-1.4.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/snappy-java-1.0.3.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/stax-api-1.0.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/velocity-1.7.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/xmlenc-0.52.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/zookeeper-3.4.5.jar:
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hbase/hb/hbase-0.94.27/bin/../lib/native/Linux-amd64-64
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase/hb
18/03/30 10:18:11 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hslave1,hmaster,hslave2:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@fa4033b
Welcome to ZooKeeper!
18/03/30 10:18:11 INFO zookeeper.ClientCnxn: Opening socket connection to server hslave2/192.168.198.96:2181. Will not attempt to authenticate using SASL (unknown error)
18/03/30 10:18:11 INFO zookeeper.ClientCnxn: Socket connection established to hslave2/192.168.198.96:2181, initiating session
18/03/30 10:18:11 INFO zookeeper.ClientCnxn: Session establishment complete on server hslave2/192.168.198.96:2181, sessionid = 0x262749682e40004, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
JLine support is enabled
[zk: hslave1,hmaster,hslave2:2181(CONNECTED) 0] 
[zk: hslave1,hmaster,hslave2:2181(CONNECTED) 0] list
ZooKeeper -server host:port cmd args
    connect host:port
    get path [watch]
    ls path [watch]
    set path data [version]
    rmr path
    delquota [-n|-b] path
    quit 
    printwatches on|off
    create [-s] [-e] path data acl
    stat path [watch]
    close 
    ls2 path [watch]
    history 
    listquota path
    setAcl path acl
    getAcl path
    sync path
    redo cmdno
    addauth scheme auth
    delete path [version]
    setquota -n|-b val path
[zk: hslave1,hmaster,hslave2:2181(CONNECTED) 1] ls /hbase
[splitlog, online-snapshot, unassigned, root-region-server, table92, backup-masters, rs, table, draining, master, shutdown, hbaseid]
[zk: hslave1,hmaster,hslave2:2181(CONNECTED) 2]

---

通过查看region信息，发现很多表的region都缺失了。

使用修复meta命令，发现问题依然存在，然后尝试修复assignments，ok。

[hbase@hmaster hb]$ hbase-0.94.27/bin/hbase hbck -fixAssignments
18/03/30 09:48:47 DEBUG util.FSUtils: Creating file=hdfs://hmaster:9000/hbase/.tmp/hbase-hbck.lock with permission=rwxrwxrwx
18/03/30 09:48:47 INFO util.FSUtils: FileSystem doesn't support getDefaultReplication
18/03/30 09:48:47 INFO util.FSUtils: FileSystem doesn't support getDefaultBlockSize
18/03/30 09:48:47 DEBUG util.HBaseFsck: Launching hbck
18/03/30 09:48:47 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 2739@hmaster
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:host.name=hmaster
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_51
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_51/jre
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hbase/hb/hbase-0.94.27/bin/../conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/hbase/hb/hbase-0.94.27/bin/..:/home/hbase/hb/hbase-0.94.27/bin/../hbase-0.94.27.jar:/home/hbase/hb/hbase-0.94.27/bin/../hbase-0.94.27-tests.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/activation-1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/asm-3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/avro-1.5.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/avro-ipc-1.5.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-beanutils-1.7.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-cli-1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-codec-1.4.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-collections-3.2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-configuration-1.6.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-digester-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-el-1.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-httpclient-3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-io-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-lang-2.5.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-logging-1.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-math-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/commons-net-1.4.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/core-3.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/guava-11.0.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/hadoop-core-1.0.4.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/hamcrest-core-1.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/high-scale-lib-1.1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/httpclient-4.1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/httpcore-4.1.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-core-asl-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jackson-xc-1.8.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jamon-runtime-2.3.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jasper-compiler-5.5.23.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jasper-runtime-5.5.23.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jaxb-api-2.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-core-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-json-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jersey-server-1.8.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jettison-1.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jetty-6.1.26.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jetty-util-6.1.26.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jruby-complete-1.6.5.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsp-2.1-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/jsr305-1.3.9.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/junit-4.11.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/libthrift-0.8.0.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/log4j-1.2.16.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/metrics-core-2.1.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/netty-3.2.4.Final.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/protobuf-java-2.4.0a.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/slf4j-api-1.4.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/snappy-java-1.0.3.2.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/stax-api-1.0.1.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/velocity-1.7.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/xmlenc-0.52.jar:/home/hbase/hb/hbase-0.94.27/bin/../lib/zookeeper-3.4.5.jar:
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hbase/hb/hbase-0.94.27/bin/../lib/native/Linux-amd64-64
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase/hb
18/03/30 09:48:47 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hslave1:2181,hmaster:2181,hslave2:2181 sessionTimeout=180000 watcher=hconnection0x0
18/03/30 09:48:47 INFO zookeeper.ClientCnxn: Opening socket connection to server hmaster/192.168.198.94:2181. Will not attempt to authenticate using SASL (unknown error)
18/03/30 09:48:47 INFO zookeeper.ClientCnxn: Socket connection established to hmaster/192.168.198.94:2181, initiating session
18/03/30 09:48:47 INFO zookeeper.ClientCnxn: Session establishment complete on server hmaster/192.168.198.94:2181, sessionid = 0x62749661730002, negotiated timeout = 40000
18/03/30 09:48:47 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@647692f6; serverName=hslave1,60020,1522374468846
18/03/30 09:48:47 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is hslave1:60020
Version: 0.94.27
18/03/30 09:48:47 INFO util.HBaseFsck: Loading regionsinfo from the .META. table
18/03/30 09:48:47 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@647692f6; serverName=hslave1,60020,1522374468846
18/03/30 09:48:47 DEBUG client.MetaScanner: Scanning -ROOT- starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@647692f6
18/03/30 09:48:47 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@647692f6; serverName=hslave1,60020,1522374468846
18/03/30 09:48:47 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@647692f6

18/03/30 09:48:47 INFO util.HBaseFsck: getHTableDescriptors == tableNames => [ac_email_account_index1, ac_ftp_account, ac_stream_account_index, sn_mailpic_porn, ac_telnet_account_index, sn_ftpresource, sn_ftpresource_index, sn_telnetbehaviorinfo, ac_telnet_account, ac_ftp_account_index, ac_vpn_account_index, ac_p2p_account, sn_telnetresource, sn_ftpbehaviorinfo, ac_email_account, sn_streambehaviorinfo, sn_ircbehaviorinfo, sn_mailresource, sn_p2pbehaviorinfo, sn_rdpbehaviorinfo, ac_irc_account_index, sn_mailbehaviorinfo, sn_vpnbehaviorinfo, http_account]
Number of Tables: 24
Number of live region servers: 2
Number of dead region servers: 0
Master: hmaster,60000,1522374465970
Number of backup masters: 0
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/-ROOT-
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/.META.
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/.archive
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/.corrupt
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/.logs
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading region dirs from hdfs://hmaster:9000/hbase/.oldlogs
18/03/30 09:48:48 DEBUG util.HBaseFsck: Loading regi