1、在创建了表mytab之后,物理机出故障,导致虚拟机异常宕机重启,Hadoop->Zookeeper->Hbase启动之后查看表
2、查看Hbase UI,只有ROOT表,META元数据表和自己创建的mytab、test表都没了,这里没有截图
3、使用hbck修复META元数据
修复不了,报错
org
.apache
.hadoop
.hbase
.NotServingRegionException
: Region is
not online
4、查看Hadoop文件是否存在
文件一切正常,有之前创建mytab表的文件
5、检查Hadoop文件是否损坏
Hadoop文件也正常,那么,region没有在线,是不是zookeeper集群存在问题
6、停止集群,删除zookeeper文件,相当于重建zookeeper集群
7、再次使用hbck修复META元数据表
这次修复貌似正常了,但是日志中提示remove了一些在cache中的文件
- hbase(main):001:0> scan 'mytab'
- ROW COLUMN+CELL
-
- ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for mytab,,99999999999999 after 7 tries.
-
- Here is some help for this command:
- Scan a table; pass table name and optionally a dictionary of scanner
- specifications. Scanner specifications may include one or more of:
- TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, TIMESTAMP, MAXLENGTH,
- or COLUMNS, CACHE
-
- If no columns are specified, all columns will be scanned.
- To scan all members of a column family, leave the qualifier empty as in
- 'col_family:'.
3、使用hbck修复META元数据
- grid@master1:~$ hbase hbck -fixMeta
- ........................................
- 15/09/11 16:05:32 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
- 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
- 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=slave3, port=60020}, attempt=10 of 14 failed; retrying after sleep of 64200 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
- at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3609)
- at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:2202)
- at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
- at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.lang.reflect.Method.invoke(Method.java:606)
- at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
- at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
- ........................................
4、查看Hadoop文件是否存在
- grid@master1:~$ hadoop fs -lsr /hbase
- drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/-ROOT-
- -rw-r--r-- 2 grid supergroup 728 2015-09-08 13:06 /hbase/-ROOT-/.tableinfo.0000000001
- drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/-ROOT-/70236052
- drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs
- -rw-r--r-- 2 grid supergroup 421 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs/hlog.1441717607322
- -rw-r--r-- 2 grid supergroup 109 2015-09-08 13:06 /hbase/-ROOT-/70236052/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/info
- -rw-r--r-- 3 grid supergroup 1596 2015-09-11 19:21 /hbase/-ROOT-/70236052/info/924e206e22754690b08c8ec8bbf84692
- drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/-ROOT-/70236052/recovered.edits
- drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/.META.
- drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/.META./1028785192
- drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs
- -rw-r--r-- 2 grid supergroup 134 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs/hlog.1441717607577
- -rw-r--r-- 2 grid supergroup 111 2015-09-08 13:06 /hbase/.META./1028785192/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/info
- -rw-r--r-- 3 grid supergroup 3443 2015-09-11 19:21 /hbase/.META./1028785192/info/d735b80efd454d9caca81e4392e8d0d2
- drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/.META./1028785192/recovered.edits
- drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.archive
- drwxr-xr-x - grid supergroup 0 2015-09-09 10:58 /hbase/.corrupt
- drwxr-xr-x - grid supergroup 0 2015-09-11 17:35 /hbase/.logs
- drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388
- -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388/slave1%2C60020%2C1441989279388.1442007288767
- drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626
- -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626/slave2%2C60020%2C1441989279626.1442007288761
- drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275
- -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275/slave3%2C60020%2C1441989280275.1442007288199
- drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.oldlogs
- -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave1%2C60020%2C1441989279388.1442003688318
- -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave2%2C60020%2C1441989279626.1442003688444
- -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave3%2C60020%2C1441989280275.1442003687774
- drwxr-xr-x - grid supergroup 0 2015-09-11 16:35 /hbase/.tmp
- -rw-r--r-- 2 grid supergroup 19 2015-09-11 16:35 /hbase/.tmp/hbase-hbck.lock
- -rw-r--r-- 2 grid supergroup 38 2015-09-08 13:06 /hbase/hbase.id
- -rw-r--r-- 2 grid supergroup 3 2015-09-08 13:06 /hbase/hbase.version
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest
- -rw-r--r-- 2 grid supergroup 671 2015-09-09 13:34 /hbase/hivetest/.tableinfo.0000000001
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e
- -rw-r--r-- 2 grid supergroup 231 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab
- -rw-r--r-- 2 grid supergroup 665 2015-09-09 17:26 /hbase/mytab/.tableinfo.0000000001
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582
- -rw-r--r-- 2 grid supergroup 230 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910
- -rw-r--r-- 2 grid supergroup 236 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185
- -rw-r--r-- 2 grid supergroup 238 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7
- -rw-r--r-- 2 grid supergroup 226 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30
- -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/cf1
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test
- -rw-r--r-- 2 grid supergroup 663 2015-09-08 13:15 /hbase/test/.tableinfo.0000000001
- drwxr-xr-x - grid supergroup 0 2015-09-08 13:15 /hbase/test/.tmp
- drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad
- -rw-r--r-- 2 grid supergroup 219 2015-09-08 13:15 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/.regioninfo
- drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1
- -rw-r--r-- 3 grid supergroup 1018 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1/b8cd45592ea64e278a12b3f0b95eb220
- drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/recovered.edits
5、检查Hadoop文件是否损坏
- grid@master1:~$ hadoop fsck /
- FSCK started by grid from /10.10.10.70 for path / at Fri Sep 11 16:14:13 UTC 2015
- .............................Status: HEALTHY
- Total size: 12829 B
- Total dirs: 70
- Total files: 29 (Files currently being written: 6)
- Total blocks (validated): 29 (avg. block size 442 B) (Total open file blocks (not validated): 6)
- Minimally replicated blocks: 29 (100.0 %)
- Over-replicated blocks: 0 (0.0 %)
- Under-replicated blocks: 0 (0.0 %)
- Mis-replicated blocks: 0 (0.0 %)
- Default replication factor: 2
- Average block replication: 2.0344827
- Corrupt blocks: 0
- Missing replicas: 0 (0.0 %)
- Number of data-nodes: 3
- Number of racks: 1
- FSCK ended at Fri Sep 11 16:14:13 UTC 2015 in 30 milliseconds
-
-
- The filesystem under path '/' is HEALTHY
6、停止集群,删除zookeeper文件,相当于重建zookeeper集群
- grid@master1:~$ stop-hbase.sh
- stopping hbase...
- grid@master1:~$ zkServer.sh stop
- JMX enabled by default
- Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
- Stopping zookeeper ... STOPPED
- grid@master1:~$ cd /data/zk/
- grid@master1:/data/zk$ ls
- myid version-2
- grid@master1:/data/zk$ ls -l
- total 8
- -rw-r--r-- 1 grid hadoop 3 Sep 8 20:33 myid
- drwxr-xr-x 2 grid hadoop 4096 Sep 11 23:10 version-2
- grid@master1:/data/zk$ rm -rf version-2/
- grid@master1:/data/zk$ cd ~
- grid@master1:~$
-
- grid@master1:~$ zkServer.sh start
- JMX enabled by default
- Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
- Starting zookeeper ... STARTED
- grid@master1:~$ zkServer.sh status
- JMX enabled by default
- Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
- Mode: follower
- grid@master1:~$ start-hbase.sh
- starting master, logging to /usr/local/hbase/logs/hbase-grid-master-master1.out
- slave1: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave1.out
- slave2: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave2.out
- slave3: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave3.out
- grid@master1:~$ jps
- 7544 Jps
- 1473 SecondaryNameNode
- 6257 QuorumPeerMain
- 1556 JobTracker
- 1283 NameNode
- 6463 HMaster
- grid@master1:~$ hbase hbck -fixMeta
- .....................................
- 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
- 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
- 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
- 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
- 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
- 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
- 15/09/11 16:36:03 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
- ......................................
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/12219480/viewspace-1797852/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/12219480/viewspace-1797852/