Hbase异常宕机，NoServerForRegionException: Unable to find region（1）

1、在创建了表mytab之后，物理机出故障，导致虚拟机异常宕机重启，Hadoop->Zookeeper->Hbase启动之后查看表

1. hbase(main):001:0> scan 'mytab'
   
2. ROW COLUMN+CELL
   
3. 
   
4. ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for mytab,,99999999999999 after 7 tries.
   
5. 
   
6. Here is some help for this command:
   
7. Scan a table; pass table name and optionally a dictionary of scanner
   
8. specifications. Scanner specifications may include one or more of:
   
9. TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, TIMESTAMP, MAXLENGTH,
   
10. or COLUMNS, CACHE
    
11. 
    
12. If no columns are specified, all columns will be scanned.
    
13. To scan all members of a column family, leave the qualifier empty as in
    
14. 'col_family:'.

2、查看Hbase UI，只有ROOT表，META元数据表和自己创建的mytab、test表都没了，这里没有截图
3、使用hbck修复META元数据

1. grid@master1:~$ hbase hbck -fixMeta
   
2. ........................................
   
3. 15/09/11 16:05:32 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
   
4. 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
   
5. 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=slave3, port=60020}, attempt=10 of 14 failed; retrying after sleep of 64200 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
   
6.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3609)
   
7.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:2202)
   
8.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   
9.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   
10.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    
11.         at java.lang.reflect.Method.invoke(Method.java:606)
    
12.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
    
13.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
    
14. ........................................

修复不了，报错 org .apache .hadoop .hbase .NotServingRegionException : Region is not online
4、查看Hadoop文件是否存在

1. grid@master1:~$ hadoop fs -lsr /hbase
   
2. drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/-ROOT-
   
3. -rw-r--r-- 2 grid supergroup 728 2015-09-08 13:06 /hbase/-ROOT-/.tableinfo.0000000001
   
4. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/.tmp
   
5. drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/-ROOT-/70236052
   
6. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs
   
7. -rw-r--r-- 2 grid supergroup 421 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs/hlog.1441717607322
   
8. -rw-r--r-- 2 grid supergroup 109 2015-09-08 13:06 /hbase/-ROOT-/70236052/.regioninfo
   
9. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/.tmp
   
10. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/info
    
11. -rw-r--r-- 3 grid supergroup 1596 2015-09-11 19:21 /hbase/-ROOT-/70236052/info/924e206e22754690b08c8ec8bbf84692
    
12. drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/-ROOT-/70236052/recovered.edits
    
13. drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/.META.
    
14. drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/.META./1028785192
    
15. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs
    
16. -rw-r--r-- 2 grid supergroup 134 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs/hlog.1441717607577
    
17. -rw-r--r-- 2 grid supergroup 111 2015-09-08 13:06 /hbase/.META./1028785192/.regioninfo
    
18. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/.tmp
    
19. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/info
    
20. -rw-r--r-- 3 grid supergroup 3443 2015-09-11 19:21 /hbase/.META./1028785192/info/d735b80efd454d9caca81e4392e8d0d2
    
21. drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/.META./1028785192/recovered.edits
    
22. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.archive
    
23. drwxr-xr-x - grid supergroup 0 2015-09-09 10:58 /hbase/.corrupt
    
24. drwxr-xr-x - grid supergroup 0 2015-09-11 17:35 /hbase/.logs
    
25. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388
    
26. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388/slave1%2C60020%2C1441989279388.1442007288767
    
27. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626
    
28. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626/slave2%2C60020%2C1441989279626.1442007288761
    
29. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275
    
30. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275/slave3%2C60020%2C1441989280275.1442007288199
    
31. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.oldlogs
    
32. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave1%2C60020%2C1441989279388.1442003688318
    
33. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave2%2C60020%2C1441989279626.1442003688444
    
34. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave3%2C60020%2C1441989280275.1442003687774
    
35. drwxr-xr-x - grid supergroup 0 2015-09-11 16:35 /hbase/.tmp
    
36. -rw-r--r-- 2 grid supergroup 19 2015-09-11 16:35 /hbase/.tmp/hbase-hbck.lock
    
37. -rw-r--r-- 2 grid supergroup 38 2015-09-08 13:06 /hbase/hbase.id
    
38. -rw-r--r-- 2 grid supergroup 3 2015-09-08 13:06 /hbase/hbase.version
    
39. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest
    
40. -rw-r--r-- 2 grid supergroup 671 2015-09-09 13:34 /hbase/hivetest/.tableinfo.0000000001
    
41. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/.tmp
    
42. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e
    
43. -rw-r--r-- 2 grid supergroup 231 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/.regioninfo
    
44. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/cf1
    
45. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab
    
46. -rw-r--r-- 2 grid supergroup 665 2015-09-09 17:26 /hbase/mytab/.tableinfo.0000000001
    
47. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/.tmp
    
48. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d
    
49. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/.regioninfo
    
50. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/cf1
    
51. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224
    
52. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/.regioninfo
    
53. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/cf1
    
54. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582
    
55. -rw-r--r-- 2 grid supergroup 230 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/.regioninfo
    
56. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/cf1
    
57. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d
    
58. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/.regioninfo
    
59. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/cf1
    
60. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910
    
61. -rw-r--r-- 2 grid supergroup 236 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/.regioninfo
    
62. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/cf1
    
63. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185
    
64. -rw-r--r-- 2 grid supergroup 238 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/.regioninfo
    
65. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/cf1
    
66. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7
    
67. -rw-r--r-- 2 grid supergroup 226 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/.regioninfo
    
68. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/cf1
    
69. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d
    
70. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/.regioninfo
    
71. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/cf1
    
72. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a
    
73. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/.regioninfo
    
74. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/cf1
    
75. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b
    
76. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/.regioninfo
    
77. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/cf1
    
78. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30
    
79. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/.regioninfo
    
80. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/cf1
    
81. drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test
    
82. -rw-r--r-- 2 grid supergroup 663 2015-09-08 13:15 /hbase/test/.tableinfo.0000000001
    
83. drwxr-xr-x - grid supergroup 0 2015-09-08 13:15 /hbase/test/.tmp
    
84. drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad
    
85. -rw-r--r-- 2 grid supergroup 219 2015-09-08 13:15 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/.regioninfo
    
86. drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1
    
87. -rw-r--r-- 3 grid supergroup 1018 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1/b8cd45592ea64e278a12b3f0b95eb220
    
88. drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/recovered.edits

文件一切正常，有之前创建mytab表的文件
5、检查Hadoop文件是否损坏

1. grid@master1:~$ hadoop fsck /
   
2. FSCK started by grid from /10.10.10.70 for path / at Fri Sep 11 16:14:13 UTC 2015
   
3. .............................Status: HEALTHY
   
4.  Total size: 12829 B
   
5.  Total dirs: 70
   
6.  Total files: 29 (Files currently being written: 6)
   
7.  Total blocks (validated): 29 (avg. block size 442 B) (Total open file blocks (not validated): 6)
   
8.  Minimally replicated blocks: 29 (100.0 %)
   
9.  Over-replicated blocks: 0 (0.0 %)
   
10.  Under-replicated blocks: 0 (0.0 %)
    
11.  Mis-replicated blocks: 0 (0.0 %)
    
12.  Default replication factor: 2
    
13.  Average block replication: 2.0344827
    
14.  Corrupt blocks: 0
    
15.  Missing replicas: 0 (0.0 %)
    
16.  Number of data-nodes: 3
    
17.  Number of racks: 1
    
18. FSCK ended at Fri Sep 11 16:14:13 UTC 2015 in 30 milliseconds
    
19. 
    
20. 
    
21. The filesystem under path '/' is HEALTHY

Hadoop文件也正常，那么，region没有在线，是不是zookeeper集群存在问题
6、停止集群，删除zookeeper文件，相当于重建zookeeper集群

1. grid@master1:~$ stop-hbase.sh
   
2. stopping hbase...
   
3. grid@master1:~$ zkServer.sh stop
   
4. JMX enabled by default
   
5. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
   
6. Stopping zookeeper ... STOPPED
   
7. grid@master1:~$ cd /data/zk/
   
8. grid@master1:/data/zk$ ls
   
9. myid version-2
   
10. grid@master1:/data/zk$ ls -l
    
11. total 8
    
12. -rw-r--r-- 1 grid hadoop 3 Sep 8 20:33 myid
    
13. drwxr-xr-x 2 grid hadoop 4096 Sep 11 23:10 version-2
    
14. grid@master1:/data/zk$ rm -rf version-2/
    
15. grid@master1:/data/zk$ cd ~
    
16. grid@master1:~$
    
17. 
    
18. grid@master1:~$ zkServer.sh start
    
19. JMX enabled by default
    
20. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
    
21. Starting zookeeper ... STARTED
    
22. grid@master1:~$ zkServer.sh status
    
23. JMX enabled by default
    
24. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
    
25. Mode: follower
    
26. grid@master1:~$ start-hbase.sh
    
27. starting master, logging to /usr/local/hbase/logs/hbase-grid-master-master1.out
    
28. slave1: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave1.out
    
29. slave2: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave2.out
    
30. slave3: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave3.out
    
31. grid@master1:~$ jps
    
32. 7544 Jps
    
33. 1473 SecondaryNameNode
    
34. 6257 QuorumPeerMain
    
35. 1556 JobTracker
    
36. 1283 NameNode
    
37. 6463 HMaster

7、再次使用hbck修复META元数据表

1. grid@master1:~$ hbase hbck -fixMeta
   
2. .....................................
   
3. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
   
4. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
   
5. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
   
6. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
   
7. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
   
8. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
   
9. 15/09/11 16:36:03 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
   
10. ......................................

这次修复貌似正常了，但是日志中提示remove了一些在cache中的文件

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/12219480/viewspace-1797852/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/12219480/viewspace-1797852/