hbase报错： org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block

2020-01-19 16:37:05,531 INFO  [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 from any node: java.io.IOException: No live nodes contain block BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
2020-01-19 16:37:05,531 WARN  [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 11284.466617084763 msec.
2020-01-19 16:37:16,816 WARN  [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2020-01-19 16:37:16,817 WARN  [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2020-01-19 16:37:16,817 WARN  [mg011.tigard.com,6000,1579413911172_ChoreService_1] hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:991)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:638)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:879)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:932)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:524)
    at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:515)
    at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:491)
    at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:174)
    at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1428)
    at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:49)
    at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

hbase报错： org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-266398130-10.186.59.129-1574389974472:blk_1082979077_9238392 file=/hbase/data/migu/download_log20200112/.tabledesc/.tableinfo.0000000001

最后查看原因是hadoop集群的几个datanode节点 宕机

解决：
重启datanode节点。

start|stop-dfs.sh 在master上使用,否则只会停掉该台机器的datanode(namenode)
start|stop-yarn.sh(会停掉ResourceManager与NodeManager)在配置ResourceManager的机器上使用,否则只会停止该台机器上的NodeManager

参考： https://www.cnblogs.com/tele-share/p/9508225.html