hbase问题-hbase表list存在，ui页面存在，但是无法disable,drop,create

问题描述：

有个问题需要您那帮忙给处理下，hbase中有一个表cheat_channel_result，无法drop操作。麻烦给删掉我们在重新创建下。问题表在UI页面仍然存在

 

 

根据问题去排查：

1.hdfs中查看表已经不存在:

[huhe001@nlkfpt-hh204-hx07w ~]$ hadoop fs -ls /hbase_huhe001/data/default/cheat_channel_result ls: `/hbase_huhe001/data/default/cheat_channel_result': No such file or directory

 

2.hbase shell中尝试disable表：

disable cheat_channel_result

发现提示表不存在

is_enabled 'cheat_channel_result' ERROR: Unknown table cheat_channel_result!

 

3.尝试执行命令：

hbase hbck -repair 'default:cheat_channel_result'  hbase hbck -fixMeta -fixAssignments 'default:cheat_channel_result' hbase hbck -fixAssignments 'default:cheat_channel_result'

发现会提示没权限，所以首先先执行命令：

hadoop fs -setfacl -R -m user:huhe001:rwx hdfs://beh/hbase_huhe001/ hadoop fs -setfacl -R -m default:user:huhe001:rwx hdfs://beh/hbase_huhe001/ hadoop fs -setfacl -R -m group:huhe001:rwx hdfs://beh/hbase_huhe001/ hadoop fs -setfacl -R -m default:group:huhe001:rwx hdfs://beh/hbase_huhe001/

 

4.查看hbase的元数据表，需要在hbase用户下进入shell命令行查看

scan 'hbase:meta' , {LIMIT=>10,FILTER=>"PrefixFilter('cheat_channel_result')"}

发现元数据中不存在cheat_channel_result的信息

 

5.执行：

hbase hbck -fixMeta -fixAssignments 'default:cheat_channel_result'

然后尝试创建表

create 'default:cheat_channel_result','t1','f1'

执行会卡主很久，执行不了

 

问题最后解决思路：

1.cm页面查看日志信息

 

 

查看日志中报错提示：

Retriable error trying to truncate table=cheat_channel_result state=TRUNCATE_TABLE_CLEAR_FS_LAYOUT java.io.IOException: Received error when attempting to archive files ([class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://beh/hbase_huhe001/.tmp/data/default/cheat_channel_result/ef422cd13c0581a66f0fa85490f9bbf1/info, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://beh/hbase_huhe001/.tmp/data/default/cheat_channel_result/ef422cd13c0581a66f0fa85490f9bbf1/recovered.edits]), cannot delete region directory. at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:148) at org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteFromFs(DeleteTableProcedure.java:341) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:104) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:47) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495)

Failed to archive class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://beh/hbase_huhe001/.tmp/data/default/cheat_channel_result/ef422cd13c0581a66f0fa85490f9bbf1/info/de35ca2b6c40445a8880fa6a5f92a35e_SeqId_9_ on try #0 org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE, inode="/hbase_huhe001/.tmp/data/default/cheat_channel_result/ef422cd13c0581a66f0fa85490f9bbf1/info/de35ca2b6c40445a8880fa6a5f92a35e_SeqId_9_":huhe001:hbase:-rwxrwxrwx at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkAccessAcl(DefaultAuthorizationProvider.java:363) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:256) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:168) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3885) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3868) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:3839) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6799) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimesInt(FSNamesystem.java:2417) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:2395) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setTimes(NameNodeRpcServer.java:1135) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setTimes(AuthorizationProviderProxyClientProtocol.java:607) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:932) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272) at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.setTimes(DFSClient.java:3233) at org.apache.hadoop.hdfs.DistributedFileSystem$28.doCall(DistributedFileSystem.java:1490) at org.apache.hadoop.hdfs.DistributedFileSystem$28.doCall(DistributedFileSystem.java:1486) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.setTimes(DistributedFileSystem.java:1486) at org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:2089) at org.apache.hadoop.hbase.backup.HFileArchiver$File.moveAndClose(HFileArchiver.java:602) at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchiveFile(HFileArchiver.java:441) at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:351) at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:363) at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:300) at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:137) at org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteFromFs(DeleteTableProcedure.java:341) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:104) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:47) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1061) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:856) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:809) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:495) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbase, access=WRITE, inode="/hbase_huhe001/.tmp/data/default/cheat_channel_result/ef422cd13c0581a66f0fa85490f9bbf1/info/de35ca2b6c40445a8880fa6a5f92a35e_SeqId_9_":huhe001:hbase:-rwxrwxrwx at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkAccessAcl(DefaultAuthorizationProvider.java:363) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:256) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:168) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3885) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3868) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:3839) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6799) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimesInt(FSNamesystem.java:2417) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:2395) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setTimes(NameNodeRpcServer.java:1135) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setTimes(AuthorizationProviderProxyClientProtocol.java:607) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:932) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272) at org.apache.hadoop.ipc.Client.call(Client.java:1504) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231) at com.sun.proxy.$Proxy21.setTimes(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setTimes(ClientNamenodeProtocolTranslatorPB.java:866) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy22.setTimes(Unknown Source) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:283) at com.sun.proxy.$Proxy23.setTimes(Unknown Source) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:283) at com.sun.proxy.$Proxy23.setTimes(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.setTimes(DFSClient.java:3231) ... 21 more

根据报错提示我们知道表问题是发生在租户执行truncate 过程中出现的，同时沟通得到验证

 

 

问题解决操作：

1.根据提示，去对应的临时路径删除

hdfs dfs -rm -r hdfs://beh/hbase_huhe001/.tmp/data/default/cheat_channel_result

2.同时查看zk中信息：

删除锁表信息：

rmr /hbase_huhe001/table-lock/cheat_channel_result

 

3.然后在hbase用户下执行：

hbase hbck -fixMeta -fixAssignments

 

4.修复完成后，在huhe001下进入shell命令行确认并删除表

hbase shell hbase(main):002:0> is_enabled 'cheat_channel_result' false 0 row(s) in 0.0640 seconds hbase(main):003:0> drop 'cheat_channel_result' 0 row(s) in 1.2840 seconds hbase(main):004:0> list 'cheat_channel_result' TABLE 0 row(s) in 0.0210 seconds

 

5.查看hbase master UI页面，原来的cheat_channel_result表也没了，问题搞定