找不到元数据文件导致kudu启动失败
报错信息
Bad status: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Could not load tablet metadata from /var/lib/kudu/master/tablet-meta/00000000000000000000000000000000: /var/lib/kudu/master/tablet-meta/00000000000000000000000000000000: No such file or directory (error 2)
分析
这个错误是因为,kudu master 启动时发现元数据目录下空空如也,没有文件,所以拒绝启动。这种情况可能是上次启动时因为时间没同步,导致启动失败。所以影响本次的启动。
解决
将报错对应的wal目录和data目录清空能恢复。
参考
https://issues.apache.org/jira/browse/KUDU-1186
kudu会不会自动均衡tablet?
kudu 集群增加节点时,tablet不会自动均衡到新节点上,需要人工运维。集群下线节点时,该节点的tablet会自动在其他节点恢复(有副本的情况下)
kudu rebalance命令报错
terminate called after throwing an instance of ‘std::regex_error‘
what(): regex_error
*** Aborted at 1558779043 (unix time) try "date -d @1558779043" if you are using GNU date ***
PC: @ 0x7ff0d6cf9207 __GI_raise
*** SIGABRT (@0x3d600005a72) received by PID 23154 (TID 0x7ff0d934ba00) from PID 23154; stack trace: ***
@ 0x7ff0d8d065d0 (unknown)
@ 0x7ff0d6cf9207 __GI_raise
@ 0x7ff0d6cfa8f8 __GI_abort
@ 0x7ff0d76087d5 __gnu_cxx::__verbose_terminate_handler()
@ 0x7ff0d7606746 (unknown)
@ 0x7ff0d7606773 std::terminate()
@ 0x7ff0d7606993 __cxa_throw
@ 0x7ff0d765bdd5 std::__throw_regex_error()
@ 0x931c32 std::__detail::_Compiler<>::_M_bracket_expression()
@ 0x931e3a std::__detail::_Compiler<>::_M_atom()
@ 0x932469 std::__detail::_Compiler<>::_M_alternative()
@ 0x9324c4 std::__detail::_Compiler<>::_M_alternative()
@ 0x932649 std::__detail::_Compiler<>::_M_disjunction()
@ 0x93297b std::__detail::_Compiler<>::_Compiler()
@ 0x932cb7 std::__detail::__compile<>()
@ 0x92bfc6 (unknown)
@ 0x92c664 std::_Function_handler<>::_M_invoke()
@ 0xde6672 kudu::tools::Action::Run()
@ 0x9957d7 kudu::tools::DispatchCommand()
@ 0x99619b kudu::tools::RunTool()
@ 0x8dee4d main
@ 0x7ff0d6ce53d5 __libc_start_main
@ 0x9284b5 (unknown)
Aborted
这个问题网上有人说把gcc升级到4.9,重新编译一下kudu能解决,我试了一下并没用。后来去kudu的官方社区搜到了这个问题。这是一个kudu cdh5版本中存在的缺陷。不过可以通过给rebalance命令指定 --move_single_replicas参数来避免报错。这个错误在cdh6中修复。详情参考 https://community.cloudera.com/t5/Support-Questions/Kudu-rebalance-crash/m-p/88462#M5486