sqoop导入大量数据到hbase出现hregionServer坠机问题解决
版本信息:
zookeeper-3.4.10
hbase-1.2.6
hadoop-2.7.3
java 1.8
当用sqoop把mysql数据导出hbase是发现导入失败,查看进程发现hregionServer挂了,查看hbase-zkpk-regionserver-slave.log、hbase-zkpk-master-master.log和zk日志和网上资料发现原因主要大量数据的写入需要大量hbase资源,然而也就有长时间的FULL GC,GC时间过长与zookeeper失去连接,zk判断regionServer 死亡,regionServer已经存在(Region server exiting)等问题,最后java.lang.RuntimeException: HRegionServer Aborted
解决方案是jvm参数调优、增加堆栈空间、zk连接次数、zk等待时间等
具体如下:
hbase-site.xml:
<property>
<name>zookeeper.session.timeout</name>
<value>300000</value>
</property>
<property>
<name>hbase.zookeeper.property.tickTime</name>
<value>6000</value>
</property>
<property>
<name>hbase.hregion.memstroe.mslab.enable</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>10000</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>240000</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>280000</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>107374182400</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>100</value>
</property>
</property>
hbase-env.sh:
export HBASE_HEAPSIZE=16G
export HBASE_LOG_DIR=${HBASE_HOME}/logs
export HBASE_OPTS="-server -Xms1g -Xmx1g -XX:NewRatio=2 -verbose:gc -Xloggc:$HBASE_HOME/logs/hbasegc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$HBASE_HOME/logs"
zoo.cfg:
# The number of milliseconds of each
tickTime=6000
# increase this if you need to handle more clients
maxClientCnxns=10000
参考
[总结型] HBase随机宕机事件处理 & JVM GC回顾
Regionserver频繁挂掉故障处理实践
HBase参数配置及说明
sqoop导入hbase
HBase RegionServer挂掉问题分析