Hadoop集群环境下HBase集群slave节点总是掉问题

HBase表格设计：

create'DATA',{NAME=>'0',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'},{NAME=>'1',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'},{NAME=>'2',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'}

 

HBase：批量put

出错：

2017年 09月 14日星期四 09:12:23 CST Startingregionserver on slave1.hadoop

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 7897

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) 7897

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

2017-09-14 09:12:25,625INFO  [main] util.VersionInfo: HBase1.2.6

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Sourcecode repositoryfile:///home/busbey/projects/hbase/hbase-assembly/target/hbase-1.2.6revision=Unknown

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Compiledby busbey on Mon May 29 02:25:32 CDT 2017

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Fromsource with checksum 7e8ce83a648e252758e9dae1fbe779c9

2017-09-14 09:12:26,747INFO  [main] util.ServerCommandLine:env:JAVA_HOME=/usr/local/jdk1.7

2017-09-14 09:12:26,747INFO  [main] util.ServerCommandLine:env:SHLVL=3

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_LOG_DIR=/home/softs/hbase/bin/../logs

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_HOME=/home/softs/hbase/bin/..

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:KDEDIRS=/usr

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_OPTS=-XX:+UseConcMarkSweepGC  -XX:PermSize=128m -XX:MaxPermSize=128m-Dhbase.log.dir=/home/softs/hbase/bin/../logs-Dhbase.log.file=hbase-root-regionserver-slave1.hadoop.log-Dhbase.home.dir=/home/softs/hbase/bin/.. -Dhbase.id.str=root-Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_ENV_INIT=true

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:SSH_CLIENT=10.9.148.193 51362 22

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:MAIL=/var/mail/root

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:SELINUX_ROLE_REQUESTED=

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_ZNODE_FILE=/tmp/hbase-root-regionserver.znode

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:PWD=/home/softs/hbase

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_MASTER_OPTS= -XX:PermSize=128m -XX:MaxPermSize=128m

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:LOGNAME=root

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:CVS_RSH=ssh

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_MANAGES_ZK=false

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:G_BROKEN_FILENAMES=1

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:QTDIR=/usr/lib64/qt-3.3

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_NICENESS=0

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_REST_OPTS=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:MALLOC_ARENA_MAX=4

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SSH_CONNECTION=10.9.148.193 51362 10.9.148.190 22

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SELINUX_USE_CURRENT_RANGE=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SHELL=/bin/bash

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine: env:SELINUX_LEVEL_REQUESTED=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_REGIONSERVER_OPTS= -XX:PermSize=128m -XX:MaxPermSize=128m

2017-09-14 09:12:26,750INFO  [main] util.ServerCommandLine:env:HBASE_ROOT_LOGGER=INFO,RFA

2017-09-14 09:12:26,752INFO  [main] util.ServerCommandLine:env:CLASSPATH=/home/softs/hbase/conf:/usr/local/jdk1.7/lib/tools.jar:/home/softs/hbase/bin/..:/home/softs/hbase/bin/../lib/activation-1.1.jar:/home/softs/hbase/bin/../lib/aopalliance-1.0.jar:/home/softs/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/asm-3.1.jar:/home/softs/hbase/bin/../lib/avro-1.7.4.jar:/home/softs/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/softs/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/softs/hbase/bin/../lib/commons-cli-1.2.jar:/home/softs/hbase/bin/../lib/commons-codec-1.9.jar:/home/softs/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/softs/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/softs/hbase/bin/../lib/commons-configuration-1.6.jar:/home/softs/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/softs/hbase/bin/../lib/commons-digester-1.8.jar:/home/softs/hbase/bin/../lib/commons-el-1.0.jar:/home/softs/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/softs/hbase/bin/../lib/commons-io-2.4.jar:/home/softs/hbase/bin/../lib/commons-lang-2.6.jar:/home/softs/hbase/bin/../lib/commons-logging-1.2.jar:/home/softs/hbase/bin/../lib/commons-math-2.2.jar:/home/softs/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/softs/hbase/bin/../lib/commons-net-3.1.jar:/home/softs/hbase/bin/../lib/disruptor-3.3.0.jar:/home/softs/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/softs/hbase/bin/../lib/guava-12.0.1.jar:/home/softs/hbase/bin/../lib/guice-3.0.jar:/home/softs/hbase/bin/../lib/guice-servlet-3.0.jar:/home/softs/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-client-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-examples-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-external-blockcache-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-prefix-tree-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-procedure-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-protocol-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-resource-bundle-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-rest-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-shell-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-thrift-1.2.6.jar:/home/softs/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/softs/hbase/bin/../lib/httpclient-4.2.5.jar:/home/softs/hbase/bin/../lib/httpcore-4.4.1.jar:/home/softs/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/softs/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/softs/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/softs/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/softs/hbase/bin/../lib/javax.inject-1.jar:/home/softs/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/softs/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/softs/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/softs/hbase/bin/../lib/jcodings-1.0.8.jar:/home/softs/hbase/bin/../lib/jersey-client-1.9.jar:/home/softs/hbase/bin/../lib/jersey-core-1.9.jar:/home/softs/hbase/bin/../lib/jersey-guice-1.9.jar:/home/softs/hbase/bin/../lib/jersey-json-1.9.jar:/home/softs/hbase/bin/../lib/jersey-server-1.9.jar:/home/softs/hbase/bin/../lib/jets3t-0.9.0.jar:/home/softs/hbase/bin/../lib/jettison-1.3.3.jar:/home/softs/hbase/bin/../lib/jetty-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/softs/hbase/bin/../lib/joni-2.1.2.jar:/home/softs/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/softs/hbase/bin/../lib/jsch-0.1.42.jar:/home/softs/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/junit-4.12.jar:/home/softs/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/softs/hbase/bin/../lib/libthrift-0.9.3.jar:/home/softs/hbase/bin/../lib/log4j-1.2.17.jar:/home/softs/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/softs/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/home/softs/hbase/bin/../lib/paranamer-2.3.jar:/home/softs/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5.jar:/home/softs/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/softs/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/softs/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/softs/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/softs/hbase/bin/../lib/xmlenc-0.52.jar:/home/softs/hbase/bin/../lib/xz-1.0.jar:/home/softs/hbase/bin/../lib/zookeeper-3.4.6.jar:

2017-09-14 09:12:26,762INFO  [main] util.ServerCommandLine:env:HBASE_THRIFT_OPTS=

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:QTINC=/usr/lib64/qt-3.3/include

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:USER=root

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:QTLIB=/usr/lib64/qt-3.3/lib

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:KDE_IS_PRELINKED=1

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_SECURITY_LOGGER=INFO,RFAS

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HOME=/root

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:LESSOPEN=||/usr/bin/lesspipe.sh %s

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_START_FILE=/tmp/hbase-root-regionserver.autorestart

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_LOGFILE=hbase-root-regionserver-slave1.hadoop.log

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:LANG=zh_CN.UTF-8

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_LOG_PREFIX=hbase-root-regionserver-slave1.hadoop

2017-09-14 09:12:26,764INFO  [main] util.ServerCommandLine:env:HBASE_IDENT_STRING=root

2017-09-14 09:12:26,764INFO  [main] util.ServerCommandLine:vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation,vmVersion=24.45-b08

2017-09-14 09:12:26,765INFO  [main] util.ServerCommandLine:vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p,-XX:+UseConcMarkSweepGC, -XX:PermSize=128m, -XX:MaxPermSize=128m, -Dhbase.log.dir=/home/softs/hbase/bin/../logs,-Dhbase.log.file=hbase-root-regionserver-slave1.hadoop.log,-Dhbase.home.dir=/home/softs/hbase/bin/.., -Dhbase.id.str=root,-Dhbase.root.logger=INFO,RFA, -Dhbase.security.logger=INFO,RFAS]

2017-09-14 09:12:26,969WARN  [main] util.NativeCodeLoader:Unable to load native-hadoop library for your platform... using builtin-javaclasses where applicable

2017-09-14 09:12:27,474INFO  [main] regionserver.RSRpcServices:regionserver/slave1.hadoop/10.9.148.190:16020 server-side HConnectionretries=350

2017-09-14 09:12:27,790INFO  [main] ipc.SimpleRpcScheduler:Using deadline as user call queue, count=3

2017-09-14 09:12:27,810INFO  [main] ipc.RpcServer:regionserver/slave1.hadoop/10.9.148.190:16020: started 10 reader(s) listeningon port=16020

2017-09-14 09:12:29,678INFO  [main] fs.HFileSystem: Addedintercepting call to namenode#getBlockLocations so can do block reorderingusing class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks

2017-09-14 09:12:29,988INFO  [main]zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16020connecting to ZooKeeperensemble=master:2181,slave1:2181,slave2:2181,slave3:2181

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:host.name=slave1.hadoop

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.version=1.7.0_45

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.vendor=Oracle Corporation

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.home=/usr/local/jdk1.7/jre

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.class.path=/home/softs/hbase/conf:/usr/local/jdk1.7/lib/tools.jar:/home/softs/hbase/bin/..:/home/softs/hbase/bin/../lib/activation-1.1.jar:/home/softs/hbase/bin/../lib/aopalliance-1.0.jar:/home/softs/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/asm-3.1.jar:/home/softs/hbase/bin/../lib/avro-1.7.4.jar:/home/softs/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/softs/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/softs/hbase/bin/../lib/commons-cli-1.2.jar:/home/softs/hbase/bin/../lib/commons-codec-1.9.jar:/home/softs/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/softs/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/softs/hbase/bin/../lib/commons-configuration-1.6.jar:/home/softs/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/softs/hbase/bin/../lib/commons-digester-1.8.jar:/home/softs/hbase/bin/../lib/commons-el-1.0.jar:/home/softs/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/softs/hbase/bin/../lib/commons-io-2.4.jar:/home/softs/hbase/bin/../lib/commons-lang-2.6.jar:/home/softs/hbase/bin/../lib/commons-logging-1.2.jar:/home/softs/hbase/bin/../lib/commons-math-2.2.jar:/home/softs/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/softs/hbase/bin/../lib/commons-net-3.1.jar:/home/softs/hbase/bin/../lib/disruptor-3.3.0.jar:/home/softs/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/softs/hbase/bin/../lib/guava-12.0.1.jar:/home/softs/hbase/bin/../lib/guice-3.0.jar:/home/softs/hbase/bin/../lib/guice-servlet-3.0.jar:/home/softs/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-client-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-examples-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-external-blockcache-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-prefix-tree-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-procedure-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-protocol-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-resource-bundle-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-rest-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-shell-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-thrift-1.2.6.jar:/home/softs/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/softs/hbase/bin/../lib/httpclient-4.2.5.jar:/home/softs/hbase/bin/../lib/httpcore-4.4.1.jar:/home/softs/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/softs/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/softs/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/softs/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/softs/hbase/bin/../lib/javax.inject-1.jar:/home/softs/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/softs/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/softs/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/softs/hbase/bin/../lib/jcodings-1.0.8.jar:/home/softs/hbase/bin/../lib/jersey-client-1.9.jar:/home/softs/hbase/bin/../lib/jersey-core-1.9.jar:/home/softs/hbase/bin/../lib/jersey-guice-1.9.jar:/home/softs/hbase/bin/../lib/jersey-json-1.9.jar:/home/softs/hbase/bin/../lib/jersey-server-1.9.jar:/home/softs/hbase/bin/../lib/jets3t-0.9.0.jar:/home/softs/hbase/bin/../lib/jettison-1.3.3.jar:/home/softs/hbase/bin/../lib/jetty-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/softs/hbase/bin/../lib/joni-2.1.2.jar:/home/softs/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/softs/hbase/bin/../lib/jsch-0.1.42.jar:/home/softs/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/junit-4.12.jar:/home/softs/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/softs/hbase/bin/../lib/libthrift-0.9.3.jar:/home/softs/hbase/bin/../lib/log4j-1.2.17.jar:/home/softs/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/softs/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/home/softs/hbase/bin/../lib/paranamer-2.3.jar:/home/softs/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5.jar:/home/softs/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/softs/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/softs/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/softs/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/softs/hbase/bin/../lib/xmlenc-0.52.jar:/home/softs/hbase/bin/../lib/xz-1.0.jar:/home/softs/hbase/bin/../lib/zookeeper-3.4.6.jar:

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.io.tmpdir=/tmp

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.compiler=<NA>

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.name=Linux

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.arch=amd64

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.version=2.6.32-696.el6.x86_64

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.name=root

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.home=/root

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.dir=/home/softs/hbase

2017-09-14 09:12:30,008INFO  [main] zookeeper.ZooKeeper:Initiating client connection,connectString=master:2181,slave1:2181,slave2:2181,slave3:2181sessionTimeout=90000 watcher=regionserver:160200x0,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase

2017-09-14 09:12:30,075INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Openingsocket connection to server slave1.hadoop/10.9.148.190:2181. Will not attemptto authenticate using SASL (unknown error)

2017-09-14 09:12:30,080INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Socketconnection established to slave1.hadoop/10.9.148.190:2181, initiating session

2017-09-14 09:12:30,125INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Sessionestablishment complete on server slave1.hadoop/10.9.148.190:2181, sessionid =0x15e7df074580001, negotiated timeout = 40000

2017-09-14 09:12:30,153INFO  [RpcServer.responder] ipc.RpcServer:RpcServer.responder: starting

2017-09-14 09:12:30,155INFO  [RpcServer.listener,port=16020]ipc.RpcServer: RpcServer.listener,port=16020: starting

2017-09-14 09:12:30,353INFO  [main] mortbay.log: Logging toorg.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2017-09-14 09:12:30,358INFO  [main] http.HttpRequestLog: Httprequest log for http.requests.regionserver is not defined

2017-09-14 09:12:30,376INFO  [main] http.HttpServer: Addedglobal filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)

2017-09-14 09:12:30,377INFO  [main] http.HttpServer: Addedglobal filter 'clickjackingprevention'(class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context regionserver

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context logs

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context static

2017-09-14 09:12:30,400INFO  [main] http.HttpServer: Jetty boundto port 16030

2017-09-14 09:12:30,400INFO  [main] mortbay.log: jetty-6.1.26

2017-09-14 09:12:31,015INFO  [main] mortbay.log: StartedSelectChannelConnector@0.0.0.0:16030

2017-09-14 09:12:31,195INFO [regionserver/slave1.hadoop/10.9.148.190:16020]zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x19a4fdf0connecting to ZooKeeperensemble=master:2181,slave1:2181,slave2:2181,slave3:2181

2017-09-14 09:12:31,203INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]zookeeper.ZooKeeper: Initiating client connection,connectString=master:2181,slave1:2181,slave2:2181,slave3:2181sessionTimeout=90000 watcher=hconnection-0x19a4fdf00x0,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase

2017-09-14 09:12:31,210INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Opening socket connection to serverslave2.hadoop/10.9.148.191:2181. Will not attempt to authenticate using SASL(unknown error)

2017-09-14 09:12:31,215INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Socket connection established toslave2.hadoop/10.9.148.191:2181, initiating session

2017-09-14 09:12:31,272INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Session establishment complete on serverslave2.hadoop/10.9.148.191:2181, sessionid = 0x25e7df015ce0002, negotiatedtimeout = 40000

2017-09-14 09:12:31,555INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ClusterId : 51cb0d44-3b86-4bfd-b195-b19ad9712430

2017-09-14 09:12:31,699INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.MemStoreFlusher: globalMemStoreLimit=194.2 M,globalMemStoreLimitLowMark=184.4 M, maxHeap=485.4 M

2017-09-14 09:12:31,728INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: CompactionChecker runs every 10sec

2017-09-14 09:12:31,839INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled

2017-09-14 09:12:31,839INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled

2017-09-14 09:12:31,854INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: reportForDuty tomaster=master.hadoop,16000,1505351542568 with port=16020,startcode=1505351548182

2017-09-14 09:12:32,153INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hfile.CacheConfig:Allocating LruBlockCache size=194.15 MB, blockSize=64 KB

2017-09-14 09:12:32,175INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hfile.CacheConfig:Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=210048,freeSize=203370992, maxSize=203581040, heapSize=210048, minSize=193401984,minFactor=0.95, multiSize=96700992, multiFactor=0.5, singleSize=48350496,singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false,cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 09:12:32,377INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.WALFactory:Instantiating WALProvider of type classorg.apache.hadoop.hbase.wal.DefaultWALProvider

2017-09-14 09:12:32,470INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: WALconfiguration: blocksize=128 MB, rollsize=121.60 MB,prefix=slave1.hadoop%2C16020%2C1505351548182.default, suffix=,logDir=hdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182,archiveDir=hdfs://master:9000/hbase/oldWALs

2017-09-14 09:12:32,790INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: Slow synccost: 267 ms, current pipeline: []

2017-09-14 09:12:32,792INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: New WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470

2017-09-14 09:12:32,846INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metricsevery 5000 milliseconds

2017-09-14 09:12:32,906INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.ReplicationSourceManager: Current list of replicators:[slave1.hadoop,16020,1504077821200, slave2.hadoop,16020,1505351523724,slave1.hadoop,16020,1505351548182, slave3.hadoop,16020,1505351532331] otherRSs: [slave2.hadoop,16020,1505351523724, slave1.hadoop,16020,1505351548182,slave3.hadoop,16020,1505351532331]

2017-09-14 09:12:33,058INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HeapMemoryManager: Starting HeapMemoryTuner chore.

2017-09-14 09:12:33,067INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker slave1.hadoop,16020,1505351548182starting

2017-09-14 09:12:33,079INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Serving as slave1.hadoop,16020,1505351548182,RpcServer on slave1.hadoop/10.9.148.190:16020, sessionid=0x15e7df074580001

2017-09-14 09:12:33,105INFO [regionserver/slave1.hadoop/10.9.148.190:16020]quotas.RegionServerQuotaManager: Quota support disabled

2017-09-14 09:13:20,702INFO  [ReplicationExecutor-0]replication.ReplicationQueuesZKImpl: Atomically movingslave1.hadoop,16020,1504077821200's WALs to my queue

2017-09-14 09:17:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0

2017-09-14 09:22:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0

2017-09-14 09:27:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0

2017-09-14 09:32:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0

2017-09-14 09:37:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0

2017-09-14 09:42:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0

2017-09-14 09:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0

2017-09-14 09:52:32,173 INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0

2017-09-14 09:57:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0

2017-09-14 10:02:06,073INFO [PriorityRpcServer.handler=1,queue=1,port=16020]regionserver.RSRpcServices: OpenCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:02:06,352INFO  [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]hfile.CacheConfig: Created cacheConfig for 0:blockCache=LruBlockCache{blockCount=0, currentSize=210048, freeSize=203370992,maxSize=203581040, heapSize=210048, minSize=193401984, minFactor=0.95,multiSize=96700992, multiFactor=0.5, singleSize=48350496, singleFactor=0.25},cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false,cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,prefetchOnOpen=false

2017-09-14 10:02:06,361INFO  [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,388INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=0,currentSize=210048, freeSize=203370992, maxSize=203581040, heapSize=210048,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 10:02:06,389INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,405INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1] hfile.CacheConfig:Created cacheConfig for 2: blockCache=LruBlockCache{blockCount=0,currentSize=210048, freeSize=203370992, maxSize=203581040, heapSize=210048,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false,cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 10:02:06,406INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,474INFO  [RS_OPEN_REGION-slave1:16020-0]regionserver.HRegion: Onlined 3f1b89971cc74cb7bfab4e4b0a3bf2c9; nextsequenceid=2

2017-09-14 10:02:06,509INFO [PostOpenDeployTasks:3f1b89971cc74cb7bfab4e4b0a3bf2c9]regionserver.HRegionServer: Post open deploy tasks for CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:02:06,636INFO [PostOpenDeployTasks:3f1b89971cc74cb7bfab4e4b0a3bf2c9]hbase.MetaTableAccessor: Updated rowCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. with server=slave1.hadoop,16020,1505351548182

2017-09-14 10:02:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0

2017-09-14 10:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0

2017-09-14 10:12:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0

2017-09-14 10:12:32,995INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470with entries=1, filesize=359 B; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920

2017-09-14 10:12:32,998INFO  [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]wal.FSHLog: Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470

2017-09-14 10:17:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0

2017-09-14 10:22:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0

2017-09-14 10:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0

2017-09-14 10:32:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0

2017-09-14 10:37:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0

2017-09-14 10:42:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0

2017-09-14 10:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0

2017-09-14 10:49:36,562INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=124.17 MB

2017-09-14 10:49:42,835INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=85700, memsize=124.2 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/d4ed3074dde745d882ea3f6d896f4dd7

2017-09-14 10:49:43,108INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/d4ed3074dde745d882ea3f6d896f4dd7,entries=494658, sequenceid=85700, filesize=63.9 M

2017-09-14 10:49:43,111INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~124.17 MB/130198992,currentsize=4.25 MB/4459608 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.in 6549ms, sequenceid=85700, compaction requested=false

2017-09-14 10:52:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=27, hits=0, hitRatio=0, cachingAccesses=27,cachingHits=0, cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0

2017-09-14 10:53:05,853INFO [regionserver/slave1.hadoop/10.9.148.190:16020.leaseChecker]regionserver.RSRpcServices: Scanner 5 lease expired on regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:54:48,951INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920with entries=149352, filesize=121.61 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887

2017-09-14 10:56:49,722INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=119.05 MB

2017-09-14 10:56:55,601INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=169579, memsize=119.1 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/f0a1cfa73bd5407fa13fea322063b3a9

2017-09-14 10:56:55,672INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/f0a1cfa73bd5407fa13fea322063b3a9,entries=477138, sequenceid=169579, filesize=61.3 M

2017-09-14 10:56:55,675INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~119.05 MB/124833880,currentsize=9.36 MB/9814568 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 5954ms,sequenceid=169579, compaction requested=false

2017-09-14 10:57:22,752INFO  [sync.3] wal.FSHLog: Slow synccost: 480 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 10:57:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=27, hits=0, hitRatio=0, cachingAccesses=27,cachingHits=0, cachingHitsRatio=0,evictions=629, evicted=0, evictedPerRun=0.0

2017-09-14 11:02:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=462.47 KB, freeSize=193.70 MB, max=194.15 MB,blockCount=2, accesses=33, hits=4, hitRatio=12.12%, , cachingAccesses=33,cachingHits=4, cachingHitsRatio=12.12%, evictions=659, evicted=0,evictedPerRun=0.0

2017-09-14 11:03:21,439INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB; 0=114.39MB

2017-09-14 11:03:27,045INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=250053, memsize=114.4 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/3906d45c77aa4d0392cfabe93def9f59

2017-09-14 11:03:27,112INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/3906d45c77aa4d0392cfabe93def9f59,entries=459096, sequenceid=250053, filesize=58.8 M

2017-09-14 11:03:27,114INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~114.39 MB/119943872,currentsize=13.92 MB/14598040 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 5675ms, sequenceid=250053,compaction requested=true

2017-09-14 11:03:27,126INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:03:27,127INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. into tmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp,totalSize=183.9 M

2017-09-14 11:03:45,559INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 (all) file(s) in 0 ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. into59850935db0840eabce90c676f6701b9(size=183.8 M), total size for store is 183.8M. This selection was in queue for 0sec, and took 18sec to execute.

2017-09-14 11:03:45,632INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.,storeName=0, fileCount=3, fileSize=183.9 M, priority=7, time=517255043298492;duration=18sec

2017-09-14 11:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=462.52 KB, freeSize=193.70 MB, max=194.15 MB,blockCount=2, accesses=2957, hits=4, hitRatio=0.14%, , cachingAccesses=35,cachingHits=4, cachingHitsRatio=11.43%, evictions=689, evicted=2,evictedPerRun=0.0029027576092630625

2017-09-14 11:07:53,465INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887with entries=150665, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399

2017-09-14 11:10:00,083 INFO  [MemStoreFlusher.0] regionserver.HRegion:Flushing 2/3 column families, memstore=128.00 MB; 2=18.66 MB; 0=109.34 MB

2017-09-14 11:10:05,464INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=326980, memsize=109.3 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/ac9ee478e810460494cfef143290c341

2017-09-14 11:10:06,214INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=326980, memsize=18.7 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/3a98a0c896774fec8719b2d2b1df2115

2017-09-14 11:10:06,272INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/ac9ee478e810460494cfef143290c341,entries=435900, sequenceid=326980, filesize=56.6 M

2017-09-14 11:10:06,309INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/3a98a0c896774fec8719b2d2b1df2115,entries=94980, sequenceid=326980, filesize=7.1 M

2017-09-14 11:10:06,319INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218448,currentsize=479.64 KB/491152 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 6235ms,sequenceid=326980, compaction requested=false

2017-09-14 11:12:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=497.18 KB, freeSize=193.66 MB, max=194.15 MB,blockCount=3, accesses=6952, hits=2933, hitRatio=42.19%, ,cachingAccesses=3516, cachingHits=2933, cachingHitsRatio=83.42%, evictions=719,evicted=2, evictedPerRun=0.0027816412039101124

2017-09-14 11:17:32,174 INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=8622, hits=2942, hitRatio=34.12%, ,cachingAccesses=5186, cachingHits=2942, cachingHitsRatio=56.73%, evictions=749,evicted=2, evictedPerRun=0.0026702270843088627

2017-09-14 11:17:50,905INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=122.52 MB

2017-09-14 11:17:57,353INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=409609, memsize=122.5 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/35e19cbf6a47452b9d485b684db1df29

2017-09-14 11:17:57,408INFO  [MemStoreFlusher.0] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/35e19cbf6a47452b9d485b684db1df29,entries=468186, sequenceid=409609, filesize=65.8 M

2017-09-14 11:17:57,410INFO  [MemStoreFlusher.0] regionserver.HRegion:Finished memstore flush of ~122.52 MB/128470400, currentsize=5.87 MB/6157376for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 6505ms,sequenceid=409609, compaction requested=true

2017-09-14 11:17:57,519INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitTransaction: Starting split of regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,195INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Running close preflush ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,196INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Flushing 3/3 column families, memstore=6.20 MB

2017-09-14 11:17:58,383INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410060, memsize=692.0 K,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/0ff7b107866e4677bd84e3b2b61dcd01

2017-09-14 11:17:58,645INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410060, memsize=5.5 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/25ccf0ebf5554b18bbbb9a5935dbdabf

2017-09-14 11:17:58,686INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/0ff7b107866e4677bd84e3b2b61dcd01,entries=2496, sequenceid=410060, filesize=387.1 K

2017-09-14 11:17:58,711INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/25ccf0ebf5554b18bbbb9a5935dbdabf,entries=27750, sequenceid=410060, filesize=2.2 M

2017-09-14 11:17:58,713INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Finished memstore flush of ~6.20 MB/6497168,currentsize=81.49 KB/83448 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.in 518ms, sequenceid=410060, compaction requested=true

2017-09-14 11:17:58,713INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Flushing 3/3 column families, memstore=81.49 KB

2017-09-14 11:17:58,770INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410115, memsize=79.1 K,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/a88c6b8fcf734c22bac0e8b57e9e433d

2017-09-14 11:17:58,828INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410115, memsize=2.4 K,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/21c20a3e4c324e5da987a6aba80d71d0

2017-09-14 11:17:58,868INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/a88c6b8fcf734c22bac0e8b57e9e433d,entries=294, sequenceid=410115, filesize=47.5 K

2017-09-14 11:17:58,898INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/21c20a3e4c324e5da987a6aba80d71d0,entries=12, sequenceid=410115, filesize=5.7 K

2017-09-14 11:17:58,900INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Finished memstore flush of ~81.49 KB/83448, currentsize=0B/0 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in187ms, sequenceid=410115, compaction requested=true

2017-09-14 11:17:58,906INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 0

2017-09-14 11:17:58,909INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 1

2017-09-14 11:17:58,911INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 2

2017-09-14 11:17:58,964INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: ClosedCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,965INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitTransaction: Preparing to split 8 storefiles for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.using 2 threads

2017-09-14 11:17:59,718INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1] hfile.CacheConfig:Created cacheConfig for 0: blockCache=LruBlockCache{blockCount=4,currentSize=640872, freeSize=202940168, maxSize=203581040, heapSize=640872,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,719INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:17:59,720INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 0: blockCache=LruBlockCache{blockCount=4,currentSize=640872, freeSize=202940168, maxSize=203581040, heapSize=640872,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,722INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:17:59,946INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,947INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttlepoint 2684354560; major period 604800000, major jitter 0.500000, min localityto compact 0.000000

2017-09-14 11:18:00,005INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:18:00,006INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,012INFO  [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]hfile.CacheConfig: Created cacheConfig for 2:blockCache=LruBlockCache{blockCount=5, currentSize=772752, freeSize=202808288,maxSize=203581040, heapSize=772752, minSize=193401984, minFactor=0.95,multiSize=96700992, multiFactor=0.5, singleSize=48350496, singleFactor=0.25},cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false,cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,prefetchOnOpen=false

2017-09-14 11:18:00,014INFO  [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,018INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 2: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:18:00,019INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,047INFO [slave1.hadoop,16020,1505351548182-daughterOpener=c5ddefa1371f4c7d731310cc082cdfb7]regionserver.HRegion: Onlined c5ddefa1371f4c7d731310cc082cdfb7; nextsequenceid=410117

2017-09-14 11:18:00,137INFO [slave1.hadoop,16020,1505351548182-daughterOpener=7726b6f586c6b46ad1334186a21be68b]regionserver.HRegion: Onlined 7726b6f586c6b46ad1334186a21be68b; nextsequenceid=410116

2017-09-14 11:18:00,139INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegionServer: Post open deploy tasks forCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:18:00,151INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]hbase.MetaTableAccessor: Updated rowCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.with server=slave1.hadoop,16020,1505351548182

2017-09-14 11:18:00,151INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegionServer: Post open deploy tasks forCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:00,154INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:18:00,154INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 5 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into tmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=306.7 M

2017-09-14 11:18:00,161INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]hbase.MetaTableAccessor: Updated row CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.with server=slave1.hadoop,16020,1505351548182

2017-09-14 11:18:00,163INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 0 in region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:00,164INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 5 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=306.7 M

2017-09-14 11:18:00,380INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitRequest: Region split, hbase:meta updated, and report tomaster. Parent=CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9., newregions: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7..Split took 2sec

2017-09-14 11:18:00,380INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitRequest: Split transaction journal:

         STARTED at 1505359077432

         PREPARED at 1505359077518

         BEFORE_PRE_SPLIT_HOOK at 1505359077519

         AFTER_PRE_SPLIT_HOOK at 1505359077520

         SET_SPLITTING at 1505359077530

         CREATE_SPLIT_DIR at 1505359078195

         CLOSED_PARENT_REGION at 1505359078965

         OFFLINED_PARENT at 1505359078965

         STARTED_REGION_A_CREATION at 1505359079332

         STARTED_REGION_B_CREATION at 1505359079430

         PONR at 1505359079518

         OPENED_REGION_A at 1505359080139

         OPENED_REGION_B at 1505359080139

         BEFORE_POST_SPLIT_HOOK at 1505359080334

         AFTER_POST_SPLIT_HOOK at 1505359080334

         COMPLETED at 1505359080334

2017-09-14 11:18:00,486WARN [B.defaultRpcServer.handler=0,queue=0,port=16020]regionserver.RSRpcServices: 7 encountered RegionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. is not online onslave1.hadoop,16020,1505351548182, closing ...

2017-09-14 11:18:33,335INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 5 (all) file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into42cb5c4b051d42c58de7f84feb6fdf45(size=151.2 M), total size for store is 151.2M. This selection was in queue for 0sec, and took 33sec to execute.

2017-09-14 11:18:33,337INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=5, fileSize=306.7 M, priority=5, time=518128079182940;duration=33sec

2017-09-14 11:18:33,365INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:33,366INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=9.3 M

2017-09-14 11:18:33,402INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 5 (all) file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into a81a9c60452b4c5497dd8055d0504a90(size=155.5 M), total size for store is155.5 M. This selection was in queue for 0sec, and took 33sec to execute.

2017-09-14 11:18:33,402INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=5, fileSize=306.7 M, priority=5, time=518128071001392; duration=33sec

2017-09-14 11:18:34,570INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into 3e87be46c78f40ddb8f0e1ce4c5ff4e9(size=9.3M), total size for store is 9.3 M. This selection was in queue for 0sec, andtook 1sec to execute.

2017-09-14 11:18:34,570INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=9.3 M, priority=7, time=518161282561103;duration=1sec

2017-09-14 11:20:36,895INFO  [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]wal.FSHLog: Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399with entries=139255, filesize=121.61 MB; new WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840

2017-09-14 11:20:36,899INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920

2017-09-14 11:20:36,912INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887

2017-09-14 11:22:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=883.21 KB, freeSize=193.29 MB, max=194.15 MB,blockCount=6, accesses=15629, hits=2973, hitRatio=19.02%, ,cachingAccesses=7171, cachingHits=2972, cachingHitsRatio=41.44%, evictions=779,evicted=3, evictedPerRun=0.00385109125636518

2017-09-14 11:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=892.95 KB, freeSize=193.28 MB, max=194.15 MB,blockCount=7, accesses=15860, hits=2975, hitRatio=18.76%, ,cachingAccesses=7402, cachingHits=2974, cachingHitsRatio=40.18%, evictions=809,evicted=3, evictedPerRun=0.0037082817871123552

2017-09-14 11:28:28,526INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.4 M, Region memstore size=96.4 M

2017-09-14 11:28:28,526INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=96.42 MB

2017-09-14 11:28:33,268INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher:Flushed, sequenceid=470830, memsize=89.3 M, hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/2e35213036664f35a5de111d46452a82

2017-09-14 11:28:33,701INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=470830, memsize=7.1 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/d31a5152e1e34e809607ec73f38c8abd

2017-09-14 11:28:33,768INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/2e35213036664f35a5de111d46452a82,entries=330288, sequenceid=470830, filesize=49.4 M

2017-09-14 11:28:33,819INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/d31a5152e1e34e809607ec73f38c8abd,entries=33966, sequenceid=470830, filesize=3.0 M

2017-09-14 11:28:33,822INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~96.42 MB/101105000,currentsize=220.79 KB/226088 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 5295ms,sequenceid=470830, compaction requested=false

2017-09-14 11:32:31,667INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840with entries=130189, filesize=121.62 MB; new WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596

2017-09-14 11:32:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1.03 MB, freeSize=193.12 MB, max=194.15 MB,blockCount=9, accesses=15864, hits=2977, hitRatio=18.77%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=839,evicted=3, evictedPerRun=0.0035756854340434074

2017-09-14 11:32:58,727INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=128.00 MB

2017-09-14 11:33:05,481INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=488903, memsize=128.0 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/cfb52cbc8dd54dbca1a143cca485d2ec

2017-09-14 11:33:05,533INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/cfb52cbc8dd54dbca1a143cca485d2ec,entries=472692, sequenceid=488903, filesize=70.8 M

2017-09-14 11:33:05,536INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218984,currentsize=245.87 KB/251768 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 6808ms, sequenceid=488903, compaction requested=false

2017-09-14 11:33:30,606INFO  [sync.2] wal.FSHLog: Slow synccost: 350 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010, 10.9.148.192:50010]

2017-09-14 11:37:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1.03 MB, freeSize=193.12 MB, max=194.15 MB,blockCount=9, accesses=15864, hits=2977, hitRatio=18.77%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=869,evicted=3, evictedPerRun=0.00345224398188293

2017-09-14 11:40:48,609INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heap pressure.Total Memstore size=184.4 M, Region memstore size=113.8 M

2017-09-14 11:40:48,610INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=113.80 MB

2017-09-14 11:40:54,396INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=541946, memsize=107.6 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/94682411f19e49019f75d8bb8f1e3984

2017-09-14 11:40:54,720INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=541946, memsize=6.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/bfeedf84f0e1402c82a28af10303e032

2017-09-14 11:40:54,774INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/94682411f19e49019f75d8bb8f1e3984,entries=397416, sequenceid=541946, filesize=59.5 M

2017-09-14 11:40:54,806INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/bfeedf84f0e1402c82a28af10303e032,entries=29256, sequenceid=541946, filesize=2.7 M

2017-09-14 11:40:54,808INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~113.80 MB/119329776,currentsize=275.40 KB/282008 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6198ms,sequenceid=541946, compaction requested=true

2017-09-14 11:40:54,817 INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:40:54,818INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=14.9 M

2017-09-14 11:40:56,495INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intoc03d152596014599bb0534b02b376bc9(size=15.0 M), total size for store is 15.0 M.This selection was in queue for 0sec, and took 1sec to execute.

2017-09-14 11:40:56,496INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=14.9 M, priority=7, time=519502733345301;duration=1sec

2017-09-14 11:42:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1021.62 KB, freeSize=193.15 MB, max=194.15 MB,blockCount=8, accesses=16100, hits=2977, hitRatio=18.49%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=899,evicted=4, evictedPerRun=0.004449388012290001

2017-09-14 11:43:31,904INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596with entries=128535, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833

2017-09-14 11:43:31,908INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399

2017-09-14 11:43:31,921INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840

2017-09-14 11:46:53,797INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=127.2 M

2017-09-14 11:46:53,798INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=127.24 MB

2017-09-14 11:46:57,445INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.7 M, Region memstore size=57.3 M

2017-09-14 11:46:57,447INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=57.32 MB

2017-09-14 11:47:03,191INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=566552, memsize=127.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/c050d9857d0b4a94ac4b092d13e871ec

2017-09-14 11:47:03,258INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=577750, memsize=53.9 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f6c5cdbf035f41f285829c0f222ae04f

2017-09-14 11:47:03,265INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/c050d9857d0b4a94ac4b092d13e871ec,entries=465870, sequenceid=566552, filesize=70.8 M

2017-09-14 11:47:03,267INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~127.24 MB/133416160,currentsize=195.73 KB/200424 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 9469ms, sequenceid=566552, compaction requested=true

2017-09-14 11:47:03,273INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:47:03,274INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=297.1 M

2017-09-14 11:47:03,583 INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=577750, memsize=3.4 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/cbf89107dec741fa88678491cd1b8d45

2017-09-14 11:47:03,666INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/f6c5cdbf035f41f285829c0f222ae04f,entries=199266, sequenceid=577750, filesize=29.8 M

2017-09-14 11:47:03,734INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/cbf89107dec741fa88678491cd1b8d45,entries=15528, sequenceid=577750, filesize=1.5 M

2017-09-14 11:47:03,754 INFO  [MemStoreFlusher.0] regionserver.HRegion:Finished memstore flush of ~57.32 MB/60106416, currentsize=112.47 KB/115168 forregion CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6307ms,sequenceid=577750, compaction requested=true

2017-09-14 11:47:03,802INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:47:03,802INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 4 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=289.8 M

2017-09-14 11:47:11,363INFO  [sync.2] wal.FSHLog: Slow synccost: 100 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1021.62 KB, freeSize=193.15 MB, max=194.15 MB,blockCount=8, accesses=21074, hits=2977, hitRatio=14.13%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=929,evicted=4, evictedPerRun=0.004305704962462187

2017-09-14 11:47:42,495INFO  [sync.4] wal.FSHLog: Slow synccost: 730 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:47:43,082INFO  [sync.1] wal.FSHLog: Slow synccost: 478 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:48:00,596INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 (all) file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into 7733f6a209b5440c8064d7ae0e78175f(size=297.0 M), total size for store is297.0 M. This selection was in queue for 0sec, and took 57sec to execute.

2017-09-14 11:48:00,597INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=3, fileSize=297.1 M, priority=7, time=519871190566306;duration=57sec

2017-09-14 11:48:00,796INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 4 (all) file(s) in 0 of CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.into afd20ea26d33422da8958863af014add(size=289.7 M), total size for store is289.7 M. This selection was in queue for 0sec, and took 56sec to execute.

2017-09-14 11:48:00,797INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=4, fileSize=289.8 M, priority=6, time=519871719170783;duration=56sec

2017-09-14 11:48:06,588INFO  [sync.1] wal.FSHLog: Slow synccost: 823 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:52:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25426, hits=2979, hitRatio=11.72%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=959,evicted=8, evictedPerRun=0.008342023007571697

2017-09-14 11:54:46,673INFO  [sync.3] wal.FSHLog: Slow synccost: 1861 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:55:40,288INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833with entries=129272, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218

2017-09-14 11:55:40,293INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596

2017-09-14 11:57:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25426, hits=2979, hitRatio=11.72%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=989,evicted=8, evictedPerRun=0.008088978938758373

2017-09-14 11:57:48,416INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=94.7 M

2017-09-14 11:57:48,416INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=94.68 MB

2017-09-14 11:57:53,120INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=637089, memsize=88.8 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/9ec71e99d8fd47ff88b0b2a6f091a779

2017-09-14 11:57:53,445INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=637089, memsize=5.9 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/e7b8472f595243dbbc217367710e9665

2017-09-14 11:57:53,488INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/9ec71e99d8fd47ff88b0b2a6f091a779,entries=329706, sequenceid=637089, filesize=48.9 M

2017-09-14 11:57:53,521INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/e7b8472f595243dbbc217367710e9665,entries=26298, sequenceid=637089, filesize=2.7 M

2017-09-14 11:57:53,524INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~94.68 MB/99279648,currentsize=198.28 KB/203040 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 5107ms, sequenceid=637089,compaction requested=true

2017-09-14 11:57:53,530INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:57:53,531INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=19.2 M

2017-09-14 11:57:55,737INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intobfb1adeac5444446a7f780c555234b58(size=19.2 M), total size for store is 19.2 M.This selection was in queue for 0sec, and took 2sec to execute.

2017-09-14 11:57:55,737INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=19.2 M, priority=7, time=520521447730712;duration=2sec

2017-09-14 12:02:04,880INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=128.00 MB

2017-09-14 12:02:11,614INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=644924, memsize=128.0 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/f974666048d74d739008b1cd889f49c0

2017-09-14 12:02:11,657INFO  [MemStoreFlusher.1] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/f974666048d74d739008b1cd889f49c0,entries=470202, sequenceid=644924, filesize=71.1 M

2017-09-14 12:02:11,658INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218592,currentsize=247.68 KB/253624 for regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 6778ms, sequenceid=644924, compaction requested=false

2017-09-14 12:02:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1019, evicted=8, evictedPerRun=0.007850834168493748

2017-09-14 12:07:15,856INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218with entries=129538, filesize=121.61 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362035790

2017-09-14 12:07:15,860INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833

2017-09-14 12:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1049, evicted=8, evictedPerRun=0.007626310922205448

2017-09-14 12:09:31,021INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.4 M, Region memstore size=111.6 M

2017-09-14 12:09:31,021INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=111.64 MB

2017-09-14 12:09:33,770INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.6 M, Region memstoresize=72.9 M

2017-09-14 12:09:33,771INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=72.91 MB

2017-09-14 12:09:39,622INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=706940, memsize=106.8 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/37f4c026cf2f40b0a344c7e3048babd3

2017-09-14 12:09:40,530INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=706940, memsize=4.8 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/4a18796ab17a44cab5886eece6f37f7d

2017-09-14 12:09:40,705INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/37f4c026cf2f40b0a344c7e3048babd3,entries=396984, sequenceid=706940, filesize=58.8 M

2017-09-14 12:09:40,810INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/4a18796ab17a44cab5886eece6f37f7d,entries=22092, sequenceid=706940, filesize=2.1 M

2017-09-14 12:09:40,828INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~111.64 MB/117063960,currentsize=171.20 KB/175304 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 9806ms,sequenceid=706940, compaction requested=true

2017-09-14 12:09:41,121INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=689755, memsize=72.9 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/73ad5e6ad61d42c89b2f09a9182fd46a

2017-09-14 12:09:41,171INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/73ad5e6ad61d42c89b2f09a9182fd46a,entries=268962, sequenceid=689755, filesize=40.3 M

2017-09-14 12:09:41,174INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~72.91 MB/76450064,currentsize=110.16 KB/112808 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 7402ms, sequenceid=689755, compaction requested=true

2017-09-14 12:10:06,563INFO  [sync.4] wal.FSHLog: Slow synccost: 174 ms, current pipeline: [10.9.148.190:50010, 10.9.148.192:50010,10.9.148.191:50010]

2017-09-14 12:12:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1079, evicted=8, evictedPerRun=0.007414272520691156

2017-09-14 12:17:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1109, evicted=8, evictedPerRun=0.007213706150650978

2017-09-14 12:18:24,734INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362035790with entries=129386, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362704656

2017-09-14 12:18:24,738INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218

2017-09-14 12:19:31,849INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=93.2 M

2017-09-14 12:19:31,850INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=93.23 MB

2017-09-14 12:19:36,804INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=747393, memsize=93.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/55cf50752f9f4615a2bc8355335fb9e3

2017-09-14 12:19:36,845INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/55cf50752f9f4615a2bc8355335fb9e3,entries=345804, sequenceid=747393, filesize=51.4 M

2017-09-14 12:19:36,848INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~93.23 MB/97763144,currentsize=166.49 KB/170488 for regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 4998ms, sequenceid=747393, compaction requested=true

2017-09-14 12:19:36,854INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 12:19:36,855INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=162.8 M

2017-09-14 12:19:52,353INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into 1204be22ee6c48a6b1d92025a09f822d(size=162.7 M), total size for store is459.7 M. This selection was in queue for 0sec, and took 15sec to execute.

2017-09-14 12:19:52,353INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=3, fileSize=162.8 M, priority=6, time=521824771615235;duration=15sec

2017-09-14 12:19:56,029INFO  [sync.4] wal.FSHLog: Slow synccost: 168 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:22:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=28313, hits=2979, hitRatio=10.52%, , cachingAccesses=7406,cachingHits=2976, cachingHitsRatio=40.18%, evictions=1139, evicted=8,evictedPerRun=0.007023705169558525

2017-09-14 12:24:04,205INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=122.86 MB

2017-09-14 12:24:10,663INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=786337, memsize=122.9 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/bdc237ad953f43508e96c4ed9747addc

2017-09-14 12:24:10,715INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/bdc237ad953f43508e96c4ed9747addc,entries=452208, sequenceid=786337, filesize=68.1 M

2017-09-14 12:24:10,717INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~122.86 MB/128828080,currentsize=5.29 MB/5549560 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6512ms,sequenceid=786337, compaction requested=true

2017-09-14 12:24:10,726INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 12:24:10,726INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=175.8 M

2017-09-14 12:24:27,231INFO  [sync.1] wal.FSHLog: Slow synccost: 143 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:24:28,437INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into 2e56407c59f64b1e96640f783dfaa18c(size=175.6M), total size for store is 465.3 M. This selection was in queue for 0sec, andtook 17sec to execute.

2017-09-14 12:24:28,437INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=3, fileSize=175.8 M, priority=6, time=522098643140405;duration=17sec

2017-09-14 12:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=31103, hits=2979, hitRatio=9.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1169, evicted=8, evictedPerRun=0.006843456067144871

2017-09-14 12:27:47,672INFO  [sync.1] wal.FSHLog: Slow synccost: 3569 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:30:05,873INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362704656with entries=129330, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805

2017-09-14 12:32:09,616INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=114.2 M

2017-09-14 12:32:09,616INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=114.24 MB

2017-09-14 12:32:11,680INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.due to global heap pressure. Total Memstore size=184.6 M, Region memstoresize=70.3 M

2017-09-14 12:32:11,680INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=70.28 MB

2017-09-14 12:32:25,470WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920

java.io.IOException: Badresponse ERROR for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920 from datanode10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,472WARN  [DataStreamer for file/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920] hdfs.DFSClient:Error Recovery for block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920in pipeline 10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: baddatanode 10.9.148.192:50010

2017-09-14 12:32:25,518WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922

java.io.IOException: Badresponse ERROR for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922from datanode 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,523WARN  [DataStreamer for file /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920]hdfs.DFSClient: DataStreamer Exception

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,523WARN  [sync.0] hdfs.DFSClient: Errorwhile syncing

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,524ERROR [sync.0] wal.FSHLog: Error syncing, request close of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,524INFO  [sync.0] wal.FSHLog: Slow synccost: 10542 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010]

2017-09-14 12:32:25,582WARN  [DataStreamer for file/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f4717749ff1c40beb6639677f2e107ceblock BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922] hdfs.DFSClient:Error Recovery for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922in pipeline 10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: baddatanode 10.9.148.192:50010

2017-09-14 12:32:25,584WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921

java.io.IOException: Badresponse ERROR for block BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921from datanode 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,585WARN  [DataStreamer for file/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/d12b4c633d2a406ea84a248b4a1a54abblock BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921]hdfs.DFSClient: Error Recovery for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921 in pipeline10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: bad datanode10.9.148.192:50010

2017-09-14 12:32:25,649INFO  [Thread-647] hdfs.DFSClient:Exception in createBlockOutputStream

java.io.IOException: Badconnect ack with firstBadLink as 10.9.148.192:50010

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2017-09-14 12:32:25,649INFO  [Thread-647] hdfs.DFSClient:Abandoning BP-200595776-10.9.148.193-1503627647735:blk_1073744578_22923

2017-09-14 12:32:25,799INFO  [Thread-647] hdfs.DFSClient:Excluding datanode 10.9.148.192:50010

2017-09-14 12:32:25,863INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Slow sync cost: 316 ms, current pipeline: [10.9.148.190:50010,10.9.148.191:50010]

2017-09-14 12:32:25,864ERROR [sync.1] wal.FSHLog: Error syncing, request close of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,865ERROR [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Failed close of WAL writerhdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805,unflushedEntries=1

org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:1650)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1667)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:858)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:709)

         atorg.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)

         at java.lang.Thread.run(Thread.java:744)

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,865FATAL [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: ABORTING region server slave1.hadoop,16020,1505351548182:Failed log close in log roller

org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException:hdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805,unflushedEntries=1

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:907)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:709)

         atorg.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:1650)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1667)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:858)

         ... 3 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,954FATAL [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []

2017-09-14 12:32:26,184INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] regionserver.HRegionServer:Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" :"java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48602144

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 289356608

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 917,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 1,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

    "TotalCallTime_TimeRangeCount_10000-30000": 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

    "QueueCallTime_TimeRangeCount_0-1": 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

   "QueueCallTime_TimeRangeCount_3-10" : 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

   "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 9,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

    "RequestSize_SizeRangeCount_10-100": 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

    "ResponseSize_SizeRangeCount_10-100": 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353375774,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277608984,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 172728,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 795950,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: STOPPED: Failed log close in log roller

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.SplitLogWorker: Sending interrupt to stop the worker thread

2017-09-14 12:32:26,284INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Stopping infoServer

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.LogRoller: LogRoller exiting.

2017-09-14 12:32:26,300INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.

2017-09-14 12:32:26,301INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker slave1.hadoop,16020,1505351548182exiting

2017-09-14 12:32:26,324INFO [regionserver/slave1.hadoop/10.9.148.190:16020] mortbay.log: StoppedSelectChannelConnector@0.0.0.0:16030

2017-09-14 12:32:26,327INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.

2017-09-14 12:32:26,327WARN [B.defaultRpcServer.handler=14,queue=2,port=16020] ipc.RpcServer:(responseTooSlow):{"processingtimems":10793,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"10.9.148.17:2345","param":"region=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,for 1 actions and 1st rowkey=LGXCD4D31H01012051498898607","starttimems":1505363534981,"queuetimems":0,"class":"HRegionServer","responsesize":970,"method":"Multi"}

2017-09-14 12:32:28,947INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=826921, memsize=62.5 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f4717749ff1c40beb6639677f2e107ce

2017-09-14 12:32:29,143INFO  [Thread-651] hdfs.DFSClient:Exception in createBlockOutputStream

java.io.IOException: Badconnect ack with firstBadLink as 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2017-09-14 12:32:29,144INFO  [Thread-651] hdfs.DFSClient:Abandoning BP-200595776-10.9.148.193-1503627647735:blk_1073744580_22927

2017-09-14 12:32:29,163INFO  [Thread-651] hdfs.DFSClient:Excluding datanode 10.9.148.192:50010

2017-09-14 12:32:29,997INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=826921, memsize=7.7 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/112a6705ac76495b8ae9200ba60d0897

2017-09-14 12:32:30,005INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=817996, memsize=114.2 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/d12b4c633d2a406ea84a248b4a1a54ab

2017-09-14 12:32:30,052INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/f4717749ff1c40beb6639677f2e107ce,entries=231420, sequenceid=826921, filesize=34.5 M

2017-09-14 12:32:30,056INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/d12b4c633d2a406ea84a248b4a1a54ab,entries=423588, sequenceid=817996, filesize=63.0 M

2017-09-14 12:32:30,057WARN [regionserver/slave1.hadoop/10.9.148.190:16020.append-pool1-t1]wal.FSHLog: Append sequenceId=818069, requesting roll of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,060FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region:CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,062FATAL [MemStoreFlusher.1] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,079INFO  [MemStoreFlusher.1]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48860760

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 231470104

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

    "ResponseSize_SizeRangeCount_1000-10000": 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

    "ProcessCallTime_TimeRangeCount_0-1": 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

    "ProcessCallTime_TimeRangeCount_100-300": 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

    "exceptions.NotServingRegionException": 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

    "exceptions.UnknownScannerException": 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" : "master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

   "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

    "ScanNext_TimeRangeCount_600000-inf": 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,082FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region:CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException:Append sequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,082FATAL [MemStoreFlusher.1] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,084INFO  [MemStoreFlusher.0] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/112a6705ac76495b8ae9200ba60d0897,entries=36204, sequenceid=826921, filesize=3.4 M

2017-09-14 12:32:30,086FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,086FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,093INFO  [MemStoreFlusher.0]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48864520

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 233210456

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

    "ResponseSize_SizeRangeCount_1000-10000": 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

    "ProcessCallTime_TimeRangeCount_0-1": 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

    "ProcessCallTime_TimeRangeCount_100-300": 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

    "exceptions.NotServingRegionException": 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

    "exceptions.UnknownScannerException": 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" : "master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

   "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

    "ScanNext_TimeRangeCount_600000-inf": 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,094INFO  [MemStoreFlusher.1]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" :"java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48862664

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 232753112

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

    "TotalCallTime_TimeRangeCount_1000-3000": 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

   "QueueCallTime_TimeRangeCount_3-10" : 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

    "ProcessCallTime_TimeRangeCount_1-3": 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

    "ProcessCallTime_TimeRangeCount_300-1000": 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

   "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,096FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,097INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Excluding unflushable regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.- trying to find a different region to flush.

2017-09-14 12:32:30,100FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,100INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=164.7 K, Region memstore size=48.8 K

2017-09-14 12:32:30,109INFO  [MemStoreFlusher.0]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48864944

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 234635400

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

    "TotalCallTime_TimeRangeCount_300-1000": 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

    "ProcessCallTime_TimeRangeCount_1-3": 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

    "ProcessCallTime_TimeRangeCount_300-1000": 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,112INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Excluding unflushable regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. - trying to find adifferent region to flush.

2017-09-14 12:32:30,112INFO [regionserver/slave1.hadoop/10.9.148.190:16020]snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManagerabruptly.

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020]flush.RegionServerFlushTableProcedureManager: Stopping region server flushprocedure manager abruptly.

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: aborting server slave1.hadoop,16020,1505351548182

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020] client.ConnectionManager$HConnectionImplementation:Closing zookeeper sessionid=0x25e7df015ce0002

2017-09-14 12:32:30,113INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting

2017-09-14 12:32:30,114INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher:Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=164.7 K, Region memstoresize=115.9 K

2017-09-14 12:32:30,114INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting

2017-09-14 12:32:30,129INFO [regionserver/slave1.hadoop/10.9.148.190:16020] zookeeper.ZooKeeper:Session: 0x25e7df015ce0002 closed

2017-09-14 12:32:30,129INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread]zookeeper.ClientCnxn: EventThread shut down

2017-09-14 12:32:30,129INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Waiting on 2 regions to close

2017-09-14 12:32:30,138INFO  [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 0

2017-09-14 12:32:30,139INFO [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 1

2017-09-14 12:32:30,139INFO [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 2

2017-09-14 12:32:30,143ERROR [RS_CLOSE_REGION-slave1:16020-0] regionserver.HRegion: Memstore size is118664

2017-09-14 12:32:30,146INFO  [RS_CLOSE_REGION-slave1:16020-0]regionserver.HRegion: ClosedCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 12:32:30,148 INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 0

2017-09-14 12:32:30,148INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 1

2017-09-14 12:32:30,151INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 2

2017-09-14 12:32:30,152ERROR [RS_CLOSE_REGION-slave1:16020-1] regionserver.HRegion: Memstore size is49952

2017-09-14 12:32:30,154INFO  [RS_CLOSE_REGION-slave1:16020-1]regionserver.HRegion: ClosedCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 12:32:30,336INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: stopping server slave1.hadoop,16020,1505351548182;all regions closed.

2017-09-14 12:32:30,336WARN [regionserver/slave1.hadoop/10.9.148.190:16020] wal.ProtobufLogWriter:Failed to write trailer, non-fatal, continuing...

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,348ERROR [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Shutdown / close of WAL failed:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

2017-09-14 12:32:30,358INFO [regionserver/slave1.hadoop/10.9.148.190:16020] regionserver.Leases:regionserver/slave1.hadoop/10.9.148.190:16020 closing leases

2017-09-14 12:32:30,358INFO [regionserver/slave1.hadoop/10.9.148.190:16020] regionserver.Leases:regionserver/slave1.hadoop/10.9.148.190:16020 closed leases

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hbase.ChoreService:Chore service for: slave1.hadoop,16020,1505351548182 had [[ScheduledChore:Name: slave1.hadoop,16020,1505351548182-MemstoreFlusherChore Period: 10000Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for regionslave1.hadoop,16020,1505351548182 Period: 120000 Unit: MILLISECONDS]] onshutdown

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Split Thread to finish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Merge Thread to finish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Large Compaction Thread tofinish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Small Compaction Thread tofinish...

2017-09-14 12:32:30,409INFO [regionserver/slave1.hadoop/10.9.148.190:16020] ipc.RpcServer: Stoppingserver on 16020

2017-09-14 12:32:30,410INFO  [RpcServer.listener,port=16020]ipc.RpcServer: RpcServer.listener,port=16020: stopping

2017-09-14 12:32:30,413INFO  [RpcServer.responder]ipc.RpcServer: RpcServer.responder: stopped

2017-09-14 12:32:30,413INFO  [RpcServer.responder]ipc.RpcServer: RpcServer.responder: stopping

2017-09-14 12:32:30,462INFO [regionserver/slave1.hadoop/10.9.148.190:16020] zookeeper.ZooKeeper:Session: 0x15e7df074580001 closed

2017-09-14 12:32:30,462INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: stopping server slave1.hadoop,16020,1505351548182;zookeeper connection closed.

2017-09-14 12:32:30,462INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: regionserver/slave1.hadoop/10.9.148.190:16020exiting

2017-09-14 12:32:30,462ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting

java.lang.RuntimeException:HRegionServer Aborted

         atorg.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)

         atorg.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)

         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

         atorg.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2697)

2017-09-14 12:32:30,463INFO  [main-EventThread] zookeeper.ClientCnxn:EventThread shut down

2017-09-14 12:32:30,511INFO  [Thread-4]regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true;fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@7ed32b58

2017-09-14 12:32:30,511INFO  [Thread-4]regionserver.ShutdownHook: Starting fs shutdown hook thread.

2017-09-14 12:32:30,530INFO  [Thread-4]regionserver.ShutdownHook: Shutdown hook finished.

 

 

 

 

 

 

 

 

 

 

 

 

重新配置hadoop、HBase、重新批量put

错误日志：

2017-08-30 15:09:14,445WARN  [DataStreamer for file/hbase/WALs/slave1.hadoop,16020,1504076498637/slave1.hadoop%2C16020%2C1504076498637..meta.1504076507048.metablock BP-200595776-10.9.148.193-1503627647735:blk_1073742324_20651]hdfs.DFSClient: DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):Client (=DFSClient_NONMAPREDUCE_-1470364669_1) is not the lease owner(=DFSClient_NONMAPREDUCE_-908925855_1:/hbase/WALs/slave1.hadoop,16020,1504076498637-splitting/slave1.hadoop%2C16020%2C1504076498637..meta.1504076507048.meta(inode 16723) [Lease.  Holder:DFSClient_NONMAPREDUCE_-1470364669_1, pending creates: 1]

         atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2624)

         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2539)

         atorg.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:863)

         atorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:530)

         atorg.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)

         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)

         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)

         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)

         at java.security.AccessController.doPrivileged(NativeMethod)

         at javax.security.auth.Subject.doAs(Subject.java:415)

         atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)

         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

 

         at org.apache.hadoop.ipc.Client.call(Client.java:1411)

         at org.apache.hadoop.ipc.Client.call(Client.java:1364)

         atorg.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

         at com.sun.proxy.$Proxy18.getAdditionalDatanode(UnknownSource)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:606)

         atorg.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

         atorg.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

         at com.sun.proxy.$Proxy18.getAdditionalDatanode(UnknownSource)

         atorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:393)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

         atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:606)

         atorg.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)

         at com.sun.proxy.$Proxy19.getAdditionalDatanode(UnknownSource)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1028)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

解决办法，调整xcievers参数

默认是4096，改为8192

vi/home/dwhftp/opt/hadoop/conf/hdfs-site.xml

<property>

<name>dfs.datanode.max.xcievers</name>

<value>8192</value>

</property>

dfs.datanode.max.xcievers参数说明：

一个 Hadoop HDFS Datanode 有一个同时处理文件的上限. 这个参数叫 xcievers (Hadoop的作者把这个单词拼错了). 在你加载之前，先确认下你有没有配置这个文件conf/hdfs-site.xml里面的xceivers参数，至少要有4096:

<property>

<name>dfs.datanode.max.xcievers</name>

<value>4096</value>

</property>

Hadoop设置更改（vihdfs-site.xml）：增加绿色部分

<configuration>

<!-- secondarynamenode -->

  <property>

   <name>dfs.namenode.secondary.http-address</name>

   <value>master.hadoop:50090</value>

  </property>

<!-- namenode name dir-->

  <property>

   <name>dfs.namenode.name.dir</name>

    <value>file:/home/softs/hadoop/tmp/dfs/name</value>

  </property>

<!-- datanode data dir-->

  <property>

   <name>dfs.datanode.data.dir</name>

   <value>file:/home/softs/hadoop/tmp/dfs/data</value>

  </property>

<!-- replication-->

  <property>

    <name>dfs.replication</name>

    <value>1</value>

  </property>

<!-- userpermissions  -->

  <property>

    <name>dfs.permissions</name>

    <value>false</value>

  </property>

  <property>

   <name>dfs.datanode.handler.count</name>

    <value>3</value>

  </property>

  <property>

   <name>dfs.client.socket-timeout</name>

    <value>60000</value>

  </property>

  <property>

    <name>dfs.support.append</name>

    <value>true</value>

  </property>

  <property>

   <name>dfs.datanode.max.xcievers</name>

    <value>8192</value>

  </property>

</configuration>

错误日志：

2017-09-01 09:39:30,827WARN [regionserver/slave1.hadoop/10.9.148.190:16020] util.Sleeper: We slept30263ms instead of 3000ms, this is likely due to a long garbage collectingpause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired

2017-09-01 09:39:30,826INFO [main-SendThread(slave2.hadoop:2181)] zookeeper.ClientCnxn: Clientsession timed out, have not heard from server in 36049ms for sessionid0x15e3200dc5c0000, closing socket connection and attempting reconnect

2017-09-01 09:39:30,826INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave3.hadoop:2181)]zookeeper.ClientCnxn: Client session timed out, have not heard from server in30345ms for sessionid 0x15e3200dc5c0003, closing socket connection andattempting reconnect

2017-09-01 09:39:46,287INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Openingsocket connection to server master.hadoop/10.9.148.193:2181. Will not attemptto authenticate using SASL (unknown error)

2017-09-01 09:39:46,287INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Opening socket connection to serverslave2.hadoop/10.9.148.191:2181. Will not attempt to authenticate using SASL(unknown error)

2017-09-01 09:39:46,287INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Socketconnection established to master.hadoop/10.9.148.193:2181, initiating session

2017-09-01 09:39:46,287INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Socket connection established toslave2.hadoop/10.9.148.191:2181, initiating session

2017-09-01 09:39:47,772INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Unable toreconnect to ZooKeeper service, session 0x15e3200dc5c0000 has expired, closingsocket connection

2017-09-01 09:39:50,548INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session0x15e3200dc5c0003 has expired, closing socket connection

2017-09-01 09:40:02,048FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: regionserver:16020-0x15e3200dc5c0000,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbaseregionserver:16020-0x15e3200dc5c0000 received expired from ZooKeeper, aborting

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-01 09:40:02,048WARN [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread]client.ConnectionManager$HConnectionImplementation: This client just lost it'ssession with ZooKeeper, closing it. It will be recreated next time someoneneeds it

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-01 09:40:02,048FATAL [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: org.apache.hadoop.hbase.YouAreDeadException:Server REPORT rejected; currently processing slave1.hadoop,16020,1504077821200as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

org.apache.hadoop.hbase.YouAreDeadException:org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)

         atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

         atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         atjava.lang.reflect.Constructor.newInstance(Constructor.java:526)

         atorg.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

         atorg.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

         atorg.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:962)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException):org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

         atorg.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:8994)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1151)

         ... 2 more

描述及解决方法：

为什么regionserver 和Zookeeper的session expired? 可能的原因有

1. 网络不好。

2. Java full GC， 这会block所有的线程。如果时间比较长，也会导致session expired.

怎么办？

1. 将Zookeeper的timeout时间加长。

2. 配置“hbase.regionserver.restart.on.zk.expire” 为true。 这样子，遇到ZooKeepersession expired ， regionserver将选择 restart 而不是 abort

具体的配置是，在hbase-site.xml中加入

<configuration>

        <property>

                <name>hbase.rootdir</name>

               <value>hdfs://master:9000/hbase</value>

        </property>

        <property>

               <name>hbase.cluster.distributed</name>

                <value>true</value>

        </property>

        <property>

                <name>hbase.master</name>

               <value>master</value>

        </property>

        <property>

               <name>hbase.zookeeper.quorum</name>

               <value>master,slave1,slave2,slave3</value>

        </property>

        <property>

               <name>hbase.zookeeper.property.dataDir</name>

               <value>/home/softs/zookeeper/data</value>

        </property>

        <property>

               <name>zookeeper.session.timeout</name>

                <value>90000</value>

                <description>ZooKeepersession timeout.

                        HBase passes this tothe zk quorum as suggested maximum time for a

                        session.  See http://hadoop.apache.org/zooke ... sions

                        “The client sends a requested timeout, theserver responds with the

                        timeout that it cangive the client. The current implementation

                        requires that thetimeout be a minimum of 2 times the tickTime

                        (as set in the serverconfiguration) and a maximum of 20 times

                        the tickTime.” Set thezk ticktime with hbase.zookeeper.property.tickime.

                        In milliseconds.

                </description>

        </property>

        <property>

               <name>hbase.regionserver.restart.on.zk.expire</name>

                <value>true</value>

                <description>

                        Zookeeper sessionexpired will force regionserver exit.

                        Enable this will makethe regionserver restart.

                </description>

        </property>

</configuration>

HBase Java Full GC错误：

错误日志：

2017-09-0109:40:02,048 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING regionserver slave1.hadoop,16020,1504077821200: regionserver:16020-0x15e3200dc5c0000,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbaseregionserver:16020-0x15e3200dc5c0000 received expired from ZooKeeper, aborting

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-0109:40:02,048 WARN [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread] client.ConnectionManager$HConnectionImplementation:This client just lost it's session with ZooKeeper, closing it. It will berecreated next time someone needs it

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-0109:40:02,048 FATAL [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: org.apache.hadoop.hbase.YouAreDeadException:Server REPORT rejected; currently processing slave1.hadoop,16020,1504077821200as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

org.apache.hadoop.hbase.YouAreDeadException:org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processingslave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

         atsun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

         atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         atjava.lang.reflect.Constructor.newInstance(Constructor.java:526)

         atorg.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

         atorg.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:962)

         atjava.lang.Thread.run(Thread.java:744)

Causedby:org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException):org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

         atorg.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:8994)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1151)

         ... 2 more

解决方法：

看起来是GC时间太长了，导致ZK 心跳 连接超时，最终通知这个server下线。

HBase集群总是掉的原因：

1.网络不好，导致RegionServer 和 HMaster通信超时，RegionServer被认为已经挂掉，从而退出集群 --网络问题，无法从软件方面解决，关于通信超时的设置下面做个简单介绍

2.Java full GC ，这个过程会block所有的线程，如果此事件过长，导致Session expired 会话过期，导致退出集群--下文会阐述

3.各节点时间不一致，导致RegionServer 退出。--hbase.master.maxclockskew 增大容忍度，默认是30s，但不要太大，毕竟时间不一致是不正常现象，可将所有节点和同一服务器时间做同步，也可以和时间服务器同步。（这个在安装zookeeper的时候，就讲过，时间最好一致，要么安装集群时钟管理软件，要么手动设置，超过30sHBase集群就难以启动了）

修改HBase（./conf/hbase-env.sh）:

#exportHBASE_OPTS="-XX:+UseConcMarkSweepGC"

exportHBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC-XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -Xmn256m"

解释：

具体参数含义如下：

-XX:+UseCompressedOops

压缩指针，解决内存占用

-XX:+UseParNewGC

设置年轻代为并行收集

-XX:+UseConcMarkSweepG

 使用CMS内存收集

-XX:+CMSClassUnloadingEnabled

相对于并行收集器，CMS收集器默认不会对永久代进行垃圾回收。如果希望对永久代进行垃圾回收，可用设置标志-XX:+CMSClassUnloadingEnabled。 在早期JVM版本中，要求设置额外的标志-XX:+CMSPermGenSweepingEnabled。注意，即使没有设置这个标志，一旦永久代耗尽空 间也会尝试进行垃圾回收，但是收集不会是并行的，而再一次进行Full GC。

-XX:+UseCMSCompactAtFullCollection

使用并发收集器时,开启对年老代的压缩.

-XX:CMSFullGCsBeforeCompaction

由于并发收集器不对内存空间进行压缩,整理,所以运行一段时间以后会产生”碎片”,使得运行效率降低.此值设置运行多少次GC以后对内存空间进行压缩,整理.

-XX:+CMSParallelRemarkEnabled

降低标记停顿

-XX:CMSInitiatingOccupancyFraction=75

使用cms作为垃圾回收使用75％后开始CMS收集

-XX:SoftRefLRUPolicyMSPerMB

每兆堆空闲空间中SoftReference的存活时间

 

 

4.推测原因1--compaction、split过于频繁：

 

              由于配置里面hfile最大文件大小设置为1G，所以compaction、split比较频繁，资源消耗比较大，导致gc暂停时间过长，出现写hdfs错误，导致regionsever挂掉

                 推测原因2-- hdfs压力过大，datanode超负荷：

 

               由于集群运行各种任务，hdfs读写压力大，datanode的负载比较高，导致regionserver写hdfs异常，宕机

         解决办法：1、hfile大小改为100G，禁止系统自己做major compaction。2、给datanode多一些内存，调整rpc线程数。

因为本人搭建的集群的slave性能很差，磁盘、内存都不是很大，所以在这里并没有设置HFile的大小

 

参考网址：http://wenda.chinahadoop.cn/question/4872