hbase 某台regionserver更换ip后出现块不同步的现象

方式一: 重新hbase集群让metedata识别新ip,使用修复元数据,重新分配块的脚本 rephbase.sh

#!/bin/bash
su - hbase <<EOF
hbase hbck -fixMeta -fixAssignments
hbase hbck -repair
hbase hbck -fixEmptyMetaCells
hbase hbck
EOF

方式二:如果一没有成功,例如出现如下错误

进入ambari hbase master uI 观察节点状态发现新节点ip已经存在并且数据在上面,但是旧节点的主机名也存在,块在新ip节点上,所以需要通过graceful脚本

graceful_stop.sh 命令为 在hbase/bin 下 执行此脚本 graceful_stop.sh 10.0.30.139
graceful_stop.sh脚本内容：

#!/usr/bin/env bash
#
#/**
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements.  See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership.  The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License.  You may obtain a copy of the License at
# *
# *     http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */

# Move regions off a server then stop it.  Optionally restart and reload.
# Turn off the balancer before running this script.
function usage {
  echo "Usage: graceful_stop.sh [--config <conf-dir>] [-d] [-e] [--restart [--reload]] [--thrift] [--rest] <hostname>"
  echo " thrift         If we should stop/start thrift before/after the hbase stop/start"
  echo " rest           If we should stop/start rest before/after the hbase stop/start"
  echo " restart        If we should restart after graceful stop"
  echo " reload         Move offloaded regions back on to the restarted server"
  echo " d|debug        Print helpful debug information"
  echo " maxthreads xx  Limit the number of threads used by the region mover. Default value is 1."
  echo " hostname       Hostname of server we are to stop"
  echo " e|failfast     Set -e so exit immediately if any command exits with non-zero status"
  exit 1
}

if [ $# -lt 1 ]; then
  usage
fi

bin=`dirname "$0"`
bin=`cd "$bin">/dev/null; pwd`
# This will set HBASE_HOME, etc.
. "$bin"/hbase-config.sh
# Get arguments
restart=
reload=
debug=
thrift=
rest=
maxthreads=1
failfast=
while [ $# -gt 0 ]
do
  case "$1" in
    --thrift)  thrift=true; shift;;
    --rest)  rest=true; shift;;
    --restart)  restart=true; shift;;
    --reload)   reload=true; shift;;
    --failfast | -e)  failfast=true; shift;;
    --debug | -d)  debug="--debug"; shift;;
    --maxthreads) shift; maxthreads=$1; shift;;
    --) shift; break;;
    -*) usage ;;
    *)  break;;	# terminate while loop
  esac
done

# "$@" contains the rest. Must be at least the hostname left.
if [ $# -lt 1 ]; then
  usage
fi

# Emit a log line w/ iso8901 date prefixed
log() {
  echo `date +%Y-%m-%dT%H:%M:%S` $1
}

# See if we should set fail fast before we do anything.
if [ "$failfast" != "" ]; then
  log "Set failfast, will exit immediately if any command exits with non-zero status"
  set -e
fi

hostname=$1
filename="/tmp/$hostname"

local=
localhostname=`/bin/hostname`

if [ "$localhostname" == "$hostname" ]; then
  local=true
fi

log "Disabling load balancer"
HBASE_BALANCER_STATE=`echo 'balance_switch false' | "$bin"/hbase --config ${HBASE_CONF_DIR} shell | tail -3 | head -1`
log "Previous balancer state was $HBASE_BALANCER_STATE"

log "Unloading $hostname region(s)"
HBASE_NOEXEC=true "$bin"/hbase --config ${HBASE_CONF_DIR} org.jruby.Main "$bin"/region_mover.rb --file=$filename $debug --maxthreads=$maxthreads unload $hostname
log "Unloaded $hostname region(s)"

# Stop the server(s). Have to put hostname into its own little file for hbase-daemons.sh
hosts="/tmp/$(basename $0).$$.tmp"
echo $hostname >> $hosts
if [ "$thrift" != "" ]; then
  log "Stopping thrift server on $hostname"
  if [ "$local" == true ]; then
    "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} stop thrift
  else
    "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} stop thrift
  fi
fi
if [ "$rest" != "" ]; then
  log "Stopping rest server on $hostname"
  if [ "$local" == true ]; then
    "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} stop rest
  else
    "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} stop rest
  fi
fi
log "Stopping regionserver on $hostname"
if [ "$local" == true ]; then
  "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} stop regionserver
else
  "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} stop regionserver
fi
if [ "$restart" != "" ]; then
  log "Restarting regionserver on $hostname"
  if [ "$local" == true ]; then
    "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} start regionserver
  else
    "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} start regionserver
  fi
  if [ "$thrift" != "" ]; then
    log "Restarting thrift server on $hostname"
    # -b 0.0.0.0 says listen on all interfaces rather than just default.
    if [ "$local" == true ]; then
      "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} start thrift -b 0.0.0.0
    else
      "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} start thrift -b 0.0.0.0
    fi
  fi
  if [ "$rest" != "" ]; then
    log "Restarting rest server on $hostname"
    if [ "$local" == true ]; then
      "$bin"/hbase-daemon.sh --config ${HBASE_CONF_DIR} start rest
    else
      "$bin"/hbase-daemons.sh --config ${HBASE_CONF_DIR} --hosts ${hosts} start rest
    fi
  fi
  if [ "$reload" != "" ]; then
    log "Reloading $hostname region(s)"
    HBASE_NOEXEC=true "$bin"/hbase --config ${HBASE_CONF_DIR} org.jruby.Main "$bin"/region_mover.rb --file=$filename $debug --maxthreads=$maxthreads load $hostname
    log "Reloaded $hostname region(s)"
  fi
fi

# Restore balancer state
if [ $HBASE_BALANCER_STATE != "false" ]; then
  log "Restoring balancer state to $HBASE_BALANCER_STATE"
  echo "balance_switch $HBASE_BALANCER_STATE" | "$bin"/hbase --config ${HBASE_CONF_DIR} shell &> /dev/null
fi

# Cleanup tmp files.
trap "rm -f  "/tmp/$(basename $0).*.tmp" &> /dev/null" EXIT

这个操作是平滑的重启regionserver进程，对服务不会有影响，他会先将需要重启的regionserver上面的所有 region迁移到其它的服务器，然后重启，最后又会将之前的region迁移回来，但我们修改一个配置时，可以用这种方式重启每一台机子，这个命令会关 闭balancer，所以最后我们要在hbase shell里面执行一下balance_switch true，对于hbase regionserver重启，不要直接kill进程，这样会造成在zookeeper.session.timeout这个时间长的中断，也不要通过 bin/hbase-daemon.sh stop regionserver去重启，如果运气不太好，-ROOT-或者.META.表在上面的话，所有的请求会全部失败。

完成后重启hbase 集群并且 balance_switch true 手工开启状态如下为正常