Hadoop NameNode HA fencing

请先参阅官方文档：http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

主要原理是使用journal log，当主namenode增加修改元数据之后，会把log同步给journal ,然后再传给standby namenode，以达到元数据一致性。同一时刻只有一个namenode可以写入journal log, 这样就不担心脑裂问题。原文如下：

importantly, when using the Quorum Journal Manager, only one NameNode will ever be allowed to write to the JournalNodes, so there is no potential for corrupting the file system metadata from a split-brain scenario

需要在HDFS.site.xml添加如下内容：

  <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
  </property>

  <property>
   <name>dfs.ha.namenodes.nameservice1</name>
    <value>nn1,nn2</value>
  </property>
 
  <property>
  <name>dfs.namenode.rpc-address.nameservice1.nn1</name>
  <value>10.40.2.91:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.nameservice1.nn2</name>
  <value>10.40.2.92:8020</value>
</property>

 <property>
  <name>dfs.namenode.http-address.nameservice1.nn1</name>
  <value>10.40.2.91:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.nameservice1.nn2</name>
  <value>10.40.2.92:50070</value>
</property>

  <property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://10.40.2.91:8485;10.40.2.92:8485;10.40.2.175:8485/nameservice1</value>
</property>

  <property>
  <name>dfs.client.failover.proxy.provider.nameservice1</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

  <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence
             shell(/data/hadoop/scripts/fence.sh)</value>
  </property>

    <property>
      <name>dfs.ha.fencing.ssh.connect-timeout</name>
      <value>30000</value>
    </property>

   <property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/data/journal/data</value>
</property>

  <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

  <property>
   <name>ha.zookeeper.quorum</name>
   <value>10.40.2.91:2181,10.40.2.92:2181,10.40.2.175:2181</value>
 </property>

core.site.xml：

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://nameservice1</value>
</property>

但是fencing的时候发现无法切换，具体原因暂时不清楚，错误如下：

2019-04-29 16:23:40,206 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to connect to 10.40.22.91 as user hdfs
com.jcraft.jsch.JSchException: Auth fail
        at com.jcraft.jsch.Session.connect(Session.java:519)
        at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:100)
        at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97)
        at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:532)
        at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505)
        at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
        at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892)
        at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:921)
        at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:820)
        at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

官方文档给了2个方案，一个是sshfence,一个是shell方式，而且fence的方法可以多个，以回车换行排序即可，如果第一个fence失败，会继续使用之后的方法。

The fencing methods used during a failover are configured as a carriage-return-separated list, which will be attempted in order until one indicates that fencing has succeeded. There are two methods which ship with Hadoop: shell and sshfence

由于sshfence失败原因不清楚，那么我自己写了一个shell来做fence.

# cat /data/hadoop/scripts/fence.sh 

#!/bin/bash
nc -z 10.40.2.92 8020
if [ $? -eq 0 ]; then
   PID=`ssh 10.40.2.92 lsof -i:8020 | grep LISTEN | awk -F' ' '{print $2}'`
   ssh 10.40.2.92 "kill -9 $PID"
   exit 0
fi

所谓fence就是隔离的意思，只要你能保证namenode状态正常即可，我采用的是查看需要切换的namenode的端口是否存在，如果存在，那么就kill掉那个进程。

2019-04-29 16:24:59,360 INFO org.apache.hadoop.ha.ShellCommandFencer: Launched fencing command '/data/hadoop/scripts/fence.sh' with pid 3724
2019-04-29 16:24:59,371 INFO org.apache.hadoop.ha.NodeFencer: ====== Fencing successful by method org.apache.hadoop.ha.ShellCommandFencer(/data/hadoop/scripts/fence.sh) ======