高可用性hadoop集群问题

按照网上的教程和apache的教程搭建高可用性hadoop，用的方法主要是zookeeper和journalnode，启动之后，两个namenode只有一个是active，另外一个是standby。

然后按照网上说的教程，kill掉一个active的namenode之后，应该能自动切换，结果没有自动切换，看了下日志

2016-12-07 09:53:05,499 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning Service Fencing Process... ======
2016-12-07 09:53:05,500 INFO org.apache.hadoop.ha.NodeFencer: Trying method 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(null)
2016-12-07 09:53:05,776 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to datanode1...
2016-12-07 09:53:05,798 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connecting to datanode1 port 22
2016-12-07 09:53:05,821 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connection established
2016-12-07 09:53:05,898 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Remote version string: SSH-2.0-OpenSSH_6.7p1 Ubuntu-5ubuntu1.4
2016-12-07 09:53:05,899 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Local version string: SSH-2.0-JSCH-0.1.42
2016-12-07 09:53:05,899 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2016-12-07 09:53:06,769 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT sent
2016-12-07 09:53:06,770 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT received
2016-12-07 09:53:06,780 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Disconnecting from datanode1 port 22
2016-12-07 09:53:06,887 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to connect to datanode1 as user hadoop
com.jcraft.jsch.JSchException: Algorithm negotiation fail
        at com.jcraft.jsch.Session.receive_kexinit(Session.java:520)
        at com.jcraft.jsch.Session.connect(Session.java:286)
        at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:100)
        at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97)
        at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:532)
        at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505)
        at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
        at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892)
        at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:910)
        at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:809)
        at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2016-12-07 09:53:06,893 WARN org.apache.hadoop.ha.NodeFencer: Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful.
2016-12-07 09:53:06,893 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fence service by any configured method.
2016-12-07 09:53:06,931 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
java.lang.RuntimeException: Unable to fence NameNode at datanode1/192.168.1.51:8020
        at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:533)
        at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505)
        at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
        at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892)
        at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:910)
        at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:809)
        at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:418)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

网上查了一下，是ssh的密钥协商失败，有的例子是说加入不支持的密码算法

后来找了一篇，说是

jsch jar包要更新到0.1.50以上版本，经测试0.1.42版本与openssh6.7配合有问题。

日志记录的openssh就是6.7版本的

jsch是jsch0.1.42

去hadoop的目录下看了 ，就是这个版本

下载0.1.54版本，替换掉0.1.42版本，在此启动运行，成功

2016-12-07 11:11:37,967 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning Service Fencing Process... ======
2016-12-07 11:11:37,968 INFO org.apache.hadoop.ha.NodeFencer: Trying method 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(null)
2016-12-07 11:11:38,239 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to datanode1...
2016-12-07 11:11:38,254 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connecting to datanode1 port 22
2016-12-07 11:11:38,274 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connection established
2016-12-07 11:11:38,353 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Remote version string: SSH-2.0-OpenSSH_6.7p1 Ubuntu-5ubuntu1.4
2016-12-07 11:11:38,354 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Local version string: SSH-2.0-JSCH-0.1.54
2016-12-07 11:11:38,354 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2016-12-07 11:11:39,198 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
2016-12-07 11:11:46,245 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2016-12-07 11:11:46,276 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT sent
2016-12-07 11:11:46,277 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEXINIT received
2016-12-07 11:11:46,278 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-
sha1
2016-12-07 11:11:46,279 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
2016-12-07 11:11:46,279 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
2016-12-07 11:11:46,280 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
2016-12-07 11:11:46,280 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@ope
nssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
2016-12-07 11:11:46,281 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@ope
nssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
2016-12-07 11:11:46,281 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: none,zlib@openssh.com
2016-12-07 11:11:46,282 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server: none,zlib@openssh.com
2016-12-07 11:11:46,282 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server:
2016-12-07 11:11:46,283 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server:
2016-12-07 11:11:46,283 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exc
hange-sha1,diffie-hellman-group1-sha1
2016-12-07 11:11:46,284 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2016-12-07 11:11:46,285 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2016-12-07 11:11:46,285 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2016-12-07 11:11:46,286 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2016-12-07 11:11:46,286 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2016-12-07 11:11:46,287 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: none
2016-12-07 11:11:46,287 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client: none
2016-12-07 11:11:46,288 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client:
2016-12-07 11:11:46,288 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client:
2016-12-07 11:11:46,292 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: server->client aes128-ctr hmac-sha1 none
2016-12-07 11:11:46,292 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: kex: client->server aes128-ctr hmac-sha1 none
2016-12-07 11:11:46,306 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_KEX_ECDH_INIT sent
2016-12-07 11:11:46,306 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: expecting SSH_MSG_KEX_ECDH_REPLY
2016-12-07 11:11:46,609 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: ssh_rsa_verify: signature true
2016-12-07 11:11:46,661 WARN org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Permanently added 'datanode1' (RSA) to the list of known hosts.
2016-12-07 11:11:46,662 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_NEWKEYS sent
2016-12-07 11:11:46,663 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_NEWKEYS received
2016-12-07 11:11:46,716 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_SERVICE_REQUEST sent
2016-12-07 11:11:46,718 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: SSH_MSG_SERVICE_ACCEPT received
2016-12-07 11:11:46,727 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentications that can continue: publickey,keyboard-interactive,password
2016-12-07 11:11:46,728 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next authentication method: publickey
2016-12-07 11:12:14,513 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentication succeeded (publickey).
2016-12-07 11:12:14,515 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connected to datanode1