免匙SSH登录失败问题（非常规）

问题描述：

上周给公司搭建大数据平台，选取三台机器，安装配置一切顺利。后来发现/home目录的挂载盘容量不够用，所以就扩容了，在扩容之前将/home/hadoop文件夹复制到其他地方，扩容后再复制回来，现在问题来了，hadoop文件夹迁移回来后，发现免匙SSH无用了。

问题排查：

1.检测权限

chmod 600 ~/.ssh/authorized_keys

chmod 700 ~/.ssh/

2.debug SSH

通过命令 ssh -vvv master 查看日志

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to master [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/hadoop/.ssh/identity type -1
debug1: identity file /home/hadoop/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /home/hadoop/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/hadoop/.ssh/id_rsa type 1
debug1: identity file /home/hadoop/.ssh/id_rsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_dsa type -1
debug1: identity file /home/hadoop/.ssh/id_dsa-cert type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa type -1
debug1: identity file /home/hadoop/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 114/256
debug2: bits set: 512/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: check_host_in_hostfile: host master filename /home/hadoop/.ssh/known_hosts
debug3: check_host_in_hostfile: host master filename /home/hadoop/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 5
debug1: Host 'master' is known and matches the RSA host key.
debug1: Found key in /home/hadoop/.ssh/known_hosts:5
debug2: bits set: 525/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/hadoop/.ssh/identity ((nil))
debug2: key: /home/hadoop/.ssh/id_rsa (0x2ae9888a6330)
debug2: key: /home/hadoop/.ssh/id_dsa ((nil))
debug2: key: /home/hadoop/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address 127.0.0.1.
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/hadoop/.ssh/identity
debug3: no such identity: /home/hadoop/.ssh/identity
debug1: Offering public key: /home/hadoop/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1645
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/hadoop/.ssh/id_dsa
debug3: no such identity: /home/hadoop/.ssh/id_dsa
debug1: Trying private key: /home/hadoop/.ssh/id_ecdsa
debug3: no such identity: /home/hadoop/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

3.从日志信息并没有看到是权限问题，通过比较可以正常免匙的机器发现以下不同：

在这期间我尝试新建了用户test，在新建的用户中配置免匙ssh依然不行，但是发现root用户的免匙SSH是ok的，现在目标又回到了权限问题，可是发现权限都是ok的。

4.在快要放弃的时候搜到了这篇帖子：http://www.linuxidc.com/Linux/2013-07/87267.htm

看了这篇文章后我立刻用ls -laZ检查了我的.ssh目录：

[hadoop@master ~]$ ls -laZ .ssh

drwx------. hadoop hadoop unconfined_u:object_r:file_t:s0  .

drwxr-xr-x. hadoop hadoop unconfined_u:object_r:file_t:s0  ..

-rw-------. hadoop hadoop unconfined_u:object_r:file_t:s0  authorized_keys

-rw-------. hadoop hadoop unconfined_u:object_r:file_t:s0  id_rsa

-rw-r--r--. hadoop hadoop unconfined_u:object_r:file_t:s0  id_rsa.pub

-rw-r--r--. hadoop hadoop unconfined_u:object_r:file_t:s0  id_rsa.pub.slave1

-rw-r--r--. hadoop hadoop unconfined_u:object_r:file_t:s0  id_rsa.pub.slave2

-rw-r--r--. hadoop hadoop unconfined_u:object_r:file_t:s0  known_hosts

我也是“果然不是ssh_home_t”。

5.解决问题：

切换到root用户，修复context:

[root@master ~]# restorecon -r -vv /home/ 

[hadoop@master ~]$ ls -laZ .ssh

drwx------. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 .

drwxr-xr-x. hadoop hadoop unconfined_u:object_r:user_home_dir_t:s0..

-rw-------. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 authorized_keys

-rw-------. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 id_rsa

-rw-r--r--. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub

-rw-r--r--. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub.slave1

-rw-r--r--. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub.slave2

-rw-r--r--. hadoop hadoop unconfined_u:object_r:ssh_home_t:s0 known_hosts

6.验证是否解决：

[hadoop@master ~]$ ssh master

Last login: Wed May 31 09:01:48 2017 from 10.0.17.19

Starting Nexus OSS...

Started Nexus OSS.

-bash: /etc/profile.d/mystart.sh: line 2: syntax error near unexpected token `&&'

-bash: /etc/profile.d/mystart.sh: line 2: ` && '

[hadoop@master ~]$ 

7.问题总结：

根本原因是我将/home/hadoop目录做了移动，可是移动回来的时候用的是root用户，即便我改回了hadoop:hadoop，可是目录下的文件夹的context变了，而ssh鉴权又非常严格必须是ssh_home_t才可以。