Hadoop伪分布式部署之ssh免密钥登陆

标签: hadoop ssh免密钥
5人阅读 评论(0) 收藏 举报
分类:

前言

在之前的章节中,我们有介绍到伪分布式的hdfsyarn和mapreduce历史服务与日志聚集SecondaryNameNode的部署。接下来我们一起探讨下hadoop的ssh免密钥登陆。

我们的hadoop环境如下
操作系统:CentOS6.4
Java版本:Oracle jdk1.7
Hadoop版本:Hadoop2.5.0
主机hostname:hadoop01.datacenter.com
hadoop目录:/opt/modules/hadoop-2.5.0

启动停止hadoop服务

在hadoop的sbin目录下,我们可以看到一些start-*.sh和stop-*.sh的脚本:

[hadoop@hadoop01 ~]$ cd /opt/modules/hadoop-2.5.0/
[hadoop@hadoop01 hadoop-2.5.0]$ ll sbin/
total 88
-rwxr-xr-x 1 hadoop hadoop 2752 Aug  7  2014 distribute-exclude.sh
-rwxr-xr-x 1 hadoop hadoop 6435 Aug  7  2014 hadoop-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1360 Aug  7  2014 hadoop-daemons.sh
-rwxr-xr-x 1 hadoop hadoop 1427 Aug  7  2014 hdfs-config.sh
-rwxr-xr-x 1 hadoop hadoop 2291 Aug  7  2014 httpfs.sh
-rwxr-xr-x 1 hadoop hadoop 4063 Aug  7  2014 mr-jobhistory-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1648 Aug  7  2014 refresh-namenodes.sh
-rwxr-xr-x 1 hadoop hadoop 2145 Aug  7  2014 slaves.sh
-rwxr-xr-x 1 hadoop hadoop 1471 Aug  7  2014 start-all.sh
-rwxr-xr-x 1 hadoop hadoop 1128 Aug  7  2014 start-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 3705 Aug  7  2014 start-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1357 Aug  7  2014 start-secure-dns.sh
-rwxr-xr-x 1 hadoop hadoop 1347 Aug  7  2014 start-yarn.sh
-rwxr-xr-x 1 hadoop hadoop 1462 Aug  7  2014 stop-all.sh
-rwxr-xr-x 1 hadoop hadoop 1179 Aug  7  2014 stop-balancer.sh
-rwxr-xr-x 1 hadoop hadoop 3206 Aug  7  2014 stop-dfs.sh
-rwxr-xr-x 1 hadoop hadoop 1340 Aug  7  2014 stop-secure-dns.sh
-rwxr-xr-x 1 hadoop hadoop 1340 Aug  7  2014 stop-yarn.sh
-rwxr-xr-x 1 hadoop hadoop 4278 Aug  7  2014 yarn-daemon.sh
-rwxr-xr-x 1 hadoop hadoop 1353 Aug  7  2014 yarn-daemons.sh
[hadoop@hadoop01 hadoop-2.5.0]$ 

现在我们使用start-yarn.sh和stop-yarn.sh启动停止yarn服务试试:

[hadoop@hadoop01 hadoop-2.5.0]$ sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-resourcemanager-hadoop01.datacenter.com.out
The authenticity of host 'hadoop01.datacenter.com (192.168.190.151)' can't be established.
RSA key fingerprint is e2:ca:19:e5:04:0e:3c:11:d3:1a:cb:1f:b0:03:e6:87.
Are you sure you want to continue connecting (yes/no)? yes
hadoop01.datacenter.com: Warning: Permanently added 'hadoop01.datacenter.com,192.168.190.151' (RSA) to the list of known hosts.
hadoop@hadoop01.datacenter.com's password: 
hadoop01.datacenter.com: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-nodemanager-hadoop01.datacenter.com.out
[hadoop@hadoop01 hadoop-2.5.0]$ 
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
hadoop@hadoop01.datacenter.com's password: 
hadoop01.datacenter.com: stopping nodemanager
no proxyserver to stop
[hadoop@hadoop01 hadoop-2.5.0]$ 

上面的启动停止过程中,有提到RSA,并且让我们输入了hadoop用户的密码,熟悉linux的同学都知道,这是用了ssh登陆hadoop01.datacenter.com这台机器。
我们可以查看这些shell脚本的代码,可以看到调用关系是start-yarn.sh->yarn-daemons.sh->slaves.sh。
在slaves.sh中,有下面这样一段:

...
# start the daemons
for slave in $SLAVE_NAMES ; do
 ssh $HADOOP_SSH_OPTS $slave $"${@// /\\ }" \
   2>&1 | sed "s/^/$slave: /" &
 if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then
   sleep $HADOOP_SLAVE_SLEEP
 fi
...

可以看出来,这里有使用ssh协议访问每个从节点。如果整个集群有成百上千个节点,那么我们在主节点使用ssh协议开启从节点的服务的时候,输入密码这个工作量非常大,不利于集群维护的便利性,所以我们可以采用免密钥登陆的方式。

ssh免密钥登陆配置

首先删除现有的ssh信息:

[hadoop@hadoop01 hadoop-2.5.0]$ cd ~/.ssh
[hadoop@hadoop01 .ssh]$ ll
total 4
-rw-r--r-- 1 hadoop hadoop 421 Apr 15 20:40 known_hosts
[hadoop@hadoop01 .ssh]$ rm known_hosts 
[hadoop@hadoop01 .ssh]$ ll
total 0
[hadoop@hadoop01 .ssh]$ 

然后不输入密码(直接按三次回车)生成私钥和公钥:

[hadoop@hadoop01 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
b6:35:ea:59:32:ed:3a:24:65:5b:8b:67:63:88:84:a9 hadoop@hadoop01.datacenter.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|     o           |
|    o . o .      |
|   . . +S=o.     |
|  E   o.==*.     |
|       o=+o.     |
|       ..*       |
|        +o.      |
+-----------------+
[hadoop@hadoop01 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Apr 15 21:15 id_rsa
-rw-r--r-- 1 hadoop hadoop  412 Apr 15 21:15 id_rsa.pub
[hadoop@hadoop01 .ssh]$ 

其中id_rsa为私钥文件,id_rsa.pub为公钥文件。
接下来我们将公钥发送给从节点hadoop01.datacenter.com:

[hadoop@hadoop01 .ssh]$ ssh-copy-id hadoop01.datacenter.com
The authenticity of host 'hadoop01.datacenter.com (192.168.190.151)' can't be established.
RSA key fingerprint is e2:ca:19:e5:04:0e:3c:11:d3:1a:cb:1f:b0:03:e6:87.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop01.datacenter.com,192.168.190.151' (RSA) to the list of known hosts.
hadoop@hadoop01.datacenter.com's password: 
Now try logging into the machine, with "ssh 'hadoop01.datacenter.com'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[hadoop@hadoop01 .ssh]$ ll
total 16
-rw------- 1 hadoop hadoop  412 Apr 15 21:19 authorized_keys
-rw------- 1 hadoop hadoop 1675 Apr 15 21:15 id_rsa
-rw-r--r-- 1 hadoop hadoop  412 Apr 15 21:15 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  421 Apr 15 21:18 known_hosts
[hadoop@hadoop01 .ssh]$ 

现在我们就可以通过无密码通过ssh登陆到hadoop01.datacenter.com节点了:

[hadoop@hadoop01 .ssh]$ ssh hadoop@hadoop01.datacenter.com       
Last login: Sun Apr 15 21:12:33 2018 from 192.168.190.1
[hadoop@hadoop01 ~]$ exit
logout
Connection to hadoop01.datacenter.com closed.
[hadoop@hadoop01 .ssh]$ 

接下来我们试一下hadoop中的start-yarn.sh和stop-yarn.sh:

[hadoop@hadoop01 .ssh]$ cd /opt/modules/hadoop-2.5.0/
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-resourcemanager-hadoop01.datacenter.com.out
hadoop01.datacenter.com: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-hadoop-nodemanager-hadoop01.datacenter.com.out
[hadoop@hadoop01 hadoop-2.5.0]$ jps
4281 ResourceManager
4708 Jps
4461 NodeManager
[hadoop@hadoop01 hadoop-2.5.0]$ sbin/stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
hadoop01.datacenter.com: stopping nodemanager
no proxyserver to stop
[hadoop@hadoop01 hadoop-2.5.0]$ jps
4843 Jps
[hadoop@hadoop01 hadoop-2.5.0]$ 

无需输入ssh登陆用户的密码,便成功启动和停止了yarn的相关服务。

总结

1、hadoop可以通过ssh协议启动和停止集群中的节点的相关服务。
2、可以通过配置无密钥登陆,来访问hadoop集群中的节点。
3、使用“ssh-keygen -t rsa”命令可以生成公私密钥对。
4、使用“ssh-copy-id 节点IP或者主机名”可以将公钥发送给相应节点。

查看评论

SSH项目实战培训精讲

-
  • 1970年01月01日 08:00

Ubuntu hadoop 伪分布式环境搭建步骤+ssh密钥(免密码登录)配置

1.关闭防火墙 查看防火墙状态 service iptables status 关闭防火墙 service iptables stop 查看防火墙开机启动状态 chkconfig iptab...
  • s646575997
  • s646575997
  • 2016-04-02 16:22:57
  • 2490

hadoop配置ssh免秘钥登录

问题:单机安装hadoop后,进行启动停止服务等操作时,需要多次输入本地密码,比较麻烦,需要配置ssh无密码登录。  问题详细描述:例如下图中关闭hadoop时候,需要输入4此本地密码。 解决方法...
  • zhenzhendeblog
  • zhenzhendeblog
  • 2016-06-08 10:12:36
  • 628

hadoop伪分布式集群ssh免密码登录

来配置我们的ssh无密码登录到slave1和slave2上1.    首先检查每个虚拟机是否安装了ssh和启动了sshd服务如果没有安装的话要执行以下代码这个命令要在root下运行,因为hadoop用...
  • Gscsd_T
  • Gscsd_T
  • 2018-03-29 14:24:15
  • 43

Hadoop实战——Hadoop架构思想、伪分布式无密登陆部署

本系列博客主要讲述Hadoop的学习心得。本篇博客主要阐述Hadoop的基本概念、处理海量数据的解决思路、Hadoop版本选择及其伪分布式集群安装,以及安装之后的hdfs、mapreduce基础演示、...
  • zhongkelee
  • zhongkelee
  • 2016-07-09 23:49:55
  • 908

linux下为hadoop开启SSH免密钥登录

hadoop节点之间的通讯是通过ssh进行的,这一点在hadoop-deamon.sh start namenode的过程中就能看出来。 SSH默认都是需要密码的,开启免密钥登录会减少很多麻烦。 ...
  • qq_20441521
  • qq_20441521
  • 2016-12-08 14:39:11
  • 798

Hadoop系列之(一)CentOS7安装配置及SSH无密码验证配置

由于最近在进行关于Hadoop的学习,开始重新搭建一套Hadoop的环境,和之前自己搭建的环境的区别在于操作系统的版本,本次采用的为CentOS7.2的版本,7相对于6的版本改动较大。一些命令都不太一...
  • triumphao
  • triumphao
  • 2016-11-21 20:50:26
  • 2479

配置ssh无密码登录、安装Java运行环境、Hadoop安装、Hadoop伪分布式配置

配置ssh无密码登录,安装Java运行环境,Hadoop安装,Hadoop伪分布式配置
  • PWK999
  • PWK999
  • 2017-12-06 16:27:30
  • 190

hadoop设置ssh免密码登录

对于需要远程管理其它机器,一般使用远程桌面或者telnet。linux一般只能是telnet。但是telnet的缺点是通信不加密,存在不安全因素,只适合内网访问。为 解决这个问题,推出了通信加密通信协...
  • zwx19921215
  • zwx19921215
  • 2014-02-21 21:59:19
  • 18792

批量配置SSH 免密钥登录脚本

[root@c3-zabbix-serv hurl]# cat sendsshkey.sh  #!/bin/bash  if [ -z $1 ]; then echo -e "\033[40;31...
  • a3470194
  • a3470194
  • 2016-01-15 14:31:38
  • 2037
    个人资料
    等级:
    访问量: 257
    积分: 98
    排名: 142万+
    文章分类
    文章存档