目标: 配置一个hadoop 1.2.1 测试环境
用的JDK是: jdk-7u65-linux-x64.gz
选的hadoop 是: hadoop-1.2.1.tar.gz
均来源Apache和oracle网站
主机规划:
Linux版本 : Centos 6.5 x64位
/boot:用来存放与Linux系统启动有关的程序,比如启动引导装载程序等,建议大小为100MB。
/usr:用来存放Linux系统中的应用程序,其相关数据较多,建议大于3GB以上。
/var:用来存放Linux系统中经常变化的数据以及日志文件,建议大于1GB以上。
/home:存放普通用户的数据,是普通用户的宿主目录,建议大小为剩下的空间。
/:Linux系统的根目录,所有的目录都挂在这个目录下面,建议大小为5GB以上。
/tmp:将临时盘在独立的分区,可避免在文件系统被塞满时影响到系统的稳定性。建议大小为500MB以上。
swap:实现虚拟内存,建议大小是物理内存的1~2倍。
Boot Loader密码是 D*****2014
安装选择 Basic Server 不然所有的包得一个个打,确实很麻烦.
配置sudo ,赋予hadoop用户权限 ,并配置sudo免密
useradd hadoop
passwd hadoop
vi /etc/sudoers
root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL 添加一行
hadoop ALL=(ALL) NOPASSWD: ALL 添加一行免密码
配置本机的解析
[hadoop@master /]$ cat /etc/hosts 10.15.5.200 master.hadoop 10.15.5.201 slave01.hadoop 10.15.5.202 slave02.hadoop
配置主机名
vi etc/sysconfig/network HOSTNAME=master.hadoop
配置IP 地址
[hadoop@master /]$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet UUID=721f9261-45d5-4335-9b47-64459173b2a9 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=none HWADDR=00:50:56:82:00:0F IPADDR=10.15.5.200 PREFIX=24 GATEWAY=10.15.5.1 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0"
挂载 安装盘,以便所需要的安装包。打上ftp包 方便安装软件
[root@master home]# mount -t auto /dev/cdrom /home/cdrom mount: block device /dev/sr0 is write-protected, mounting read-only [root@master Packages]# rpm -ivh ftp-0.17-54.el6.x86_64.rpm warning: ftp-0.17-54.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Preparing... ########################################### [100%] 1:ftp ########################################### [100%]
目标: master和slaves hadoop账户无密码互访
原理:
可以粗糙的理解为,我制造一把钥匙,把样子发给你,你在门禁系统里记下我钥匙的样子,我拿这把钥匙去开门时候你的门禁系统直接放行。
Master上,登录hadoop用户,并在/home/hadoop目录下操作
[hadoop@master ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 8d:81:09:c8:45:3f:c0:fb:3b:a0:cf:95:b6:dd:e9:b1 hadoop@master.hadoop The key's randomart p_w_picpath is: +--[ RSA 2048]----+ | . == | | o .+ o | | .= . | | . . + | | . S . | | . .. | | . .+. . | | .. ooo . + | | .o ....E | +-----------------+
该隐藏文件夹下,生成了一个 id_rsa.pub文件
[hadoop@master .ssh]$ ls -al /home/hadoop/.ssh total 16 drwx------. 2 hadoop hadoop 4096 Jul 30 22:09 . drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 .. -rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa -rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub [hadoop@master .ssh]$ cat id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== hadoop@master.hadoop
把这个文件的内容,追加到 authorized_keys 的文件里
[ hadoop@master .ssh]$ cat id_rsa.pub >> authorized_keys [ hadoop@master .ssh]$ ls authorized_keys id_rsa id_rsa.pub [ hadoop@master .ssh]$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAugxDdYElWX8URmERVTPsNtSW2mZoEMZyzmIKEe3UQjMbgYOCx0jV15/vtnOXJf4K+S6hccajih2oemcC4BmMI99nFWcyd9zcrvfJvn/DZHnw0Yog6MyMd9Qw2BQWUL265dhW2FNCaECuyG2U1CxR0W9wZlVz54JlToCRX6yUvWZzzGqUw/oR3ZWE7pUpQiOHV0ZNypUtbWEw/zj7N01LbVskNLqyY164APIVfZBONPMXjs/H6B8/VcxsA0lDwAqndnMxJ1iQHFkMntNgqcLkhs8oAmvl+/A6NH2I0ZBW+VOCUIjbNKNRO9BKwGvUQUZGcKthEPU0jK5ErsS6RPBrmQ== hadoop@master.hadoop
此时需要修改authorized_keys的文件权限 ,不安全的设置安全设置,会让你不能使用RSA功能
[ hadoop@master .ssh]$ ls -al total 20 drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 . drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 .. -rw-rw-r--. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys -rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa -rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub [ hadoop@master .ssh]$ sudo chmod 600 authorized_keys [ hadoop@master .ssh]$ ls -al total 20 drwx------. 2 hadoop hadoop 4096 Jul 30 22:20 . drwx------. 3 hadoop hadoop 4096 Jul 30 22:09 .. -rw-------. 1 hadoop hadoop 402 Jul 30 22:20 authorized_keys -rw-------. 1 hadoop hadoop 1671 Jul 30 22:09 id_rsa -rw-r--r--. 1 hadoop hadoop 402 Jul 30 22:09 id_rsa.pub
修改 /etc/ssh/sshd_config 并且确认其中三个选项 然后sudo service sshd restart 服务
[ hadoop@master .ssh]$ sudo cat /etc/ssh/sshd_config RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys
在所有Slave上建立 .ssh文件夹,改变权限
[hadoop@slave02 ~]$ mkdir .ssh [hadoop@slave02 ~]$ ls -al total 32 drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 . drwxr-xr-x. 4 root root 4096 Jul 30 20:10 .. -rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history -rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout -rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile -rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc drwxrwxr-x. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh -rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo [hadoop@slave02 ~]$ sudo chmod 700 .ssh [hadoop@slave02 ~]$ ls -al total 32 drwx------. 3 hadoop hadoop 4096 Jul 30 22:47 . drwxr-xr-x. 4 root root 4096 Jul 30 20:10 .. -rw-------. 1 hadoop hadoop 401 Jul 30 20:58 .bash_history -rw-r--r--. 1 hadoop hadoop 18 Jul 18 2013 .bash_logout -rw-r--r--. 1 hadoop hadoop 176 Jul 18 2013 .bash_profile -rw-r--r--. 1 hadoop hadoop 124 Jul 18 2013 .bashrc drwx------. 2 hadoop hadoop 4096 Jul 30 22:47 .ssh -rw-------. 1 hadoop hadoop 557 Jul 30 20:28 .viminfo [hadoop@slave02 ~]$
把id_rsa.pub发送给slaves
[ hadoop@master .ssh]$ scp id_rsa.pub hadoop@slave01.hadoop :/home/hadoop/.ssh/id_rsa_frommaster.pub The authenticity of host 'slave01.hadoop (10.15.5.201)' can't be established. RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'slave01.hadoop,10.15.5.201' (RSA) to the list of known hosts. hadoop@slave01.hadoop 's password: id_rsa.pub 100% 402 0.4KB/s 00:00 [ hadoop@master .ssh]$ scp id_rsa.pub hadoop@slave02.hadoop :/home/hadoop/.ssh/id_rsa_frommaster.pub The authenticity of host 'slave02.hadoop (10.15.5.202)' can't be established. RSA key fingerprint is 76:14:2f:f9:d9:03:07:17:7c:d1:ad:1e:af:55:45:00. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'slave02.hadoop,10.15.5.202' (RSA) to the list of known hosts. hadoop@slave02.hadoop 's password: id_rsa.pub 100% 402 0.4KB/s 00:00
把这个值追加到slaves上的authorized_keys里面,并修改安全权限
[hadoop@slave02 .ssh]$ cat id_rsa_frommaster.pub >> authorized_keys [hadoop@slave02 .ssh]$ sudo chmod 600 authorized_keys
修改slaves主机上的 /etc/ssh/sshd_config内容,并重启sshd服务
[hadoop@slave01 .ssh]$ sudo cat /etc/ssh/sshd_config RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys [hadoop@slave01 .ssh]$ sudo service sshd restart Stopping sshd: [ OK ] Starting sshd: [ OK ]
最终验证方法
[hadoop@master .ssh]$ ssh slave01.hadoop Last login: Wed Jul 30 23:00:25 2014 from master.hadoop [hadoop@slave01 ~]$ logout Connection to slave01.hadoop closed. [hadoop@master .ssh]$ ssh slave02.hadoop [hadoop@slave02 ~]$ logout Connection to slave02.hadoop closed.
同理: 从slaves 到master
slaves 生成公钥
把slaves把Slave的公钥追加到Master的".ssh"文件夹下的"authorized_keys"中
[hadoop@slave01 .ssh]$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqkRUrgiyD3W36WPRpL9ifJjgnVe+1R4x4mRhimKXOfmW+D3dcbi9FwE2j0/H+NaMdLWjQYw685ITdfHni0X5La7YXy6EIe0FQB/NxDKVslc44RuzJQJKQkosqSY/HGVfefF7OzIFCEcqVQDqN+OpWlrbXNtmO1UUhG2TFvj3MsbgwWHf7fc+uSN7Y7bMzLJpKhAVEjljjyTaHekJ8WMNzgT160vgr0mIZawDlRxKRlS2HtWQnndf74ZJDCeQKgJA6RUKuqBLop5x/0lvHBbUkn3MADnJUlIzd5pJW/aFEVySypfkGZTspj3+m8gNfQkqYJiB6sBfFgUYt1iPL/GNRw== hadoop@slave01.hadoop
[hadoop@slave02 ~]$ cat .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3mHGmqOGjIkC/Elc/4CopgvKCQ7AdtqV87dhQPWnDdlljePVY/GbCWYqp4h8CmtmsMNreTva8RLBPSIP3jGMrbvNDCCuMo+mZGa034yR6JwekI9zVikzYscCTqWCk6w5Hs3u/pnb1Ym6a46ho+DNEm42qIaYzrhRn9Fc1F9Hd3/dXRq0KZlO/5xMKuhFt1GKy+gs+l7MiK6y7pTISx+ox/mDjDQfzxfpguRO68XX54+dD0gONSb/MaVVMdqiWK3Fh88ouN23sKi/cnzD1vLSm55kpT6ZRo792qcFmMe7CIYc6dwdkKZBgkdheayWy5PPfsZrGfDd/erVCWrmofDhiw== hadoop@slave02.hadoop
可以用cat >>的方式追加,或者vi直接写入。钥匙么,让master知道就行了.
转载于:https://blog.51cto.com/ciscolang/1538827