hadoop集群的搭建
安装一台Linux虚拟机
-
配置网络
vi /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0 TYPE=Ethernet ONBOOT=yes //开机自启 NM_CONTROLLED=yes BOOTPROTO=static //静态ip IPADDR=192.168.2.11 //ip地址 PERFIX=24 //子网掩码长度 GATEWAY=192.168.2.1 //网关 DNS1=192.168.2.1 //首选DNS DNS2=114.114.114.114 //次选DNS
要将之前存在的HWADDR与UUID两行删除,因为之后要进行克隆
2.配置主机名
vi /etc/sysconfig/networkHOSTNAME=hadoop01
3.安装ssh客户端
yum install -y openssh-clients
4关闭防火墙
service iptables stop //关闭防火墙 chkconfig iptables off //开机不自启
5.重启后测试网络
ping www.baidu.com
6.安装ssh服务
克隆
-
克隆虚拟机 (此处只列一台)
然后完成 -
修改克隆出来的虚拟机网卡
vi /etc/udev/rules.d/70-persistent-net.rules
将NAME="eth0"的三行删除,将NAME="eth1"改为NAME=“eth0” -
配置网络
vi /etc/sysconfig/network-scripts/ifcfg-eth0
仅需修改ip地址
hosts映射
vi /etc/hosts
免密登录
-
方法一: (建议)
一对多免密
(前提是在 /etc/hosts映射文件里添加过这些免密机器的映射,并且这些机器的登录密码相同)autossh.sh 自己建一个脚本,复制如下代码,将PWD_1改为自己虚拟机设置的密码(本人为123456)
#!/bin/bash #yum安装expect yum -y install expect #PWD_1是登陆密码,可以自己设定 PWD_1=123456 ips=$(cat /etc/hosts |grep -v "::" | grep -v "127.0.0.1") key_generate() { expect -c "set timeout -1; spawn ssh-keygen -t rsa; expect { {Enter file in which to save the key*} {send -- \r;exp_continue} {Enter passphrase*} {send -- \r;exp_continue} {Enter same passphrase again:} {send -- \r;exp_continue} {Overwrite (y/n)*} {send -- n\r;exp_continue} eof {exit 0;} };" } auto_ssh_copy_id () { expect -c "set timeout -1; spawn ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@$1; expect { {Are you sure you want to continue connecting *} {send -- yes\r;exp_continue;} {*password:} {send -- $2\r;exp_continue;} eof {exit 0;} };" } # rm -rf ~/.ssh key_generate for ip in $ips do auto_ssh_copy_id $ip $PWD_1 done
给脚本添加执行权限
chmod u+x autossh.sh
然后运行脚本
./autossh.sh
-
方法二:
一对一免密,参考个人博客
https://blog.csdn.net/qq_38671360/article/details/83515380
安装jdk
tar -zxvf jdk-8u102-linux-x64.tar.gz -C /usr/local/
安装hadoop
-
解压下载的压缩包
tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local/
-
修改配置文件
cd hadoop-2.7.3.tar.gz/etc/hadoop/
最简化配置如下vi hadoop-env.sh
将=后的值改为绝对路径# The java implementation to use. export JAVA_HOME=/usr/local/jdk1.8.0_102
vi core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop01:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop-2.7.3/tmp</value> </property> </configuration>
vi hdfs-site.xml
<configuration> <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoop-2.7.3/data/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoop-2.7.3/data/data</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.secondary.http.address</name> <value>hadoop01:50090</value> </property> </configuration>
将mapred-site.xml.template复制一份并改名为mapred-site.xml
cp mapred-site.xml.template mapred-site.xmlvi mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
vi yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop01</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
vi slaves
hadoop02 hadoop03
配置环境变量
vi /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_102
export HADOOP_HOME=/usr/local/hadoop-2.7.3
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
使环境变量生效
source /etc/profile
把第一台的 /etc/profile、hosts、安装好的jdk和hadoop发送给另外几台
scp -r /etc/profile hadoop02:/etc/
scp -r /etc/hosts hadoop02:/etc/
scp -r /usr/local/jdk1.8.0_102/ hadoop02:/usr/local/
scp -r /usr/local/hadoop-2.7.3/ hadoop02:/usr/local/
启动集群
-
初始化HDFS:
在hadoop01进行操作(操作一次就ok)hadoop namenode -format
-
启动hdfs集群:
start-dfs.sh
-
启动yarn集群:
start-yarn.sh
-
通过jps命令查看启动后的进程:
hadoop01:
hadoop02:
通过网页查看启动是否成功:
hadoop01:50070
到此一个最简配置的hadoop集群就搭建完成了!( * ^ ▽ ^ * )