搭建Hadoop分布式集群一
条件
hadoop-2.6.0.tar.gz
结构图
步骤:
1.Hadoop文件上传,解压缩
上传 hadoop-2.6.0.tar.gz 到/opt目录下
解压到/usr/local目录下
tar -zxf hadoop-2.6.0.tar.gz -C /usr/local
一些常用的目录解读
2.修改配置文件
cd /usr/local/hadoop-2.6.0/etc/hadoop
- core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/log/hadoop/tmp</value>
</property>
</configuration>
- hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
- mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
- hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
- yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
</configuration>
- yarn-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
- slaves
删除localhost,添加:
slave1
slave2
slave3
3.设置IP映射
vim /etc/hosts
添加
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com
4.克隆slave等三台主机
克隆slave1,slave2,slave3
同样克隆出slave2和slave3
更改三台主机的设置
如slave1:
(1)删除文件
rm -rf /etc/udev/rules.d/70-persistent-net.rules
(2)记住值(机器不同,值不同)
(3)更改主机ip
cd /etc/sysconfig/network-scripts/
vim ifcfg-eth0
看是否成功Ping 通
service network restart
ping www.baidu.com
(4)修改机器名
vim /etc/sysconfig/network
(5)重启虚拟机
reboot
(6)在master验证
ping slave1
克隆虚拟机,如果在配置前已提前克隆,则需拷贝hadoop安装文件到集群slave节点
scp -r /usr/local/hadoop-2.6.0 slave1:/usr/local
scp -r /usr/local/hadoop-2.6.0 slave2:/usr/local
scp -r /usr/local/hadoop-2.6.0 slave3:/usr/local
5.配置SSH无密码登录
(1)使用ssh-keygen产生公钥与私钥对。
输入命令“ssh-keygen -t rsa”,接着按三次Enter键
[root@master ~]# ssh-keygen -t rsa
生成私有密钥id_rsa和公有密钥id_rsa.pub两个文件
(2)用ssh-copy-id将公钥复制到远程机器中
ssh-copy-id -i /root/.ssh/id_rsa.pub master//依次输入yes,123456(root用户的密码)
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
(3)验证是否设置无密码登录
依次输入
ssh slave1
ssh slave2
ssh slave3
参考书籍
《教材-Hadoop大数据开发基础》 ----人民邮电出版社