hadoop搭建

Hadoop安装文档 1. 配置主机名 [root@H3CUBG1XA_MASTER_V1 ~]# hostnamectl set-hostname master [root@H3CUBG1XA_MASTER_V1 ~]# bash [root@H3CUBG1XA_SLAVE1_V1 ~]# hostnamectl set-hostname slave1 [root@H3CUBG1XA_SLAVE1_V1 ~]# bash [root@H3CUBG1XA_SLAVE2_V1 ~]# hostnamectl set-hostname slave2 [root@H3CUBG1XA_SLAVE2_V1 ~]# bash 2. 配置IP地址映射 [root@master ~]# vi /etc/hosts 删除原先的所有内容 新增下面三行内容 192.168.90.157 master 192.168.90.71 slave1 192.168.90.185 slave2 分发到slave1 scp /etc/hosts root@slave1:/etc/ 输入 yes 输入密码 password 分发到slave2 scp /etc/hosts root@slave2:/etc/ 输入 yes 输入密码 password 3. 关闭防火墙 [root@master ~]# systemctl disable firewalld.service --now [root@master ~]# setenforce 0 [root@slave1 ~]# systemctl disable firewalld.service --now [root@slave1 ~]# setenforce 0 [root@slave2 ~]# systemctl disable firewalld.service --now [root@slave2 ~]# setenforce 0 4. 配置免密登录 切换到hadoop用户 [root@master ~]# su - hadoop [root@slave1 ~]# su - hadoop [root@slave2 ~]# su - hadoop 删除干净ssh [hadoop@master ~]# rm -rf ~/.ssh [hadoop@slave1 ~]# rm -rf ~/.ssh [hadoop@slave2 ~]# rm -rf ~/.ssh 生成密钥,一路enter [hadoop@master ~]# ssh-keygen [hadoop@slave1 ~]# ssh-keygen [hadoop@slave2 ~]# ssh-keygen 每个节点把密钥移动到第一个节点上 [hadoop@master ~]# ssh-copy-id master 输入yes 输入密码 password [hadoop@slave1 ~]# ssh-copy-id master 输入yes 输入密码 password [hadoop@slave2 ~]# ssh-copy-id master 输入yes 输入密码 password master节点查看是否有三个钥匙 [hadoop@master ~]# cat ~/.ssh/authorized_keys master节点复制密钥给第二个和第三个节点 [hadoop@master ~]# scp -r ~/.ssh/authorized_keys slave1:~/.ssh/ 输入yes 输入密码 password [hadoop@master ~]# scp -r ~/.ssh/authorized_keys slave2:~/.ssh/ 输入yes 输入密码 password 5. 配置JDK 切换到root用户 [hadoop@master ~]# exit [hadoop@slave1 ~]# exit [hadoop@slave2 ~]# exit 解压jdk安装包 [root@master ~]# tar -zxvf /opt/software/jdk-8u152-linux-x64.tar.gz -C /usr/local/src/ 改名 [root@master ~]# mv /usr/local/src/jdk1.8.0_152/ /usr/local/src/java 配环境变量 [root@master ~]# vi /etc/profile 再文件最下方加入 # JAVA_HOME export JAVA_HOME=/usr/local/src/java export PATH=$PATH:$JAVA_HOME/bin 保存 使环境变量立即生效 [root@master ~]# source /etc/profile 分发环境变量文件到slave1和slave2 [root@master ~]# scp /etc/profile slave1:/etc/ 输入yes 输入密码 password [root@master ~]# scp /etc/profile slave2:/etc/ 输入yes 输入密码 password 分发jdk解压包到slave1和slave2 [root@master ~]# scp -r /usr/local/src/java slave1:/usr/local/src/ 输入密码 password [root@master ~]# scp -r /usr/local/src/java slave2:/usr/local/src/ 输入yes 输入密码 password 6. 配置Hadoop 解压hadoop安装包 [root@master ~]# tar -zxvf /opt/software/hadoop-2.7.1.tar.gz -C /usr/local/src/ 改名 [root@master ~]# mv /usr/local/src/hadoop-2.7.1 /usr/local/src/hadoop 设置hadoop环境变量 [root@master ~]# vi /etc/profile 再文件最下方加入 # HADOOP_HOME export HADOOP_HOME=/usr/local/src/hadoop export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin 分发新的环境变量配置文件到slave1和slave2 [root@master ~]# scp /etc/profile slave1:/etc/ 输入密码 password [root@master ~]# scp /etc/profile slave2:/etc/ 输入密码 password 切换到修改文件的配置目录 [root@master ~]# cd /usr/local/src/hadoop/etc/hadoop/ 1.修改hadoop-env.sh文件 [root@master hadoop]# vi hadoop-env.sh 在第25行 export JAVA_HOME=/usr/local/src/java 2.修改core-site.xml文件 [root@master hadoop]# vi core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/src/hadoop/tmp</value> </property> 3.修改hdfs-site.xml文件 [root@master hadoop]# vi hdfs-site.xml <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/src/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/src/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> 4.修改yarn-site.xml文件 [root@master hadoop]# vi yarn-site.xml <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> 5.修改mapred-site.xml.template文件 [root@master hadoop]# mv mapred-site.xml.template mapred-site.xml [root@master hadoop]# vi mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> 6.修改slaves文件 [root@master hadoop]# vi slaves 删除原文件的所有内容 加入 master slave1 slave2 创建数据存储目录 [root@master hadoop]# mkdir /usr/local/src/hadoop/tmp [root@master hadoop]# mkdir -p /usr/local/src/hadoop/dfs/name [root@master hadoop]# mkdir -p /usr/local/src/hadoop/dfs/data 更改hadoop目录所属用户权限 [root@master ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/ 更改java目录所属用户权限 [root@master ~]# chown -R hadoop:hadoop /usr/local/src/java/ 分发hadoop解压包到slave1和slave2 [root@master ~]# scp -r /usr/local/src/hadoop slave1:/usr/local/src/ 输入密码 password [root@master ~]# scp -r /usr/local/src/hadoop slave2:/usr/local/src/ 输入密码 password slave1和slave2修改hadoop解压包权限 [root@slave1 ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/ [root@slave2 ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/ slave1和slave2修改java解压包权限 [root@slave1 ~]# chown -R hadoop:hadoop /usr/local/src/java/ [root@slave2 ~]# chown -R hadoop:hadoop /usr/local/src/java/ 切换到hadoop用户 [root@master ~]# su - hadoop [root@slave1 ~]# su - hadoop [root@slave2 ~]# su - hadoop 使环境变量生效 [hadoop@master ~]# source /etc/profile [hadoop@slave1 ~]# source /etc/profile [hadoop@slave2 ~]# source /etc/profile 7. 初始化 [hadoop@master ~]# hadoop namenode -format 启动进程 [hadoop@master ~]# start-all.sh 查看master、slave1、slave2的进程 master下 NameNode DataNode SecondaryNameNode ResourceManager NodeManager slave1下 DataNode NodeManager slave2下 DataNode NodeManager 浏览器查看 http://master:50070 8. xcall命令 1.切换目录 [root@master ~]# cd /usr/local/bin 2.创建文件 [root@master bin]# vi xcall 3.加入下面的内容 #!/bin/bash pcount=$# if((pcount==0)); then echo "command can not be null !" exit fi user=`whoami` source /etc/profile for host in master slave1 slave2 do echo ================ $host ================= ssh $host $@ done 4.修改权限 [root@master bin]# chmod 777 xcall 5.添加JPS软链接 [root@master bin]# ln -s /usr/local/src/java/bin/jps jps 6.分发xcall文件到slave1和slave2 [root@master bin]# scp /usr/local/bin/xcall slave1:/usr/local/bin/ [root@master bin]# scp /usr/local/bin/xcall slave2:/usr/local/bin/ 7.slave1添加软链接 [root@slave1 ~]# cd /usr/local/bin [root@slave1 bin]# ln -s /usr/local/bin/java/bin/jps jps 8.slave2添加软链接 [root@slave2 ~]# cd /usr/local/bin [root@slave2 bin]# ln -s /usr/local/bin/java/bin/jps jpsS