环境:
centos7 桌面版。
地址:192.168.1.60 hadoop1 , 192.168.1.61 hadoop2 , 192.168.1.62 hadoop3
1.三台服务器关闭防火墙
> systemctl status firewalld
> systemctl stop firewalld
>systemctl disables firewalld
2 . 免密登录
三天服务器在root用户下都生成密钥
>ssh-keygen -t rsa
全部默认回车 ,可看到生成一个 ".ssh"的文件夹
>cd .ssh
有两个文件 私钥:id_rsa(自己保留) 公钥:id_rsa.pub (发送给别的服务器)
> scp id_rsa.pub root@192.168.1.61:/root/.ssh/id_rsa.pub.hadoop1
> scp id_rsa.pub root@192.168.1.62:/root/.ssh/id_rsa.pub.hadoop1
期间输入“yes”,和 相关服务器密码
分别在hadoop2,hadoop3中操作:
>touch authorized_keys
>chmod 600 authorized_keys
>cat id_ras.pub.hadoop1 >> authotized_keys
然后hadoop1 就可以免密登录hadoop2. hadoop3.了
3 。 安装java
查看centos7系统中自带的java安装包
>rpm -qa |grep java
删除自带的java版本 rpm -e --nodeps 文件名
查看留下的安装包
>cd /root/yao
>tar -zxvf jdk-8u131-linux-x64.tar.gz
> cd jdk1.8.0_131
> pwd
/root/yao/jdk1.8.0_131 (将该路径写到环境变量里边)
>vim /etc/profile
export JAVA_HOME=/root/yao/jdk1.8.0_131
export PATH=$PATH:$JAVA_HOME/bin
刷新该文件 。使修改生效
>source /etc/profile
查看当前使用java
>java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
(表示安装成功)
4 。安装hadoop
在hadoop1中安装
>cd /root/yao
>wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.9.3.tar.gz
> tar -zxvf hadoop-2.6.0-cdh5.9.3.tar.gz
> cd hadoop-2.6.0-cdh5.9.3/
>vim slaves
hadoop2
hadoop3
>cd ../../
>mkdir data
>mkdir tmp
>mkdir dfs
>cd dfs
>mkdir name
>mkdir data
>cd ../../
> cd etc/hadoop
>vim core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/yao/hadoop-2.6.0-cdh5.9.3/data/tmp/</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:8020</value>
</property>
</configuration>
>vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop1:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/root/yao/hadoop-2.6.0-cdh5.9.3/data/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/root/yao/hadoop-2.6.0-cdh5.9.3/data/tmp/dfs/data</value>
</property>
</configuration>
>cp mapred-site.xml.template mapred-site.xml
>vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
> vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
>vim hadoop-env.sh
export JAVA_HOME=/root/yao/jdk1.8.0_131
将hadoop1中配置的整个hadoop文件夹复制到hadoop1 ,hadoop2中
>cd /root/yao
>scp -r hadoop-2.6.0-cdh5.9.3 hadoop2:/root/yao/
>scp -r hadoop-2.6.0-cdh5.9.3 hadoop3:/root/yao/
5 .启动 hadoop集群
>cd /root/yao/hadoop-2.6.0-cdh5.9.3/
>bin/hdfs namenode -format
>sbin/start-dfs.sh
>sbin/start-yarn.sh
6 。 检测
web页面检测
在浏览器中打开:
HDFS管理界面: http://192.168.1.60:50070/
YARN管理界面: http://192.168.1.60:8088/
基准测试
7. hadoop官方使用文档
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.9.3/
8 。启动过程中出现的警告
a .Unable to load native-hadoop library for your platform
参考:https://blog.csdn.net/jack85986370/article/details/51902871
因为apache 提供的hadoop本地库32位 ,自己手动换成64位包。
http://dl.bintray.com/sequenceiq/sequenceiq-bin/
(1)下载相应版本的64lib包
(2)解压到lib/native 和lib目录下
> tar -xvf /root/yao/hadoop-native-64-2.6.0.tar -C /root/yao/hadoop-2.6.0-cdh5.9.3/lib/native
> tar -xvf /root/yao/hadoop-native-64-2.6.0.tar -C /root/yao/hadoop-2.6.0-cdh5.9.3/lib
(3) 添加到环境变量
>vim /etc/profile
export HADOOP_HOME=/root/yao/hadoop-2.6.0-cdh5.9.3/
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
>source /etc/profile
(4)检查
>cd /root/yao/hadoop-2.6.0-cdh5.9.3/bin
>./hadoop checknative -a
问题解决
b.common.Util: Path /should be specified as a URI inconfiguration files. Please update hdfs configura
参考:https://blog.csdn.net/qq_22937011/article/details/52382794?locationNum=14&fps=1
> vim hdfs-site.xml
(将原value值的路径前加上 : “file: ”)
<name>dfs.datanode.data.dir</name>
<value>file:/root/yao/hadoop-2.6.0-cdh5.9.3/data/tmp/dfs/data</value>
问题解决
至此 : 搭建hadoop集群完成