一、修改hosts文件
在主节点,就是第一台主机的命令行下;
vim /etc/hosts
我的是三台云主机:
在原文件的基础上加上;
ip1 master worker0 namenode
ip2 worker1 datanode1
ip3 worker2 datanode2
其中的ipN代表一个可用的集群IP,ip1为master的主节点,ip2和iip3为从节点。
二、ssh互信(免密码登录)
注意我这里配置的是root用户,所以以下的家目录是/root
如果你配置的是用户是xxxx,那么家目录应该是/home/xxxxx/
#在主节点执行下面的命令:
ssh-keygen -t rsa -P '' #一路回车直到生成公钥
scp /root/.ssh/id_rsa.pub root@worker1:/root/.ssh/id_rsa.pub.master #从master节点拷贝id_rsa.pub到worker主机上,并且改名为id_rsa.pub.master scp /root/.ssh/id_rsa.pub root@worker1:/root/.ssh/id_rsa.pub.master #同上,以后使用workerN代表worker1和worker2.
scp /etc/hosts root@workerN:/etc/hosts #统一hosts文件,让几个主机能通过host名字来识别彼此
#在对应的主机下执行如下命令: cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys #master主机 cat /root/.ssh/id_rsa.pub.master >> /root/.ssh/authorized_keys #workerN主机
这样master主机就可以无密码登录到其他主机,这样子在运行master上的启动脚本时和使用scp命令时候,就可以不用输入密码了。
三、安装基础环境(JAVA和SCALA环境)
1.Java1.8环境搭建:
配置master的java环境
卸载本机自带的jdk:
查找rpm -qa | grep jdk; 卸载 yum -y remove java-1.4.2-gcj-compat-1.4.2.0-40jpp.115
#下载jdk1.8的rpm包
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.rpm rpm -ivh jdk-8u112-linux-x64.rpm
#增加JAVA_HOME
vim etc/profile
#增加如下行: #Java home export JAVA_HOME=/usr/java/jdk1.8.0_112/
#刷新配置:
source /etc/profile #当然reboot也是可以的
配置workerN主机的java环境
#使用scp命令进行拷贝 scp jdk-8u112-linux-x64.rpm root@workerN:/root
#其他的步骤如master节点配置一样
2.Scala2.11.8环境搭建:
Master节点:
#下载scala安装包: wget -O "scala-2.11.8.rpm" "https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.rpm"
#安装rpm包: rpm -ivh scala-2.11.8.rpm
#增加SCALA_HOME vim /etc/profile
#增加如下内容; #Scala Home export SCALA_HOME=/usr/share/scala #刷新配置 source /etc/profile
WorkerN节点;
#使用scp命令进行拷贝 scp scala-2.11.8.rpm root@workerN:/root #其他的步骤如master节点配置一样
四、Hadoop2.8.2完全分布式搭建
MASTER节点:
1.下载二进制包:
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.8.2/hadoop-2.8.2.tar.gz
2.解压并移动至相应目录
我的习惯是将软件放置/opt目录下:
tar -xvf /opt/hadoop-2.8.2.tar.gz mv hadoop-2.8.2 /opt
3.修改相应的配置文件:
(1)/etc/profile:
增加如下内容:
#hadoop enviroment export HADOOP_HOME=/opt/hadoop-2.8.2/ export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH" export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
(2)$HADOOP_HOME/etc/hadoop/hadoop-env.sh
修改JAVA_HOME 如下:
export JAVA_HOME=/usr/java/jdk1.8.0_112/
(3)$HADOOP_HOME/etc/hadoop/slaves(注意:只针对master配置如下slaves,其他slaves清楚掉)
worker1
worker2
(4)$HADOOP_HOME/etc/hadoop/core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop-2.8.2/tmp</value> </property> </configuration>
(5)$HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/opt/hadoop-2.8.2/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/opt/hadoop-2.8.2/hdfs/data</value> </property> </configuration>
(6)$HADOOP_HOME/etc/hadoop/mapred-site.xml
复制template,生成xml:
cp mapred-site.xml.template mapred-site.xml
内容:
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:19888</value> </property> </configuration>
(7)$HADOOP_HOME/etc/hadoop/yarn-site.xml
<!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:8088</value> </property>
至此master节点的hadoop搭建完毕
再启动之前我们需要
格式化一下namenode,注意如果多次执行格式化会导致ClusterId不一致,此时应删除hadoop-2.8.2/hdfs下所有目录
hdfs namenode -format
WorkerN节点:
(1)复制master节点的hadoop文件夹到worker上:
scp -r /opt/hadoop-2.8.2 root@workerN:/opt #注意这里的N要改为1或者2
(2)修改/etc/profile:
过程如master一样
五、Spark2.2.1完全分布式环境搭建:
MASTER节点:
1.下载文件:
wget -O "spark-2.2.1-bin-hadoop2.7.tgz" "http://d3kbcqa49mib13.cloudfront.net/spark-2.2.1-bin-hadoop2.7.tgz"
2.解压并移动至相应的文件夹;
tar -xvf /opt/spark-2.2.1-bin-hadoop2.7.tgz mv spark-2.2.1-bin-hadoop2.7 /opt
3.修改相应的配置文件:
(1)/etc/profie
#Spark enviroment export SPARK_HOME=/opt/spark-2.1.2-bin-hadoop2.7/ export PATH="$SPARK_HOME/bin:$PATH"
(2)$SPARK_HOME/conf/spark-env.sh
cp spark-env.sh.template spark-env.sh
#配置内容如下:
export SCALA_HOME=/usr/share/scala export JAVA_HOME=/usr/java/jdk1.8.0_112/ export SPARK_MASTER_IP=master export SPARK_WORKER_MEMORY=1g export HADOOP_CONF_DIR=/opt/hadoop-2.8.2/etc/hadoop
(3)$SPARK_HOME/conf/slaves
cp slaves.template slaves
配置内容如下
master
worker1
worker2
WorkerN节点:
将配置好的spark文件复制到workerN节点
scp -r /opt/spark-2.2.1-bin-hadoop2.7 root@workerN:/opt
修改/etc/profile,增加spark相关的配置,如MASTER节点一样
六、启动集群的脚本
启动集群脚本start-cluster.sh如下:
#!/bin/bash echo -e "\033[31m ========Start The Cluster======== \033[0m" echo -e "\033[31m Starting Hadoop Now !!! \033[0m" /opt/hadoop-2.8.2/sbin/start-dfs.sh /opt/hadoop-2.8.2/sbin/start-yarn.sh echo -e "\033[31m Starting Spark Now !!! \033[0m" /opt/spark-2.1.2-bin-hadoop2.7/sbin/start-all.sh echo -e "\033[31m The Result Of The Command \"jps\" : \033[0m" jps echo -e "\033[31m ========END======== \033[0m"
截图如下:
关闭集群脚本stop-cluser.sh如下:
#!/bin/bash echo -e "\033[31m ===== Stoping The Cluster ====== \033[0m" echo -e "\033[31m Stoping Spark Now !!! \033[0m" /opt/spark-2.1.0-bin-hadoop2.7/sbin/stop-all.sh
/opt/hadoop-2.8.2/sbin/stop-dfs.sh /opt/hadoop-2.8.2/sbin/stop-yarn.shecho -e " \033[31m Stopting Hadoop Now !!! \033[0m "/opt/hadoop- 2.7. 3/sbin/stop-all. sh echo -e " \033[31m The Result Of The Command \"jps\" : \033[0m " jps echo -e " \033[31m ======END======== \033[0m "
截图如下:
七、常见问题处理
1、 hdfs nodename -format 格式化一次即可,如果需要格式多次需要删除调/hadoop-2.8.2/hdfs下所有文件夹,防止ClusterID不一致
2、 Hadoop jps出现process information unavailable提示解决办法
3、process information unavailable
八、测试一下集群:
1、集群状态查看
这里我都用最简单最常用的Wordcount来测试好了!
2. 关闭集群防火墙
关闭防火墙
[root@lvxinghao ~]# systemctl stop firewalld[root@lvxinghao ~]# systemctl status firewalld
禁用防火墙(系统启动时不启动防火墙服务)
[root@lvxinghao ~]# systemctl disable firewalld
[root@lvxinghao ~]# systemctl is-enabled firewalld
3、测试hadoop
在/opt目录下新建文件wordcount.txt并输入内容,测试的源文件的内容为:
Hello hadoop
hello spark
hello bigdata
然后执行下列命令:
hadoop fs -mkdir -p /Hadoop/Input hadoop fs -put /opt/wordcount.txt /Hadoop/Input hadoop jar /opt/hadoop-2.8.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.2.jar wordcount /Hadoop/Input /Hadoop/Output
等待mapreduce执行完毕后,查看结果;
查看文件目录指令:hadoop fs -ls -R /
hadoop fs -cat /Hadoop/Output/*
hadoop集群搭建成功!
4.测试spark
为了避免麻烦这里我们使用spark-shell,做一个简单的worcount的测试
用于在测试hadoop的时候我们已经在hdfs上存储了测试的源文件,下面就是直接拿来用就好了!
spark-shell
val file=sc.textFile("hdfs://master:9000/Hadoop/Input/wordcount.txt") val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_) rdd.collect() rdd.foreach(println)
5、文件目录读取:
spark:
val alldata = sc.textFile("data/Flag/*/part-*")
println(alldata.count())
退出的话使用如下命令:
:quit