Spark 分布式环境搭建
准备
- 两台华为云服务器(一台Master,一台Worker)
- Ubantu20系统
- 软件 :
jdk1.8
hadoop-2.7.3.tar
scala-2.13.0.tgz
spark-2.1.0-bin-hadoop2.7.tgz
网络配置
vim /etc/hosts
配置网络别名
192.168.0.237 Master
192.168.0.14 Worker1
配置ssh无密码登录worker
为了避免缓存干扰,可以先删除Master和worker1的.ssh文件夹
rm -rf ~/.ssh
ssh worker1
#此时在worker1
rm -rf ~/.ssh
在服务器上使用ssh-keygen -t rsa
重新生成.ssh文件夹
#在Master执行,生成公钥
scp /root/.ssh/id_rsa.pub root@worker1:/root/id_rsa.pub.master
#从master节点拷贝id_rsa.pub到worker主机上,并且改名为id_rsa.pub.master
scp /etc/hosts root@worker1:/etc/hosts
#在对应的主机下执行如下命令:
cat /root/id_rsa.pub.master >> /root/.ssh/authorized_keys
#worker1主机
安装基础环境
我的所有软件都安装在/usr/local/目录下,重点是配置/etc/profile文件
export JAVA_HOME=/usr/local/jdk1.8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=.:${JAVA_HOME}/bin:$PATH
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
export HADOOP_HOME=/usr/local/hadoop
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/usr/local/spark
使用source /etc/profile
使其生效,再将配置scp拷贝到wroker,在source命令让worker节点配置生效
配置完后,可以用scp 把配置好的软件从Master拷贝到Worker
scp -r 软件目录 用户名@IP地址:拷贝目标
scp -r /usr/local/hadoop root@Worker:/usr/local/
各种配置文件
1. Hadoop配置
1. $HADOOP_HOME/etc/hadoop/hadoop-env.sh
修改JAVA_HOME变量
export JAVA_HOME=/usr/java/jdk1.8.0_112/
2. $HADOOP_HOME/etc/hadoop/slaves
worker1
3. $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>
4. $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/hdfs/data</value>
</property>
</configuration>
5. $HADOOP_HOME/etc/hadoop/mapred-site.xml
复制template,生成xml:
cp mapred-site.xml.template mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:19888</value>
</property>
</configuration>
6. $HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
至此master节点的hadoop搭建完毕,再启动之前我们需要格式化一下namenode
hadoop namenode -format
2. Spark
1. $SPARK_HOME/conf/spark-env.sh
cp spark-env.sh.template spark-env.sh
export SCALA_HOME=/usr/local/scala
export JAVA_HOME=/usr/local/jdk1.8/
export SPARK_MASTER_IP=master
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
2. $SPARK_HOME/conf/slaves
cp slaves.template slaves
master
worker1
再把配好的spark scp到worker1节点
启动服务
封装成启动脚本
#!/bin/bash
echo -e "\033[31m ========Start The Cluster======== \033[0m"
echo -e "\033[31m Starting Hadoop Now !!! \033[0m"
/usr/local/hadoop/sbin/start-all.sh
echo -e "\033[31m Starting Spark Now !!! \033[0m"
/usr/local/spark/sbin/start-all.sh
echo -e "\033[31m The Result Of The Command \"jps\" : \033[0m"
jps #打印正在运行的java进程
echo -e "\033[31m ========END======== \033[0m"
输入http://ip:8080 可以进入spark管理系统
输入http://ip:8088 可以进入hadoop的管理系统
3. 测试
上传测试文件到HDFS,创建目录时,可能会报namenode unsage错误,如果报错,运行hadoop dfsadmin -safemode leave
hadoop fs -mkdir -p /Hadoop/Input
hadoop fs -put 测试文件 /Hadoop/Input
测试Hadoop
hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar 测试文件 /Hadoop/Input /Hadoop/Output
hadoop fs -cat /Hadoop/Output/* #查看输出结果
测试Spark
spark-shell 开启交互式窗口
val file=sc.textFile("hdfs://master:9000/Hadoop/Input/测试文件")
val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
rdd.collect()
rdd.foreach(println)
查看workCount的结果
关闭脚本
#!/bin/bash
echo -e "\033[31m ===== Stoping The Cluster ====== \033[0m"
echo -e "\033[31m Stoping Spark Now !!! \033[0m"
/usr/local/spark/sbin/stop-all.sh
echo -e "\033[31m Stopting Hadoop Now !!! \033[0m"
/usr/local/hadoop/sbin/stop-all.sh
echo -e "\033[31m The Result Of The Command \"jps\" : \033[0m"
jps
echo -e "\033[31m ======END======== \033[0m"
~
~