hadoop与spark共存伪分布式系统搭建

1 安装部署Hadoop

关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

配置主机名

vi /etc/hosts
ip bigdata2

免密码登陆（不对称：公钥、私钥-----> 字符串）

ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub root@bigdata2

安装jdk

mkdir tools
mkdir training
tar -zxvf jdk-8u181-linux-x64.tar.gz -C /root/training/

设置环境变量(zsh终端)

vi ~/.zshrc

export JAVA_HOME=/root/training/jdk1.8.0_241
export PATH=$JAVA_HOME/bin:$PATH

source ~/.zshrc

hadoop部署

tar -zxvf hadoop-3.1.2.tar.gz -C ~/training/

设置环境变量

vim ~/.zshrc

export HADOOP_HOME=/root/training/hadoop-3.1.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

export HDFS_DATANODE_USER=root
export HDFS_DATANODE_SECURE_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

source ~/.zshrc

配置hadoop
HDFS：NameNode、SecondaryNameNode、DataNode
Yarn：ResourceManager、NodeManager

        hdfs-site.xml  
                <!--配置数据块的冗余度-->
                <property>
                    <name>dfs.replication</name>
                    <value>1</value>
                </property>
 
                <!--禁用了HDFS的权限-->
                <property>
                    <name>dfs.permissions</name>
                    <value>false</value>
                </property>
 
        core-site.xml
         
                <!--NameNode的地址-->
                <property>
                    <name>fs.defaultFS</name>
                    <value>hdfs://bigdata111:9000</value>
                </property>      
         
                <!--HDFS对应的操作系统目录-->
                <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/root/training/hadoop-3.1.2/tmp</value>
                </property>  
 
        mapred-site.xml
 
                <!--MapReduce运行的框架-->
                <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
                </property>  
 
                <property>
                    <name>mapreduce.map.env</name>
                    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
                </property>  
 
                <property>
                    <name>mapreduce.reduce.env</name>
                    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
                </property>  
                 
                <property>
                    <name>yarn.app.mapreduce.am.env</name>
                    <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
                </property>      
 
        yarn-site.xml
                <!--配置Yarn的主节点-->
                <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>bigdata111</value>
                </property>  
 
                 
                <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
                </property>  

HDFS格式化

hdfs namenode -format

启动hadoop，由于hadoop与spark的start-all.sh与stop-all.sh相同，为了避免冲突，将hadoop中start-all.sh改为start-hadoop-all.sh,stop-all.sh改为stop-hadoop-all.sh

start-hadoop-all.sh = start-dfs.sh + start-yarn.sh

hadoop3中：

HDFS端口：9870
Yarn：8088

hadoop2中

HDFS端口：50070
Yarn：8088

2 安装部署spark

spark的安装教程
启动spark

start_all

关闭spark

stop_all

spark网页端访问

http://ip:8080/