伪分布式搭建（hadoop，spark，scala）

一.安装伪分布式hadoop

1.安装java环境

（1）.下载jdk安装包

官网地址：https://www.oracle.com/java /technologies /javase-jdk8-downloads.html

（2）.卸载自带的openjdk

rpm -qa | grep java
#卸载相关服务
rpm -e --nodeps 
java -version

（3）.安装jdk

tar -zxvf jdk-8u152-linux-x64.tar.gz -C /usr/local/src/
ls /usr/local/src

（4）.设置java变量

vi /etc/profile
export JAVA_HOME=/usr/local/src/jdk1.8.0_152
export PATH=$PATH:$JAVA_HOME/bin

2.实现免密登录

（1）.创建ssh密钥

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

（2）.将master公钥id-dsa复制到master进行公钥验证，实现本机免密登录

ssh-copy-id -i /root/.ssh/id_dsa.pub master
ssh master    #登录
exit          #退出

3.hadoop环境的安装和配置

（1）.解压hadoop安装包

（2）.修改环境变量

vim /etc/profile
    export HADOOP_HOME=/opt/hadoop
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source /etc/profile

（3）.编辑/usr/local/src/hadoop-2.7.1/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/src/jdk1.8.0_152

（4）.编辑/usr/local/src/hadoop-2.7.1/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop/tmp</value>
    </property>
</configuration>

（5）.编辑/usr/local/src/hadoop-2.7.1/etc/hadoop/hdfs-site.xml

复制/usr/local/src/hadoop-2.7.1/etc/hadoop/mapred-site.xml.template改名为

/usr/local/src/hadoop-2.7.1/etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>

（6）.编辑/usr/local/src/hadoop-2.7.1/etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

（7）.编辑/usr/local/src/hadoop-2.7.1/etc/hadoop/slaves

master

（8）.格式化hdfs

hdfs namenode -format

（9）.启动集群jps查看，登录网页

start-all.sh
jps

二.安装伪分布式spark

1.解压安装包

tar -zxf spark-2.0.0-bin-hadoop2.6.tgz -C /usr/local/src

2.复制和重复名后，打开spark-env.sh编辑

cd /usr/local/src/spark-3.2.1-bin-hadoop2.7/conf
cp /usr/local/srcspark-3.2.1-bin-hadoop2.7/conf/spark-env.sh.template /usr/local/srcspark-3.2.1-bin-hadoop2.7/conf/spark-env.sh
vim spark-env.sh
    export JAVA_HOME=/usr/local/src/jdk1.8.0_152
    export HADOOP_HOME=/opt/software/hadoop-2.7.1
    export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
    export SPARK_MASTER_IP=master
    export SPART_LOCAL_IP=master

3.进入spark目录下的/sbin下启动spark集群，jsp查看

cd /usr/local/spark-2.0.0-bin-hadoop2.6/sbin/
./start-all.sh
jps

cd /usr/local/spark-2.0.0-bin-hadoop2.6/
./bin/spark-shell

4.查看网页http：//192.168.32.128：8080

三.安装伪分布式scala

1.解压安装

包

tar -zxf scala-2.11.8.tgz -C /usr/local/src

2.配置scala环境变量

vim /etc/profile
    export SCALA_HOME=/usr/local/src/scala-2.11.8
    export PATH=$PATH:$SCALA_HOME/bin

source /etc/profile
scala