基于hadoop2.6.4的spark1.6.2的伪分布式

1、解压scala2.1.0，scala2.1.0与spark21.6.2匹配

tar -zxf scala-2.10.4.tgz -C /home/hadoop/soft-install/bigdata/

vi /etc/profile

export SCALA_HOME=/home/hadoop/soft-install/bigdata/scala-2.10.4

PATH=$PATH:$SCALA_HOME/bin

刷新环境变量

source /etc/profile

测试是否成功

scala -version

出现版本号即成功

2、解压spark压缩包

tar -zxf spark-1.6.2-bin-hadoop2.6.tgz -C /home/hadoop/soft-install/bigdata/

配置环境变量 vi /etc/profile

export SPARK_HOME=/home/hadoop/soft-install/bigdata/spark-1.6.2-bin-hadoop2.6

$SPARK_HOME/bin:$SPARK_HOME/sbin

刷新环境变量 source /etc/profile

3、配置文件

进入配置文件目录 cd /home/hadoop/soft-install/bigdata/spark-1.6.2-bin-hadoop2.6/conf

1、配置slaves mv slaves.template slaves

vi slaves

把localhost修改成主机名，默认也行

2、配置 spark-env.sh mv spark-env.sh.template spark-env.sh

vi spark-env.sh

export JAVA_HOME=/home/hadoop/soft-install/bigdata/jdk1.8.0_333 export SCALA_HOME=/home/hadoop/soft-install/bigdata/scala-2.10.4 export HADOOP_HOME=/home/hadoop/soft-install/bigdata/hadoop-2.6.4 export HADOOP_CONF_DIR=/home/hadoop/soft-install/bigdata/hadoop-2.6.4/etc/hadoop

export SPARK_MASTER_IP=hadoop export SPARK_MASTER_PORT=7077

/home/hadoop/soft-install/bigdata/spark-1.6.2-bin-hadoop2.6/sbin/start-all.s

4、jps查看进程 出现master和work即成功