1.准备工作
(1)安装工具包(自行下载)
(2)window与linux连接操作软件(自行下载)
(3)统一安装路径
所有安装包我放在 /home/study/spark/package 目录下
所有安装我均安装在 /home/study/spark 目录下
2.在/home/study/spark下安装JDK,并修改环境变量
(1)sudo vi /etc/profile
注:因为修改权限的问题需要sudo执行,否则可能会出现不能保存的问题
export JAVA_HOME=/home/study/spark/jdk1.8.0_111/
export JRE_HOME=/home/study/spark/jdk1.8.0_111/jre
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
(2)保存更新 /etc/profile 文件
source /etc/profile
(3)检测是否安装成功
java -version
3..在/home/study/spark下安装Hadoop,并修改环境变量
cd /home/study/spark
tar -xzvf ./package/hadoop-2.6.0.tar.gz
(1)sudo vi /etc/profile,加入Hadoop变量
export HADOOP_HOME=/home/study/spark/hadoop-2.6.0
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
(2)修改/home/study/spark/hadoop-2.6.0/etc/hadoop/hadoop-env.sh 配置文件
cd /home/study/spark/hadoop-2.6.0/etc/hadoop
vi hadoop-env.sh
在文件最后加入Java环境
export JAVA_HOME=/home/study/spark/jdk1.8.0_111/
(3)修改/home/study/spark/hadoop-2.6.0/etc/hadoop下的其他配置文件
①修改:core-site.xm
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration
②修改 hdfs-site.xm
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value> # 配置的备份方式,在单机hadoop中,设置为1(默认为3)
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/study/spark/hadoopdata/hdfs/namenode</value>
</property> #这里路径要换成自己数据存放路径
<property>
<name>dfs.data.dir</name>
<value>file:///home/study/spark/hadoopdata/hdfs/datanode</value>
</property> #这里路径要换成自己数据存放路径
</configuration>
③修改 mapred-site.xml
注:若没有mapred-site.xml文件,将 mapred-site.xml.template文件重命名为mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
④修改 yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
(4)初始化hadoop(在bin目录下hdfs)
cd /home/study/spark/hadoop-2.6.0/bin
./hdfs namenode -format
(5)启动Hadoop
cd /home/study/spark/hadoop-2.6.0/sbin
./start-all.sh
(6)停止Hadoop
cd /home/study/spark/hadoop-2.6.0/sbin
./stop-all.sh
(7)jps查看后台进程
注:截图中出现master是因为,写帖子时已安装完spark
(8)window下浏览器中打开端口,如http://xxx.xxx.xxx.xxx:8088
注:本文实在云主机上搭建的,故xxx.xxx.xxx.xxx为云主机的ip地址,如果本地可为localhost或者127.0.0.1
(9)其他端口
port 8088: cluster and all applications
port 50070: Hadoop NameNode
port 50090: Secondary NameNode
port 50075: DataNode
4.安装scala
(1)解压scala,安装/home/study/spark目录
cd /home/study/spark
tar -xzvf ./package/scala-2.10.6.tgz
sudo vi /etc/profile
(2)在 /etc/profile 文件下,添加scala环境变量
export SCALA_HOME=/home/study/spark/scala-2.10.6
export PATH=$SCALA_HOME/bin:$PATH
(3)保存更新 /etc/profile
source /etc/profile
(4)检测
scala -version
5.在/home/study/spark下安装spark
(1)解压,并通过mv指令修改spark-2.0.1-bin-hadoop2.7文件名为 spark-2.0.1
cd /home/study/spark
tar -xzvf ./package/spark-2.0.1-bin-hadoop2.7.tgz
mv spark-2.0.1-bin-hadoop2.7 spark-2.0.1
sudo vi /etc/profile
(2)在 /etc/profile 文件中添加环境变量
export SPARK_HOME=/home/study/spark/spark-2.0.1
export PATH=$SPARK_HOME/bin:$PATH
(3)保存更新 /etc/profile
source /etc/profile
(4)
在/home/study/spark/spark-2.0.1/conf目录下复制并重命名 spark-env.sh.template 为 spark-env.sh :
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
在 spark-env.sh 中添加:
export JAVA_HOME=/home/study/spark/jdk1.8.0_111/
export SCALA_HOME=/home/study/spark/scala-2.10.6
export SPARK_MASTER_IP=localhost
export SPARK_WORKER_MEMORY=1G
- 启动spark
cd /home/study/spark/spark-2.0.1/sbin
./start-all.sh
停止spark
cd /home/study/spark/spark-2.0.1/sbin
./stop-all.sh
(5)window下浏览器中打开端口,如http://xxx.xxx.xxx.xxx:8088
注:本文实在云主机上搭建的,故xxx.xxx.xxx.xxx为云主机的ip地址,如果本地可为localhost或者127.0.0.1
6.测试spark