续上篇Hbase
下载spark2.11.0和scala
以下是对spark的版本描述
Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.1 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
Note that support for Java 7 and Python 2.6 are deprecated as of Spark 2.0.0, and support for Scala 2.10 and versions of Hadoop before 2.6 are deprecated as of Spark 2.1.0, and may be removed in Spark 2.2.0.
下载包
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz
wget https://downloads.lightbend.com/scala/2.11.11/scala-2.11.11.tgz
创建文件夹
mkdir -p /opt/scala
mkdir -p /opt/spark
解压包
tar -zxvf scala-2.11.11.tgz -C /opt/scala
tar -zxvf spark-2.1.1-bin-hadoop2.7.tgz -C /opt/spark
分别创建用户级的环境变量文件
/etc/profile.d/scala.sh
export SCALA_HOME=/opt/scala/current
/etc/profile.d/spark.sh
export SPARK_HOME=/opt/spark/current
export PATH=$PATH:${SPARK_HOME}/bin
修改spark配置文件
cp ./conf/spark-env.sh.template ./conf/spark-env.sh
编辑spark-env.sh
export SCALA_HOME=${SCALA_HOME}
export JAVA_HOME=${JAVA_HOME}
export SPARK_MASTER_IP=master
export SPARK_WORKER_MEMORY=500m
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
cp slaves.template slaves
编辑slaves
master
slave01
slave02
赋权
chown -R hadoop:hadoop /opt/scala
chown -R hadoop:hadoop /opt/spark
并把对应文件夹scp到其余机器上
在hadoop用户下创建软连接
ln -s /opt/scala/scala-2.11.11 ./current
ln -s /opt/spark/spark-2.1.1-bin-hadoop2.7 ./current
测试
spark-shell
val file=sc.textFile("hdfs://iZuf68ho3sfplkorf9r8akZ:9000/stella/input/wordcount.txt")
val rdd = file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
rdd.collect()
rdd.foreach(println)
详细安装可以参考,转载: http://www.cnblogs.com/purstar/p/6293605.html