创建用户
addusr
- sbt方式编译
- Maven编译
- 打包方式编译make-distribution
用maven编译spark1.3.1
装maven
tar -zxf apache-maven-3.0.5-bin.tar.gz
maven的默认工作目录为
cd ~/.m2
配置环境变量
vim ~/.bashrc
export MAVEN_HOME=/usr/maven
export PATH=$PATH:$MAVEN_HOME/bin
解压spark
[root@localhost data01]# tar -zxf spark-1.3.1.tgz
mvn编译
3.1编译的注意事项 设置域名解析
$ sudo vi /etc/resolv.conf
内容如下:
nameserver 8.8.8.8
nameserver 8.8.4.4
3.2编译
有pom。xml文件说明可以用mvn编译
3.3设置mvn内存使用
参考doc http://spark.apache.org/docs/1.3.0/building-spark.html
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
3.4在spark-1.3.1目录下编译
参考此处
# Apache Hadoop 2.4.X or 2.5.X
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=VERSION -DskipTests clean package
Versions of Hadoop after 2.5.X may or may not work with the -Phadoop-2.4 profile (they were
released after this version of Spark).
第一种编译方式:
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package
编译后不会生成jar包
第二种打包编译方式:
make-distribution.sh
修改里面的参数
VERSION=1.3.1
SPARK_HADOOP_VERSION=2.6.0
SPARK_HIVE=1
#VERSION=$("$MVN" help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | tail -n 1)
#SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
# | grep -v "INFO"\
# | tail -n 1)
#SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
# | grep -v "INFO"\
# | fgrep --count "<id>hive</id>";\
# # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
# # because we use "set -o pipefail"
# echo -n)
./make-distribution.sh --tgz -Phadoop-2.4 -Dhadoop.version=2.6.0 -Pyarn -Phive-0.13.1 -Phive-thriftserver
注意这里默认的scala版本是 : scala-2.10.4.tgz
简单测试
解压编译好的`spark-1.3.1-bin-2.6.0.tgz `
bin/spark-submit --master local --class org.apache.spark.examples.SparkPi lib/spark-examples-1.3.1-hadoop2.6.0.jar
PI
Spark context available as sc.
15/05/28 23:04:56 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
read
读取本地文件
val textFile = sc.textFile("README.md")
textFile.take(10)
读取hdfs
val hdfsTextFile = sc.textFile("hdfs://localhost:9000/input/test")//这里的路径要查看core-site
hdfsTextFile.flatMap(x => x.split(" ")).map(x => (x,1)).reduceByKey(_ + _).collect