spark学习2 编译的

创建用户

addusr

1. sbt方式编译
2. Maven编译
3. 打包方式编译make-distribution

用maven编译spark1.3.1

官方编译文档

装maven

tar -zxf apache-maven-3.0.5-bin.tar.gz

maven的默认工作目录为

cd  ~/.m2

配置环境变量

vim ~/.bashrc
export MAVEN_HOME=/usr/maven
export PATH=$PATH:$MAVEN_HOME/bin

解压spark

[root@localhost data01]# tar -zxf spark-1.3.1.tgz

mvn编译

3.1编译的注意事项 设置域名解析

$ sudo vi /etc/resolv.conf
内容如下：
nameserver 8.8.8.8
nameserver 8.8.4.4

3.2编译
有pom。xml文件说明可以用mvn编译

3.3设置mvn内存使用
参考doc http://spark.apache.org/docs/1.3.0/building-spark.html

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

3.4在spark-1.3.1目录下编译
参考此处

# Apache Hadoop 2.4.X or 2.5.X
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=VERSION -DskipTests clean package

Versions of Hadoop after 2.5.X may or may not work with the -Phadoop-2.4 profile (they were
released after this version of Spark).

第一种编译方式：

mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package

编译后不会生成jar包

第二种打包编译方式：

make-distribution.sh

修改里面的参数

VERSION=1.3.1
SPARK_HADOOP_VERSION=2.6.0
SPARK_HIVE=1

#VERSION=$("$MVN" help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | tail -n 1)
#SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | tail -n 1)
#SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | fgrep --count "<id>hive</id>";\
#    # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
#    # because we use "set -o pipefail"
#    echo -n)

./make-distribution.sh --tgz -Phadoop-2.4 -Dhadoop.version=2.6.0 -Pyarn -Phive-0.13.1 -Phive-thriftserver

注意这里默认的scala版本是 ： scala-2.10.4.tgz

简单测试

解压编译好的`spark-1.3.1-bin-2.6.0.tgz `

 bin/spark-submit  --master local --class org.apache.spark.examples.SparkPi lib/spark-examples-1.3.1-hadoop2.6.0.jar

PI

Spark context available as sc.
15/05/28 23:04:56 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

read

读取本地文件

val textFile = sc.textFile("README.md")
textFile.take(10)

读取hdfs

val hdfsTextFile = sc.textFile("hdfs://localhost:9000/input/test")//这里的路径要查看core-site
hdfsTextFile.flatMap(x => x.split(" ")).map(x => (x,1)).reduceByKey(_ + _).collect