软件版本:
jdk:1.8
maven:3.61 http://maven.apache.org/download.cgi
spark:2.42 https://archive.apache.org/dist/spark/spark-2.4.2/
hadoop版本:hadoop-2.6.0-cdh5.7.0(spark编译支持的hadoop版本,不需要安装)
配置maven:
#配置环境变量 [root@hadoop004 soft]# cat /etc/profile.d/maven.sh MAVEN_HOME=/usr/local/maven export PATH=$MAVEN_HOME/bin:$PATH #确认maven版本 [root@hadoop004 maven]# mvn --version Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00) Maven home: /usr/local/maven Java version: 1.8.0_111, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_111/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-862.3.2.el7.x86_64", arch: "amd64", family: "unix" #配置mvn的本地存放地址:settings.xml文件 <localRepository>/usr/local/maven/repo</localRepository> #配置mvn下载源为阿里云的maven仓库,加速下载 <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror>
配置Spark:
tar xf spark-2.4.2.tgz cd spark-2.4.2.tgz #修改pom.xml文件,添加clouder仓库 <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository>
执行编译命令:
#在spark目录下执行 ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 注:本次编译时长大约为35分钟,中间无任何报错; 注:默认使用的scala版本为最新的,如果要指定scala版本,通过以下方式修改 比如把scala版本改为 2.10 ./dev/change-scala-version.sh 2.10
参数说明:
--name:生成压缩包的后缀名字;前缀默认为spark版本的名字,本例为:spark-2.4.2-bin
--tgz:采用压缩格式为tar,压缩的后缀名为.tgz
-Pyarn:表示spark需要运行在yarn上面
-Phadoop-2.6:表示spark使用hadoop的profile的id
-Dhadoop.version=2.6.0-cdh5.7.0:表示spark使用hadoop的版本;如果不指定,默认使用的是2.2.0的hadoop
-Phive -Phive-thriftserver:表示支持hive
其它参数:
-DskipTests:跳过测试
生成的文件:
在spark目录下:spark-2.4.2-bin-2.6.0-cdh5.7.0.tgz
使用编译后的spark部署:
tar xf spark-2.4.2-bin-2.6.0-cdh5.7.0.tgz ln -s spark-2.4.2-bin-2.6.0-cdh5.7.0 spark #配置spark的环境变量 [hadoop@hadoop001 ~]$ vim .bash_profile export SPARK_HOME=/home/hadoop/app/spark-2.4.2-bin-2.6.0-cdh5.7.0 export PATH=${SPARK_HOME}/bin:$PATH [hadoop@hadoop001 ~]$ source .bash_profile #运行spark测试 [hadoop@hadoop001 ~]$ spark-shell 19/04/29 10:51:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hadoop001:4040 Spark context available as 'sc' (master = local[*], app id = local-1556506274719). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.2 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111) Type in expressions to have them evaluated. Type :help for more information. scala>
转载于:https://blog.51cto.com/14284607/2386484