spark2如何集成到cdh里

最近做性能测试需要spark2测试下和spark1.6性能有多大差别，官方文档里写着可以集成，但是自己怎么搞都不行，折磨了3天的时间，目前终于把spark2集成到集群里了

我安装的是最新版本的

下载spark2安装包

wget http://archive.cloudera.com/beta/spark2/parcels/latest/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel
[root@namenode01 parcel-repo]# wget http://archive.cloudera.com/beta/spark2/parcels/2.0.0.cloudera.beta2/manifest.json
[root@namenode01 parcel-repo]# chown cloudera-scm.cloudera-scm manifest.json SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel
[root@namenode01 parcel-repo]# echo "9501a3b45add128d9d3fedcccc4797518a87769b" > SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234-el7.parcel.sha

[root@namenode01 csd]# pwd
/opt/cloudera/csd
wget http://archive.cloudera.com/beta/spark2/csd/SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar
[root@namenode01 csd]# ls
SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar
[root@namenode01 csd]# chown cloudera-scm:cloudera-scm SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar ;chmod 644 SPARK2_ON_YARN-2.0.0.cloudera.beta2.jar 

几面上重启cm

之后把之前的旧的文件cp到spark2目录下，只需修改下spark2的配置文件即可。

不要纠结spark2是否在cm管理界面的添加服务里是否可以看到spark2服务。

[root@namenode01 bin]# cp  /etc/spark/conf/classpath.txt  /opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234 /etc/spark2/conf.dist/
[root@namenode01 bin]# cp  /etc/spark/conf/spark-env.sh  /opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta2-1.cdh5.7.0.p0.110234 /etc/spark2/conf.dist/

 vi /etc/spark2/conf/spark-env.sh
export SPARK_HOME=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
这个地方要改成spark2的

[root@datanode01 ~]# spark2-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://10.1.8.101:4040
Spark context available as 'sc' (master = local[*], app id = local-1497409518846).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them evaluated.
Type :help for more information.


scala>