CDH集成Spark2

下载组件

CDH5.12默认集成spark1.6，更换为spark2需要下载四个文件：

• SPARK2_ON_YARN-2.2.0.cloudera1.jar

  http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.2.0.cloudera1.jar

• SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel

  http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera1/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel

• SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel.sha1

  http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera1/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel.sha1

• manifest.json

  http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera1/manifest.json

部署

部署需要将这四个文件放到指定的位置，同时替换权限。首先将四个文件全部上传到/opt目录下。

[root@hadoop01 ~]# mkdir -p /opt/cloudera/csd
[root@hadoop01 ~]# cd /opt

[root@hadoop01 opt]# mv SPARK2_ON_YARN-2.2.0.cloudera1.jar /opt/cloudera/csd
[root@hadoop01 opt]# mv SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel /opt/cloudera/parcel-repo
[root@hadoop01 opt]# mv SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel.sha1 /opt/cloudera/parcel-repo/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-xenial.parcel.sha
[root@hadoop01 opt]# mv /opt/cloudera/parcel-repo/manifest.json /opt/cloudera/parcel-repo/manifest.json_back
[root@hadoop01 opt]# mv manifest.json /opt/cloudera/parcel-repo

[root@hadoop01 opt]# chown -R cloudera-scm:cloudera-scm /opt/cloudera

Python大数据搜索

CDH分配

然后启动CDH服务，分配spark2.