SparkR的编译安装

在编译从github下载SparkR-pkg-master.zip 解压后编译

#SPARK_VERSION=1.5.2 SPARK_HADOOP_VERSION=2.6.0 ./install-dev.sh 

* installing *source* package ‘SparkR’ ...

** libs
** arch - 
./sbt/sbt assembly
Launching sbt from sbt/sbt-launch-0.13.6.jar
Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.6.jar
make: *** [target/scala-2.10/sparkr-assembly-0.1.jar] Error 1
ERROR: compilation failed for package ‘SparkR’

* removing ‘/root/SparkR-pkg-master/lib/SparkR‘

意思是说sbt-launch-0.13.6.jar这个包不对，怎么会不对呢

#ll /root/SparkR-pkg-master/pkg/src/sbt/sbt-launch-0.13.6.jar
-rw-r--r-- 1 root root 0 Dec 15 22:23 /root/SparkR-pkg-master/pkg/src/sbt/sbt-launch-0.13.6.jar

确实是下载到的包不对

解决方法：

vim  SparkR-pkg-master/pkg/src/sbt/sbt

1、有两个URL要修改一下 修改如下

URL1=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar

URL2=https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar

2、修改命令

   curl  --progress-bar  ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}

改为
    curl  --silent -L ${URL1} > ${JAR} || curl --silent -L ${URL2} > ${JAR}

完了之后再次编译

# SPARK_VERSION=1.5.2 SPARK_HADOOP_VERSION=2.6.0 ./install-dev.sh 
* installing *source* package ‘SparkR’ ...
** libs
** arch - 
./sbt/sbt assembly
Attempting to fetch sbt
Launching sbt from sbt/sbt-launch-0.13.6.jar
Getting org.scala-sbt sbt 0.13.6 ...
downloading https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.6/jars/sbt.jar ...
[SUCCESSFUL ] org.scala-sbt#sbt;0.13.6!sbt.jar (5691ms)
downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar ...
[SUCCESSFUL ] org.scala-lang#scala-library;2.10.4!scala-library.jar (6320ms)
downloading https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.13.6/jars/main.jar ...
[SUCCESSFUL ] org.scala-sbt#main;0.13.6!main.jar (36212ms)

开始下包了.......

或者不使用sbt的方法编译，使用maven

# USE_YARN=1 USE_MAVEN=1 SPA_YARN_VEVERSION=2.6.0 SPARK_VERSION=1.5.2 SPARK_HADOOP_VERSION=2.6.0 ./install-dev.sh 

# USE_YARN=1 USE_MAVEN=1 SPA_YARN_VEVERSION=2.6.0 SPARK_VERSION=1.5.2 SPARK_HADOOP_VERSION=2.6.0 ./install-dev.sh 
* installing *source* package ‘SparkR’ ...
** libs
** arch - 
mvn -Dhadoop.version=2.6.0 -Dspark.version=1.5.2 -DskipTests "-Pyarn" -Dyarn.version=2.4.0 clean package shade:shade
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for edu.berkeley.cs.amplab:sparkr:jar:0.1

..............

最后编译好之后主要是  lib/SparR   这个文件然后拷贝到R的库下边，原来有的替换掉。

#cp -rf lib/SparkR   /usr/local/lib64/R/library/

> library(SparkR)
[SparkR] Initializing with classpath /usr/local/lib64/R/library/SparkR/sparkr-assembly-0.1.jar

>