【spark】win7-64位下编译spark1.6.0

1：设置setting.xml中maven仓库为http://maven.oschina.net/content/groups/public/  (此仓库需要maven3.3.3以上)

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<interactiveMode>false</interactiveMode>
 <mirrors>
 <mirror>
     <id>nexus-osc</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus osc</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
 </mirror>
 <mirror>
     <id>nexus-osc-thirdparty</id>
     <mirrorOf>thirdparty</mirrorOf>
     <name>Nexus osc thirdparty</name>
     <url>http://maven.oschina.net/content/repositories/thirdparty/</url>
 </mirror>
 </mirrors>
  <profiles>
  <profile>
  <id>jdk-1.7</id>
  <activation>
      <jdk>1.7</jdk>
  </activation>
  <repositories>
      <repository>
          <id>nexus</id>
          <name>local private nexus</name>
          <url>http://maven.oschina.net/content/groups/public/</url>
          <releases>
              <enabled>true</enabled>
          </releases>
          <snapshots>
              <enabled>false</enabled>
          </snapshots>
      </repository>
  </repositories>
  <pluginRepositories>
      <pluginRepository>
          <id>nexus</id>
          <name>local private nexus</name>
          <url>http://maven.oschina.net/content/groups/public/</url>
          <releases>
              <enabled>true</enabled>
          </releases>
          <snapshots>
              <enabled>false</enabled>
          </snapshots>
      </pluginRepository>
  </pluginRepositories>
  </profile>
  </profiles>
</settings>

2：下载spark1.6源码
3：下载apache-maven-3.3.9
4：set MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m

如果配置内存太低会导致OutOfMemory或者OutOfPerm异常
5：cd E:\DM\Spark\spark-1.6.0
6：添加-Pnetlib-lgpl允许使用netlib的外部BLAS dll(参见breeze安装文章)

mvn -Pyarn -Pnetlib-lgpl -Dhadoop.version=2.3.0 -Dyarn.version=2.3.0 -DskipTests clean package 
7：当使用eclipse scala IDE编译时，需要设置对应项目的scala compiler类库为scala.2.10才能编译成功

8：默认的spark版本由于版权原因，不能加载netlib的外部BLAS dll，如果不想重新编译spark

可以压缩以下jar的内容到spark-assembly-1.6.0-hadoop2.3.0.jar(同时参见breeze安装文章)

jniloader-1.1

native_ref-java-1.1

native_system-java-1.1

netlib-native_ref-win-x86_64-1.1-natives

netlib-native_system-win-x86_64-1.1-natives

压缩进去后，启动spark将不会出现

16/03/08 19:27:03 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
16/03/08 19:27:03 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS