spark在eclipse中遇到的问题及处理措施_标记下

环境：cdh5.9  spark1.6

本地开发环境：下载了最新的ide：scala-SDK-4.6.0-vfinal-2.12-win32.win32.x86_64 

开始没有使用maven，而是使用eclipse直接加jar包的方式。

下面截图报错：

More than one scala library found in the build path (G:/scala-SDK-4.6.0-vfinal-2.12-win32.win32.x86_64/eclipse/plugins/org.scala-lang.scala-library_2.12.2.v20170412-161608-VFINAL-21d12e9.jar, G:/spark-assembly-1.6.0-cdh5.9.0-hadoop2.6.0-cdh5.9.0.jar).At least one has an incompatible version. Please update the project build path so it contains only one compatible scala library.

报错有多个scala版本，spark1.6 scala版本是2.10.5  

下面修改了版本为2.10.6

悲催的是问题反而严重了：

各种查询很无语，没有找到合适的方法，后来发现spark-assembly-1.6.0-cdh5.9.0-hadoop2.6.0-cdh5.9.0.jar  包中打包了scala的相关信息：

将scala包部分删除，问题解决。

后来和朋友聊天发现使用maven没有遇到这样的问题，改进现有版本使用maven

下面贴下maven的pom.xml

 <repositories>
    <repository>
      <id>cloudera.com</id>
      <name>Cloudera Maven Repository</name>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository>
  </repositories>							


  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.10.6</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.2</version>
    </dependency>
    <dependency>
      <groupId>commons-configuration</groupId>
      <artifactId>commons-configuration</artifactId>
      <version>1.10</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>1.2.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>1.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_2.10</artifactId>
      <version>3.0.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-spark</artifactId>
      <version>1.2.0-cdh5.9.0</version>
    </dependency>
    <dependency>
      <groupId>org.scala-tools</groupId>
      <artifactId>maven-scala-plugin</artifactId>
      <version>2.15.2</version>
    </dependency>
    <!--<dependency>-->
      <!--<groupId>it.unimi.dsi</groupId>-->
      <!--<artifactId>fastutil</artifactId>-->
      <!--<version>7.0.13</version>-->
    <!--</dependency>-->
  </dependencies>