Windows IDEA 项目(Scala+Sbt、Scala+Maven)创建与远程部署到Linux(远程部署其它项目也适用)

一、scala+sbt

注：sbt为创建项目时IDEA自动安装的
前提：先安装Scala插件

1. scala-sbt项目创建：

• File => New => project
  

• Next

  • 为避免后续部署到Linux运行出现问题，选择Scala版本时注意Linux安装的Spart版本是基于哪个Scala版本构建的

  Note that, Spark 2.x is pre-built with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12. Spark 3.0+ is pre-built with Scala 2.12.

• Finish，等待"dump project structure from sbt"，即把sbt所需要的项目结构从远程服务器拉取到本地
  
  • 完成后，scala文件夹会变为蓝色，然后可以开始创建.scala文件

2. 编辑build.sbt

• 基于你所需要编写的程序，编辑build.sbt，比如添加

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.1"

• 添加后会出现红色框的部分，不建议直接refresh，因为下载资源比较慢
  

  • 解决方法(参考网页)，仅供参考(本人下载spark-sql用了11分钟，不知道是否是解决了，毕竟11分钟时间挺长)
    

    • C:/user(用户)/xxx/.sbt
    • 创建repositories，内容为

      [repositories]
      local
      huaweicloud-maven: https://repo.huaweicloud.com/repository/maven/
      aliyun-nexus: https://maven.aliyun.com/nexus/content/groups/public/
      jcenter: https://jcenter.bintray.com/
      maven-central: https://repo1.maven.org/maven2/
      typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
      sbt-plugin-repo: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
      

    • IDEA：File => Setting
      • VM parameters添加：-Dsbt.override.build.repos=true
      • Apply，Ok

附：

二、scala+maven

• File => New => project
  
• Next
  
• Finish
• 工程名称上右键 => Add Framework Surport => Scala (同样，选择Scala版本时注意Linux安装的Spart版本是基于哪个Scala版本构建的)

Add Framework Surport 找不到 Scala：参考博客

• 配置依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>SampleScalaMaven</artifactId>
    <version>1.0-SNAPSHOT</version>

    <repositories>
        <repository>
            <id>jboss</id>
            <name>JBoss Repository</name>
            <url>http://repository.jboss.com/maven2/</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency> <!-- Spark dependency -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.0.1</version>
        </dependency>
    </dependencies>
    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>2.12.10</scalaVersion>
                    <args>
                        <arg>-target:jvm-1.8</arg>
                    </args>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

注：以下这部分依赖不能少

少了会出现的问题：当使用Maven打包项目为jar并上传到Linux运行的使用，会出现Failed to load class SencondarySort

• Refresh => 完成(Down)

• 附：IDEA 自带 Maven 配置国内源

  • 打开"IDEA安装目录(如：IntelliJ IDEA 2020.1)\plugins\maven\lib\maven3\conf\settings.xml"，找到mirrors，编辑如下：

<mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
	 <mirror>
		<id>alimaven</id>
		<name>aliyun maven</name>
		<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
		<mirrorOf>central</mirrorOf>
	 </mirror>
</mirrors>

三、远程部署到Linux

• Tools => Deployment => Configuration
  
• + => SFTP
  
• SSH Configuration
  
• 填写相关SSH配置信息
  
• Test Connection
  
• Ok => Ok
• Root path
  
• Mapprings => Deployment path(相对于Linux用户主目录的，即上图/home/hadoop)
  
  • 部署路径为：/home/hadoop/Myapp/sample_scala_maven
• Test Connection
  
• Ok => Ok
• Tools => Option => 把本地已改变的文件上传到默认服务器 => Always => Ok
  
• 个人建议：打包完成之后，手动Upload整个项目，避免远程jar包未更新
  

---

参考：window+idea搭建远程调试spark环境