mac上IDEA编辑spark提交遇到的问题

最新推荐文章于 2024-05-04 11:42:57 发布

Starblingbling

最新推荐文章于 2024-05-04 11:42:57 发布

阅读量469

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/gzwdz778/article/details/79154794

版权

spark 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

1.问题1:mac上使用IDEA打包好scala编写的spark项目后，把jar包放到集群上使用spark-submit提交时，报错说主类找不到，解决如下：

（2）在pom文件中添加如下：

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>3.2.2</version>
            <executions>
                <execution>
                    <id>eclipse-add-source</id>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                </execution>
                <execution>
                    <id>scala-compile-first</id>
                    <phase>process-resources</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>scala-test-compile-first</id>
                    <phase>process-test-resources</phase>
                    <goals>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>attach-scaladocs</id>
                    <phase>verify</phase>
                    <goals>
                        <goal>doc-jar</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <scalaVersion>2.10.4</scalaVersion>
                <recompileMode>incremental</recompileMode>
                <useZincServer>true</useZincServer>
                <args>
                    <arg>-unchecked</arg>
                    <arg>-deprecation</arg>
                    <arg>-feature</arg>
                </args>
                <!--<jvmArgs>-->
                <!--<jvmArg>-Xms1024m</jvmArg>-->
                <!--<jvmArg>-Xmx1024m</jvmArg>-->
                <!--<jvmArg>-XX:PermSize=${PermGen}</jvmArg>-->
                <!--<jvmArg>-XX:MaxPermSize=${MaxPermGen}</jvmArg>-->
                <!--<jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>-->
                <!--</jvmArgs>-->
                <javacArgs>
                    <javacArg>-source</javacArg>
                    <javacArg>${java.version}</javacArg>
                    <javacArg>-target</javacArg>
                    <javacArg>${java.version}</javacArg>
                    <javacArg>-Xlint:all,-serial,-path</javacArg>
                </javacArgs>
            </configuration>
        </plugin>
    </plugins>
</build>

(2)使用maven重新编译，打包，步骤，如下图：

2.问题2:使用spark-submit提交时，问题1顺利解决，新错误是：Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes，解决如下：

zip -d 你的jar文件 META-INF/*.RSA META-INF/*.DSA META-INF/*.SF