spark在yarn上运行遇见的错误

最新推荐文章于 2020-12-30 22:58:57 发布

安纳西丶小雨

最新推荐文章于 2020-12-30 22:58:57 发布

阅读量1.7k

点赞数

分类专栏：报错

本文链接：https://blog.csdn.net/weixin_44784775/article/details/92853869

版权

报错专栏收录该内容

2 篇文章 0 订阅

订阅专栏

Spark代码运行出错

19/06/19 22:29:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/19 22:29:30 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/06/19 22:29:31 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/06/19 22:29:31 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/06/19 22:29:31 INFO yarn.Client: Setting up container launch context for our AM
19/06/19 22:29:31 INFO yarn.Client: Setting up the launch environment for our AM container
19/06/19 22:29:31 INFO yarn.Client: Preparing resources for our AM container
19/06/19 22:29:34 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/06/19 22:29:44 INFO yarn.Client: Uploading resource file:/tmp/spark-4ebf7d5b-61e2-4768-95d4-dfb97d199e56/__spark_libs__5213362222344047075.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/__spark_libs__5213362222344047075.zip
19/06/19 22:30:01 INFO yarn.Client: Uploading resource file:/home/hadoop01/count/spark-study-core-1.0-SNAPSHOT-jar-with-dependencies.jar -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/spark-study-core-1.0-SNAPSHOT-jar-with-dependencies.jar
19/06/19 22:30:12 INFO yarn.Client: Uploading resource file:/tmp/spark-4ebf7d5b-61e2-4768-95d4-dfb97d199e56/__spark_conf__1038658387679626348.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/__spark_conf__.zip
19/06/19 22:30:12 INFO spark.SecurityManager: Changing view acls to: hadoop01
19/06/19 22:30:12 INFO spark.SecurityManager: Changing modify acls to: hadoop01
19/06/19 22:30:12 INFO spark.SecurityManager: Changing view acls groups to: 
19/06/19 22:30:12 INFO spark.SecurityManager: Changing modify acls groups to: 
19/06/19 22:30:12 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop01); groups with view permissions: Set(); users  with modify permissions: Set(hadoop01); groups with modify permissions: Set()
19/06/19 22:30:12 INFO yarn.Client: Submitting application application_1560952784300_0002 to ResourceManager
19/06/19 22:30:12 INFO impl.YarnClientImpl: Submitted application application_1560952784300_0002
19/06/19 22:30:13 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:14 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1560954612808
         final status: UNDEFINED
         tracking URL: http://hadoop02:8088/proxy/application_1560952784300_0002/
         user: hadoop01
19/06/19 22:30:15 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:16 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:17 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:18 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
。。。。。。
19/06/19 22:31:34 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:31:35 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:31:36 INFO yarn.Client: Application report for application_1560952784300_0002 (state: FAILED)
19/06/19 22:31:36 INFO yarn.Client:
client token: N/A
         diagnostics: Application application_1560952784300_0002 failed 2 times due to AM Container for appattempt_1560952784300_0002_000002 exited with  exitCode: 15
For more detailed output, check application tracking page:http://hadoop02:8088/cluster/app/application_1560952784300_0002Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e54_1560952784300_0002_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
        at org.apache.hadoop.util.Shell.run(Shell.java:482)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1560954612808
         final status: FAILED
         tracking URL: http://hadoop02:8088/cluster/app/application_1560952784300_0002
         user: hadoop01
Exception in thread "main" org.apache.spark.SparkException: Application application_1560952784300_0002 finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1168)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/06/19 22:31:36 INFO util.ShutdownHookManager: Shutdown hook called
19/06/19 22:31:36 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-4ebf7d5b-61e2-4768-95d4-dfb97d199e56
./spark-submit-wc-02.sh: line 14: hdfs://bd1901/spark/wordcount: No such file or directory

重写了项目中的pom.xml

<build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

删除了代码中的conf.setMaster(“local”)
就运行顺利了

[hadoop01@hdoop01 count]$ sh spark-submit-wc-02.sh 
Warning: Skip remote jar hdfs://bd1901/share/jars/original-spark-study-core-1.0-SNAPSHOT.jar.
19/06/20 13:18:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/20 13:18:21 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
19/06/20 13:18:21 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/06/20 13:18:21 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/06/20 13:18:21 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/06/20 13:18:21 INFO yarn.Client: Setting up container launch context for our AM
19/06/20 13:18:21 INFO yarn.Client: Setting up the launch environment for our AM container
19/06/20 13:18:21 INFO yarn.Client: Preparing resources for our AM container
19/06/20 13:18:23 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/06/20 13:18:31 INFO yarn.Client: Uploading resource file:/tmp/spark-37fb97f4-9d68-46c4-bd3a-77aef2777da4/__spark_libs__4568974335995054152.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560997628871_0009/__spark_libs__4568974335995054152.zip
19/06/20 13:18:40 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://bd1901/share/jars/original-spark-study-core-1.0-SNAPSHOT.jar
19/06/20 13:18:40 INFO yarn.Client: Uploading resource file:/tmp/spark-37fb97f4-9d68-46c4-bd3a-77aef2777da4/__spark_conf__1879285324732297533.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560997628871_0009/__spark_conf__.zip
19/06/20 13:18:40 INFO spark.SecurityManager: Changing view acls to: hadoop01
19/06/20 13:18:40 INFO spark.SecurityManager: Changing modify acls to: hadoop01
19/06/20 13:18:40 INFO spark.SecurityManager: Changing view acls groups to: 
19/06/20 13:18:40 INFO spark.SecurityManager: Changing modify acls groups to: 
19/06/20 13:18:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop01); groups with view permissions: Set(); users  with modify permissions: Set(hadoop01); groups with modify permissions: Set()
19/06/20 13:18:40 INFO yarn.Client: Submitting application application_1560997628871_0009 to ResourceManager
19/06/20 13:18:40 INFO impl.YarnClientImpl: Submitted application application_1560997628871_0009
19/06/20 13:18:41 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:41 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1561007852323
         final status: UNDEFINED
         tracking URL: http://hadoop03:8088/proxy/application_1560997628871_0009/
         user: hadoop01
19/06/20 13:18:42 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:43 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:44 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:45 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:46 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:47 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:48 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:49 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:50 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:51 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:52 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:53 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:54 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:55 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:56 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:57 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:58 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:18:59 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:00 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:01 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:02 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:03 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:04 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:05 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:06 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:07 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:08 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:09 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:10 INFO yarn.Client: Application report for application_1560997628871_0009 (state: ACCEPTED)
19/06/20 13:19:11 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:11 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.2.101
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1561007852323
         final status: UNDEFINED
         tracking URL: http://hadoop03:8088/proxy/application_1560997628871_0009/
         user: hadoop01
19/06/20 13:19:12 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:13 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:14 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:15 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:16 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:17 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:18 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:19 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:20 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:21 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:22 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:23 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:24 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:25 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:27 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:28 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:29 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:30 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:31 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:32 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:33 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:34 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:35 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:36 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:37 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:38 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:39 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:40 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:41 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:42 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:43 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:44 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:45 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:46 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:47 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:48 INFO yarn.Client: Application report for application_1560997628871_0009 (state: RUNNING)
19/06/20 13:19:49 INFO yarn.Client: Application report for application_1560997628871_0009 (state: FINISHED)
19/06/20 13:19:49 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.2.101
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1561007852323
         final status: SUCCEEDED
         tracking URL: http://hadoop03:8088/proxy/application_1560997628871_0009/
         user: hadoop01
19/06/20 13:19:49 INFO util.ShutdownHookManager: Shutdown hook called
19/06/20 13:19:49 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-37fb97f4-9d68-46c4-bd3a-77aef2777da4

安纳西丶小雨

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
spark在yarn上运行遇见的错误

Spark代码运行出错19/06/19 22:29:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable19/06/19 22:29:30 INFO yarn.Client: Requ...
复制链接

扫一扫