Spark2.1.1中用各种模式运行计算圆周率的官方Demo

最新推荐文章于 2024-06-15 20:39:43 发布
最新推荐文章于 2024-06-15 20:39:43 发布
阅读量1.1k 收藏
点赞数
分类专栏： 大数据

1使用单机local模式提交任务

2使用独立的Spark集群模式提交任务

3使用Spark集群 +Hadoop集群的模式提交任务

        3.1用yarn-client模式执行计算程序

                3.1.1操作步骤和方法

                3.1.2常见错误解决

                          3.1.2.1 Yarn applicationhas already ended!

                                      3.1.2.1.1主要错误信息

                                      3.1.2.1.2错误原因

                                      3.1.2.1.3解决方法

                         3.1.2.2 Required executormemory (1024+384 MB)

                                      3.1.2.2.1主要错误信息

                                      3.1.2.2.2错误原因

                                      3.1.2.2.3解决方法

       3.2用yarn-cluster模式执行计算程序

               3.2.1操作步骤和方法

            关键字： Linux  CentOS  Hadoop  Spark  yarn-client   yarn-cluster

          版本号：CentOS7   Hadoop2.8.0    Spark2.1.1

          说明：本文主要使用Spark单机模式(也就是Local模式)、Spark独立集群模式（也就是Standalone模式）、Spark集群+Haddop集群模式（也就是Yarn-Client模式和Yarn-Cluster模式）来运行Spark官方提供的计算圆周率的Demo。

1  使用单机local模式提交任务

          local模式也就是本地模式，也就是在本地机器上单机执行程序。使用这个模式的话，并不需要启动Hadoop集群，也不需要启动Spark集群，只要有一台机器上安装了JDK、Scala、Spark即可运行。

        进入到Spark2.1.1的安装目录，命令是：

[plain]  view plain  copy
  1. cd  /opt/spark/spark-2.1.1-bin-hadoop2.7  

        执行命令，用单机模式运行计算圆周率的Demo：

[plain]  view plain  copy
  1. ./bin/spark-submit  --class   org.apache.spark.examples.SparkPi  --master   local  examples/jars/spark-examples_2.11-2.1.1.jar  

如图：


稍等几秒，计算即可完成

如图：


完整信息是：

[plain]  view plain  copy
  1. [root@hserver1 ~]# cd /opt/spark/spark-2.1.1-bin-hadoop2.7  
  2. [root@hserver1 spark-2.1.1-bin-hadoop2.7]# ./bin/spark-submit--class org.apache.spark.examples.SparkPi --master localexamples/jars/spark-examples_2.11-2.1.1.jar  
  3. 17/05/17 16:31:38 INFO spark.SparkContext: Running Spark version2.1.1  
  4. 17/05/17 16:31:39 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable  
  5. 17/05/17 16:31:39 INFO spark.SecurityManager: Changing view acls to:root  
  6. 17/05/17 16:31:39 INFO spark.SecurityManager: Changing modify aclsto: root  
  7. 17/05/17 16:31:39 INFO spark.SecurityManager: Changing view aclsgroups to:  
  8. 17/05/17 16:31:39 INFO spark.SecurityManager: Changing modify aclsgroups to:  
  9. 17/05/17 16:31:39 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups withview permissions: Set(); users  withmodify permissions: Set(root); groups with modify permissions: Set()  
  10. 17/05/17 16:31:40 INFO util.Utils: Successfully started service'sparkDriver' on port 41012.  
  11. 17/05/17 16:31:40 INFO spark.SparkEnv: Registering MapOutputTracker  
  12. 17/05/17 16:31:40 INFO spark.SparkEnv: RegisteringBlockManagerMaster  
  13. 17/05/17 16:31:40 INFO storage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information  
  14. 17/05/17 16:31:40 INFO storage.BlockManagerMasterEndpoint:BlockManagerMasterEndpoint up  
  15. 17/05/17 16:31:40 INFO storage.DiskBlockManager: Created localdirectory at /tmp/blockmgr-48295dbe-32ae-4bdb-a4e2-9fada0294094  
  16. 17/05/17 16:31:40 INFO memory.MemoryStore: MemoryStore started withcapacity 413.9 MB  
  17. 17/05/17 16:31:40 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator  
  18. 17/05/17 16:31:41 INFO util.log: Logging initialized @4545ms  
  19. 17/05/17 16:31:41 INFO server.Server: jetty-9.2.z-SNAPSHOT  
  20. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5118388b{/jobs,null,AVAILABLE,@Spark}  
  21. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@15a902e7{/jobs/json,null,AVAILABLE,@Spark}  
  22. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7876d598{/jobs/job,null,AVAILABLE,@Spark}  
  23. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/job/json,null,AVAILABLE,@Spark}  
  24. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5af28b27{/stages,null,AVAILABLE,@Spark}  
  25. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@71104a4{/stages/json,null,AVAILABLE,@Spark}  
  26. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4985cbcb{/stages/stage,null,AVAILABLE,@Spark}  
  27. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72f46e16{/stages/stage/json,null,AVAILABLE,@Spark}  
  28. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3c9168dc{/stages/pool,null,AVAILABLE,@Spark}  
  29. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@332a7fce{/stages/pool/json,null,AVAILABLE,@Spark}  
  30. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@549621f3{/storage,null,AVAILABLE,@Spark}  
  31. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@54361a9{/storage/json,null,AVAILABLE,@Spark}  
  32. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@32232e55{/storage/rdd,null,AVAILABLE,@Spark}  
  33. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5217f3d0{/storage/rdd/json,null,AVAILABLE,@Spark}  
  34. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@37ebc9d8{/environment,null,AVAILABLE,@Spark}  
  35. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@293bb8a5{/environment/json,null,AVAILABLE,@Spark}  
  36. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@2416a51{/executors,null,AVAILABLE,@Spark}  
  37. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6fa590ba{/executors/json,null,AVAILABLE,@Spark}  
  38. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6e9319f{/executors/threadDump,null,AVAILABLE,@Spark}  
  39. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72e34f77{/executors/threadDump/json,null,AVAILABLE,@Spark}  
  40. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7bf9b098{/static,null,AVAILABLE,@Spark}  
  41. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@389adf1d{/,null,AVAILABLE,@Spark}  
  42. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@77307458{/api,null,AVAILABLE,@Spark}  
  43. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1fc0053e{/jobs/job/kill,null,AVAILABLE,@Spark}  
  44. 17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@290b1b2e{/stages/stage/kill,null,AVAILABLE,@Spark}  
  45. 17/05/17 16:31:41 INFO server.ServerConnector: StartedSpark@32fe9d0a{HTTP/1.1}{0.0.0.0:4040}  
  46. 17/05/17 16:31:41 INFO server.Server: Started @4957ms  
  47. 17/05/17 16:31:41 INFO util.Utils: Successfully started service'SparkUI' on port 4040.  
  48. 17/05/17 16:31:41 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, andstarted at http://192.168.27.143:4040  
  49. 17/05/17 16:31:41 INFO spark.SparkContext: Added JARfile:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1495009901648  
  50. 17/05/17 16:31:41 INFO executor.Executor: Starting executor IDdriver on host localhost  
  51. 17/05/17 16:31:41 INFO util.Utils: Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 41396.  
  52. 17/05/17 16:31:41 INFO netty.NettyBlockTransferService: Servercreated on 192.168.27.143:41396  
  53. 17/05/17 16:31:41 INFO storage.BlockManager: Usingorg.apache.spark.storage.RandomBlockReplicationPolicy for block replicationpolicy  
  54. 17/05/17 16:31:41 INFO storage.BlockManagerMaster: RegisteringBlockManager BlockManagerId(driver, 192.168.27.143, 41396, None)  
  55. 17/05/17 16:31:41 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.143:41396 with 413.9 MB RAM,BlockManagerId(driver, 192.168.27.143, 41396, None)  
  56. 17/05/17 16:31:41 INFO storage.BlockManagerMaster: RegisteredBlockManager BlockManagerId(driver, 192.168.27.143, 41396, None)  
  57. 17/05/17 16:31:41 INFO storage.BlockManager: Initialized BlockManager:BlockManagerId(driver, 192.168.27.143, 41396, None)  
  58. 17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4e6d7365{/metrics/json,null,AVAILABLE,@Spark}  
  59. 17/05/17 16:31:42 INFO internal.SharedState: Warehouse path is 'file:/opt/spark/spark-2.1.1-bin-hadoop2.7/spark-warehouse/'.  
  60. 17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3e58d65e{/SQL,null,AVAILABLE,@Spark}  
  61. 17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@63998bf4{/SQL/json,null,AVAILABLE,@Spark}  
  62. 17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@62a8fd44{/SQL/execution,null,AVAILABLE,@Spark}  
  63. 17/05/17 16:31:42 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f5b5ca4{/SQL/execution/json,null,AVAILABLE,@Spark}  
  64. 17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4cfbaf4{/static/sql,null,AVAILABLE,@Spark}  
  65. 17/05/17 16:31:44 INFO spark.SparkContext: Starting job: reduce atSparkPi.scala:38  
  66. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: Got job 0 (reduce atSparkPi.scala:38) with 2 output partitions  
  67. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: Final stage:ResultStage 0 (reduce at SparkPi.scala:38)  
  68. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: Parents of finalstage: List()  
  69. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: Missing parents:List()  
  70. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: SubmittingResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has nomissing parents  
  71. 17/05/17 16:31:44 INFO memory.MemoryStore: Block broadcast_0 storedas values in memory (estimated size 1832.0 B, free 413.9 MB)  
  72. 17/05/17 16:31:44 INFO memory.MemoryStore: Block broadcast_0_piece0stored as bytes in memory (estimated size 1167.0 B, free 413.9 MB)  
  73. 17/05/17 16:31:44 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.143:41396 (size: 1167.0 B, free:413.9 MB)  
  74. 17/05/17 16:31:44 INFO spark.SparkContext: Created broadcast 0 frombroadcast at DAGScheduler.scala:996  
  75. 17/05/17 16:31:44 INFO scheduler.DAGScheduler: Submitting 2 missingtasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)  
  76. 17/05/17 16:31:44 INFO scheduler.TaskSchedulerImpl: Adding task set0.0 with 2 tasks  
  77. 17/05/17 16:31:45 INFO scheduler.TaskSetManager: Starting task 0.0in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL,6026 bytes)  
  78. 17/05/17 16:31:45 INFO executor.Executor: Running task 0.0 in stage0.0 (TID 0)  
  79. 17/05/17 16:31:45 INFO executor.Executor: Fetchingspark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar with timestamp1495009901648  
  80. 17/05/17 16:31:45 INFO client.TransportClientFactory: Successfullycreated connection to /192.168.27.143:41012 after 106 ms (0 ms spent inbootstraps)  
  81. 17/05/17 16:31:45 INFO util.Utils: Fetchingspark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar to/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724/userFiles-d086a0e3-7a29-4d7e-9667-6156186a89c8/fetchFileTemp942491911381944893.tmp  
  82. 17/05/17 16:31:45 INFO executor.Executor: Addingfile:/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724/userFiles-d086a0e3-7a29-4d7e-9667-6156186a89c8/spark-examples_2.11-2.1.1.jarto class loader  
  83. 17/05/17 16:31:46 INFO executor.Executor: Finished task 0.0 in stage0.0 (TID 0). 1114 bytes result sent to driver  
  84. 17/05/17 16:31:46 INFO scheduler.TaskSetManager: Starting task 1.0in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL,6026 bytes)  
  85. 17/05/17 16:31:46 INFO executor.Executor: Running task 1.0 in stage0.0 (TID 1)  
  86. 17/05/17 16:31:46 INFO executor.Executor: Finished task 1.0 in stage0.0 (TID 1). 1041 bytes result sent to driver  
  87. 17/05/17 16:31:46 INFO scheduler.TaskSetManager: Finished task 0.0in stage 0.0 (TID 0) in 1182 ms on localhost (executor driver) (1/2)  
  88. 17/05/17 16:31:46 INFO scheduler.TaskSetManager: Finished task 1.0in stage 0.0 (TID 1) in 162 ms on localhost (executor driver) (2/2)  
  89. 17/05/17 16:31:46 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool  
  90. 17/05/17 16:31:46 INFO scheduler.DAGScheduler: ResultStage 0 (reduceat SparkPi.scala:38) finished in 1.284 s  
  91. 17/05/17 16:31:46 INFO scheduler.DAGScheduler: Job 0 finished:reduce at SparkPi.scala:38, took 2.135304 s  
  92. Pi is roughly 3.1400557002785012  
  93. 17/05/17 16:31:46 INFO server.ServerConnector: StoppedSpark@32fe9d0a{HTTP/1.1}{0.0.0.0:4040}  
  94. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@290b1b2e{/stages/stage/kill,null,UNAVAILABLE,@Spark}  
  95. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1fc0053e{/jobs/job/kill,null,UNAVAILABLE,@Spark}  
  96. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@77307458{/api,null,UNAVAILABLE,@Spark}  
  97. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@389adf1d{/,null,UNAVAILABLE,@Spark}  
  98. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7bf9b098{/static,null,UNAVAILABLE,@Spark}  
  99. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72e34f77{/executors/threadDump/json,null,UNAVAILABLE,@Spark}  
  100. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6e9319f{/executors/threadDump,null,UNAVAILABLE,@Spark}  
  101. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6fa590ba{/executors/json,null,UNAVAILABLE,@Spark}  
  102. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@2416a51{/executors,null,UNAVAILABLE,@Spark}  
  103. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@293bb8a5{/environment/json,null,UNAVAILABLE,@Spark}  
  104. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@37ebc9d8{/environment,null,UNAVAILABLE,@Spark}  
  105. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5217f3d0{/storage/rdd/json,null,UNAVAILABLE,@Spark}  
  106. 17/05/17 16:31:46 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32232e55{/storage/rdd,null,UNAVAILABLE,@Spark}  
  107. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@54361a9{/storage/json,null,UNAVAILABLE,@Spark}  
  108. 17/05/17 16:31:46 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@549621f3{/storage,null,UNAVAILABLE,@Spark}  
  109. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@332a7fce{/stages/pool/json,null,UNAVAILABLE,@Spark}  
  110. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@3c9168dc{/stages/pool,null,UNAVAILABLE,@Spark}  
  111. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72f46e16{/stages/stage/json,null,UNAVAILABLE,@Spark}  
  112. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4985cbcb{/stages/stage,null,UNAVAILABLE,@Spark}  
  113. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@71104a4{/stages/json,null,UNAVAILABLE,@Spark}  
  114. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5af28b27{/stages,null,UNAVAILABLE,@Spark}  
  115. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/job/json,null,UNAVAILABLE,@Spark}  
  116. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7876d598{/jobs/job,null,UNAVAILABLE,@Spark}  
  117. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@15a902e7{/jobs/json,null,UNAVAILABLE,@Spark}  
  118. 17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5118388b{/jobs,null,UNAVAILABLE,@Spark}  
  119. 17/05/17 16:31:46 INFO ui.SparkUI: Stopped Spark web UI athttp://192.168.27.143:4040  
  120. 17/05/17 16:31:46 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpointstopped!  
  121. 17/05/17 16:31:46 INFO memory.MemoryStore: MemoryStore cleared  
  122. 17/05/17 16:31:46 INFO storage.BlockManager: BlockManager stopped  
  123. 17/05/17 16:31:46 INFO storage.BlockManagerMaster:BlockManagerMaster stopped  
  124. 17/05/17 16:31:46 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!  
  125. 17/05/17 16:31:46 INFO spark.SparkContext: Successfully stoppedSparkContext  
  126. 17/05/17 16:31:46 INFO util.ShutdownHookManager: Shutdown hook called  
  127. 17/05/17 16:31:46 INFO util.ShutdownHookManager: Deleting directory/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724  
  128. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  

2  使用独立的Spark集群模式提交任务

            这种模式也就是Standalone模式，使用独立的Spark集群模式提交任务，需要先启动Spark集群，但是不需要启动Hadoop集群。启动Spark集群的方法是进入$SPARK_HOME/sbin目录下，执行start-all.sh脚本，启动成功后，可以访问下面的地址看是否成功：

        http://Spark的Marster机器的IP:8080/

如图：

   

      说明：Spark集群启动成功后。开始执行下面的一系列命令。

      进入到Spark2.1.1的安装目录，命令是：

[plain]  view plain  copy
  1. cd   /opt/spark/spark-2.1.1-bin-hadoop2.7  

        执行命令，用Standalone模式运行计算圆周率的Demo：

[plain]  view plain  copy
  1. ./bin/spark-submit   --class   org.apache.spark.examples.SparkPi   --master   spark://192.168.27.143:7077   examples/jars/spark-examples_2.11-2.1.1.jar  

如图：


稍等几秒就OK了

如图：


完整信息是：

[plain]  view plain  copy
  1. [root@hserver1 sbin]# cd /opt/spark/spark-2.1.1-bin-hadoop2.7  
  2. [root@hserver1 spark-2.1.1-bin-hadoop2.7]# ./bin/spark-submit--class org.apache.spark.examples.SparkPi --master spark://192.168.27.143:7077examples/jars/spark-examples_2.11-2.1.1.jar  
  3. 17/05/17 17:05:44 INFO spark.SparkContext: Running Spark version2.1.1  
  4. 17/05/17 17:05:45 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable  
  5. 17/05/17 17:05:45 INFO spark.SecurityManager: Changing view acls to:root  
  6. 17/05/17 17:05:45 INFO spark.SecurityManager: Changing modify aclsto: root  
  7. 17/05/17 17:05:45 INFO spark.SecurityManager: Changing view aclsgroups to:  
  8. 17/05/17 17:05:45 INFO spark.SecurityManager: Changing modify aclsgroups to:  
  9. 17/05/17 17:05:45 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups withview permissions: Set(); users  withmodify permissions: Set(root); groups with modify permissions: Set()  
  10. 17/05/17 17:05:46 INFO util.Utils: Successfully started service'sparkDriver' on port 39853.  
  11. 17/05/17 17:05:46 INFO spark.SparkEnv: Registering MapOutputTracker  
  12. 17/05/17 17:05:46 INFO spark.SparkEnv: RegisteringBlockManagerMaster  
  13. 17/05/17 17:05:46 INFO storage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information  
  14. 17/05/17 17:05:46 INFO storage.BlockManagerMasterEndpoint:BlockManagerMasterEndpoint up  
  15. 17/05/17 17:05:46 INFO storage.DiskBlockManager: Created localdirectory at /tmp/blockmgr-50829b10-3ea0-4feb-b495-c23ada2aa90e  
  16. 17/05/17 17:05:46 INFO memory.MemoryStore: MemoryStore started withcapacity 413.9 MB  
  17. 17/05/17 17:05:46 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator  
  18. 17/05/17 17:05:47 INFO util.log: Logging initialized @4733ms  
  19. 17/05/17 17:05:47 INFO server.Server: jetty-9.2.z-SNAPSHOT  
  20. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7876d598{/jobs,null,AVAILABLE,@Spark}  
  21. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/json,null,AVAILABLE,@Spark}  
  22. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5af28b27{/jobs/job,null,AVAILABLE,@Spark}  
  23. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@71104a4{/jobs/job/json,null,AVAILABLE,@Spark}  
  24. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4985cbcb{/stages,null,AVAILABLE,@Spark}  
  25. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72f46e16{/stages/json,null,AVAILABLE,@Spark}  
  26. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3c9168dc{/stages/stage,null,AVAILABLE,@Spark}  
  27. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@332a7fce{/stages/stage/json,null,AVAILABLE,@Spark}  
  28. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@549621f3{/stages/pool,null,AVAILABLE,@Spark}  
  29. 17/05/17 17:05:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54361a9{/stages/pool/json,null,AVAILABLE,@Spark}  
  30. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@32232e55{/storage,null,AVAILABLE,@Spark}  
  31. 17/05/17 17:05:47 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5217f3d0{/storage/json,null,AVAILABLE,@Spark}  
  32. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@37ebc9d8{/storage/rdd,null,AVAILABLE,@Spark}  
  33. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@293bb8a5{/storage/rdd/json,null,AVAILABLE,@Spark}  
  34. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@2416a51{/environment,null,AVAILABLE,@Spark}  
  35. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6fa590ba{/environment/json,null,AVAILABLE,@Spark}  
  36. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6e9319f{/executors,null,AVAILABLE,@Spark}  
  37. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72e34f77{/executors/json,null,AVAILABLE,@Spark}  
  38. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7bf9b098{/executors/threadDump,null,AVAILABLE,@Spark}  
  39. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@389adf1d{/executors/threadDump/json,null,AVAILABLE,@Spark}  
  40. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@77307458{/static,null,AVAILABLE,@Spark}  
  41. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1fc0053e{/,null,AVAILABLE,@Spark}  
  42. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@290b1b2e{/api,null,AVAILABLE,@Spark}  
  43. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@47874b25{/jobs/job/kill,null,AVAILABLE,@Spark}  
  44. 17/05/17 17:05:47 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@33617539{/stages/stage/kill,null,AVAILABLE,@Spark}  
  45. 17/05/17 17:05:47 INFO server.ServerConnector: StartedSpark@64da2a7{HTTP/1.1}{0.0.0.0:4040}  
  46. 17/05/17 17:05:47 INFO server.Server: Started @5201ms  
  47. 17/05/17 17:05:47 INFO util.Utils: Successfully started service'SparkUI' on port 4040.  
  48. 17/05/17 17:05:47 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, andstarted at http://192.168.27.143:4040  
  49. 17/05/17 17:05:47 INFO spark.SparkContext: Added JARfile:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:39853/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1495011947714  
  50. 17/05/17 17:05:47 INFO client.StandaloneAppClient$ClientEndpoint:Connecting to master spark://192.168.27.143:7077...  
  51. 17/05/17 17:05:48 INFO client.TransportClientFactory: Successfullycreated connection to /192.168.27.143:7077 after 73 ms (0 ms spent inbootstraps)  
  52. 17/05/17 17:05:48 INFO cluster.StandaloneSchedulerBackend: Connectedto Spark cluster with app ID app-20170517170548-0000  
  53. 17/05/17 17:05:48 INFO util.Utils: Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 42511.  
  54. 17/05/17 17:05:48 INFO netty.NettyBlockTransferService: Servercreated on 192.168.27.143:42511  
  55. 17/05/17 17:05:48 INFO storage.BlockManager: Usingorg.apache.spark.storage.RandomBlockReplicationPolicy for block replicationpolicy  
  56. 17/05/17 17:05:48 INFO storage.BlockManagerMaster: RegisteringBlockManager BlockManagerId(driver, 192.168.27.143, 42511, None)  
  57. 17/05/17 17:05:48 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.143:42511 with 413.9 MB RAM,BlockManagerId(driver, 192.168.27.143, 42511, None)  
  58. 17/05/17 17:05:48 INFO storage.BlockManagerMaster: RegisteredBlockManager BlockManagerId(driver, 192.168.27.143, 42511, None)  
  59. 17/05/17 17:05:48 INFO storage.BlockManager: InitializedBlockManager: BlockManagerId(driver, 192.168.27.143, 42511, None)  
  60. 17/05/17 17:05:48 INFO client.StandaloneAppClient$ClientEndpoint:Executor added: app-20170517170548-0000/0 onworker-20170517170420-192.168.27.141-42670 (192.168.27.141:42670) with 1 cores  
  61. 17/05/17 17:05:48 INFO cluster.StandaloneSchedulerBackend: Grantedexecutor ID app-20170517170548-0000/0 on hostPort 192.168.27.141:42670 with 1cores, 1024.0 MB RAM  
  62. 17/05/17 17:05:48 INFO client.StandaloneAppClient$ClientEndpoint:Executor added: app-20170517170548-0000/1 onworker-20170517170421-192.168.27.142-42367 (192.168.27.142:42367) with 1 cores  
  63. 17/05/17 17:05:48 INFO cluster.StandaloneSchedulerBackend: Grantedexecutor ID app-20170517170548-0000/1 on hostPort 192.168.27.142:42367 with 1cores, 1024.0 MB RAM  
  64. 17/05/17 17:05:48 INFO client.StandaloneAppClient$ClientEndpoint:Executor updated: app-20170517170548-0000/0 is now RUNNING  
  65. 17/05/17 17:05:48 INFO client.StandaloneAppClient$ClientEndpoint:Executor updated: app-20170517170548-0000/1 is now RUNNING  
  66. 17/05/17 17:05:49 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6974a715{/metrics/json,null,AVAILABLE,@Spark}  
  67. 17/05/17 17:05:49 INFO cluster.StandaloneSchedulerBackend:SchedulerBackend is ready for scheduling beginning after reachedminRegisteredResourcesRatio: 0.0  
  68. 17/05/17 17:05:49 INFO internal.SharedState: Warehouse path is'file:/opt/spark/spark-2.1.1-bin-hadoop2.7/spark-warehouse/'.  
  69. 17/05/17 17:05:49 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@2228db21{/SQL,null,AVAILABLE,@Spark}  
  70. 17/05/17 17:05:49 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@241a0c3a{/SQL/json,null,AVAILABLE,@Spark}  
  71. 17/05/17 17:05:49 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3f4f9acd{/SQL/execution,null,AVAILABLE,@Spark}  
  72. 17/05/17 17:05:49 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4bf324f9{/SQL/execution/json,null,AVAILABLE,@Spark}  
  73. 17/05/17 17:05:49 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ffc5491{/static/sql,null,AVAILABLE,@Spark}  
  74. 17/05/17 17:05:51 INFO spark.SparkContext: Starting job: reduce atSparkPi.scala:38  
  75. 17/05/17 17:05:51 INFO scheduler.DAGScheduler: Got job 0 (reduce atSparkPi.scala:38) with 2 output partitions  
  76. 17/05/17 17:05:51 INFO scheduler.DAGScheduler: Final stage:ResultStage 0 (reduce at SparkPi.scala:38)  
  77. 17/05/17 17:05:51 INFO scheduler.DAGScheduler: Parents of finalstage: List()  
  78. 17/05/17 17:05:51 INFO scheduler.DAGScheduler: Missing parents:List()  
  79. 17/05/17 17:05:52 INFO scheduler.DAGScheduler: SubmittingResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has nomissing parents  
  80. 17/05/17 17:05:52 INFO memory.MemoryStore: Block broadcast_0 storedas values in memory (estimated size 1832.0 B, free 413.9 MB)  
  81. 17/05/17 17:05:52 INFO memory.MemoryStore: Block broadcast_0_piece0stored as bytes in memory (estimated size 1167.0 B, free 413.9 MB)  
  82. 17/05/17 17:05:52 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.143:42511 (size: 1167.0 B, free:413.9 MB)  
  83. 17/05/17 17:05:52 INFO spark.SparkContext: Created broadcast 0 frombroadcast at DAGScheduler.scala:996  
  84. 17/05/17 17:05:52 INFO scheduler.DAGScheduler: Submitting 2 missingtasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)  
  85. 17/05/17 17:05:52 INFO scheduler.TaskSchedulerImpl: Adding task set0.0 with 2 tasks  
  86. 17/05/17 17:05:55 INFOcluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executorNettyRpcEndpointRef(null) (192.168.27.141:49512) with ID 0  
  87. 17/05/17 17:05:55 INFO scheduler.TaskSetManager: Starting task 0.0in stage 0.0 (TID 0, 192.168.27.141, executor 0, partition 0, PROCESS_LOCAL,6030 bytes)  
  88. 17/05/17 17:05:55 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.141:38597 with 413.9 MB RAM,BlockManagerId(0, 192.168.27.141, 38597, None)  
  89. 17/05/17 17:05:56 INFOcluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executorNettyRpcEndpointRef(null) (192.168.27.142:42822) with ID 1  
  90. 17/05/17 17:05:56 INFO scheduler.TaskSetManager: Starting task 1.0in stage 0.0 (TID 1, 192.168.27.142, executor 1, partition 1, PROCESS_LOCAL,6030 bytes)  
  91. 17/05/17 17:05:56 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.142:36281 with 413.9 MB RAM,BlockManagerId(1, 192.168.27.142, 36281, None)  
  92. 17/05/17 17:05:57 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.141:38597 (size: 1167.0 B, free:413.9 MB)  
  93. 17/05/17 17:05:57 INFO scheduler.TaskSetManager: Finished task 0.0in stage 0.0 (TID 0) in 2273 ms on 192.168.27.141 (executor 0) (1/2)  
  94. 17/05/17 17:05:58 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.142:36281 (size: 1167.0 B, free:413.9 MB)  
  95. 17/05/17 17:05:58 INFO scheduler.DAGScheduler: ResultStage 0 (reduceat SparkPi.scala:38) finished in 5.860 s  
  96. 17/05/17 17:05:58 INFO scheduler.TaskSetManager: Finished task 1.0in stage 0.0 (TID 1) in 2686 ms on 192.168.27.142 (executor 1) (2/2)  
  97. 17/05/17 17:05:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool  
  98. 17/05/17 17:05:58 INFO scheduler.DAGScheduler: Job 0 finished:reduce at SparkPi.scala:38, took 6.955772 s  
  99. Pi is roughly 3.1422557112785565  
  100. 17/05/17 17:05:58 INFO server.ServerConnector: StoppedSpark@64da2a7{HTTP/1.1}{0.0.0.0:4040}  
  101. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@33617539{/stages/stage/kill,null,UNAVAILABLE,@Spark}  
  102. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@47874b25{/jobs/job/kill,null,UNAVAILABLE,@Spark}  
  103. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@290b1b2e{/api,null,UNAVAILABLE,@Spark}  
  104. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1fc0053e{/,null,UNAVAILABLE,@Spark}  
  105. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@77307458{/static,null,UNAVAILABLE,@Spark}  
  106. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@389adf1d{/executors/threadDump/json,null,UNAVAILABLE,@Spark}  
  107. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7bf9b098{/executors/threadDump,null,UNAVAILABLE,@Spark}  
  108. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72e34f77{/executors/json,null,UNAVAILABLE,@Spark}  
  109. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6e9319f{/executors,null,UNAVAILABLE,@Spark}  
  110. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6fa590ba{/environment/json,null,UNAVAILABLE,@Spark}  
  111. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@2416a51{/environment,null,UNAVAILABLE,@Spark}  
  112. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@293bb8a5{/storage/rdd/json,null,UNAVAILABLE,@Spark}  
  113. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@37ebc9d8{/storage/rdd,null,UNAVAILABLE,@Spark}  
  114. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5217f3d0{/storage/json,null,UNAVAILABLE,@Spark}  
  115. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@32232e55{/storage,null,UNAVAILABLE,@Spark}  
  116. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@54361a9{/stages/pool/json,null,UNAVAILABLE,@Spark}  
  117. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@549621f3{/stages/pool,null,UNAVAILABLE,@Spark}  
  118. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@332a7fce{/stages/stage/json,null,UNAVAILABLE,@Spark}  
  119. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@3c9168dc{/stages/stage,null,UNAVAILABLE,@Spark}  
  120. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72f46e16{/stages/json,null,UNAVAILABLE,@Spark}  
  121. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4985cbcb{/stages,null,UNAVAILABLE,@Spark}  
  122. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@71104a4{/jobs/job/json,null,UNAVAILABLE,@Spark}  
  123. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5af28b27{/jobs/job,null,UNAVAILABLE,@Spark}  
  124. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/json,null,UNAVAILABLE,@Spark}  
  125. 17/05/17 17:05:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7876d598{/jobs,null,UNAVAILABLE,@Spark}  
  126. 17/05/17 17:05:58 INFO ui.SparkUI: Stopped Spark web UI athttp://192.168.27.143:4040  
  127. 17/05/17 17:05:58 INFO cluster.StandaloneSchedulerBackend: Shuttingdown all executors  
  128. 17/05/17 17:05:58 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint:Asking each executor to shut down  
  129. 17/05/17 17:05:58 INFO spark.MapOutputTrackerMasterEndpoint:MapOutputTrackerMasterEndpoint stopped!  
  130. 17/05/17 17:05:59 INFO memory.MemoryStore: MemoryStore cleared  
  131. 17/05/17 17:05:59 INFO storage.BlockManager: BlockManager stopped  
  132. 17/05/17 17:05:59 INFO storage.BlockManagerMaster:BlockManagerMaster stopped  
  133. 17/05/17 17:05:59 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!  
  134. 17/05/17 17:05:59 INFO spark.SparkContext: Successfully stoppedSparkContext  
  135. 17/05/17 17:05:59 INFO util.ShutdownHookManager: Shutdown hookcalled  
  136. 17/05/17 17:05:59 INFO util.ShutdownHookManager: Deleting directory/tmp/spark-f6058b61-f5cb-4f5f-8d01-204ae28eac2c  
  137. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  

3  使用Spark 集群+Hadoop集群的模式提交任务

             这种模式也叫On-Yarn模式，主要包括yarn-Client和yarn-Cluster两种模式。在这种模式下提交任务，需要先启动Hadoop集群，然后在启动Spark集群。

             启动Hadoop集群的方法这里不说了，有需要可以参考该博文：

          http://blog.csdn.net/pucao_cug/article/details/71698903

            Hadoop集群启动成功后，在启动Spark集群，启动方式看本文的第2章即可。

           Spark集群的安装，有需要可以参考该博文;

http://blog.csdn.net/pucao_cug/article/details/72353701

          Hadoop集群和Spark集群都启动成功后，可以接着往下操作。

3.1  用yarn-client模式执行计算程序

3.1.1   操作步骤和方法

  进入到Spark2.1.1的安装目录，命令是：

[plain]  view plain  copy
  1. cd   /opt/spark/spark-2.1.1-bin-hadoop2.7  

         执行命令，用yarn-client模式运行计算圆周率的Demo：

[plain]  view plain  copy
  1. ./bin/spark-submit   --class  org.apache.spark.examples.SparkPi  --master  yarn-client    examples/jars/spark-examples_2.11-2.1.1.jar  

如图：

稍等几秒后，即可完成计算

如图：

     完整的输出信息是：

[plain]  view plain  copy
  1. [root@hserver1 sbin]# cd   /opt/spark/spark-2.1.1-bin-hadoop2.7  
  2. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#./bin/spark-submit --class org.apache.spark.examples.SparkPi --masteryarn-client examples/jars/spark-examples_2.11-2.1.1.jar  
  3. Warning: Master yarn-client is deprecatedsince 2.0. Please use master "yarn" with specified deploy modeinstead.  
  4. 17/05/17 19:17:23 INFO spark.SparkContext:Running Spark version 2.1.1  
  5. 17/05/17 19:17:24 WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable  
  6. 17/05/17 19:17:24 INFO spark.SecurityManager:Changing view acls to: root  
  7. 17/05/17 19:17:24 INFOspark.SecurityManager: Changing modify acls to: root  
  8. 17/05/17 19:17:24 INFOspark.SecurityManager: Changing view acls groups to:  
  9. 17/05/17 19:17:24 INFOspark.SecurityManager: Changing modify acls groups to:  
  10. 17/05/17 19:17:24 INFOspark.SecurityManager: SecurityManager: authentication disabled; ui aclsdisabled; users  with view permissions:Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groupswith modify permissions: Set()  
  11. 17/05/17 19:17:25 INFO util.Utils:Successfully started service 'sparkDriver' on port 41095.  
  12. 17/05/17 19:17:25 INFO spark.SparkEnv:Registering MapOutputTracker  
  13. 17/05/17 19:17:25 INFO spark.SparkEnv:Registering BlockManagerMaster  
  14. 17/05/17 19:17:25 INFOstorage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information  
  15. 17/05/17 19:17:25 INFOstorage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up  
  16. 17/05/17 19:17:25 INFOstorage.DiskBlockManager: Created local directory at/tmp/blockmgr-fd8ef40f-5646-4eb4-b014-f687e8abcb40  
  17. 17/05/17 19:17:25 INFO memory.MemoryStore:MemoryStore started with capacity 413.9 MB  
  18. 17/05/17 19:17:25 INFO spark.SparkEnv:Registering OutputCommitCoordinator  
  19. 17/05/17 19:17:26 INFO util.log: Logginginitialized @4897ms  
  20. 17/05/17 19:17:26 INFO server.Server:jetty-9.2.z-SNAPSHOT  
  21. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@a486d78{/jobs,null,AVAILABLE,@Spark}  
  22. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@cdc3aae{/jobs/json,null,AVAILABLE,@Spark}  
  23. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7ef2d7a6{/jobs/job,null,AVAILABLE,@Spark}  
  24. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5dcbb60{/jobs/job/json,null,AVAILABLE,@Spark}  
  25. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4c36250e{/stages,null,AVAILABLE,@Spark}  
  26. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@21526f6c{/stages/json,null,AVAILABLE,@Spark}  
  27. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@49f5c307{/stages/stage,null,AVAILABLE,@Spark}  
  28. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@299266e2{/stages/stage/json,null,AVAILABLE,@Spark}  
  29. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5471388b{/stages/pool,null,AVAILABLE,@Spark}  
  30. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@66ea1466{/stages/pool/json,null,AVAILABLE,@Spark}  
  31. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@1601e47{/storage,null,AVAILABLE,@Spark}  
  32. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3bffddff{/storage/json,null,AVAILABLE,@Spark}  
  33. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@66971f6b{/storage/rdd,null,AVAILABLE,@Spark}  
  34. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@50687efb{/storage/rdd/json,null,AVAILABLE,@Spark}  
  35. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@517bd097{/environment,null,AVAILABLE,@Spark}  
  36. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@142eef62{/environment/json,null,AVAILABLE,@Spark}  
  37. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a9cc6cb{/executors,null,AVAILABLE,@Spark}  
  38. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5990e6c5{/executors/json,null,AVAILABLE,@Spark}  
  39. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@56e07a08{/executors/threadDump,null,AVAILABLE,@Spark}  
  40. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@35d6ca49{/executors/threadDump/json,null,AVAILABLE,@Spark}  
  41. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1950e8a6{/static,null,AVAILABLE,@Spark}  
  42. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@47289387{/,null,AVAILABLE,@Spark}  
  43. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@12cd9150{/api,null,AVAILABLE,@Spark}  
  44. 17/05/17 19:17:26 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@114a85c2{/jobs/job/kill,null,AVAILABLE,@Spark}  
  45. 17/05/17 19:17:26 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@f415a95{/stages/stage/kill,null,AVAILABLE,@Spark}  
  46. 17/05/17 19:17:26 INFOserver.ServerConnector: Started Spark@21d5c1a0{HTTP/1.1}{0.0.0.0:4040}  
  47. 17/05/17 19:17:26 INFO server.Server:Started @5240ms  
  48. 17/05/17 19:17:26 INFO util.Utils: Successfullystarted service 'SparkUI' on port 4040.  
  49. 17/05/17 19:17:26 INFO ui.SparkUI: BoundSparkUI to 0.0.0.0, and started at http://192.168.27.143:4040  
  50. 17/05/17 19:17:26 INFO spark.SparkContext:Added JAR file:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:41095/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1495019846524  
  51. 17/05/17 19:17:28 INFO client.RMProxy:Connecting to ResourceManager at hserver1/192.168.27.143:8032  
  52. 17/05/17 19:17:29 INFO yarn.Client:Requesting a new application from cluster with 2 NodeManagers  
  53. 17/05/17 19:17:29 INFO yarn.Client:Verifying our application has not requested more than the maximum memorycapability of the cluster (2048 MB per container)  
  54. 17/05/17 19:17:29 INFO yarn.Client: Willallocate AM container, with 896 MB memory including 384 MB overhead  
  55. 17/05/17 19:17:29 INFO yarn.Client: Settingup container launch context for our AM  
  56. 17/05/17 19:17:29 INFO yarn.Client: Settingup the launch environment for our AM container  
  57. 17/05/17 19:17:29 INFO yarn.Client:Preparing resources for our AM container  
  58. 17/05/17 19:17:33 WARN yarn.Client: Neitherspark.yarn.jars nor spark.yarn.archive is set, falling back to uploadinglibraries under SPARK_HOME.  
  59. 17/05/17 19:17:39 INFO yarn.Client:Uploading resourcefile:/tmp/spark-73613bca-fc94-455a-a152-d9b38f3940fd/__spark_libs__4346568075628334835.zip->hdfs://hserver1:9000/user/root/.sparkStaging/application_1495019556890_0001/__spark_libs__4346568075628334835.zip  
  60. 17/05/17 19:17:47 INFO yarn.Client:Uploading resourcefile:/tmp/spark-73613bca-fc94-455a-a152-d9b38f3940fd/__spark_conf__4541417269674378465.zip->hdfs://hserver1:9000/user/root/.sparkStaging/application_1495019556890_0001/__spark_conf__.zip  
  61. 17/05/17 19:17:47 INFOspark.SecurityManager: Changing view acls to: root  
  62. 17/05/17 19:17:47 INFOspark.SecurityManager: Changing modify acls to: root  
  63. 17/05/17 19:17:47 INFOspark.SecurityManager: Changing view acls groups to:  
  64. 17/05/17 19:17:47 INFOspark.SecurityManager: Changing modify acls groups to:  
  65. 17/05/17 19:17:47 INFOspark.SecurityManager: SecurityManager: authentication disabled; ui aclsdisabled; users  with view permissions:Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groupswith modify permissions: Set()  
  66. 17/05/17 19:17:47 INFO yarn.Client:Submitting application application_1495019556890_0001 to ResourceManager  
  67. 17/05/17 19:17:47 INFO impl.YarnClientImpl:Submitted application application_1495019556890_0001  
  68. 17/05/17 19:17:47 INFOcluster.SchedulerExtensionServices: Starting Yarn extension services with appapplication_1495019556890_0001 and attemptId None  
  69. 17/05/17 19:17:49 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  70. 17/05/17 19:17:49 INFO yarn.Client:  
  71.         client token: N/A  
  72.         diagnostics: [Wed May 17 19:17:48 +0800 2017] Scheduler has assigned acontainer for AM, waiting for AM container to be launched  
  73.         ApplicationMaster host: N/A  
  74.         ApplicationMaster RPC port: -1  
  75.         queue: default  
  76.         start time: 1495019867505  
  77.         final status: UNDEFINED  
  78.         tracking URL: http://hserver1:8088/proxy/application_1495019556890_0001/  
  79.         user: root  
  80. 17/05/17 19:17:50 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  81. 17/05/17 19:17:51 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  82. 17/05/17 19:17:52 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  83. 17/05/17 19:17:53 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  84. 17/05/17 19:17:54 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  85. 17/05/17 19:17:55 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  86. 17/05/17 19:17:56 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  87. 17/05/17 19:17:57 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  88. 17/05/17 19:17:58 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  89. 17/05/17 19:17:59 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  90. 17/05/17 19:18:00 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  91. 17/05/17 19:18:01 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  92. 17/05/17 19:18:02 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  93. 17/05/17 19:18:03 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  94. 17/05/17 19:18:04 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  95. 17/05/17 19:18:05 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  96. 17/05/17 19:18:06 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  97. 17/05/17 19:18:06 INFOcluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMasterregistered as NettyRpcEndpointRef(null)  
  98. 17/05/17 19:18:07 INFOcluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> hserver1, PROXY_URI_BASES ->http://hserver1:8088/proxy/application_1495019556890_0001),/proxy/application_1495019556890_0001  
  99. 17/05/17 19:18:07 INFO ui.JettyUtils:Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter  
  100. 17/05/17 19:18:07 INFO yarn.Client:Application report for application_1495019556890_0001 (state: ACCEPTED)  
  101. 17/05/17 19:18:08 INFO yarn.Client:Application report for application_1495019556890_0001 (state: RUNNING)  
  102. 17/05/17 19:18:08 INFO yarn.Client:  
  103.         client token: N/A  
  104.         diagnostics: N/A  
  105.         ApplicationMaster host: 192.168.27.142  
  106.         ApplicationMaster RPC port: 0  
  107.         queue: default  
  108.         start time: 1495019867505  
  109.         final status: UNDEFINED  
  110.         tracking URL: http://hserver1:8088/proxy/application_1495019556890_0001/  
  111.         user: root  
  112. 17/05/17 19:18:08 INFOcluster.YarnClientSchedulerBackend: Application application_1495019556890_0001has started running.  
  113. 17/05/17 19:18:08 INFO util.Utils:Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 40716.  
  114. 17/05/17 19:18:08 INFOnetty.NettyBlockTransferService: Server created on 192.168.27.143:40716  
  115. 17/05/17 19:18:08 INFO storage.BlockManager:Using org.apache.spark.storage.RandomBlockReplicationPolicy for blockreplication policy  
  116. 17/05/17 19:18:08 INFOstorage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver,192.168.27.143, 40716, None)  
  117. 17/05/17 19:18:08 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.143:40716 with 413.9 MB RAM,BlockManagerId(driver, 192.168.27.143, 40716, None)  
  118. 17/05/17 19:18:08 INFOstorage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver,192.168.27.143, 40716, None)  
  119. 17/05/17 19:18:08 INFOstorage.BlockManager: Initialized BlockManager: BlockManagerId(driver,192.168.27.143, 40716, None)  
  120. 17/05/17 19:18:08 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@27960b1e{/metrics/json,null,AVAILABLE,@Spark}  
  121. 17/05/17 19:18:08 INFOcluster.YarnClientSchedulerBackend: SchedulerBackend is ready for schedulingbeginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)  
  122. 17/05/17 19:18:09 INFOinternal.SharedState: Warehouse path is 'file:/opt/spark/spark-2.1.1-bin-hadoop2.7/spark-warehouse/'.  
  123. 17/05/17 19:18:09 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@2fc49538{/SQL,null,AVAILABLE,@Spark}  
  124. 17/05/17 19:18:09 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@d5e575c{/SQL/json,null,AVAILABLE,@Spark}  
  125. 17/05/17 19:18:09 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1cd6b1bd{/SQL/execution,null,AVAILABLE,@Spark}  
  126. 17/05/17 19:18:09 INFOhandler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b41e1bf{/SQL/execution/json,null,AVAILABLE,@Spark}  
  127. 17/05/17 19:18:09 INFOhandler.ContextHandler: Startedo.s.j.s.ServletContextHandler@67e0fd6d{/static/sql,null,AVAILABLE,@Spark}  
  128. 17/05/17 19:18:10 INFO spark.SparkContext:Starting job: reduce at SparkPi.scala:38  
  129. 17/05/17 19:18:10 INFOscheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 outputpartitions  
  130. 17/05/17 19:18:10 INFOscheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)  
  131. 17/05/17 19:18:10 INFOscheduler.DAGScheduler: Parents of final stage: List()  
  132. 17/05/17 19:18:10 INFOscheduler.DAGScheduler: Missing parents: List()  
  133. 17/05/17 19:18:10 INFOscheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map atSparkPi.scala:34), which has no missing parents  
  134. 17/05/17 19:18:10 INFO memory.MemoryStore:Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free413.9 MB)  
  135. 17/05/17 19:18:11 INFO memory.MemoryStore:Block broadcast_0_piece0 stored as bytes in memory (estimated size 1167.0 B,free 413.9 MB)  
  136. 17/05/17 19:18:11 INFOstorage.BlockManagerInfo: Added broadcast_0_piece0 in memory on192.168.27.143:40716 (size: 1167.0 B, free: 413.9 MB)  
  137. 17/05/17 19:18:11 INFO spark.SparkContext:Created broadcast 0 from broadcast at DAGScheduler.scala:996  
  138. 17/05/17 19:18:11 INFOscheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0(MapPartitionsRDD[1] at map at SparkPi.scala:34)  
  139. 17/05/17 19:18:11 INFOcluster.YarnScheduler: Adding task set 0.0 with 2 tasks  
  140. 17/05/17 19:18:26 WARNcluster.YarnScheduler: Initial job has not accepted any resources; check yourcluster UI to ensure that workers are registered and have sufficient resources  
  141. 17/05/17 19:18:34 INFOcluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executorNettyRpcEndpointRef(null) (192.168.27.141:44490) with ID 1  
  142. 17/05/17 19:18:34 INFOscheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, hserver2,executor 1, partition 0, PROCESS_LOCAL, 6037 bytes)  
  143. 17/05/17 19:18:35 INFO storage.BlockManagerMasterEndpoint:Registering block manager hserver2:32930 with 413.9 MB RAM, BlockManagerId(1,hserver2, 32930, None)  
  144. 17/05/17 19:18:40 INFOstorage.BlockManagerInfo: Added broadcast_0_piece0 in memory on hserver2:32930(size: 1167.0 B, free: 413.9 MB)  
  145. 17/05/17 19:18:43 INFOscheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, hserver2,executor 1, partition 1, PROCESS_LOCAL, 6037 bytes)  
  146. 17/05/17 19:18:43 INFOscheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 9093 ms onhserver2 (executor 1) (1/2)  
  147. 17/05/17 19:18:43 INFOscheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 367 ms onhserver2 (executor 1) (2/2)  
  148. 17/05/17 19:18:43 INFOcluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed,from pool  
  149. 17/05/17 19:18:43 INFOscheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in32.711 s  
  150. 17/05/17 19:18:43 INFOscheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took33.620405 s  
  151. Pi is roughly 3.139235696178481  
  152. 17/05/17 19:18:43 INFOserver.ServerConnector: Stopped Spark@21d5c1a0{HTTP/1.1}{0.0.0.0:4040}  
  153. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@f415a95{/stages/stage/kill,null,UNAVAILABLE,@Spark}  
  154. 17/05/17 19:18:43 INFO handler.ContextHandler:Stoppedo.s.j.s.ServletContextHandler@114a85c2{/jobs/job/kill,null,UNAVAILABLE,@Spark}  
  155. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@12cd9150{/api,null,UNAVAILABLE,@Spark}  
  156. 17/05/17 19:18:43 INFO handler.ContextHandler:Stopped o.s.j.s.ServletContextHandler@47289387{/,null,UNAVAILABLE,@Spark}  
  157. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1950e8a6{/static,null,UNAVAILABLE,@Spark}  
  158. 17/05/17 19:18:43 INFO handler.ContextHandler:Stoppedo.s.j.s.ServletContextHandler@35d6ca49{/executors/threadDump/json,null,UNAVAILABLE,@Spark}  
  159. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@56e07a08{/executors/threadDump,null,UNAVAILABLE,@Spark}  
  160. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5990e6c5{/executors/json,null,UNAVAILABLE,@Spark}  
  161. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4a9cc6cb{/executors,null,UNAVAILABLE,@Spark}  
  162. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@142eef62{/environment/json,null,UNAVAILABLE,@Spark}  
  163. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@517bd097{/environment,null,UNAVAILABLE,@Spark}  
  164. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@50687efb{/storage/rdd/json,null,UNAVAILABLE,@Spark}  
  165. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@66971f6b{/storage/rdd,null,UNAVAILABLE,@Spark}  
  166. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@3bffddff{/storage/json,null,UNAVAILABLE,@Spark}  
  167. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1601e47{/storage,null,UNAVAILABLE,@Spark}  
  168. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@66ea1466{/stages/pool/json,null,UNAVAILABLE,@Spark}  
  169. 17/05/17 19:18:43 INFO handler.ContextHandler:Stoppedo.s.j.s.ServletContextHandler@5471388b{/stages/pool,null,UNAVAILABLE,@Spark}  
  170. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@299266e2{/stages/stage/json,null,UNAVAILABLE,@Spark}  
  171. 17/05/17 19:18:43 INFO handler.ContextHandler:Stoppedo.s.j.s.ServletContextHandler@49f5c307{/stages/stage,null,UNAVAILABLE,@Spark}  
  172. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@21526f6c{/stages/json,null,UNAVAILABLE,@Spark}  
  173. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4c36250e{/stages,null,UNAVAILABLE,@Spark}  
  174. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5dcbb60{/jobs/job/json,null,UNAVAILABLE,@Spark}  
  175. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7ef2d7a6{/jobs/job,null,UNAVAILABLE,@Spark}  
  176. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@cdc3aae{/jobs/json,null,UNAVAILABLE,@Spark}  
  177. 17/05/17 19:18:43 INFOhandler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@a486d78{/jobs,null,UNAVAILABLE,@Spark}  
  178. 17/05/17 19:18:44 INFO ui.SparkUI: StoppedSpark web UI at http://192.168.27.143:4040  
  179. 17/05/17 19:18:44 INFOcluster.YarnClientSchedulerBackend: Interrupting monitor thread  
  180. 17/05/17 19:18:44 INFOcluster.YarnClientSchedulerBackend: Shutting down all executors  
  181. 17/05/17 19:18:44 INFOcluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shutdown  
  182. 17/05/17 19:18:44 INFO cluster.SchedulerExtensionServices:Stopping SchedulerExtensionServices  
  183. (serviceOption=None,  
  184.  services=List(),  
  185.  started=false)  
  186. 17/05/17 19:18:44 INFOcluster.YarnClientSchedulerBackend: Stopped  
  187. 17/05/17 19:18:44 INFOspark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!  
  188. 17/05/17 19:18:44 INFO memory.MemoryStore:MemoryStore cleared  
  189. 17/05/17 19:18:44 INFOstorage.BlockManager: BlockManager stopped  
  190. 17/05/17 19:18:44 INFOstorage.BlockManagerMaster: BlockManagerMaster stopped  
  191. 17/05/17 19:18:44 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!  
  192. 17/05/17 19:18:44 INFO spark.SparkContext:Successfully stopped SparkContext  
  193. 17/05/17 19:18:44 INFOutil.ShutdownHookManager: Shutdown hook called  
  194. 17/05/17 19:18:44 INFOutil.ShutdownHookManager: Deleting directory/tmp/spark-73613bca-fc94-455a-a152-d9b38f3940fd  
  195. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  

3.1.2  常见错误解决

        在yarn-client模式下，需要访问spark集群和hadoop集群，如果配置不对，容易出问题。常见错误如下：

3.1.2.1  Yarn application has already ended!
     3.1.2.1.1 主要错误信息

[plain]  view plain  copy
  1.  Exceptionin thread "main" org.apache.spark.SparkException: Yarn applicationhas already ended! It might have been killed or unable to launch applicationmaster.  
  2.        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)  
  3.        atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)  
  4.        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)  
  5.        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)  
  6.        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)  
  7.        atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)  
  8.        atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)  
  9.        at scala.Option.getOrElse(Option.scala:121)  
  10.        atorg.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)  
  11.        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)  
  12.        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)  
  13.        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
  14.        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
  15.        atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
  16.        at java.lang.reflect.Method.invoke(Method.java:498)  
  17.        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)  
  18.        atorg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)  
  19.        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)  
  20.        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)  
  21.        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)  
  22. 17/05/17 15:16:15 INFOutil.ShutdownHookManager: Shutdown hook called  
  23. 17/05/17 15:16:15 INFOutil.ShutdownHookManager: Deleting directory/tmp/spark-3cc28864-370d-4a14-a892-f31a67bd2023  

完整输出如下：

[plain]  view plain  copy
  1. [root@hserver1 sbin]# cd /opt/spark/spark-2.1.1-bin-hadoop2.7  
  2. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  ./bin/spark-submit --classorg.apache.spark.examples.SparkPi --master yarn-clientexamples/jars/spark-examples_2.11-2.1.1.jar  
  3. Warning: Master yarn-client is deprecated since 2.0. Please usemaster "yarn" with specified deploy mode instead.  
  4. 17/05/17 20:40:03 INFO spark.SparkContext: Running Spark version2.1.1  
  5. 17/05/17 20:40:04 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable  
  6. 17/05/17 20:40:04 INFO spark.SecurityManager: Changing view acls to:root  
  7. 17/05/17 20:40:04 INFO spark.SecurityManager: Changing modify aclsto: root  
  8. 17/05/17 20:40:04 INFO spark.SecurityManager: Changing view aclsgroups to:  
  9. 17/05/17 20:40:04 INFO spark.SecurityManager: Changing modify aclsgroups to:  
  10. 17/05/17 20:40:04 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups withview permissions: Set(); users  withmodify permissions: Set(root); groups with modify permissions: Set()  
  11. 17/05/17 20:40:05 INFO util.Utils: Successfully started service 'sparkDriver'on port 38765.  
  12. 17/05/17 20:40:05 INFO spark.SparkEnv: Registering MapOutputTracker  
  13. 17/05/17 20:40:05 INFO spark.SparkEnv: RegisteringBlockManagerMaster  
  14. 17/05/17 20:40:05 INFO storage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information  
  15. 17/05/17 20:40:05 INFO storage.BlockManagerMasterEndpoint:BlockManagerMasterEndpoint up  
  16. 17/05/17 20:40:05 INFO storage.DiskBlockManager: Created localdirectory at /tmp/blockmgr-59574c4e-a88b-4bbb-8e81-27c68d536de4  
  17. 17/05/17 20:40:05 INFO memory.MemoryStore: MemoryStore started withcapacity 413.9 MB  
  18. 17/05/17 20:40:06 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator  
  19. 17/05/17 20:40:06 INFO util.log: Logging initialized @4932ms  
  20. 17/05/17 20:40:06 INFO server.Server: jetty-9.2.z-SNAPSHOT  
  21. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@a486d78{/jobs,null,AVAILABLE,@Spark}  
  22. 17/05/17 20:40:06 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@cdc3aae{/jobs/json,null,AVAILABLE,@Spark}  
  23. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7ef2d7a6{/jobs/job,null,AVAILABLE,@Spark}  
  24. 17/05/17 20:40:06 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5dcbb60{/jobs/job/json,null,AVAILABLE,@Spark}  
  25. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4c36250e{/stages,null,AVAILABLE,@Spark}  
  26. 17/05/17 20:40:06 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@21526f6c{/stages/json,null,AVAILABLE,@Spark}  
  27. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@49f5c307{/stages/stage,null,AVAILABLE,@Spark}  
  28. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@299266e2{/stages/stage/json,null,AVAILABLE,@Spark}  
  29. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5471388b{/stages/pool,null,AVAILABLE,@Spark}  
  30. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@66ea1466{/stages/pool/json,null,AVAILABLE,@Spark}  
  31. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1601e47{/storage,null,AVAILABLE,@Spark}  
  32. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3bffddff{/storage/json,null,AVAILABLE,@Spark}  
  33. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@66971f6b{/storage/rdd,null,AVAILABLE,@Spark}  
  34. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@50687efb{/storage/rdd/json,null,AVAILABLE,@Spark}  
  35. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@517bd097{/environment,null,AVAILABLE,@Spark}  
  36. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@142eef62{/environment/json,null,AVAILABLE,@Spark}  
  37. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4a9cc6cb{/executors,null,AVAILABLE,@Spark}  
  38. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5990e6c5{/executors/json,null,AVAILABLE,@Spark}  
  39. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@56e07a08{/executors/threadDump,null,AVAILABLE,@Spark}  
  40. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@35d6ca49{/executors/threadDump/json,null,AVAILABLE,@Spark}  
  41. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1950e8a6{/static,null,AVAILABLE,@Spark}  
  42. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@47289387{/,null,AVAILABLE,@Spark}  
  43. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@12cd9150{/api,null,AVAILABLE,@Spark}  
  44. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@114a85c2{/jobs/job/kill,null,AVAILABLE,@Spark}  
  45. 17/05/17 20:40:06 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@f415a95{/stages/stage/kill,null,AVAILABLE,@Spark}  
  46. 17/05/17 20:40:06 INFO server.ServerConnector: StartedSpark@21d5c1a0{HTTP/1.1}{0.0.0.0:4040}  
  47. 17/05/17 20:40:06 INFO server.Server: Started @5350ms  
  48. 17/05/17 20:40:06 INFO util.Utils: Successfully started service'SparkUI' on port 4040.  
  49. 17/05/17 20:40:06 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, andstarted at http://192.168.27.143:4040  
  50. 17/05/17 20:40:06 INFO spark.SparkContext: Added JARfile:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:38765/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1495024806742  
  51. 17/05/17 20:40:08 INFO client.RMProxy: Connecting to ResourceManagerat hserver1/192.168.27.143:8032  
  52. 17/05/17 20:40:09 INFO yarn.Client: Requesting a new applicationfrom cluster with 2 NodeManagers  
  53. 17/05/17 20:40:09 INFO yarn.Client: Verifying our application hasnot requested more than the maximum memory capability of the cluster (2048 MBper container)  
  54. 17/05/17 20:40:09 INFO yarn.Client: Will allocate AM container, with896 MB memory including 384 MB overhead  
  55. 17/05/17 20:40:09 INFO yarn.Client: Setting up container launchcontext for our AM  
  56. 17/05/17 20:40:09 INFO yarn.Client: Setting up the launchenvironment for our AM container  
  57. 17/05/17 20:40:09 INFO yarn.Client: Preparing resources for our AMcontainer  
  58. 17/05/17 20:40:13 WARN yarn.Client: Neither spark.yarn.jars norspark.yarn.archive is set, falling back to uploading libraries underSPARK_HOME.  
  59. 17/05/17 20:40:18 INFO yarn.Client: Uploading resourcefile:/tmp/spark-5e00aae5-dc98-48fd-83b4-978714e002d4/__spark_libs__2259912550909667048.zip->hdfs://hserver1:9000/user/root/.sparkStaging/application_1495024538278_0001/__spark_libs__2259912550909667048.zip  
  60. 17/05/17 20:40:27 INFO yarn.Client: Uploading resourcefile:/tmp/spark-5e00aae5-dc98-48fd-83b4-978714e002d4/__spark_conf__5098178835931124874.zip->hdfs://hserver1:9000/user/root/.sparkStaging/application_1495024538278_0001/__spark_conf__.zip  
  61. 17/05/17 20:40:27 INFO spark.SecurityManager: Changing view acls to:root  
  62. 17/05/17 20:40:27 INFO spark.SecurityManager: Changing modify aclsto: root  
  63. 17/05/17 20:40:27 INFO spark.SecurityManager: Changing view aclsgroups to:  
  64. 17/05/17 20:40:27 INFO spark.SecurityManager: Changing modify aclsgroups to:  
  65. 17/05/17 20:40:27 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups withview permissions: Set(); users  withmodify permissions: Set(root); groups with modify permissions: Set()  
  66. 17/05/17 20:40:27 INFO yarn.Client: Submitting applicationapplication_1495024538278_0001 to ResourceManager  
  67. 17/05/17 20:40:28 INFO impl.YarnClientImpl: Submitted applicationapplication_1495024538278_0001  
  68. 17/05/17 20:40:28 INFO cluster.SchedulerExtensionServices: StartingYarn extension services with app application_1495024538278_0001 and attemptIdNone  
  69. 17/05/17 20:40:29 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  70. 17/05/17 20:40:29 INFO yarn.Client:  
  71.          client token: N/A  
  72.          diagnostics: [Wed May17 20:40:28 +0800 2017] Scheduler has assigned a container for AM, waiting forAM container to be launched  
  73.          ApplicationMasterhost: N/A  
  74.          ApplicationMaster RPCport: -1  
  75.          queue: default  
  76.          start time:1495024828093  
  77.          final status:UNDEFINED  
  78.          tracking URL:http://hserver1:8088/proxy/application_1495024538278_0001/  
  79.          user: root  
  80. 17/05/17 20:40:30 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  81. 17/05/17 20:40:31 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  82. 17/05/17 20:40:32 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  83. 17/05/17 20:40:33 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  84. 17/05/17 20:40:34 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  85. 17/05/17 20:40:35 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  86. 17/05/17 20:40:36 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  87. 17/05/17 20:40:37 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  88. 17/05/17 20:40:38 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  89. 17/05/17 20:40:39 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  90. 17/05/17 20:40:40 INFO yarn.Client: Application report for application_1495024538278_0001(state: ACCEPTED)  
  91. 17/05/17 20:40:41 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  92. 17/05/17 20:40:42 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  93. 17/05/17 20:40:43 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  94. 17/05/17 20:40:44 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  95. 17/05/17 20:40:45 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  96. 17/05/17 20:40:46 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  97. 17/05/17 20:40:47 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  98. 17/05/17 20:40:48 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  99. 17/05/17 20:40:49 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  100. 17/05/17 20:40:50 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  101. 17/05/17 20:40:51 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  102. 17/05/17 20:40:52 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  103. 17/05/17 20:40:53 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  104. 17/05/17 20:40:54 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  105. 17/05/17 20:40:55 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  106. 17/05/17 20:40:56 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  107. 17/05/17 20:40:57 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: ACCEPTED)  
  108. 17/05/17 20:40:58 INFO yarn.Client: Application report forapplication_1495024538278_0001 (state: FAILED)  
  109. 17/05/17 20:40:58 INFO yarn.Client:  
  110.          client token: N/A  
  111.          diagnostics:Application application_1495024538278_0001 failed 2 times due to AM Containerfor appattempt_1495024538278_0001_000002 exited with  exitCode: -103  
  112. Failing this attempt.Diagnostics: Container[pid=2675,containerID=container_1495024538278_0001_02_000001] is running beyondvirtual memory limits. Current usage: 107.7 MB of 1 GB physical memory used;2.2 GB of 2.1 GB virtual memory used. Killing container.  
  113. Dump of the process-tree for container_1495024538278_0001_02_000001:  
  114.         |- PID PPID PGRPIDSESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)RSSMEM_USAGE(PAGES) FULL_CMD_LINE  
  115.         |- 2675 2674 2675 2675(bash) 0 0 115843072 301 /bin/bash -c /opt/java/jdk1.8.0_121/bin/java -server-Xmx512m -Djava.io.tmpdir=/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/tmp-Dspark.yarn.app.container.log.dir=/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.27.143:38765'--properties-file/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/__spark_conf__/__spark_conf__.properties1>/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001/stdout2>/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001/stderr  
  116.         |- 2680 2675 2675 2675(java) 514 18 2283139072 27263 /opt/java/jdk1.8.0_121/bin/java -server -Xmx512m-Djava.io.tmpdir=/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/tmp-Dspark.yarn.app.container.log.dir=/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.27.143:38765--properties-file /root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/__spark_conf__/__spark_conf__.properties  
  117.    
  118. Container killed on request. Exit code is 143  
  119. Container exited with a non-zero exit code 143  
  120. For more detailed output, check the application tracking page:http://hserver1:8088/cluster/app/application_1495024538278_0001 Then click onlinks to logs of each attempt.  
  121. . Failing the application.  
  122.          ApplicationMasterhost: N/A  
  123.          ApplicationMaster RPCport: -1  
  124.          queue: default  
  125.          start time:1495024828093  
  126.          final status: FAILED  
  127.          tracking URL: http://hserver1:8088/cluster/app/application_1495024538278_0001  
  128.          user: root  
  129. 17/05/17 20:40:58 INFO yarn.Client: Deleted staging directoryhdfs://hserver1:9000/user/root/.sparkStaging/application_1495024538278_0001  
  130. 17/05/17 20:40:58 ERROR spark.SparkContext: Error initializingSparkContext.  
  131. org.apache.spark.SparkException: Yarn application has already ended!It might have been killed or unable to launch application master.  
  132.         atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)  
  133.         atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)  
  134.         atorg.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)  
  135.         atorg.apache.spark.SparkContext.<init>(SparkContext.scala:509)  
  136.         atorg.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)  
  137.         atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)  
  138.         atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)  
  139.         atscala.Option.getOrElse(Option.scala:121)  
  140.         atorg.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)  
  141.         atorg.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)  
  142.         at org.apache.spark.examples.SparkPi.main(SparkPi.scala)  
  143.         atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
  144.         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
  145.         atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
  146.         atjava.lang.reflect.Method.invoke(Method.java:498)  
  147.         atorg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)  
  148.         atorg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)  
  149.         atorg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)  
  150.         atorg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)  
  151.         atorg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)  
  152. 17/05/17 20:40:58 INFO server.ServerConnector: StoppedSpark@21d5c1a0{HTTP/1.1}{0.0.0.0:4040}  
  153. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@f415a95{/stages/stage/kill,null,UNAVAILABLE,@Spark}  
  154. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@114a85c2{/jobs/job/kill,null,UNAVAILABLE,@Spark}  
  155. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@12cd9150{/api,null,UNAVAILABLE,@Spark}  
  156. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@47289387{/,null,UNAVAILABLE,@Spark}  
  157. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1950e8a6{/static,null,UNAVAILABLE,@Spark}  
  158. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@35d6ca49{/executors/threadDump/json,null,UNAVAILABLE,@Spark}  
  159. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@56e07a08{/executors/threadDump,null,UNAVAILABLE,@Spark}  
  160. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5990e6c5{/executors/json,null,UNAVAILABLE,@Spark}  
  161. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4a9cc6cb{/executors,null,UNAVAILABLE,@Spark}  
  162. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@142eef62{/environment/json,null,UNAVAILABLE,@Spark}  
  163. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@517bd097{/environment,null,UNAVAILABLE,@Spark}  
  164. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@50687efb{/storage/rdd/json,null,UNAVAILABLE,@Spark}  
  165. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@66971f6b{/storage/rdd,null,UNAVAILABLE,@Spark}  
  166. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@3bffddff{/storage/json,null,UNAVAILABLE,@Spark}  
  167. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1601e47{/storage,null,UNAVAILABLE,@Spark}  
  168. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@66ea1466{/stages/pool/json,null,UNAVAILABLE,@Spark}  
  169. 17/05/17 20:40:58 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5471388b{/stages/pool,null,UNAVAILABLE,@Spark}  
  170. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@299266e2{/stages/stage/json,null,UNAVAILABLE,@Spark}  
  171. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@49f5c307{/stages/stage,null,UNAVAILABLE,@Spark}  
  172. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@21526f6c{/stages/json,null,UNAVAILABLE,@Spark}  
  173. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4c36250e{/stages,null,UNAVAILABLE,@Spark}  
  174. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5dcbb60{/jobs/job/json,null,UNAVAILABLE,@Spark}  
  175. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7ef2d7a6{/jobs/job,null,UNAVAILABLE,@Spark}  
  176. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@cdc3aae{/jobs/json,null,UNAVAILABLE,@Spark}  
  177. 17/05/17 20:40:58 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@a486d78{/jobs,null,UNAVAILABLE,@Spark}  
  178. 17/05/17 20:40:58 INFO ui.SparkUI: Stopped Spark web UI athttp://192.168.27.143:4040  
  179. 17/05/17 20:40:58 WARNcluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to requestexecutors before the AM has registered!  
  180. 17/05/17 20:40:58 INFO cluster.YarnClientSchedulerBackend: Shuttingdown all executors  
  181. 17/05/17 20:40:58 INFOcluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shutdown  
  182. 17/05/17 20:40:58 INFO cluster.SchedulerExtensionServices: StoppingSchedulerExtensionServices  
  183. (serviceOption=None,  
  184.  services=List(),  
  185.  started=false)  
  186. 17/05/17 20:40:58 INFO cluster.YarnClientSchedulerBackend: Stopped  
  187. 17/05/17 20:40:58 INFO spark.MapOutputTrackerMasterEndpoint:MapOutputTrackerMasterEndpoint stopped!  
  188. 17/05/17 20:40:58 INFO memory.MemoryStore: MemoryStore cleared  
  189. 17/05/17 20:40:58 INFO storage.BlockManager: BlockManager stopped  
  190. 17/05/17 20:40:58 INFO storage.BlockManagerMaster:BlockManagerMaster stopped  
  191. 17/05/17 20:40:58 WARN metrics.MetricsSystem: Stopping aMetricsSystem that is not running  
  192. 17/05/17 20:40:58 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!  
  193. 17/05/17 20:40:58 INFO spark.SparkContext: Successfully stoppedSparkContext  
  194. Exception in thread "main"org.apache.spark.SparkException: Yarn application has already ended! It mighthave been killed or unable to launch application master.  
  195.         atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)  
  196.         atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)  
  197.         atorg.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)  
  198.         atorg.apache.spark.SparkContext.<init>(SparkContext.scala:509)  
  199.         atorg.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)  
  200.         atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)  
  201.         atorg.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)  
  202.         atscala.Option.getOrElse(Option.scala:121)  
  203.         atorg.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)  
  204.         atorg.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)  
  205.         atorg.apache.spark.examples.SparkPi.main(SparkPi.scala)  
  206.         atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
  207.         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
  208.         atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
  209.         atjava.lang.reflect.Method.invoke(Method.java:498)  
  210.         atorg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)  
  211.         atorg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)  
  212.         atorg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)  
  213.         atorg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)  
  214.         atorg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)  
  215. 17/05/17 20:40:58 INFO util.ShutdownHookManager: Shutdown hookcalled  
  216. 17/05/17 20:40:58 INFO util.ShutdownHookManager: Deleting directory/tmp/spark-5e00aae5-dc98-48fd-83b4-978714e002d4  
  217. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  

3.1.2.1.2  错误原因

   错误提示中很难看出什么原因，但是从完整的输出信息中，可以看到一个地址：

        http://hserver1:8088/cluster/app/application_1495024538278_0001

访问该地址

如图：

错误原因上图中我已用红框标出，完整的信息是：

[plain]  view plain  copy
  1. Application application_1495024538278_0001 failed 2 timesdue to AM Container for appattempt_1495024538278_0001_000002 exited withexitCode: -103  
  2. Failing this attempt.Diagnostics:Container [pid=2675,containerID=container_1495024538278_0001_02_000001] isrunning beyond virtual memory limits. Current usage: 107.7MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killingcontainer.  
  3. Dump of the process-tree forcontainer_1495024538278_0001_02_000001 :  
  4. |- PID PPID PGRPID SESSID CMD_NAMEUSER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)RSSMEM_USAGE(PAGES) FULL_CMD_LINE  
  5. |- 2675 2674 2675 2675 (bash) 0 0115843072 301 /bin/bash -c /opt/java/jdk1.8.0_121/bin/java -server -Xmx512m-Djava.io.tmpdir=/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/tmp-Dspark.yarn.app.container.log.dir=/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.27.143:38765'--properties-file/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/__spark_conf__/__spark_conf__.properties1>/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001/stdout2> /opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001/stderr  
  6. |- 2680 2675 2675 2675 (java) 514 182283139072 27263 /opt/java/jdk1.8.0_121/bin/java -server -Xmx512m-Djava.io.tmpdir=/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/tmp-Dspark.yarn.app.container.log.dir=/opt/hadoop/hadoop-2.8.0/logs/userlogs/application_1495024538278_0001/container_1495024538278_0001_02_000001org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.27.143:38765--properties-file/root/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1495024538278_0001/container_1495024538278_0001_02_000001/__spark_conf__/__spark_conf__.properties  
  7. Container killed on request. Exit codeis 143  
  8. Container exited with a non-zero exitcode 143  
  9. For more detailed output, check theapplication tracking page:http://hserver1:8088/cluster/app/application_1495024538278_0001 Then click onlinks to logs of each attempt.  
  10. . Failing the application.  

3.1.2.1.3 解决方法

         原因很明确了，和虚拟内存有关，解决方法是修改Hadoop中的yarn-site.xml文件，我安装的是hadoop2.8.0,该文件在我的机器的/opt/hadoop/hadoop-2.8.0/etc/hadoop这个目录下，好了修改这个文件，在里面加入配置即可解决该问题：

[plain]  view plain  copy
  1. <property>   
  2.    <name>yarn.nodemanager.vmem-check-enabled</name>   
  3.    <value>false</value>   
  4. </property>  

        配置完成，重启Hadoop和Spark集群，问题即可解决。

3.1.2.2  Required executor memory (1024+384 MB)
3.1.2.2.1  主要错误信息

[plain]  view plain  copy
  1. Exceptionin thread "main" java.lang.IllegalArgumentException: Requiredexecutor memory (1024+384 MB) is above the max threshold (1024 MB) of thiscluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb'and/or 'yarn.nodemanager.resource.memory-mb'.  
  2.        atorg.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:334)  
  3.        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:168)  
  4.        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)  
  5.        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1226)  
  6.        at org.apache.spark.deploy.yarn.Client.main(Client.scala)  
  7.        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
  8.        atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
  9.        atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
  10.        at java.lang.reflect.Method.invoke(Method.java:498)  
  11.        atorg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)  
  12.        atorg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)  
  13.        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)  
  14.        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)  
  15.        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)  

3.1.2.2.2  错误原因

       报错已经很明确，配置允许使用的内存太小了，Hadoop中配置允许yarn组件使用的内存只有1024M，这里需要1024+384M。

3.1.2.2.3 解决方法

     打开Hadoop中yarn相关的配置文件yarn-site.xml，我的是在/opt/hadoop/hadoop-2.8.0/etc/hadoop这个目录下， 将yarn.scheduler.maximum-allocation-mb和yarn.nodemanager.resource.memory-mb的值配大一点，至少配个2048M吧。

[plain]  view plain  copy
  1. <property>  
  2.         <name>yarn.scheduler.maximum-allocation-mb</name>  
  3.         <value>2048</value>  
  4.         <discription>每个节点可用内存,单位MB,默认8182MB</discription>  
  5.    </property>  
  6.    <property>  
  7.        <name>yarn.nodemanager.vmem-pmem-ratio</name>  
  8.         <value>2.1</value>  
  9.    </property>  
  10.   <property>  
  11.        <name>yarn.nodemanager.resource.memory-mb</name>  
  12.         <value>2048</value>  
  13. </property>  

     配置完成，重启Hadoop和Spark集群，问题即可解决。

3.2  用yarn-cluster模式执行计算程序

3.2.1 操作步骤和方法

进入到Spark2.1.1的安装目录，命令是：

[plain]  view plain  copy
  1. cd   /opt/spark/spark-2.1.1-bin-hadoop2.7  

执行命令，用yarn-cluster模式运行计算圆周率的Demo：

[plain]  view plain  copy
  1. ./bin/spark-submit  --class  org.apache.spark.examples.SparkPi  --master  yarn-cluster   examples/jars/spark-examples_2.11-2.1.1.jar  

如图：

稍等几秒，计算即可完成

完整的输出信息是：

[plain]  view plain  copy
  1. [root@hserver1 sbin]# cd   /opt/spark/spark-2.1.1-bin-hadoop2.7  
  2. [root@hserver1 spark-2.1.1-bin-hadoop2.7]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster  examples/jars/spark-examples_2.11-2.1.1.jar  
  3. Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.  
  4. 17/05/17 21:33:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
  5. 17/05/17 21:33:36 INFO client.RMProxy: Connecting to ResourceManager at hserver1/192.168.27.143:8032  
  6. 17/05/17 21:33:36 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers  
  7. 17/05/17 21:33:36 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container)  
  8. 17/05/17 21:33:36 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead  
  9. 17/05/17 21:33:36 INFO yarn.Client: Setting up container launch context for our AM  
  10. 17/05/17 21:33:36 INFO yarn.Client: Setting up the launch environment for our AM container  
  11. 17/05/17 21:33:36 INFO yarn.Client: Preparing resources for our AM container  
  12. 17/05/17 21:33:41 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.  
  13. 17/05/17 21:33:46 INFO yarn.Client: Uploading resource file:/tmp/spark-c61cd5df-54bd-49f8-b15e-903a60fc2285/__spark_libs__8183600565694238476.zip -> hdfs://hserver1:9000/user/root/.sparkStaging/application_1495027881637_0001/__spark_libs__8183600565694238476.zip  
  14. 17/05/17 21:33:56 INFO yarn.Client: Uploading resource file:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar -> hdfs://hserver1:9000/user/root/.sparkStaging/application_1495027881637_0001/spark-examples_2.11-2.1.1.jar  
  15. 17/05/17 21:33:57 INFO yarn.Client: Uploading resource file:/tmp/spark-c61cd5df-54bd-49f8-b15e-903a60fc2285/__spark_conf__2604839061015961106.zip -> hdfs://hserver1:9000/user/root/.sparkStaging/application_1495027881637_0001/__spark_conf__.zip  
  16. 17/05/17 21:33:57 INFO spark.SecurityManager: Changing view acls to: root  
  17. 17/05/17 21:33:57 INFO spark.SecurityManager: Changing modify acls to: root  
  18. 17/05/17 21:33:57 INFO spark.SecurityManager: Changing view acls groups to:   
  19. 17/05/17 21:33:57 INFO spark.SecurityManager: Changing modify acls groups to:   
  20. 17/05/17 21:33:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()  
  21. 17/05/17 21:33:57 INFO yarn.Client: Submitting application application_1495027881637_0001 to ResourceManager  
  22. 17/05/17 21:33:58 INFO impl.YarnClientImpl: Submitted application application_1495027881637_0001  
  23. 17/05/17 21:33:59 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  24. 17/05/17 21:33:59 INFO yarn.Client:   
  25.          client token: N/A  
  26.          diagnostics: AM container is launched, waiting for AM container to Register with RM  
  27.          ApplicationMaster host: N/A  
  28.          ApplicationMaster RPC port: -1  
  29.          queue: default  
  30.          start time: 1495028037628  
  31.          final status: UNDEFINED  
  32.          tracking URL: http://hserver1:8088/proxy/application_1495027881637_0001/  
  33.          user: root  
  34. 17/05/17 21:34:00 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  35. 17/05/17 21:34:01 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  36. 17/05/17 21:34:02 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  37. 17/05/17 21:34:03 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  38. 17/05/17 21:34:04 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  39. 17/05/17 21:34:05 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  40. 17/05/17 21:34:06 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  41. 17/05/17 21:34:07 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  42. 17/05/17 21:34:08 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  43. 17/05/17 21:34:09 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  44. 17/05/17 21:34:10 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  45. 17/05/17 21:34:11 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  46. 17/05/17 21:34:12 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  47. 17/05/17 21:34:13 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  48. 17/05/17 21:34:14 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  49. 17/05/17 21:34:15 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  50. 17/05/17 21:34:16 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  51. 17/05/17 21:34:17 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  52. 17/05/17 21:34:18 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  53. 17/05/17 21:34:19 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  54. 17/05/17 21:34:20 INFO yarn.Client: Application report for application_1495027881637_0001 (state: ACCEPTED)  
  55. 17/05/17 21:34:21 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  56. 17/05/17 21:34:21 INFO yarn.Client:   
  57.          client token: N/A  
  58.          diagnostics: N/A  
  59.          ApplicationMaster host: 192.168.27.142  
  60.          ApplicationMaster RPC port: 0  
  61.          queue: default  
  62.          start time: 1495028037628  
  63.          final status: UNDEFINED  
  64.          tracking URL: http://hserver1:8088/proxy/application_1495027881637_0001/  
  65.          user: root  
  66. 17/05/17 21:34:22 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  67. 17/05/17 21:34:23 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  68. 17/05/17 21:34:24 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  69. 17/05/17 21:34:25 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  70. 17/05/17 21:34:26 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  71. 17/05/17 21:34:27 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  72. 17/05/17 21:34:28 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  73. 17/05/17 21:34:29 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  74. 17/05/17 21:34:30 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  75. 17/05/17 21:34:31 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  76. 17/05/17 21:34:32 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  77. 17/05/17 21:34:33 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  78. 17/05/17 21:34:34 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  79. 17/05/17 21:34:35 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  80. 17/05/17 21:34:36 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  81. 17/05/17 21:34:37 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  82. 17/05/17 21:34:38 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  83. 17/05/17 21:34:39 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  84. 17/05/17 21:34:40 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  85. 17/05/17 21:34:41 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  86. 17/05/17 21:34:42 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  87. 17/05/17 21:34:43 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  88. 17/05/17 21:34:44 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  89. 17/05/17 21:34:45 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  90. 17/05/17 21:34:46 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  91. 17/05/17 21:34:47 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  92. 17/05/17 21:34:48 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  93. 17/05/17 21:34:49 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  94. 17/05/17 21:34:50 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  95. 17/05/17 21:34:51 INFO yarn.Client: Application report for application_1495027881637_0001 (state: RUNNING)  
  96. 17/05/17 21:34:52 INFO yarn.Client: Application report for application_1495027881637_0001 (state: FINISHED)  
  97. 17/05/17 21:34:52 INFO yarn.Client:   
  98.          client token: N/A  
  99.          diagnostics: N/A  
  100.          ApplicationMaster host: 192.168.27.142  
  101.          ApplicationMaster RPC port: 0  
  102.          queue: default  
  103.          start time: 1495028037628  
  104.          final status: SUCCEEDED  
  105.          tracking URL: http://hserver1:8088/proxy/application_1495027881637_0001/  
  106.          user: root  
  107. 17/05/17 21:34:52 INFO util.ShutdownHookManager: Shutdown hook called  
  108. 17/05/17 21:34:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-c61cd5df-54bd-49f8-b15e-903a60fc2285  
  109. [root@hserver1 spark-2.1.1-bin-hadoop2.7]#  

         注意，使用yarn-cluster模式计算，结果没有输出在控制台，结果写在了Hadoop集群的日志中，如何查看计算结果？注意到刚才的输出中有地址：

      http://hserver1:8088/proxy/application_1495027881637_0001/

直接用浏览器访问

如图：

      

点击Logs链接

如图：

点击打开内容

如图：

       

王忠海
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark demo 运行
MusicDancing的博客
04-07 2307
1. package com.zz.spark.sparksql import org.apache.spark.sql.{DataFrame, SparkSession} /** * 自定义DataFrame注册成数据表，查询数据表 */ object Demo1 { def main(args: Array[String]): Unit = { val spark = SparkSession.builder() .appName(
Spark 测试程序计算圆周率
Running_Tiger的博客
03-06 2851
Spark 测试程序计算圆周率1. 普通模式提交任务该算法是利用蒙特·卡罗算法求圆周率PI，通过计算机模拟大量的随机数，最终会计算出比较精确的π。bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://node-1:7077 \ --executor-memory 1G \ --total-exe
Flink on YARN（下）：常见问题与排查思路
Ververica的博客
09-26 2998
杨弢（搏远），阿里巴巴计算平台事业部技术专家，Apache Hadoop Committer，目前专注于 YARN、Flink、YuniKorn 等开源项目的资源调度方向...
一文了解Spark引擎的优势及应用场景
最新发布
架构师小吴的博客
06-15 1657
而对数据计算复杂（有推荐、分类、聚类算法场景）且时延要求高的场景，如迭代计算, 交互式计算, 流计算、有机器学习算法需求，图计算需求，且成本投入可以接受的情况下使用Spark SQL,Spark SQL读取的数据都是存入到内存中，因此对机器的内存有要求，且要求内存较大, 相对较贵.总结一下，hiveSQL 应用于大规模的离线跑批，但是对时间要求不高，而Spark SQL 应用于迭代式计算和交互式数据挖掘、Spark Streaming 支持流式数据计算、Spark MLlibk提供的机器学习功能的程序库。
Spark在Eclipse运行的demo
Carson的专栏
12-26 164
刚接触Spark，很多还都不是太懂，查看了别人写的中文API：Spark中文文档 根据里面的参数写了一个最简单的调用，如果maven已配好的话，可以直接运行 &lt;dependency&gt; &lt;groupId&gt;org.apache.spark&lt;/groupId&gt; &lt;artifactId&gt;spark-core_2.10&lt;/artifactId&...
spark运行简单的demo程序
热门推荐
archer的技术故事
01-04 1万+
spark运行简单的demo程序 使用spark可以直接在命令行中启动spark-shell，然后在spark-shell中使用scala进行数据的处理。现在要介绍的是使用ide进行处理程序的编写。 前提： 1、已经安装好spark能够运行起来。 2、了解基本的scala语法 好，下面上货。 1、首先用maven创建一个简单的quickstart程序 2、在app目录下创建创
spark-圆周率
山海
07-22 450
计算圆周率： bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://node01:7077 \ --executor-memory 1G \ --total-executor-cores 2 \ examples/jars/spark-examples_2.11-2.0.2.jar \ 10 ...
spark-2.1.1.zip
05-02
Apache Spark 是一个开源的大数据处理框架，专注于速度、易用性和高级分析能力。Spark 2.1.1 版本是这个项目的一个稳定版本，它提供了...通过下载并解压"spark-2.1.1.zip"，您可以开始探索这个强大框架的各种可能性。
spark2.1.1的源码包
06-17
Spark 2.1.1 源码包是大数据处理框架Spark的核心代码库，它包含了Spark的所有组件和模块，如Spark Core、Spark SQL、Spark Streaming、Spark MLlib以及Spark GraphX等。对于开发者和研究者而言，深入理解Spark源码有...
spark-2.1.1-bin-hadoop2.7
09-11
- **资源管理**：Spark 2.1.1可以与YARN（Hadoop的资源管理系统）协同工作，有效管理和分配集群资源，实现跨节点的分布式计算。 - **MapReduce兼容**：Spark可以处理由Hadoop MapReduce产生的数据，进一步扩大了其...
spark-2.1.1.tgz
04-28
这个tgz文件"spark-2.1.1.tgz"是一个压缩包，用于在不同的操作系统上安装和运行Spark。 首先，我们来了解一下Spark的核心概念。Spark主要由以下几个组件构成： 1. **Spark Core**：这是Spark的基础，负责任务调度...
spark example 2.2.0版本 maven项目
01-22
该项目可直接导入eclipse使用，编译后无错误，是学习java+spark编程的快速方法
spark-2.1.1-bin-hadoop2.7.tgz.7z
04-28
这个特定的压缩包"spark-2.1.1-bin-hadoop2.7.tgz.7z"是为Linux系统设计的，它包含了Spark 2.1.1版本，并且已经与Hadoop 2.7.2版本进行了预编译集成，这意味着它可以无缝地与Hadoop生态系统交互。 Hadoop 2.7.2是一...
Spark实现PI圆周率计算
qq_42155078的博客
05-11 1025
1.采用蒙特卡洛概率的方法计算Pi圆周率 以中心点为圆心，圆的半径为1，圆的面积之比与正方形的面积之比为：π/4那么我们可以通过落入圆形中的点数占的比例来估计π的值 利用Spark进行代码实现： object Demo13_Pi { def main(args: Array[String]): Unit = { val list: Seq[Int] = 0 to 1000000 //创建spark 环境 val conf: SparkConf = new Spark
spark2.1.1 demo
Zzhou1990的博客
09-05 540
在spark之前版本中使用sparkContext作为进入spark切入点,操作rdd。 spark2.0中引入了SparkSession 作为全新切入点用来操作DataSet和Datafarme。 创建sparksession: SparkSession spark = SparkSession .builder() ...
Dubbox安装步骤（包含github源码下载与编译）
weixin_34377919的博客
04-10 209
2019独角兽企业重金招聘Python工程师标准>>> ...
环信 iOS Demo 试用小记
王洪伟的专栏
09-13 1万+
一、下载SDK和Demo下载地址： http://www.easemob.com/sdk/直接下载：http://downloads.easemob.com/downloads/iOSSDK-20150710.zip二、解压、用xcode打开（Xcode为6.4版本）修改 Team账号为自己的开发者账号三、编译出现21个警告，在真机上运行正常。
测试demo的执行
wyqwilliam的博客
04-26 1114
在将测试demo由一台服务器上传到另一台服务器之后，往往会出现与之前的权限不同，出现了权限权限不够的情况，这时候就需要修改了权限之后再进行执行。 修改权限的命令：chmod 777 文件名 执行命令：./demo 如果想要给该文件夹及该文件夹下的所有文件夹，及所有文件赋权限： chmod -R 777 asr_demo【文件夹】 ...
spark 2.2.0学习笔记4之SparkSQLDemo
undergrowth的专栏
12-23 883
spark 2.2.0学习笔记4之SparkSQLDemoInfospark sql—-Spark 用来操作结构化/半结构化数据的程序包 从各种数据源读取数据/支持各种方式的sql查询 hive查询 支持udf(用户自定义函数) SchemaRDD(DataFrame)—-存放Row 对象的RDD，每个Row 对象代表一行记录 SELECT SUM(user.favouritesCount)
spark2.1.1对应的org.apache.spark
03-26
您可以在以下地址中找到Spark 2.1.1的官方文档： https://spark.apache.org/docs/2.1.1/ 该文档包含各种Spark功能的详细说明、API参考、示例代码等等。如果您需要了解Spark的更多信息或解决问题，这是一个很好的...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值