Spark2.1.1中用各种模式运行计算圆周率的官方Demo

最新推荐文章于 2022-05-30 08:31:32 发布
最新推荐文章于 2022-05-30 08:31:32 发布
阅读量1.7w 收藏 42
点赞数 14
分类专栏： Linux 运维 CentOS hadoop spark yarn 文章标签： linux CentOS hadoop spark yarn
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/pucao_cug/article/details/72453382
版权


1使用单机local模式提交任务

2使用独立的Spark集群模式提交任务

3使用Spark集群 +Hadoop集群的模式提交任务

        3.1用yarn-client模式执行计算程序

                3.1.1操作步骤和方法

                3.1.2常见错误解决

                          3.1.2.1 Yarn applicationhas already ended!

                                      3.1.2.1.1主要错误信息

                                      3.1.2.1.2错误原因

                                      3.1.2.1.3解决方法

                         3.1.2.2 Required executormemory (1024+384 MB)

                                      3.1.2.2.1主要错误信息

                                      3.1.2.2.2错误原因

                                      3.1.2.2.3解决方法

       3.2用yarn-cluster模式执行计算程序

               3.2.1操作步骤和方法

            关键字： Linux  CentOS  Hadoop  Spark  yarn-client   yarn-cluster

          版本号：CentOS7   Hadoop2.8.0    Spark2.1.1

          说明：本文主要使用Spark单机模式(也就是Local模式)、Spark独立集群模式（也就是Standalone模式）、Spark集群+Haddop集群模式（也就是Yarn-Client模式和Yarn-Cluster模式）来运行Spark官方提供的计算圆周率的Demo。

1  使用单机local模式提交任务

          local模式也就是本地模式，也就是在本地机器上单机执行程序。使用这个模式的话，并不需要启动Hadoop集群，也不需要启动Spark集群，只要有一台机器上安装了JDK、Scala、Spark即可运行。

        进入到Spark2.1.1的安装目录，命令是：

cd  /opt/spark/spark-2.1.1-bin-hadoop2.7

        执行命令，用单机模式运行计算圆周率的Demo：

./bin/spark-submit  --class   org.apache.spark.examples.SparkPi  --master   local  examples/jars/spark-examples_2.11-2.1.1.jar

如图：


稍等几秒，计算即可完成

如图：


完整信息是：

[root@hserver1 ~]# cd /opt/spark/spark-2.1.1-bin-hadoop2.7
[root@hserver1 spark-2.1.1-bin-hadoop2.7]# ./bin/spark-submit--class org.apache.spark.examples.SparkPi --master localexamples/jars/spark-examples_2.11-2.1.1.jar
17/05/17 16:31:38 INFO spark.SparkContext: Running Spark version2.1.1
17/05/17 16:31:39 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable
17/05/17 16:31:39 INFO spark.SecurityManager: Changing view acls to:root
17/05/17 16:31:39 INFO spark.SecurityManager: Changing modify aclsto: root
17/05/17 16:31:39 INFO spark.SecurityManager: Changing view aclsgroups to:
17/05/17 16:31:39 INFO spark.SecurityManager: Changing modify aclsgroups to:
17/05/17 16:31:39 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups withview permissions: Set(); users  withmodify permissions: Set(root); groups with modify permissions: Set()
17/05/17 16:31:40 INFO util.Utils: Successfully started service'sparkDriver' on port 41012.
17/05/17 16:31:40 INFO spark.SparkEnv: Registering MapOutputTracker
17/05/17 16:31:40 INFO spark.SparkEnv: RegisteringBlockManagerMaster
17/05/17 16:31:40 INFO storage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/05/17 16:31:40 INFO storage.BlockManagerMasterEndpoint:BlockManagerMasterEndpoint up
17/05/17 16:31:40 INFO storage.DiskBlockManager: Created localdirectory at /tmp/blockmgr-48295dbe-32ae-4bdb-a4e2-9fada0294094
17/05/17 16:31:40 INFO memory.MemoryStore: MemoryStore started withcapacity 413.9 MB
17/05/17 16:31:40 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator
17/05/17 16:31:41 INFO util.log: Logging initialized @4545ms
17/05/17 16:31:41 INFO server.Server: jetty-9.2.z-SNAPSHOT
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5118388b{/jobs,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@15a902e7{/jobs/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7876d598{/jobs/job,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/job/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5af28b27{/stages,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@71104a4{/stages/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4985cbcb{/stages/stage,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72f46e16{/stages/stage/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3c9168dc{/stages/pool,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@332a7fce{/stages/pool/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@549621f3{/storage,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@54361a9{/storage/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@32232e55{/storage/rdd,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@5217f3d0{/storage/rdd/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@37ebc9d8{/environment,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@293bb8a5{/environment/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@2416a51{/executors,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6fa590ba{/executors/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@6e9319f{/executors/threadDump,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@72e34f77{/executors/threadDump/json,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@7bf9b098{/static,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@389adf1d{/,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@77307458{/api,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@1fc0053e{/jobs/job/kill,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@290b1b2e{/stages/stage/kill,null,AVAILABLE,@Spark}
17/05/17 16:31:41 INFO server.ServerConnector: StartedSpark@32fe9d0a{HTTP/1.1}{0.0.0.0:4040}
17/05/17 16:31:41 INFO server.Server: Started @4957ms
17/05/17 16:31:41 INFO util.Utils: Successfully started service'SparkUI' on port 4040.
17/05/17 16:31:41 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, andstarted at http://192.168.27.143:4040
17/05/17 16:31:41 INFO spark.SparkContext: Added JARfile:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1495009901648
17/05/17 16:31:41 INFO executor.Executor: Starting executor IDdriver on host localhost
17/05/17 16:31:41 INFO util.Utils: Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 41396.
17/05/17 16:31:41 INFO netty.NettyBlockTransferService: Servercreated on 192.168.27.143:41396
17/05/17 16:31:41 INFO storage.BlockManager: Usingorg.apache.spark.storage.RandomBlockReplicationPolicy for block replicationpolicy
17/05/17 16:31:41 INFO storage.BlockManagerMaster: RegisteringBlockManager BlockManagerId(driver, 192.168.27.143, 41396, None)
17/05/17 16:31:41 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.143:41396 with 413.9 MB RAM,BlockManagerId(driver, 192.168.27.143, 41396, None)
17/05/17 16:31:41 INFO storage.BlockManagerMaster: RegisteredBlockManager BlockManagerId(driver, 192.168.27.143, 41396, None)
17/05/17 16:31:41 INFO storage.BlockManager: Initialized BlockManager:BlockManagerId(driver, 192.168.27.143, 41396, None)
17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4e6d7365{/metrics/json,null,AVAILABLE,@Spark}
17/05/17 16:31:42 INFO internal.SharedState: Warehouse path is 'file:/opt/spark/spark-2.1.1-bin-hadoop2.7/spark-warehouse/'.
17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@3e58d65e{/SQL,null,AVAILABLE,@Spark}
17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@63998bf4{/SQL/json,null,AVAILABLE,@Spark}
17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@62a8fd44{/SQL/execution,null,AVAILABLE,@Spark}
17/05/17 16:31:42 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f5b5ca4{/SQL/execution/json,null,AVAILABLE,@Spark}
17/05/17 16:31:42 INFO handler.ContextHandler: Startedo.s.j.s.ServletContextHandler@4cfbaf4{/static/sql,null,AVAILABLE,@Spark}
17/05/17 16:31:44 INFO spark.SparkContext: Starting job: reduce atSparkPi.scala:38
17/05/17 16:31:44 INFO scheduler.DAGScheduler: Got job 0 (reduce atSparkPi.scala:38) with 2 output partitions
17/05/17 16:31:44 INFO scheduler.DAGScheduler: Final stage:ResultStage 0 (reduce at SparkPi.scala:38)
17/05/17 16:31:44 INFO scheduler.DAGScheduler: Parents of finalstage: List()
17/05/17 16:31:44 INFO scheduler.DAGScheduler: Missing parents:List()
17/05/17 16:31:44 INFO scheduler.DAGScheduler: SubmittingResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has nomissing parents
17/05/17 16:31:44 INFO memory.MemoryStore: Block broadcast_0 storedas values in memory (estimated size 1832.0 B, free 413.9 MB)
17/05/17 16:31:44 INFO memory.MemoryStore: Block broadcast_0_piece0stored as bytes in memory (estimated size 1167.0 B, free 413.9 MB)
17/05/17 16:31:44 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.143:41396 (size: 1167.0 B, free:413.9 MB)
17/05/17 16:31:44 INFO spark.SparkContext: Created broadcast 0 frombroadcast at DAGScheduler.scala:996
17/05/17 16:31:44 INFO scheduler.DAGScheduler: Submitting 2 missingtasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)
17/05/17 16:31:44 INFO scheduler.TaskSchedulerImpl: Adding task set0.0 with 2 tasks
17/05/17 16:31:45 INFO scheduler.TaskSetManager: Starting task 0.0in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL,6026 bytes)
17/05/17 16:31:45 INFO executor.Executor: Running task 0.0 in stage0.0 (TID 0)
17/05/17 16:31:45 INFO executor.Executor: Fetchingspark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar with timestamp1495009901648
17/05/17 16:31:45 INFO client.TransportClientFactory: Successfullycreated connection to /192.168.27.143:41012 after 106 ms (0 ms spent inbootstraps)
17/05/17 16:31:45 INFO util.Utils: Fetchingspark://192.168.27.143:41012/jars/spark-examples_2.11-2.1.1.jar to/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724/userFiles-d086a0e3-7a29-4d7e-9667-6156186a89c8/fetchFileTemp942491911381944893.tmp
17/05/17 16:31:45 INFO executor.Executor: Addingfile:/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724/userFiles-d086a0e3-7a29-4d7e-9667-6156186a89c8/spark-examples_2.11-2.1.1.jarto class loader
17/05/17 16:31:46 INFO executor.Executor: Finished task 0.0 in stage0.0 (TID 0). 1114 bytes result sent to driver
17/05/17 16:31:46 INFO scheduler.TaskSetManager: Starting task 1.0in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL,6026 bytes)
17/05/17 16:31:46 INFO executor.Executor: Running task 1.0 in stage0.0 (TID 1)
17/05/17 16:31:46 INFO executor.Executor: Finished task 1.0 in stage0.0 (TID 1). 1041 bytes result sent to driver
17/05/17 16:31:46 INFO scheduler.TaskSetManager: Finished task 0.0in stage 0.0 (TID 0) in 1182 ms on localhost (executor driver) (1/2)
17/05/17 16:31:46 INFO scheduler.TaskSetManager: Finished task 1.0in stage 0.0 (TID 1) in 162 ms on localhost (executor driver) (2/2)
17/05/17 16:31:46 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool
17/05/17 16:31:46 INFO scheduler.DAGScheduler: ResultStage 0 (reduceat SparkPi.scala:38) finished in 1.284 s
17/05/17 16:31:46 INFO scheduler.DAGScheduler: Job 0 finished:reduce at SparkPi.scala:38, took 2.135304 s
Pi is roughly 3.1400557002785012
17/05/17 16:31:46 INFO server.ServerConnector: StoppedSpark@32fe9d0a{HTTP/1.1}{0.0.0.0:4040}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@290b1b2e{/stages/stage/kill,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@1fc0053e{/jobs/job/kill,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@77307458{/api,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@389adf1d{/,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7bf9b098{/static,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72e34f77{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6e9319f{/executors/threadDump,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@6fa590ba{/executors/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@2416a51{/executors,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@293bb8a5{/environment/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@37ebc9d8{/environment,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5217f3d0{/storage/rdd/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32232e55{/storage/rdd,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@54361a9{/storage/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@549621f3{/storage,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@332a7fce{/stages/pool/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@3c9168dc{/stages/pool,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@72f46e16{/stages/stage/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4985cbcb{/stages/stage,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@71104a4{/stages/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5af28b27{/stages,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@4a3e3e8b{/jobs/job/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@7876d598{/jobs/job,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@15a902e7{/jobs/json,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO handler.ContextHandler: Stoppedo.s.j.s.ServletContextHandler@5118388b{/jobs,null,UNAVAILABLE,@Spark}
17/05/17 16:31:46 INFO ui.SparkUI: Stopped Spark web UI athttp://192.168.27.143:4040
17/05/17 16:31:46 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpointstopped!
17/05/17 16:31:46 INFO memory.MemoryStore: MemoryStore cleared
17/05/17 16:31:46 INFO storage.BlockManager: BlockManager stopped
17/05/17 16:31:46 INFO storage.BlockManagerMaster:BlockManagerMaster stopped
17/05/17 16:31:46 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!
17/05/17 16:31:46 INFO spark.SparkContext: Successfully stoppedSparkContext
17/05/17 16:31:46 INFO util.ShutdownHookManager: Shutdown hook called
17/05/17 16:31:46 INFO util.ShutdownHookManager: Deleting directory/tmp/spark-e43f7320-c6f3-4937-8b56-612019cde724
[root@hserver1 spark-2.1.1-bin-hadoop2.7]#

2  使用独立的Spark集群模式提交任务

            这种模式也就是Standalone模式，使用独立的Spark集群模式提交任务，需要先启动Spark集群，但是不需要启动Hadoop集群。启动Spark集群的方法是进入$SPARK_HOME/sbin目录下，执行start-all.sh脚本，启动成功后，可以访问下面的地址看是否成功：

        http://Spark的Marster机器的IP:8080/

如图：

   

      说明：Spark集群启动成功后。开始执行下面的一系列命令。

最低0.47元/天 解锁文章
陈某1987
关注
  • 14
    点赞
  • 踩
  • 42
    收藏
    觉得还不错? 一键收藏
  • 15
    评论
专栏目录
Spark实现PI圆周率计算
qq_42155078的博客
05-11 1023
1.采用蒙特卡洛概率的方法计算Pi圆周率 以中心点为圆心，圆的半径为1，圆的面积之比与正方形的面积之比为：π/4那么我们可以通过落入圆形中的点数占的比例来估计π的值 利用Spark进行代码实现： object Demo13_Pi { def main(args: Array[String]): Unit = { val list: Seq[Int] = 0 to 1000000 //创建spark 环境 val conf: SparkConf = new Spark
探秘 Spark 开发实践：SparkDemo 项目全面解析
最新发布
gitblog_00063的博客
06-03 390
探秘 Spark 开发实践：SparkDemo 项目全面解析 1、项目介绍 项目地址:https://gitcode.com/baifendian/SparkDemo SparkDemo 是一个为开发者量身定制的开源项目，它包含了丰富的 Spark 示例，覆盖了 Python、Java、Scala 和 R 四种主流编程语言，并涉及到 Spark 的核心组件如 Streaming、Sql、Mllib...
一篇文章读懂：Spark运行模式
不断学习，不断进步，提高自己
05-30 1491
本文主要讲解Spark性能调优与原理分析
Spark集群的三种模式
ZYN的博客
02-08 2623
文章目录1、Spark的由来1.1 Hadoop的发展1.2 MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1 Standalone模式部署配置历史服务器配置高可用运行模式3.2 Yarn模式安装部署配置历史服务器运行模式4、WordCount案例 1、Spark的由来 定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可扩展的大数据分析计算引擎。 1.1 Hadoop的发展 Hadoop1.x存在的问题： Na
spark-2.1.1.zip
05-02
Apache Spark 是一个开源的大数据处理框架，专注于速度、易用性和高级分析能力。Spark 2.1.1 版本是这个项目的一个稳定版本，它提供了...通过下载并解压"spark-2.1.1.zip"，您可以开始探索这个强大框架的各种可能性。
spark2.1.1的源码包
06-17
Spark 2.1.1 源码包是大数据处理框架Spark的核心代码库，它包含了Spark的所有组件和模块，如Spark Core、Spark SQL、Spark Streaming、Spark MLlib以及Spark GraphX等。对于开发者和研究者而言，深入理解Spark源码有...
spark-2.1.1-bin-hadoop2.7
09-11
- **资源管理**：Spark 2.1.1可以与YARN（Hadoop的资源管理系统）协同工作，有效管理和分配集群资源，实现跨节点的分布式计算。 - **MapReduce兼容**：Spark可以处理由Hadoop MapReduce产生的数据，进一步扩大了其...
spark-2.1.1-bin-hadoop2.7.tgz.7z
04-28
这个特定的压缩包"spark-2.1.1-bin-hadoop2.7.tgz.7z"是为Linux系统设计的，它包含了Spark 2.1.1版本，并且已经与Hadoop 2.7.2版本进行了预编译集成，这意味着它可以无缝地与Hadoop生态系统交互。 Hadoop 2.7.2是一...
spark-2.1.1.tgz
04-28
这个tgz文件"spark-2.1.1.tgz"是一个压缩包，用于在不同的操作系统上安装和运行Spark。 首先，我们来了解一下Spark的核心概念。Spark主要由以下几个组件构成： 1. **Spark Core**：这是Spark的基础，负责任务调度...
sparkGraphx图计算DEMO
qq_16234927的博客
07-18 637
spark图计算简单来说就是构建边集合，点集合，然后把边集合，点集合放到graphx中进行计算。 下面我用scala写一下spark图计算的demo。 package sparksql import org.apache.spark.graphx.{Edge, Graph} import org.apache.spark.{SparkConf, SparkContext} object spa...
Linux(Ubuntu14.04)+MySQL Community Server 5.7.17安装(5.7.17deb安装包)
pucao_cug的专栏
03-23 5054
Linux(Ubuntu14.04)+MySQL Community Server 5.7.17的安装，使用deb安装包进行安装
Linux安装zookeeper集群(CentOS7+Zookeeper3.4.10)
热门推荐
pucao_cug的专栏
05-15 2万+
Linux安装zookeeper集群(CentOS7+Zookeeper3.4.10) 版本号：CentOS7 Zookeeper3.4.10 JDK1.8
Spark-bug总结之一次task长时间卡住不动,一直Running的问题解决
一个记录普通男孩在ＩＴ界学习思考感悟的地方
02-28 5766
目录 现象 排查 现象 spark提交任务后，某一个Stage卡住，通过spark界面看到，executor运行正常，卡住的Stage的task已经分配至executor，但task 一直在running并且数据量不大，task不结束，同时log中也无异常报出。 20/07/27 07:40:13 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 15841@ip-10-3-1-141.ec2.i
spark的官方例子，集合了JAVA和SCALA双版本
yingdajun的博客
05-03 1517
这里是我把JAVA和SCALA双语言的Spark实现源码集成起来，上传到Github上面 免得之后换新电脑忘记了。 https://github.com/yingdajun/realrealsparkdemo
本地模式和集群模式运行spark程序
Above the cloud
02-02 2609
一、本地模式运行spark程序 二、集群模式运行spark程序
spark自带demo的详解
俺叫赵小邪的博客
11-21 6975
import scala.Tuple2;import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import
spark2.1.1对应的org.apache.spark
03-26
.0.documentation在哪里可以找到？ 您可以在以下地址中找到Spark 2.1.1的官方文档： https://spark.apache.org/docs/2.1.1/ 该文档包含各种Spark功能的详细说明、API参考、示例代码等等。如果您需要了解Spark的更多信息或解决问题，这是一个很好的起点。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 15
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值