在amazon ec2上部署spark cluster

最新推荐文章于 2024-02-17 19:01:15 发布
最新推荐文章于 2024-02-17 19:01:15 发布
阅读量2.1k 收藏 1
点赞数
分类专栏： Linux 文章标签： spark amazon ec2 hdfs sacala sbt
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/alexwang1983/article/details/50643160
版权

1，利用spark自带的ec2脚本来生成spark

在spark的安装目录下，执行如下命令

$ ./ec2/spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> --vpc-id=<vpc-id> --subnet-id=<subnet-id> launch <cluster-name>, where <keypair> is the name of your EC2 key pair (that you gave it when you created it), <key-file> is the private key file for your key pair,<num-slaves> is the number of slave nodes to launch (try 1 at first), <vpc-id> is the name of your VPC, <subnet-id> is the name of your subnet, and <cluster-name> is the name to give to your cluster.

比如:

$ export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtU export AWS_ACCESS_KEY_ID=ABCDEFG1234567890123

$ ./ec2/spark-ec2 --key-pair=spark_study --identity-file=/home/ubuntu/spark_study.pem --region=ap-northeast-1 --zone=ap-northeast-1a  launch my-spark-cluster

2,在本地生成scala程序

首先创建如下目录

./src

./src/main

./src/main/scala


然后创建文件，./src/main/scala/SimpleApp.scala

内容为

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/home/ubuntu/spark-1.6.0-bin-hadoop2.6/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

我们使用sbt来生成scala的jar包,首先需要安装sbt。

通过如下命令安装sbt

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
sudo apt-get install sbt


创建如下文件 ./simple.sbt

内容为

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.3"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0"

然后执行 

sbt package

可以看到在target目录下生成了一个jar包，这个包我们可以在cluster上面执行。


3，将jar包上传到spark的master结点上。

4，注意我们的程序里面的logfile路径

/home/ubuntu/spark-1.6.0-bin-hadoop2.6/README.md

在spark cluster上面，默认会去读取hdfs的路径，这就需要现在hdfs上面创建一个文本文件

首先登陆到master的机器结点，然后进入hdfs的安装目录

cd ~/ephemeral-hdfs

创建一个文本文件~/README.md，然后把文本文件拷贝到hdfs的 
/home/ubuntu/spark-1.6.0-bin-hadoop2.6/
目录下。

通过如下命令拷贝.

bin/hadoop fs -mkdir /home/ubuntu/spark-1.6.0-bin-hadoop2.6/

bin/hadoop fs -put ~/README.md /home/ubuntu/spark-1.6.0-bin-hadoop2.6/

然后可以通过如下命令检查是否拷贝正确

bin/hadoop fs -ls /home/ubuntu/spark-1.6.0-bin-hadoop2.6/

这时应该可以看到README.md文件。这就是我们拷贝到HDFS上的文件。


5，提交运行拷贝到master结点的jar包

进入spark的安装路径，然后执行如下命令

./bin/spark-submit --class SimpleApp --master spark://ec2-52-192-126-225.ap-northeast-1.compute.amazonaws.com:7077 --deploy-mode client /home/ec2-user/simple-project_2.10-1.0.jar

我们可以看到运行结果 

16/02/07 12:55:31 INFO spark.SecurityManager: Changing view acls to: root
16/02/07 12:55:31 INFO spark.SecurityManager: Changing modify acls to: root
16/02/07 12:55:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/02/07 12:55:32 INFO util.Utils: Successfully started service 'sparkDriver' on port 36092.
16/02/07 12:55:32 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/02/07 12:55:32 INFO Remoting: Starting remoting
16/02/07 12:55:33 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@172.31.12.26:33336]
16/02/07 12:55:33 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 33336.
16/02/07 12:55:33 INFO spark.SparkEnv: Registering MapOutputTracker
16/02/07 12:55:33 INFO spark.SparkEnv: Registering BlockManagerMaster
16/02/07 12:55:33 INFO storage.DiskBlockManager: Created local directory at /mnt/spark/blockmgr-76f22bc5-a78a-4847-ab29-b7292f7a7cff
16/02/07 12:55:33 INFO storage.DiskBlockManager: Created local directory at /mnt2/spark/blockmgr-dd59300b-9590-41c3-b94a-d76a3c7fe8db
16/02/07 12:55:33 INFO storage.MemoryStore: MemoryStore started with capacity 511.5 MB
16/02/07 12:55:33 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/02/07 12:55:33 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/02/07 12:55:33 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/02/07 12:55:33 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/02/07 12:55:33 INFO ui.SparkUI: Started SparkUI at http://ec2-52-192-126-225.ap-northeast-1.compute.amazonaws.com:4040
16/02/07 12:55:33 INFO spark.HttpFileServer: HTTP File server directory is /mnt/spark/spark-6ce5d4ee-c69e-40d1-9053-5d613461f9bf/httpd-77ec3777-a82b-4ef3-9fca-5a4c790a5747
16/02/07 12:55:33 INFO spark.HttpServer: Starting HTTP Server
16/02/07 12:55:33 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/02/07 12:55:33 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:56378
16/02/07 12:55:33 INFO util.Utils: Successfully started service 'HTTP file server' on port 56378.
16/02/07 12:55:33 INFO spark.SparkContext: Added JAR file:/home/ec2-user/simple-project_2.10-1.0.jar at http://172.31.12.26:56378/jars/simple-project_2.10-1.0.jar with timestamp 1454849733709
16/02/07 12:55:33 INFO client.AppClient$ClientEndpoint: Connecting to master spark://ec2-52-192-126-225.ap-northeast-1.compute.amazonaws.com:7077...
16/02/07 12:55:34 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160207125534-0003
16/02/07 12:55:34 INFO client.AppClient$ClientEndpoint: Executor added: app-20160207125534-0003/0 on worker-20160207085403-172.31.2.135-39140 (172.31.2.135:39140) with 2 cores
16/02/07 12:55:34 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160207125534-0003/0 on hostPort 172.31.2.135:39140 with 2 cores, 6.0 GB RAM
16/02/07 12:55:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 47128.
16/02/07 12:55:34 INFO netty.NettyBlockTransferService: Server created on 47128
16/02/07 12:55:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/02/07 12:55:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.31.12.26:47128 with 511.5 MB RAM, BlockManagerId(driver, 172.31.12.26, 47128)
16/02/07 12:55:34 INFO storage.BlockManagerMaster: Registered BlockManager
16/02/07 12:55:34 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160207125534-0003/0 is now RUNNING
16/02/07 12:55:34 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/02/07 12:55:35 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 46.3 KB, free 46.3 KB)
16/02/07 12:55:35 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.4 KB, free 50.7 KB)
16/02/07 12:55:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.12.26:47128 (size: 4.4 KB, free: 511.5 MB)
16/02/07 12:55:35 INFO spark.SparkContext: Created broadcast 0 from textFile at SimpleApp.scala:11
16/02/07 12:55:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/07 12:55:35 WARN snappy.LoadSnappy: Snappy native library not loaded
16/02/07 12:55:35 INFO mapred.FileInputFormat: Total input paths to process : 1
16/02/07 12:55:35 INFO spark.SparkContext: Starting job: count at SimpleApp.scala:12
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Got job 0 (count at SimpleApp.scala:12) with 2 output partitions
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (count at SimpleApp.scala:12)
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Missing parents: List()
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at filter at SimpleApp.scala:12), which has no missing parents
16/02/07 12:55:35 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.1 KB, free 53.9 KB)
16/02/07 12:55:35 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1891.0 B, free 55.7 KB)
16/02/07 12:55:35 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.31.12.26:47128 (size: 1891.0 B, free: 511.5 MB)
16/02/07 12:55:35 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/02/07 12:55:35 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at filter at SimpleApp.scala:12)
16/02/07 12:55:35 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/02/07 12:55:38 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (ip-172-31-2-135.ap-northeast-1.compute.internal:38142) with ID 0
16/02/07 12:55:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ip-172-31-2-135.ap-northeast-1.compute.internal, partition 0,NODE_LOCAL, 2286 bytes)
16/02/07 12:55:38 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, ip-172-31-2-135.ap-northeast-1.compute.internal, partition 1,NODE_LOCAL, 2286 bytes)
16/02/07 12:55:38 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-172-31-2-135.ap-northeast-1.compute.internal:46994 with 4.1 GB RAM, BlockManagerId(0, ip-172-31-2-135.ap-northeast-1.compute.internal, 46994)
16/02/07 12:55:38 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-172-31-2-135.ap-northeast-1.compute.internal:46994 (size: 1891.0 B, free: 4.1 GB)
16/02/07 12:55:39 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-31-2-135.ap-northeast-1.compute.internal:46994 (size: 4.4 KB, free: 4.1 GB)
16/02/07 12:55:39 INFO storage.BlockManagerInfo: Added rdd_1_0 in memory on ip-172-31-2-135.ap-northeast-1.compute.internal:46994 (size: 3.1 KB, free: 4.1 GB)
16/02/07 12:55:39 INFO storage.BlockManagerInfo: Added rdd_1_1 in memory on ip-172-31-2-135.ap-northeast-1.compute.internal:46994 (size: 2.9 KB, free: 4.1 GB)
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1477 ms on ip-172-31-2-135.ap-northeast-1.compute.internal (1/2)
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1502 ms on ip-172-31-2-135.ap-northeast-1.compute.internal (2/2)
16/02/07 12:55:39 INFO scheduler.DAGScheduler: ResultStage 0 (count at SimpleApp.scala:12) finished in 3.923 s
16/02/07 12:55:39 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Job 0 finished: count at SimpleApp.scala:12, took 4.132609 s
16/02/07 12:55:39 INFO spark.SparkContext: Starting job: count at SimpleApp.scala:13
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Got job 1 (count at SimpleApp.scala:13) with 2 output partitions
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (count at SimpleApp.scala:13)
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Missing parents: List()
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at filter at SimpleApp.scala:13), which has no missing parents
16/02/07 12:55:39 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 58.8 KB)
16/02/07 12:55:39 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1893.0 B, free 60.7 KB)
16/02/07 12:55:39 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.31.12.26:47128 (size: 1893.0 B, free: 511.5 MB)
16/02/07 12:55:39 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at filter at SimpleApp.scala:13)
16/02/07 12:55:39 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, ip-172-31-2-135.ap-northeast-1.compute.internal, partition 0,PROCESS_LOCAL, 2286 bytes)
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, ip-172-31-2-135.ap-northeast-1.compute.internal, partition 1,PROCESS_LOCAL, 2286 bytes)
16/02/07 12:55:39 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-172-31-2-135.ap-northeast-1.compute.internal:46994 (size: 1893.0 B, free: 4.1 GB)
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 65 ms on ip-172-31-2-135.ap-northeast-1.compute.internal (1/2)
16/02/07 12:55:39 INFO scheduler.DAGScheduler: ResultStage 1 (count at SimpleApp.scala:13) finished in 0.078 s
16/02/07 12:55:39 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 79 ms on ip-172-31-2-135.ap-northeast-1.compute.internal (2/2)
16/02/07 12:55:39 INFO scheduler.DAGScheduler: Job 1 finished: count at SimpleApp.scala:13, took 0.100431 s
16/02/07 12:55:39 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
Lines with a: 32, Lines with b: 11
16/02/07 12:55:39 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/02/07 12:55:39 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/02/07 12:55:39 INFO ui.SparkUI: Stopped Spark web UI at http://ec2-52-192-126-225.ap-northeast-1.compute.amazonaws.com:4040
16/02/07 12:55:39 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors
16/02/07 12:55:39 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down
16/02/07 12:55:39 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/02/07 12:55:39 INFO storage.MemoryStore: MemoryStore cleared
16/02/07 12:55:39 INFO storage.BlockManager: BlockManager stopped
16/02/07 12:55:39 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/02/07 12:55:39 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/02/07 12:55:39 INFO spark.SparkContext: Successfully stopped SparkContext
16/02/07 12:55:39 INFO util.ShutdownHookManager: Shutdown hook called
16/02/07 12:55:39 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-6ce5d4ee-c69e-40d1-9053-5d613461f9bf/httpd-77ec3777-a82b-4ef3-9fca-5a4c790a5747
16/02/07 12:55:39 INFO util.ShutdownHookManager: Deleting directory /mnt2/spark/spark-8d522cb1-beec-4d21-9a8c-e9c1cf635ea7
16/02/07 12:55:39 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-6ce5d4ee-c69e-40d1-9053-5d613461f9bf


需要继续研究的是当前是在主节点通过client的模式来部署程序的，当使用cluster模式来部署的时候，会报错。这个需要继续研究了。
alexwang1983
关注
专栏目录
spark完全分布式部署
需要远程指导仿真实验、代码有问题的，请后台私信或者关注公众号
06-30 3387
本实验任务主要完成基于ubuntu环境的Spark完全分布式部署、配置和调试工作。通过完成本实验任务，要求学生熟练掌握Spark完全分布式部署方法，为后续实验的开展奠定Spark平台基础，也为从事大数据平台运维工程师、大数据技术支持工程师等岗位工作奠定夯实的技能基础。掌握Spark完全分布式环境的部署本次环境是：Ubuntu16.04spark有以下几种安装模式，每种安装模式都有自己不同的优点和长处。   local(本地模式)： 常用于本地开发测试，本地还分为local单线程和local-cluster多
spark伪分布部署
需要远程指导仿真实验、代码有问题的，请后台私信或者关注公众号
06-30 656
本实验任务主要完成基于ubuntu环境的Spark伪分布部署、配置和调试工作。通过完成本实验任务，要求学生熟练掌握Spark伪分布部署方法，为后续实验的开展奠定Spark平台基础，也为从事大数据平台运维工程师、大数据技术支持工程师等岗位工作奠定夯实的技能基础。掌握Spark伪分布环境的搭建配置本次环境是：Ubuntu16.04 所需root密码为simpleware-sshspark有以下几种安装模式，每种安装模式都有自己不同的优点和长处。   local(本地模式)： 常用于本地开发测试，本地还分为loc
Spark学习笔记（一）Spark Cluster Mode
Mike的博客
03-18 839
Spark Cluster Mode 首先了解一些概念： Application #用户通过Spark构建的程序，由Driver程序和executors程序组成。 Application jar #个人理解是包含了Application所依赖的其他包的jar Driver program #master，SC生成的运行在程序入口（main()）之下的程序，个人理解Driv...
【Spark学习】Apache Spark部署之Amazon EC2
weixin_30564785的博客
12-02 134
摘要: Spark版本：1.1.1 转载于:https://www.cnblogs.com/zhangningbo/p/4138014.html
在ec2上部署spark-部署成功后无法启动
taizitj的专栏
11-23 1051
好不容易脚本运行成功，也能连接上去看到spark，然而，并没有ui可以显示，也没有启动。 鼓捣半天，首先，屌丝程序员用的1g内存的免费套餐，spark配置了master和worker都1g的虚拟机，其实我并不确定这个会不会导致启动失败，因为我启动失败的原因报错是因为连了鬼畜的机器。好吧，刚用aws，完全不知道那鬼畜的域名（或者是ip别名，还没有弄清那个主机名称是什么，就先叫域名吧）对应的机器是采
spark Cluster Mode
JackLi31742的博客
11-12 232
1、SparkContext 和cluster manager（standalone、yarn等）建立连接，给APP分配资源 2、获得executor 3、发送jar给executor 4、sparkContext发送task给executor note： 1、每个application都是独立的，driver端调度task，executor端task运行在不同的jvm上 术语 Term...
Spark Cluster Mode Overview
蓝色水彼的博客
04-04 268
本文档描述spark如何在集群运行，便于理解过程中包含的组件。请阅读application submission guide来了解如何向集群提交程序。 Components spark程序在集群中作为独立的线程集合运行，由主程序（driver程序）中的SparkContext对象来协调。 特别地，SparkContext可以连接几种不同的集群管理者（如standalone集群管理、Mesos...
在aws的ec2上部署spark
taizitj的专栏
11-22 2933
在spark官网上有详细的在ec2上部署spark的教程。 在并发编程网上也有相关中文教程。 但是在使用情况下还是遇到了不少问题。 首先，accessid和密钥对不是一个概念，我就是弄错了，所以一直登录不上。 然后，在运行脚本过程中，一些错误忘记记录了，主要是一个连接spark集群的错误： Copying SSH key /home/tj/private/aws_spark.pem to
flintrock:用于启动Apache Spark集群的命令行工具
05-07
假设您已经，这是在EC2上启动集群的快速方法。 Flintrock与Amazon Linux配合使用效果最佳。 您可以获取最新的AMI ID。 flintrock launch test-cluster \ --num-slaves 1 \ --spark-version 3.0.1 \ --ec2-key-...
2.1 Amazon EC2 Instance
JessicaWin
06-01 920
An instance is a virtual server in the cloud. Its configuration at launch is a copy of the AMI that you specified when you launched the instance. Aninstance typeessentially determines the hardware of the host computer used for your instance.
Spark的Cluster模式架构图
圈圈任
02-28 907
一、在Spark的官网文件中，可以看到Spark的Cluster模式架构图，浏览器输入:https://spark.apache.org/docs/latest/cluster-overview.html（1）DriverProgram 就是程序员设计的Spark程序，在Spark中必须定义SparkContext，它是开发Spark应用程序的入口（2）SparkContext是通过Cluster...
【spark】【集群模式】【Cluster】【概述 】
最新发布
wq6qeg88的博客
02-17 997
具体来说，要在集群上运行，SparkContext可以连接到几种类型的集群管理器（Spark自己的独立集群管理器，Mesos，YARN或Kubernetes），这些集群管理器在应用程序之间分配资源。连接后，Spark会获取集群中节点上的执行器，这些执行器是为应用程序运行计算和存储数据的进程。Spark可以控制跨应用程序（在集群管理器级别）和应用程序内部（如果在同一SparkContext上发生多个计算）的资源分配。Spark应用程序在集群上作为独立的进程集运行，由主程序中的。监控指南还介绍了其他监控选项。
Spark cluster 安装
然光 不学无術 暗于大理
10-03 132
使用 Ubuntu 机器做主服务 Master 192.168.50.15 (Ubuntu) Worker 1 192.168.50.15 (Ubuntu) Worker 2 192.168.50.16 (Windows 10) 先安装 JDK 和下载 Apache Spark sudo apt-get install openjdk-11-jdk wget https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-had
Spark----Cluster模式（standalone和Spark on Yarn）的安装配置
WQY992的博客
07-26 711
Cluster模式 standalone，Yarn，Messos（国内很少用） 说明 如果现有的集群框架中存在MR的相关应用，同时相关的MR无法转换为Spark应用的，集群选型为Spark On Yarn； 如果现有的集群框架中只有Spark应用，那么建议采用spark Standalone模式； spark可以运行在不同的计算引擎上，所以集群模式安装分不同种情况： 说明：spark的Stand...
spark-cluster及yarn-client说明
bsf5521的博客
08-02 299
spark on yarn 的两种模式 1）yarn-cluster：我们一般用于生产使用 2）yarn-client：适用于交互、调试，能够立即看到app的输出 Yarn-cluster和yarn-client的区别在于appMaster：yarn appMaster，每个yarn app实例有一个appMaster进程，是为app启动的第一个container；负责从ResourceMa
Spark cluster 工作机制
Codedemo的博客
05-17 217
Spark-cluster 工作任务执行流程模型 在YARN Cluster模式下，任务提交后会和ResourceManager通讯申请启动ApplicationMaster，随后ResourceManager分配container，在合适的NodeManager上启动ApplicationMaster，此时的ApplicationMaster就是Drive...
spark cluster和spark client的区别---转载
z7one
08-05 3007
Spark支持Yarn,Mesos,Standalone三种集群部署模式，它们的共同点：Master服务(Yarn ResourceManager,Mesos master,Spark standalone)来决定哪些应用可以运行以及在哪什么时候运行，Slave服务(Yarn NodeManger)运行在每个节点上，节点上实际运行着Executor进程，此外还监控着它们的运行状态以及资源的消耗 Spark On Yarn模式，又分cluster和client模式。cluster和client的最大区别在于
Spark有两种常见的提交方式：client 模式和 cluster 模式对机器 CPU 的影响
qq_43688472的博客
08-30 874
总体而言，Client 模式下对机器的 CPU 影响较大，因为 Driver 运行在客户端节点上，而 Cluster 模式下对机器的 CPU 影响相对均匀，因为任务在整个集群中运行。客户端只负责提交应用程序，并不参与应用程序的实际运行。在 Client 模式中，机器的 CPU 负担主要集中在 Driver 进程上，因为 Driver 负责调度和监控整个应用程序的运行。在 Cluster 模式中，机器的 CPU 负担在整个集群中分布，因为 Driver 和 Executor 都在各自的节点上运行。
Spark Cluster与Application中的重要概念以及如何提交应用spark-submit
俺叫赵小邪的博客
11-23 1089
一、Spark的运行机制： 一个Spark应用包含两个部分： （1） Driver Program（一个）：里面包含最重要的SparkContext。 （2）Executor（多个）：一个里面包含多个Task(任务)和Cache(缓存)。1、构建Spark Application运行环境 在Driver Program中新建SparkContext(包含spar
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值