Master HA彻底解密
视频学习来源:DT-大数据梦工厂 IMF传奇行动视频(后附王家林老师联系方式)
本期内容:
1、MasterHA解析
2、Master HA的四种方式
3、Master HA的内部工作机制
4、Master HA的源码解析
实际提交程序的时候,提交给作为Leader的Master;
程序在运行前是粗粒度的资源分配模式,一般一个Master挂掉后,并不影响集群的运行。
一、Master HA解析
1、生产环境下一般采用Zookeeper做HA,且建议为三台Master,Zookeeper会自动化管理Masters的切换;
2、采用Zookeeper做HA的时候,Zookeeper会负责保存整个Spark集群运行的时候的元数据:Workers、Drivers、Executor;
3、Zookeeper遇到当前Active级别的Master出现故障的时候会从StandbyMaster中选取出一台作为Active级别的Master,但是需要注意被选举后到成为真正的ActiveMaster之间需要从Zookeeper中获取集群当前运行状态的元数据信息并进行恢复;
4、在Master切换的过程中,所有的已经在运行的程序皆运行正常!因为SparkApplication在运行前已经通过Cluster Manager获取了计算资源,所以在运行时Job本身的调度和处理和Master是没有任何关系的!
5、在Master切换过程中唯一的影响是不能提交新的Job:一方面是不能够提交新的应用程序给集群,因为只有Active Master才能接受新的程序的提交请求;另外一方面,已经运行的程序中也不能够因为因为Active操作触发新的Job的提交请求;
二、Master HA的四大方式
1、Master HA的四种方式分别是:Zookeeper、FILESYSTEM(对实时性、延迟性要求没有那么高)、CUSTOM、NONE;
2、需要说明的是:
a)Zookeeper是自动管理Master;
b)FILESYSTEM的方式在Master出现故障后需要手动重新启动机器,机器启动后会立即成为Active级别的Master来对外提供服务(接收应用程序提交的请求、接收新的Job运行的请求);
c)CUSTOM等的方式运行用户自定义MasterHA的实现,这对于高级用户特别有用;
d)None,这是默认情况,当我们下载安装了Spark集群中就是采用
val (persistenceEngine_, leaderElectionAgent_) = RECOVERY_MODE match {
case "ZOOKEEPER" =>
logInfo("Persisting recovery state to ZooKeeper")
val zkFactory =
new ZooKeeperRecoveryModeFactory(conf, serializer)
(zkFactory.createPersistenceEngine(), zkFactory.createLeaderElectionAgent(this))
case "FILESYSTEM" =>
val fsFactory =
new FileSystemRecoveryModeFactory(conf, serializer)
(fsFactory.createPersistenceEngine(), fsFactory.createLeaderElectionAgent(this))
case "CUSTOM" =>
val clazz = Utils.classForName(conf.get("spark.deploy.recoveryMode.factory"))
val factory = clazz.getConstructor(classOf[SparkConf], classOf[Serializer])
.newInstance(conf, serializer)
.asInstanceOf[StandaloneRecoveryModeFactory]
(factory.createPersistenceEngine(), factory.createLeaderElectionAgent(this))
case _ =>
(new BlackHolePersistenceEngine(), new MonarchyLeaderAgent(this))
}
persistenceEngine = persistenceEngine_
leaderElectionAgent = leaderElectionAgent_
}
4、PersistenceEngine中有一个至关重要的方法persist来实现数据持久化,readPersistData来恢复获取;
/**
* Returns the persisted data sorted by their respective ids (which implies that they're
* sorted by time of creation).
*/
final def readPersistedData(
rpcEnv: RpcEnv): (Seq[ApplicationInfo], Seq[DriverInfo], Seq[WorkerInfo]) = {
rpcEnv.deserialize { () =>
(read[ApplicationInfo]("app_"), read[DriverInfo]("driver_"), read[WorkerInfo]("worker_"))
}
}
/**
* Defines how the object is serialized and persisted. Implementation will
* depend on the store used.
*/
def persist(name: String, obj: Object)
/**
* Defines how the object referred by its name is removed from the store.
*/
def unpersist(name: String)
5
、
FILESYSTEM
和
NONE
的方式均采用
MonarchyElectionAgent
的方式来完成
Leader
选举
private def completeRecovery() {
// Ensure "only-once" recovery semantics using a short synchronization period.
if (state != RecoveryState.RECOVERING) { return }
state = RecoveryState.COMPLETING_RECOVERY
// Kill off any workers and apps that didn't respond to us.
workers.filter(_.state == WorkerState.UNKNOWN).foreach(removeWorker)
apps.filter(_.state == ApplicationState.UNKNOWN).foreach(finishApplication)
// Reschedule drivers which were not claimed by any workers
drivers.filter(_.worker.isEmpty).foreach { d =>
logWarning(s"Driver ${d.id} was not found after master recovery")
if (d.desc.supervise) {
logWarning(s"Re-launching ${d.id}")
relaunchDriver(d)
} else {
removeDriver(d.id, DriverState.ERROR, None)
logWarning(s"Did not re-launch ${d.id} because it was not supervised")
}
}
DT大数据梦工厂
新浪微博:www.weibo.com/ilovepains/
微信公众号:DT_Spark
博客:http://.blog.sina.com.cn/ilovepains
TEL:18610086859
Email:18610086859@vip.126.com