目标:简单模拟Spark的通信机制,多个worker向Master注册并发送心跳,Master定时清理超时的worker。具体流程如下:
1、启动Master并开启清空超时Worker的定时任务
2、Worker启动的时候,在preStart方法中连接Master,并向Master注册自己的相关信息
3、Master收到worker的注册并返回自己的url给Worker,表示该Worker注册成功
4、Worker收到注册成功的消息后,定时给Master发生心跳消息
流程图:
通信消息的样本类
//client向server发生注册消息的
case class RegisterWork(val id:String,val memory:Int,val cores: Int)
//server向client反馈注册成功的信息
case class RegisteredWorker(val clientHost:String)
//该伴生对象用于worker本地发生消息给自己
case object SendHeartBeat
//client发生心跳给server
case class HeartBeat(val id_client:String)
//该伴生对象用于server定时检测超时的client
case object CheckTimeOutWork
class WorkerInfo(val id:String,val memory: Int,val cores: Int){
var lastHeartBeatTine: Long = 0
}
ResourceManager节点实现Master代码:
import akka.actor.{Actor, ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import scala.collection.mutable
//server
class Master(var host: String,var port : Int) extends Actor{
val id2WorkInfo = new mutable.HashMap[String,WorkerInfo]()
//为了便于一些额外的逻辑,比如按Worker的剩余可用memory进行排序
val workes = new mutable.HashSet[WorkerInfo]()
override def preStart(): Unit = {
//millis是毫秒的单位,其在包scala.concurrent.duration下
import scala.concurrent.duration._