模拟Spark的Master和Worker通信
- 一个Master
- 管理Worker
- 若干个Worker(Worker可以按需添加)
- 注册
- 发送心跳
图解
实现思路
-
构建Master,Worker阶段
-
构建Master ActorSystem Actor
-
构建Worker ActorSystem Actor
-
-
Worker注册阶段
- Worker进程向Master注册(将自己的ID,CPU核数,内存大小(M)发送给Master)
-
Worker定时发送心跳阶段
- Worker定期向Master发送心跳消息
-
Master定时心跳监测阶段
- Master定期检查Worker心跳,将一些超时的Worker移除,并对Worker按照内存进行倒序排序
-
多个Worker测试阶段
- 启动多个Worker,查看是否注册成功
1.工程搭建
1.分别搭建以下几个项目
工程名 | 说明 |
---|---|
spark-demo-common | 存放公共的消息、实体类 |
spark-demo-master | Akka Master节点 |
spark-demo-worker | Akka Worker节点 |
2.导入依赖(在附录一中)
3.导入配置文件(在附录二中)
修改Master的端口为7000
修改Worker的端口为7100
4.整体架构
- spark-demo-common结构
- spark-demo-master结构
- spark-demo-worker结构
2.代码参考(仔细看代码注释,非常之详细)
spark-demo-common代码参考
- Entities(Worker基本信息)
package spark.common
// Worker基本信息
case class WorkerInfo(workerid:String,
cpu:Int,
mem:Int,
lastHeartBeatTime:Long)
- MessagePackage(封装Worker注册消息,注册成功消息,心跳消息)
package spark.common
// 封装Worker注册消息
// 1. wokerid
// 2. cpu核数
// 3. mem内存大小(M)
case class WorkerRegisterMessage(workerid:String,
cpu:Int,
mem:Int)
// 注册成功消息
case object RegisterSuccessMessage
// 心跳消息
case class WorkerHeartBeatMessage(workerid:String,
cpu:Int,
mem:Int)
spark-demo-master代码参考
- MasterActor
package spark.actor
import java.util.Date
import akka.actor.Actor
import spark.common.{RegisterSuccessMessage, WorkerHeartBeatMessage, WorkerInfo, WorkerRegisterMessage}
import spark.master.ConfigUtil
//akka.tcp://actorSystem@127.0.0.1:7000
object MasterActor extends Actor {
//3.1 定义一个Map集合.用来存储WorkerActor的注册信息
private var regWorkerActorMap = collection.mutable.Map[String, WorkerInfo]()
//7.Master定时自检的动作,检查超时的Worker,并移除这些WorkerActor
override def preStart(): Unit = {
//7.1 导入时间单位隐式转换和定时器隐式参数
import scala.concurrent.duration._
import context.dispatcher
//7.2开启定时任务
context.system.scheduler.schedule(0 seconds, ConfigUtil.`master.check.heartbeat.interval` seconds) {
// 7.3 获取所有的超时的worker对象
val timeOutWorker = regWorkerActorMap.filter {
//参数的格式: workId=>workerInfo(worderId,cpu,mem,LastHearBeatTime)
keyval =>
//7.3.1 获取当前worker对象的最后一次心跳时间,
var LastHearBeatTime = keyval._2.lastHeartBeatTime
//7.3.2 超时公式(当前时间-最后一次心跳时间)>超时时间(配置文件中的数据*1000)
if (new Date().getTime - LastHearBeatTime > ConfigUtil.`master.check.heartbeat.timeout` * 1000) true else false
}
// 7.4 从regWorkerActorMap中移除哪些超时的worker对象
if (!timeOutWorker.isEmpty) {
regWorkerActorMap --= timeOutWorker.map(_._1) //双列集合是根据键删除此键值对的
}
// 7.5 对所有存活下来的worker对象,按照内存大小进行降序排序,
// 7.5.1从regWorkerActorMap获取所有的workInfo对象
val workerList = regWorkerActorMap.map(_._2).toList
// 7.5.2 对Workerlist按照内存大小降序排序
val workerGroupList = workerList.sortBy(_.mem).reverse
// 7.5.3打印排序后的结果
println("MasterActor存储的活着的Worker对象如下")
println(workerList)
}
}
override def receive = {
// case x =>println(x)
//1. 接收workerActor提交的注册信息
case WorkerRegisterMessage(workerid, cpu, mem) => {
//能走到这里,肯定是worker的合法的注册信息
//2. 打印接收到的注册信息
println(s"MasterActor:接收到worker的注册信息是,${workerid},${cpu},${mem}")
//3. 保存WorkerActor提交的注册信息
//3.2 具体的保存动作
regWorkerActorMap += workerid -> WorkerInfo(workerid, cpu, mem, new Date().getTime)
// 4. 为了查看是否成功
println(s"MasterActor:存储的Worker有:${regWorkerActorMap}")
//5. 获取WorkerActor的引用,给他一个回执信息,提示注册成功
val workerActor = context.actorSelection("akka.tcp://actorSystrm@127.0.0.1:7001/user/workerActor")
workerActor ! RegisterSuccessMessage
}
//6. 打印接收到的心跳信息
case WorkerHeartBeatMessage(workerid, cpu, mem) => {
//6.1. 接收workerActor提交的注册信息
println(s"MasterActor:接收到${workerid}的心跳信息")
//6.2 更新该worker的最后一次心跳信息
regWorkerActorMap += workerid -> WorkerInfo(workerid, cpu, mem, new Date().getTime)
}
}
}
- ConfigUtil
package spark.master
import com.typesafe.config.{Config, ConfigFactory}
//针对Master的工具类
object ConfigUtil {
//1 .获取当前项目的配置信息对象
private val config: Config = ConfigFactory.load()
//2.获取检查worker心跳的时间周期
val `master.check.heartbeat.interval` = config.getInt("master.check.heartbeat.interval")
//3.获取worker心跳的超时时间
val `master.check.heartbeat.timeout` = config.getInt("master.check.heartbeat.timeout")
}
- Master
package spark.master
import akka.actor.{ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import spark.actor.MasterActor
object Master {
def main(args: Array[String]): Unit = {
//创建ActorSystem
val actorSystem = ActorSystem("actorSystem", ConfigFactory.load())
//通过ActorSystem关联MasterActor
val masterActor = actorSystem.actorOf(Props(MasterActor), "masterActor")
//测试代码
// masterActor !"hello"
}
}
spark-demo-worker代码参考
- WorkerActor
package spark.actor
import java.util.UUID
import akka.actor.{Actor, ActorSelection}
import com.typesafe.config.ConfigUtil
import spark.common.{RegisterSuccessMessage, WorkerHeartBeatMessage, WorkerRegisterMessage}
import spark.worker.ConfigUtils
import scala.util.Random
//akka.tcp://actorSystrm@127.0.0.1:7001
object WorkerActor extends Actor {
//1.定义一些成员变量,用来记录MasterActor的引用,和提交给Master的注册信息(workid,cpu,mem)
private var masterActorRef: ActorSelection = _ //MasterActor的运用
private var workerId: String = _ //当前WorkerActor的id
private var cpu: Int = _ //当前WorkerActor的核数
private var mem: Int = _ //当前WorkerActor的内存大小(单位是MB)
private var CPU_LIST: List[Int] = List(1, 2, 3, 4, 6, 7) //cpu取值范围
private var MEM_LIST: List[Int] = List(512, 1024, 2048, 4096) //内存的取值范围
//2.在preStart()方法中,对上诉的成员变进行赋值,并将注册的信息封装成:RegisterMessage对象,发送给MasterWorker
//preStart()方法中的内容是在actor的start()方法之前执行
override def preStart(): Unit = {
// 2.1 获取MasterActor的引用
masterActorRef = context.actorSelection("akka.tcp://actorSystem@127.0.0.1:7000/user/masterActor")
// 2.2 创建一个随机数对象
var r = new Random()
// 2.3 给WorkerId,cpu,men变量赋值
workerId = UUID.randomUUID().toString
cpu = CPU_LIST(r.nextInt(CPU_LIST.size))
mem = MEM_LIST(r.nextInt(MEM_LIST.size))
// 2.4 将上诉的成员变量封装成RegisterMeessage对象
// 2.5 将上诉封装好的RegisterMeessage发送个MasterWorker
val workerRegisterMessage = WorkerRegisterMessage(workerId, cpu, mem)
masterActorRef ! workerRegisterMessage
}
override def receive = {
case RegisterSuccessMessage => {
//1. 打印接收到的注册成功消息
println("WorkerActor:接收到注册成功消息!")
//2. 导入时间单位隐式转换,和定时器的隐式参数
import scala.concurrent.duration._
import context.dispatcher
//3. 定时给Master发送心跳信息
//3.1 开启定时任务
context.system.scheduler.schedule(0 seconds,ConfigUtils.`worker.heartbeat.interval` seconds){
//3.2 给MasterActor发送心跳信息,
masterActorRef ! WorkerHeartBeatMessage(workerId,cpu,mem )
}
}
}
}
- ConfigUtils
package spark.worker
import com.typesafe.config.{Config, ConfigFactory}
object ConfigUtils {
//获取worker的心跳周期
//1. 获取配置信息对象
private val config: Config = ConfigFactory.load()
//2. 获取worker心跳的具体周期
val `worker.heartbeat.interval` = config.getInt("worker.heartbeat.interval")
def main(args: Array[String]): Unit = {
println(`worker.heartbeat.interval`)
}
}
- Worker
package spark.worker
import akka.actor.{ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import spark.actor.WorkerActor
object Worker {
def main(args: Array[String]): Unit = {
//1.获取actorSystem对象,用来管理我们自定义的Actor对象
val actorSystrm = ActorSystem("actorSystrm", ConfigFactory.load())
//2.通过actorSystem对象,关联自定义的actor对象
val workerActor = actorSystrm.actorOf(Props(WorkerActor), "workerActor")
//3.测试代码
// workerActor!"hello world"
}
}
3.最终实现效果
- 启动master
- 启动worker
附录一(三个模块中都要添加)
注意:在master和worker模块中要额外添加common的依赖
<dependency>
<groupId>spark</groupId>
<artifactId>spark-demo-common</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.11.8</scala.version>
<scala.compat.version>2.11</scala.compat.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.11</artifactId>
<version>2.3.14</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-remote_2.11</artifactId>
<version>2.3.14</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
附录二
master的application.conf配置文件
akka.actor.provider = "akka.remote.RemoteActorRefProvider"
akka.remote.netty.tcp.hostname = "127.0.0.1"
akka.remote.netty.tcp.port = "7000"
#配置检查Worker心跳的时间周期(单位: 秒)
master.check.heartbeat.interval = 6
#配置worker心跳超时的时间(秒)
master.check.heartbeat.timeout = 15
worker的application.conf配置文件
akka.actor.provider = "akka.remote.RemoteActorRefProvider"
akka.remote.netty.tcp.hostname = "127.0.0.1"
akka.remote.netty.tcp.port = "7001"
worker.heartbeat.interval = 5