简易的Spark底层通信框架实现

最新推荐文章于 2022-05-27 16:14:07 发布

不忘初心$$

最新推荐文章于 2022-05-27 16:14:07 发布

阅读量474

点赞数 1

分类专栏： spark&&scala

本文链接：https://blog.csdn.net/qq_42786792/article/details/103197322

版权

spark&&scala 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

模拟Spark的Master和Worker通信

一个Master
- 管理Worker
若干个Worker（Worker可以按需添加）
- 注册
- 发送心跳

图解
在这里插入图片描述
实现思路

构建Master,Worker阶段
- 构建Master ActorSystem Actor
- 构建Worker ActorSystem Actor
Worker注册阶段
- Worker进程向Master注册(将自己的ID,CPU核数,内存大小(M)发送给Master)
Worker定时发送心跳阶段
- Worker定期向Master发送心跳消息
Master定时心跳监测阶段
- Master定期检查Worker心跳,将一些超时的Worker移除,并对Worker按照内存进行倒序排序
多个Worker测试阶段
- 启动多个Worker,查看是否注册成功

1.工程搭建

1.分别搭建以下几个项目

工程名	说明
spark-demo-common	存放公共的消息、实体类
spark-demo-master	Akka Master节点
spark-demo-worker	Akka Worker节点

2.导入依赖(在附录一中)
3.导入配置文件(在附录二中)
修改Master的端口为7000
修改Worker的端口为7100
4.整体架构

spark-demo-common结构
spark-demo-master结构
spark-demo-worker结构

2.代码参考(仔细看代码注释,非常之详细)

spark-demo-common代码参考

Entities(Worker基本信息)

package spark.common

// Worker基本信息
case class WorkerInfo(workerid:String,
                      cpu:Int,
                      mem:Int,
                      lastHeartBeatTime:Long)

MessagePackage(封装Worker注册消息,注册成功消息,心跳消息)

package spark.common

// 封装Worker注册消息
// 1. wokerid
// 2. cpu核数
// 3. mem内存大小(M)
case class WorkerRegisterMessage(workerid:String,
                                 cpu:Int,
                                 mem:Int)

// 注册成功消息
case object RegisterSuccessMessage

// 心跳消息
case class WorkerHeartBeatMessage(workerid:String,
                                  cpu:Int,
                                  mem:Int)

spark-demo-master代码参考

MasterActor

package spark.actor

import java.util.Date

import akka.actor.Actor
import spark.common.{RegisterSuccessMessage, WorkerHeartBeatMessage, WorkerInfo, WorkerRegisterMessage}
import spark.master.ConfigUtil

//akka.tcp://actorSystem@127.0.0.1:7000
object MasterActor extends Actor {
  //3.1 定义一个Map集合.用来存储WorkerActor的注册信息
  private var regWorkerActorMap = collection.mutable.Map[String, WorkerInfo]()

  //7.Master定时自检的动作,检查超时的Worker,并移除这些WorkerActor
  override def preStart(): Unit = {
    //7.1 导入时间单位隐式转换和定时器隐式参数
    import scala.concurrent.duration._
    import context.dispatcher
    //7.2开启定时任务
    context.system.scheduler.schedule(0 seconds, ConfigUtil.`master.check.heartbeat.interval` seconds) {
      //  7.3 获取所有的超时的worker对象
      val timeOutWorker = regWorkerActorMap.filter {
        //参数的格式: workId=>workerInfo(worderId,cpu,mem,LastHearBeatTime)
        keyval =>
          //7.3.1 获取当前worker对象的最后一次心跳时间,
          var LastHearBeatTime = keyval._2.lastHeartBeatTime
          //7.3.2 超时公式(当前时间-最后一次心跳时间)>超时时间(配置文件中的数据*1000)
          if (new Date().getTime - LastHearBeatTime > ConfigUtil.`master.check.heartbeat.timeout` * 1000) true else false
      }
      //  7.4 从regWorkerActorMap中移除哪些超时的worker对象
      if (!timeOutWorker.isEmpty) {
        regWorkerActorMap --= timeOutWorker.map(_._1) //双列集合是根据键删除此键值对的
      }
      //  7.5 对所有存活下来的worker对象,按照内存大小进行降序排序,
      //  7.5.1从regWorkerActorMap获取所有的workInfo对象
      val workerList = regWorkerActorMap.map(_._2).toList
      // 7.5.2 对Workerlist按照内存大小降序排序
      val workerGroupList = workerList.sortBy(_.mem).reverse
      // 7.5.3打印排序后的结果
      println("MasterActor存储的活着的Worker对象如下")
      println(workerList)
    }
  }

  override def receive = {
    //    case x =>println(x)
    //1. 接收workerActor提交的注册信息
    case WorkerRegisterMessage(workerid, cpu, mem) => {
      //能走到这里,肯定是worker的合法的注册信息
      //2. 打印接收到的注册信息
      println(s"MasterActor:接收到worker的注册信息是,${workerid},${cpu},${mem}")
      //3. 保存WorkerActor提交的注册信息
      //3.2 具体的保存动作
      regWorkerActorMap += workerid -> WorkerInfo(workerid, cpu, mem, new Date().getTime)
      // 4. 为了查看是否成功
      println(s"MasterActor:存储的Worker有:${regWorkerActorMap}")
      //5. 获取WorkerActor的引用,给他一个回执信息,提示注册成功
      val workerActor = context.actorSelection("akka.tcp://actorSystrm@127.0.0.1:7001/user/workerActor")
      workerActor ! RegisterSuccessMessage

    }
    //6. 打印接收到的心跳信息
    case WorkerHeartBeatMessage(workerid, cpu, mem) => {
      //6.1. 接收workerActor提交的注册信息
      println(s"MasterActor:接收到${workerid}的心跳信息")
      //6.2 更新该worker的最后一次心跳信息
      regWorkerActorMap += workerid -> WorkerInfo(workerid, cpu, mem, new Date().getTime)
    }
  }

}

ConfigUtil

package spark.master

import com.typesafe.config.{Config, ConfigFactory}

//针对Master的工具类
object ConfigUtil {
  //1 .获取当前项目的配置信息对象
    private val config: Config = ConfigFactory.load()
  //2.获取检查worker心跳的时间周期
  val `master.check.heartbeat.interval` = config.getInt("master.check.heartbeat.interval")
  //3.获取worker心跳的超时时间
  val `master.check.heartbeat.timeout` = config.getInt("master.check.heartbeat.timeout")
}

Master

package spark.master

import akka.actor.{ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import spark.actor.MasterActor

object Master {
  def main(args: Array[String]): Unit = {
    //创建ActorSystem
    val actorSystem = ActorSystem("actorSystem", ConfigFactory.load())
    //通过ActorSystem关联MasterActor
    val masterActor = actorSystem.actorOf(Props(MasterActor), "masterActor")
    //测试代码
    //    masterActor !"hello"
  }
}

spark-demo-worker代码参考

WorkerActor

package spark.actor

import java.util.UUID

import akka.actor.{Actor, ActorSelection}
import com.typesafe.config.ConfigUtil
import spark.common.{RegisterSuccessMessage, WorkerHeartBeatMessage, WorkerRegisterMessage}
import spark.worker.ConfigUtils

import scala.util.Random

//akka.tcp://actorSystrm@127.0.0.1:7001
object WorkerActor extends Actor {
  //1.定义一些成员变量,用来记录MasterActor的引用,和提交给Master的注册信息(workid,cpu,mem)
  private var masterActorRef: ActorSelection = _ //MasterActor的运用
  private var workerId: String = _ //当前WorkerActor的id
  private var cpu: Int = _ //当前WorkerActor的核数
  private var mem: Int = _ //当前WorkerActor的内存大小(单位是MB)
  private var CPU_LIST: List[Int] = List(1, 2, 3, 4, 6, 7) //cpu取值范围
  private var MEM_LIST: List[Int] = List(512, 1024, 2048, 4096) //内存的取值范围
  //2.在preStart()方法中,对上诉的成员变进行赋值,并将注册的信息封装成:RegisterMessage对象,发送给MasterWorker
  //preStart()方法中的内容是在actor的start()方法之前执行
  override def preStart(): Unit = {
    //    2.1 获取MasterActor的引用
    masterActorRef = context.actorSelection("akka.tcp://actorSystem@127.0.0.1:7000/user/masterActor")
    //    2.2 创建一个随机数对象
    var r = new Random()
    //    2.3 给WorkerId,cpu,men变量赋值
    workerId = UUID.randomUUID().toString
    cpu = CPU_LIST(r.nextInt(CPU_LIST.size))
    mem = MEM_LIST(r.nextInt(MEM_LIST.size))
    //    2.4 将上诉的成员变量封装成RegisterMeessage对象
    //    2.5 将上诉封装好的RegisterMeessage发送个MasterWorker
    val workerRegisterMessage = WorkerRegisterMessage(workerId, cpu, mem)
    masterActorRef ! workerRegisterMessage
  }

  override def receive = {
    case RegisterSuccessMessage => {
      //1. 打印接收到的注册成功消息
      println("WorkerActor:接收到注册成功消息!")
      //2. 导入时间单位隐式转换,和定时器的隐式参数
      import  scala.concurrent.duration._
      import context.dispatcher
      //3. 定时给Master发送心跳信息
      //3.1 开启定时任务
      context.system.scheduler.schedule(0 seconds,ConfigUtils.`worker.heartbeat.interval` seconds){
        //3.2 给MasterActor发送心跳信息,
        masterActorRef ! WorkerHeartBeatMessage(workerId,cpu,mem )
      }
    }
  }
}

ConfigUtils

package spark.worker

import com.typesafe.config.{Config, ConfigFactory}

object ConfigUtils {
  //获取worker的心跳周期
  //1. 获取配置信息对象
  private val config: Config = ConfigFactory.load()
  //2. 获取worker心跳的具体周期
   val `worker.heartbeat.interval` = config.getInt("worker.heartbeat.interval")

  def main(args: Array[String]): Unit = {
    println(`worker.heartbeat.interval`)
  }
}

Worker

package spark.worker

import akka.actor.{ActorSystem, Props}
import com.typesafe.config.ConfigFactory
import spark.actor.WorkerActor

object Worker {
  def main(args: Array[String]): Unit = {
    //1.获取actorSystem对象,用来管理我们自定义的Actor对象
    val actorSystrm = ActorSystem("actorSystrm", ConfigFactory.load())
    //2.通过actorSystem对象,关联自定义的actor对象
    val workerActor = actorSystrm.actorOf(Props(WorkerActor), "workerActor")

    //3.测试代码
    //    workerActor!"hello world"
  }
}

3.最终实现效果

启动master
启动worker

附录一(三个模块中都要添加)

注意:在master和worker模块中要额外添加common的依赖

       <dependency>
            <groupId>spark</groupId>
            <artifactId>spark-demo-common</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.11.8</scala.version>
        <scala.compat.version>2.11</scala.compat.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>com.typesafe.akka</groupId>
            <artifactId>akka-actor_2.11</artifactId>
            <version>2.3.14</version>
        </dependency>

        <dependency>
            <groupId>com.typesafe.akka</groupId>
            <artifactId>akka-remote_2.11</artifactId>
            <version>2.3.14</version>
        </dependency>

    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass></mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

附录二

master的application.conf配置文件

akka.actor.provider = "akka.remote.RemoteActorRefProvider"
akka.remote.netty.tcp.hostname = "127.0.0.1"
akka.remote.netty.tcp.port = "7000"

#配置检查Worker心跳的时间周期(单位: 秒)
master.check.heartbeat.interval = 6

#配置worker心跳超时的时间(秒)
master.check.heartbeat.timeout = 15

worker的application.conf配置文件

akka.actor.provider = "akka.remote.RemoteActorRefProvider"
akka.remote.netty.tcp.hostname = "127.0.0.1"
akka.remote.netty.tcp.port = "7001"
worker.heartbeat.interval = 5

不忘初心$$

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
简易的Spark底层通信框架实现

模拟Spark的Master和Worker通信一个Master管理Worker若干个Worker（Worker可以按需添加）注册发送心跳图解实现思路构建Master,Worker阶段构建Master ActorSystem Actor构建Worker ActorSystem ActorWorker注册阶段Worker进程向Mast...
复制链接

扫一扫

专栏目录