【Spark】Idea模拟客户端和服务端分布式发送计算任务

最新推荐文章于 2024-07-17 14:29:35 发布

不爱研究的研究僧

最新推荐文章于 2024-07-17 14:29:35 发布

阅读量258

点赞数

分类专栏： Spark 文章标签：分布式 spark

本文链接：https://blog.csdn.net/weixin_43955488/article/details/129836324

版权

Spark 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

目标：

模拟两个客户端，两个服务端，将任务Task分成两个任务，分别交给两个服务端处理

（1，2，3，4）=>（1，2）、（3，4）

实现：

Task类为总任务，实际计算的任务是SubTask

因为需要在网络中传输，需要对类序列化

注意：

1、var datas: List[Int] = _ 要用var，因为初始化是_，之后task的值赋给它

class Task extends Serializable {
  val datas = List(1, 2, 3, 4)
  val logic : Int => Int = _ * 2
}

class SubTask extends Serializable {
  var datas: List[Int] = _
  var logic : Int => Int = _

  def compute()= {
    datas.map(logic)
  }
}

1、Driver

定义两个client，用Socket连接客户端和服务端

package test

import java.io.{ObjectOutputStream, OutputStream}
import java.net.{ServerSocket, Socket}

object Driver {
  def main(args: Array[String]): Unit = {
    val client1 = new Socket("localhost", 9999);
    val client2 = new Socket("localhost", 8888);

    val task = new Task()

    val out1: OutputStream = client1.getOutputStream
    val objOut1 = new ObjectOutputStream(out1)

    val subTask1 = new SubTask()
    subTask1.logic = task.logic
    subTask1.datas = task.datas.take(2)

    objOut1.writeObject(subTask1)
    objOut1.flush()
    objOut1.close()
    client1.close()

    val out2: OutputStream = client2.getOutputStream
    val objOut2 = new ObjectOutputStream(out2)

    val subTask2 = new SubTask()
    subTask2.logic = task.logic
    subTask2.datas = task.datas.takeRight(2)

    objOut2.writeObject(subTask2)
    objOut2.flush()
    objOut2.close()
    client2.close()
    println("发送完毕")
  }
}

2、Executor1

package test

import java.io.{InputStream, ObjectInputStream}
import java.net.{ServerSocket, Socket}

object Executor {
  def main(args: Array[String]): Unit = {
    val server = new ServerSocket(9999)
    println("等待接收数据")
    val client: Socket = server.accept()
    val in: InputStream = client.getInputStream
    val objIn = new ObjectInputStream(in)
    val task: SubTask = objIn.readObject().asInstanceOf[SubTask]
    val ints: List[Int] = task.compute()
    println("9999端口数据：" + ints)
    objIn.close()
    client.close()
    server.close()
  }
}

3、Executor2

package test

import java.io.{InputStream, ObjectInputStream}
import java.net.{ServerSocket, Socket}

object Executor {
  def main(args: Array[String]): Unit = {
    val server = new ServerSocket(9999)
    println("等待接收数据")
    val client: Socket = server.accept()
    val in: InputStream = client.getInputStream
    val objIn = new ObjectInputStream(in)
    val task: SubTask = objIn.readObject().asInstanceOf[SubTask]
    val ints: List[Int] = task.compute()
    println("9999端口数据：" + ints)
    objIn.close()
    client.close()
    server.close()
  }
}