第一题、百元喝酒
作业要求:每瓶啤酒2元,3个空酒瓶或者5个瓶盖可换1瓶啤酒。100元最多可喝多少瓶啤酒?(不允许借啤酒)思路:利用递归算法,一次性买完,然后递归算出瓶盖和空瓶能换的啤酒数
package com.wangbr.homework
/**
* @author: wangbr
* @date: 2021-01-29 21:40
* 作业要求:每瓶啤酒2元,3个空酒瓶或者5个瓶盖可换1瓶啤酒。100元最多可喝多少瓶啤酒?
* (不允许借啤酒)思路:利用递归算法,一次性买完,然后递归算出瓶盖和空瓶能换的啤酒数
*/
object Demo1 {
def main(args: Array[String]): Unit = {
// 第一次买共几瓶
var one:Int = 100/2;
println(s"总喝瓶数:${one+process(0,one,one)}")
}
/*
* 总数,上次剩余瓶子,上次剩余瓶盖
*/
def process(sum:Int,bottle:Int,cap:Int):Int = {
if(bottle<3 && cap<5) return sum
var a1 = bottle/3
var a2 = bottle%3
var b1 = cap/5
var b2 = cap%5
println(f"总数=${a1+b1+sum} 上次剩余瓶子=${a1+b1+a2}%02d 上次剩余瓶盖=${a1+b1+b2}%02d")
return process(a1+b1+sum,a1+b1+a2,a1+b1+b2)
}
}
第二题、人机猜拳
作业需求
- 选取对战角色
- 开始对战,用户出拳,与对手进行比较,提示胜负信息
- 猜拳结束算分,平局都加一分,获胜加二分,失败不加分
- 循环对战,当输入“n”时,终止对战,并显示对战结果
- 游戏结束后显示得分
package com.wangbr.homework
import scala.util.control.Breaks._
/**
* @author: wangbr
* @date: 2021-01-29 23:18
* 人机猜拳
* 1.1 作业需求
* 1. 选取对战角色
* 2. 开始对战,用户出拳,与对手进行比较,提示胜负信息
* 3. 猜拳结束算分,平局都加一分,获胜加二分,失败不加分
* 4. 循环对战,当输入“n”时,终止对战,并显示对战结果
* 5. 游戏结束后显示得分
*/
object Demo2 {
def main(args: Array[String]): Unit = {
println("欢迎参加游戏")
//1. 选取对战角色
println("开始选取角色 孙悟空输入[1] 唐僧输入[2]")
var role=scala.io.StdIn.readInt()
while(role !=1 && role !=2){
println("没有这个选项,请重新输入:")
role=scala.io.StdIn.readInt()
}
println(s"您选择的是[${if(role==1)"孙悟空" else "唐僧"}]")
println()
//2. 开始对战,用户出拳,与对手进行比较,提示胜负信息
println("游戏开始")
println("游戏介绍:1.石头输入[1] 2.剪刀输入[2] 3.布输入[3] 4.退出输入[n]")
var result = ""
var score:Int = 0
breakable{
while(true){
println("游请出拳")
result = scala.io.StdIn.readLine()
if(result != "1" && result != "2" && result != "3"){
//4 . 循环对战,当输入“n”时,终止对战,并显示对战结果
if(result == "n") break
println("您输入错误!请重新输入")
}else{
//3. 猜拳结束算分,平局都加一分,获胜加二分,失败不加分
var computer = scala.util.Random.nextInt(3)+1+""
var sco = if(computer == result) 1 else {if((result.toInt-computer.toInt)==1||(result.toInt-computer.toInt)== -2 ) 0 else 2}
println(s"你出的是[${if(result == "1")"石头"else if(result == "2")"剪刀" else "布"}]," +
s"机器出的是[${if(computer == "1")"石头"else if(computer == "2")"剪刀" else "布"}]," +
s"结果您是[${if(sco==2)"胜利" else if(sco == 1) "平局" else "失败"}]")
score += sco
}
}
}
//5. 游戏结束后显示得分
println(s"您的分数是[$score]")
println("bye bye")
}
}
第三题、用户位置时长统计
现有如下数据需要处理:
字段:用户ID,位置ID,开始时间,停留时长(分钟)
4行样例数据:
UserA,LocationA,8,60
UserA,LocationA,9,60
UserB,LocationB,10,60
UserB,LocationB,11,80
样例数据中的数据含义是:用户UserA,在LocationA位置,从8点开始,停留了60钟
处理要求:
1、对同一个用户,在同一个位置,连续的多条记录进行合并
2、合并原则:开始时间取最早时间,停留时长累计求和
package com.wangbr.homework
/**
* @author: wangbr
* @date: 2021-02-03 18:31
*/
case class UserInfo(userName:String,location:String,startTime:Int,duration:Int)
object LocationDemo {
def main(args: Array[String]): Unit = {
val userInfoList:List[UserInfo] = List(
UserInfo("UserA", "LocationA", 8, 60),
UserInfo("UserA", "LocationA", 9, 60),
UserInfo("UserA", "LocationB", 10, 60),
UserInfo("UserA", "LocationB", 11, 80)
)
val userMap = userInfoList.groupBy(t => t.userName+","+t.location)
val orderByUserMap = userMap.mapValues(t =>t.sortBy(x=>x.startTime))
var firstTime = 0
val totalMap = orderByUserMap.mapValues(t =>{
firstTime = t.head.startTime
var sum = t.map(x=> x.duration).sum
sum
})
totalMap.foreach{
case (datas,sumTime)=>println(s"$datas,$firstTime,$sumTime")
}
}
}
第四题、Actor间通讯
作业要求:
1、编写 2 个 Actor ,分别是 AActor 和 BActor
2、AActor 和 BActor 之间可以相互发送消息
package com.wangbr.actor
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
/**
* @author: wangbr
* @date: 2021-02-05 19:19
* 作业要求:
* 1、编写 2 个 Actor ,分别是 AActor 和 BActor
* 2、AActor 和 BActor 之间可以相互发送消息
*/
class AActor extends Actor{
override def receive: Receive = {
case 0 => println("我打")
case 10 => {
context.stop(self)
context.system.terminate()
}
case i: Int => println(s"AActor(黄飞鸿):挺猛,看我佛山无影脚...第$i 脚")
}
}
class BActor extends Actor{
override def receive: Receive = {
case 0 => println("start OK!")
case 10 => context.stop(self)
case i: Int => println(s"AActor(乔峰):挺猛,看降龙十八掌...第$i 掌")
}
}
object ABActor {
private val myFactory = ActorSystem("myFactory")
private val AActorRef: ActorRef = myFactory.actorOf(Props[AActor], "AActor")
private val BActorRef: ActorRef = myFactory.actorOf(Props[BActor], "BActor")
var flag = 0
def main(args: Array[String]): Unit = {
println("AActor 出招了...")
while (flag<=10){
BActorRef ! flag
AActorRef ! flag
flag += 1
}
Thread.sleep(100)
println("比赛结束")
}
}
第五题、模拟Spark中Master与Worker进程通讯
项目的意义
深入理解 Spark 的 Master 和 Worker 的通讯机制
为了方便同学们看 Spark 的底层源码,命名的方式和源码保持一致.(如: 通讯消息类命名就是 一样的)
加深对主从服务心跳检测机制(HeartBeat)的理解,方便以后 spark 源码二次开发。
项目的需求
worker注册到Master,Master完成注册,并回复worker注册成功
worker定时发送心跳,并在Master接收到
Master接收到worker心跳后,要更新该worker的最近一次发送心跳的时间
给 Master 启动定时任务,定时检测注册的 worker 有哪些没有更新心跳,并将其从 hashmap 中删除
master worker 进行分布式部署(Linux 系统)
MessageProtocol.scala 协议集合中心(样例类)
package com.bestksl.akka.sparkmasterworker.common
// worker注册信息 //MessageProtocol.scala
case class RegisterWorkerInfo(id: String, cpu: Int, ram: Int)
// 这个是WorkerInfo, 这个信息将来是保存到master的 hm(该hashmap是用于管理worker)
// 将来这个WorkerInfo会扩展(比如增加worker上一次的心跳时间)
class WorkerInfo(val id: String, val cpu: Int, val ram: Int) {
var lastHeartBeat: Long = System.currentTimeMillis()
}
// 当worker注册成功,服务器返回一个RegisteredWorkerInfo 对象
case object RegisteredWorkerInfo
//worker每隔一定时间由定时器发给自己的一个消息
case object SendHeartBeat
//worker每隔一定时间由定时器触发,而向master发现的协议消息
case class HeartBeat(id: String)
//master给自己发送一个触发检查超时worker的信息
case object StartTimeOutWorker
// master给自己发消息,检测worker,对于心跳超时的.
case object RemoveTimeOutWorker
SparkMaster.scala
package com.bestksl.akka.sparkmasterworker.master
import akka.actor.{Actor, ActorSystem, Props}
import com.bestksl.akka.sparkmasterworker.common.{HeartBeat, RegisterWorkerInfo, RegisteredWorkerInfo, RemoveTimeOutWorker, StartTimeOutWorker, WorkerInfo}
import com.typesafe.config.ConfigFactory
import scala.collection.mutable
import scala.concurrent.duration._
class SparkMaster extends Actor {
//定义个hm,管理workers
val workers: mutable.Map[String, WorkerInfo] = mutable.Map[String, WorkerInfo]()
override def receive: Receive = {
case "start" =>
println("master服务器启动了...")
//这里开始。。
self ! StartTimeOutWorker
case RegisterWorkerInfo(id, cpu, ram) =>
//接收到worker注册信息
if (!workers.contains(id)) {
//创建WorkerInfo 对象
val workerInfo = new WorkerInfo(id, cpu, ram)
//加入到workers
workers += ((id, workerInfo))
println("服务器的workers=" + workers)
//回复一个消息,说注册成功
sender() ! RegisteredWorkerInfo
}
case HeartBeat(id) =>
//更新对应的worker的心跳时间
//1.从workers取出WorkerInfo
val workerInfo = workers(id)
workerInfo.lastHeartBeat = System.currentTimeMillis()
println("master更新了 " + id + " 心跳时间...")
case StartTimeOutWorker =>
println("开始了定时检测worker心跳的任务")
import context.dispatcher
//说明
//1. 0 millis 不延时,立即执行定时器
//2. 9000 millis 表示每隔3秒执行一次
//3. self:表示发给自己
//4. RemoveTimeOutWorker 发送的内容
context.system.scheduler.schedule(0 millis, 9000 millis, self, RemoveTimeOutWorker)
//对RemoveTimeOutWorker消息处理
//这里需求检测哪些worker心跳超时(now - lastHeartBeat > 6000),并从map中删除
case RemoveTimeOutWorker =>
//首先将所有的 workers 的 所有WorkerInfo
val workerInfos = workers.values
val nowTime = System.currentTimeMillis()
//先把超时的所有workerInfo,删除即可
workerInfos.filter(workerInfo => (nowTime - workerInfo.lastHeartBeat) > 6000)
.foreach(workerInfo => workers.remove(workerInfo.id))
println("当前有 " + workers.size + " 个worker存活的")
}
}
object SparkMaster {
def main(args: Array[String]): Unit = {
//这里我们分析出有3个host,port,sparkMasterActor
if (args.length != 3) {
println("请输入参数 host port sparkMasterActor名字")
sys.exit()
}
val host = args(0)
val port = args(1)
val name = args(2)
//先创建ActorSystem
val config = ConfigFactory.parseString(
s"""
|akka.actor.provider="akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname=$host
|akka.remote.netty.tcp.port=$port
""".stripMargin)
val sparkMasterSystem = ActorSystem("SparkMaster", config)
//创建SparkMaster -actor
val sparkMasterRef = sparkMasterSystem.actorOf(Props[SparkMaster], s"$name")
//启动SparkMaster
sparkMasterRef ! "start"
}
}
SparkWorker.scala
package com.bestksl.akka.sparkmasterworker.worker
import akka.actor.{Actor, ActorSelection, ActorSystem, Props}
import com.bestksl.akka.sparkmasterworker.common.{HeartBeat, RegisterWorkerInfo, RegisteredWorkerInfo, SendHeartBeat}
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration._
class SparkWorker(masterHost: String, masterPort: Int, masterName: String) extends Actor {
//masterProxy是Master的代理/引用ref
var masterPorxy: ActorSelection = _
val id: String = java.util.UUID.randomUUID().toString
override def preStart(): Unit = {
println("preStart()调用")
//初始化masterPorxy
masterPorxy = context.actorSelection(s"akka.tcp://SparkMaster@$masterHost:$masterPort/user/$masterName")
println("masterProxy=" + masterPorxy)
}
override def receive: Receive = {
case "start" =>
println("worker启动了")
//发出一个注册消息
masterPorxy ! RegisterWorkerInfo(id, 16, 16 * 1024)
case RegisteredWorkerInfo =>
println("workerid= " + id + " 注册成功~")
//当注册成功后,就定义一个定时器,每隔一定时间,发送SendHeartBeat给自己
import context.dispatcher
//说明
//1. 0 millis 不延时,立即执行定时器
//2. 3000 millis 表示每隔3秒执行一次
//3. self:表示发给自己
//4. SendHeartBeat 发送的内容
context.system.scheduler.schedule(0 millis, 3000 millis, self, SendHeartBeat)
case SendHeartBeat =>
println("worker = " + id + "给master发送心跳")
masterPorxy ! HeartBeat(id)
}
}
object SparkWorker {
def main(args: Array[String]): Unit = {
if (args.length != 6) {
println("请输入参数 workerHost workerPort workerName masterHost masterPort masterName")
sys.exit()
}
val workerHost = args(0)
val workerPort = args(1)
val workerName = args(2)
val masterHost = args(3)
val masterPort = args(4)
val masterName = args(5)
val config = ConfigFactory.parseString(
s"""
|akka.actor.provider="akka.remote.RemoteActorRefProvider"
|akka.remote.netty.tcp.hostname=$workerHost
|akka.remote.netty.tcp.port=$workerPort
""".stripMargin)
//创建ActorSystem
val sparkWorkerSystem = ActorSystem("SparkWorker", config)
//创建SparkWorker 的引用/代理
val sparkWorkerRef = sparkWorkerSystem.actorOf(Props(new SparkWorker(masterHost, masterPort.toInt, masterName)), s"$workerName")
//启动actor
sparkWorkerRef ! "start"
}
}