第90讲:基于Scala的Actor之上的分布式并发消息驱动框架Akka初体验
akka在业界使用非常广泛
spark背后就是由akka驱动的
要写消息驱动的编程模型都首推akka
下面将用30讲讲解akka
本讲主要讲两部分内容:
1.akka的重大意义
2.akka与scala的actor
Spark源码中使用akka使用鉴赏:
在spark中有200行左右代码封装了akka的使用
spark是分布式的计算框架,有master和slave主从节点通信时都是使用akka。
客户端提交程序时也是使用akka。所以如果要掌握spark必须要理解和掌握akka。
第91讲:Akka第一个案例动手实战架构设计
The quick brown fox tried to jump over the lazy dog and fell on the dog
A dog is a man's best friend
Map Task -> Reduce Task --|
|-> Aggregate Task
Map Task -> Reduce Task --|
Master Actor
|
--------------------|---------------------
| | |
Map Actor Reduce Actor Aggregate Actor
Master Actor给MapActor发一个字符串
Map Actor根据规则对单词计数,计数完成后把结果传递给MasterActor
MasterActor把MapData以消息发给reduceActor,
reduceActor会reduceByKey,把相同单词(key相同)计数相加。
计算完后再把数据传回给MasterActor。
如果有多条字符串就会有多组reduce结果。
MasterActor再把结果发给AggregateActor,进行最后统计
MasterActor要获得结果需要给AggregateActor发一个空消息,
AggregateActor收到消息就会把所有统计结果发给MasterActor
这就是mapReduce计算模型。
与hadoop的mapreduce不同的是这是基于actor的。
MapActor对map产生的结果进行本地化统计,
AggregateActor才相当于hadoop的reducer。
后面先通过java使用akka。
第92讲:Akka第一个案例动手实战开发环境的搭建
第93讲:Akka第一个案例动手实战开发消息实体类
MapData:
package akka.dt.app.java.messages;
import java.util.List;
/**
* sunrunzhi
* 2018/11/12 9:46
* 用来让MapActor处理数据后存储在MapData实体中
* 然后方便将结果交给ReduceActor
*/
public class MapData {
private List dataList;
public List getDataList(){return dataList;}
public MapData(List dataList){
this.dataList=dataList;
}
}
ReduceData:
package akka.dt.app.java.messages;
import java.util.HashMap;
/**
* sunrunzhi
* 2018/11/12 13:35
*/
public class ReduceData {
private HashMap reduceDataList;
public HashMap getReduceDataList(){
return reduceDataList;
}
public ReduceData(HashMap reduceDataList){
this.reduceDataList=reduceDataList;
}
}
Result:
package akka.dt.app.java.messages;
/**
* sunrunzhi
* 2018/11/12 9:58
*/
public class Result {
/*传入的字符串先交给MapActor进行切分,然后交给ReduceActor进行本地统计,
最后交给AggregateActor进行全局的统计,
想要获得这个结果,通过MasterActor发一个消息Result,Result本身为空,不需要有任何内容。
这个消息交给MasterActor,MasterActor收到消息时,如果消息是result类型的话转过来会告诉AggregateActor,
再转发给AggregateActor。*/
}
WordCount:
package akka.dt.app.java.messages;
/**
* sunrunzhi
* 2018/11/12 9:52
* WordCount-javaBean
*/
public class WordCount {
private String word;
private Integer count;
public WordCount(String inWord,Integer inCount){
word=inWord;
count=inCount;
}
public String getWord(){return word;}
public Integer getCount(){return count;}
}
HelloAkka:
package akka.dt.app.java.messages;
import akka.actor.ActorRef;
import akka.actor.ActorSystem;
import akka.actor.Props;
import akka.dt.app.java.actors.MasterActor;
/**
* sunrunzhi
* 2018/11/9 20:14
*/
public class HelloAkka {
public static void main(String[] args)throws Exception{
ActorSystem _system=ActorSystem.create("HelloAkka");
ActorRef master=_system.actorOf(new Props(MasterActor.class),"master");
master.tell("Hi,My name is Rocky. I'm so so so so happy to me here.");
master.tell("Today,I'm going to read a news article for so you.");
master.tell("I hope I hope you'll like it.");
Thread.sleep(500);
master.tell(new Result());
Thread.sleep(500);
_system.shutdown();
}
}
第94讲:Akka第一个案例动手实战MapActor、ReduceActor、AggregateActor代码详解
AggregateActor:
package akka.dt.app.java.actors;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.ReduceData;
import akka.dt.app.java.messages.Result;
import java.util.HashMap;
import java.util.Map;
/**
* sunrunzhi
* 2018/11/9 20:18
*/
public class AggregateActor extends UntypedActor{
private Map finalReducedMap=new HashMap();
@Override
public void onReceive(Object message)throws Exception{
//AggregateActor收到消息有两种,一种是ReduceData类型,一种是Result类型
if(message instanceof ReduceData){
ReduceData reduceData=(ReduceData)message;
aggregateInMemoryReduce(reduceData.getReduceDataList());
}else if(message instanceof Result){
System.out.println(finalReducedMap.toString());
}else{
unhandled(message);
}
}
private void aggregateInMemoryReduce(Map reduceList){
Integer count=null;
for (String key:reduceList.keySet()){
if(finalReducedMap.containsKey(key)){
count=reduceList.get(key)+finalReducedMap.get(key);
finalReducedMap.put(key,count);
}else{
finalReducedMap.put(key,reduceList.get(key));
}
}
}
}
MapActor:
package akka.dt.app.java.actors;
import akka.actor.ActorRef;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.MapData;
import akka.dt.app.java.messages.WordCount;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.StringTokenizer;
/**
* sunrunzhi
* 2018/11/9 20:17
*/
public class MapActor extends UntypedActor {
String[] STOP_WORDS = {"a", "is"};
private ActorRef reduceActor = null;
private List STOP_WORDS_LIST = Arrays.asList(STOP_WORDS);
public MapActor(ActorRef inReduceActor) {
reduceActor = inReduceActor;
}
@Override
public void onReceive(Object message) throws Exception {
if (message instanceof String) {
String work = (String) message;
//map the words in the sentence
MapData data = evaluateExpression(work);//产生MapData实体
// send the result to ReduceActor
reduceActor.tell(data);
} else {
unhandled(message);
}
}
private MapData evaluateExpression(String line) {
List dataList = new ArrayList();
StringTokenizer parser = new StringTokenizer(line);
while (parser.hasMoreTokens()){
String word =parser.nextToken().toLowerCase();
if (!STOP_WORDS_LIST.contains(word)) {
dataList.add(new WordCount(word, Integer.valueOf(1)));
}
}
return new MapData(dataList);
}
}
ReduceActor:
package akka.dt.app.java.actors;
import akka.actor.ActorRef;
import akka.actor.UntypedActor;
import akka.dt.app.java.messages.MapData;
import akka.dt.app.java.messages.ReduceData;
import akka.dt.app.java.messages.WordCount;
import com.sun.javafx.collections.MappingChange;
import java.util.HashMap;
import java.util.List;
/**
* sunrunzhi
* 2018/11/9 20:17
*/
public class ReduceActor extends UntypedActor{
private ActorRef aggregateActor =null;
public ReduceActor(ActorRef inAggregateActor) {
aggregateActor=inAggregateActor;
}
@Override
public void onReceive(Object message) throws Exception{
//AggregateActor收到的消息有两种,一种是ReduceData类型,一种是Result类型
if(message instanceof MapData){
MapData mapData =(MapData)message;
//reduce the incoming data
ReduceData reduceData=reduce(mapData.getDataList());
//forward the result to aggregate actor
aggregateActor.tell(reduceData);
}else {
unhandled(message);
}
}
private ReduceData reduce(List dataList){
HashMap reducedMap=new HashMap();
for(WordCount wordCount:dataList){
if(reducedMap.containsKey(wordCount.getWord())){
Integer value=(Integer)reducedMap.get(wordCount.getWord());
value++;
reducedMap.put(wordCount.getWord(),value);
}else{
reducedMap.put(wordCount.getWord(),Integer.valueOf(1));
}
}
return new ReduceData(reducedMap);
}
}
MasterActor:
package akka.dt.app.java.actors;
import akka.actor.*;
import akka.dt.app.java.messages.Result;
import java.util.Arrays;
import java.util.List;
/**
* sunrunzhi
* 2018/11/9 20:18
*/
public class MasterActor extends UntypedActor {
private ActorRef aggregaterActor = getContext().actorOf(
new Props(AggregateActor.class), "aggregate");
private ActorRef reduceActor=getContext().actorOf(
new Props(new UntypedActorFactory() {
public UntypedActor create() {
return new ReduceActor(aggregaterActor);
}
}),"reduce");
private ActorRef mapActor=getContext().actorOf(
new Props(new UntypedActorFactory(){
public UntypedActor create(){
return new MapActor(reduceActor);
}
}),"map");
@Override
public void onReceive(Object message) throws Exception{
if (message instanceof String) {
mapActor.tell(message);
} else if (message instanceof Result) {
aggregaterActor.tell(message);
} else{
unhandled(message);
}
}
}
第95讲:Akka第一个案例动手实战MasterActor代码详解
第96讲:Akka第一个案例动手实战main方法实现中ActorSystem等代码详解
第97讲:使用SBT开发Akka第一个案例环境搭建详解
SBT import失败:
一步一个坑。凑人~~~
之前调整的项目有报莫名其妙的BUG。
第一个BUG。
解决方案:标识红色处打上对号。
总有些奇奇怪怪的BUG,莫名其妙的又没有了........
第98讲:使用SBT开发时动手解决rt.jar中CharSequence is broken等问题
第99讲:手动Artifacts打包并运行SBT开发Akka第一个案例
第100讲:使用SBT开发Akka第一个案例源码解析消息、main入口、MasterActor
第101讲:使用SBT开发Akka第一个案例源码解析MapActor、ReduceActor、AggregateActor