JStorm源码分析(二)Storm任务提交流程

最新推荐文章于 2020-07-18 15:53:26 发布

置顶 Oeljeklaus

最新推荐文章于 2020-07-18 15:53:26 发布

阅读量948

点赞数

分类专栏：大数据大数据框架知识文章标签： Storm JStorm 源码分析大数据

本文链接：https://blog.csdn.net/Oeljeklaus/article/details/80992310

版权

大数据同时被 2 个专栏收录

39 篇文章 0 订阅

订阅专栏

大数据框架知识

35 篇文章 6 订阅

订阅专栏

JStorm源码分析(二):Storm任务提交流程

JStorm源码分析文件

对于整个JStorm源码分析系列，我将JStorm源码分析的文件放在我的GitHub上JStorm源码剖析,欢迎大家fork和star

Storm任务提交流程: 1.Client端提交Topology到nimbus

调用命令:

storm jar WordCount.jar com.stone.WordCountMain wordcount

实际上是调用:

java -client WordCount.jar com.stone.WordCountMain wordcount

2.通过TopologyBuilder将Spout、Bolt按照一定的逻辑顺序构建Topology程序

TopologyBuilderbuilder=newTopologyBuilder(); 
//RandomSentenceSpout类，在已知的英文句子中，随机发送一条句子出去。 
builder.setSpout("spout1",newRandomSentenceSpout(),3); 
//SplitSentenceBolt类，主要是将一行一行的文本内容切割成单词
builder.setBolt("split1",newSplitSentenceBolt(),9).shuffleGrouping("spout1"); 
//WordCountBolt类，对单词出现的次数进行统计 
builder.setBolt("count2",newWordCountBolt(),3).shuffleGrouping("split1");

3.调用TopologyBuilder的c reateT opology()方法，获取Storm T opology实例对象。源码如下:

//STONE_NOTE调用TopologyBuilder的此方法，创建StormTopology的实例对象 publicStormTopologycreateTopology(){
Map<String,Bolt>boltSpecs=newHashMap<String,Bolt>(); Map<String,SpoutSpec>spoutSpecs=newHashMap<String,SpoutSpec>(); for(StringboltId:_bolts.keySet()){
IRichBoltbolt=_bolts.get(boltId); ComponentCommoncommon=getComponentCommon(boltId,bolt); boltSpecs.put(boltId,newBolt(ComponentObject.serialized_java(Utils.javaS
erialize(bolt)),common)); }
for(StringspoutId:_spouts.keySet()){ IRichSpoutspout=_spouts.get(spoutId); ComponentCommoncommon=getComponentCommon(spoutId,spout); spoutSpecs.put(spoutId,newSpoutSpec(ComponentObject.serialized_java(Util
s.javaSerialize(spout)),common));
}
//STONE_NOTE将Spout和Bolt的相关信息都封装在对应的map中，然后获取StormTopolog y实例对象
returnnewStormTopology(spoutSpecs,boltSpecs,newHashMap<String,StateSpout Spec>());
}

4.开始提交任务，具体过程如下:

(1)调用StormSubmitter.submitTopologyWithProgressBar(“WordCount”,conf,builder .createTopology())提交任务。 submitTopologyWithProgressBar的源码如下:

//STONE_NOTE调用此方法提交任务
publicstaticvoidsubmitTopologyWithProgressBar(Stringname,MapstormConf,Storm Topologytopology,SubmitOptionsopts)throwsAlreadyAliveException,
InvalidTopologyException{
/** *removeprogressbarinjstorm */
//STONE_NOTE调用submitTopology方法，传入Topology的名称、配置参数、实例对象
submitTopology(name,stormConf,topology,opts); }

(2)在submitTopologyWithProgressBar方法中，调用了StormSubmitter的submitTopology(name,stormConf,topology,opts)方法。

submitTopology方法的源码如下:

public static void   submitTopology(Stringname,MapstormConf,StormTopologytopology,SubmitOptionsopts)throwsAlreadyAliveException,InvalidTopologyException{ 
   	//STONE_NOTE检验Stormconf，必须是json-serializableJson的序列化对象 
    if(!Utils.isValidConf(stormConf)){
		thrownewIllegalArgumentException("Stormconfisnotvalid.Mustbejson- serializable");
	} 
    //STONE_NOTE利用stormConf创建一个hashmap的实例，并传给stormConf 
    stormConf=newHashMap(stormConf); 
    //STONE_NOTE获得命令行参数，并放入stormConf中 
    stormConf.putAll(Utils.readCommandLineOpts()); 
    Mapconf=Utils.readStormConfig();
	conf.putAll(stormConf);
	putUserInfo(conf,stormConf);
	try{
		StringserConf=Utils.to_json(stormConf); 
        if(localNimbus!=null){
		LOG.info("Submittingtopology"+name+"inlocalmode"); 
            //STONE_NOTE如果localNimbus不为空的话，调用本地模式运行 
        localNimbus.submitTopology(name,null,serConf,topology);
		}else{ 
            //STONE_NOTE通过Topology的配置信息，获取到NimbusClient 
            NimbusClient client=NimbusClient.getConfiguredClient(conf); try{
            //STONE_NOTE检测Topology的名称在集群上是否存在 
            if(topologyNameExists(client,conf,name)){
			//STONE_NOTE如果已经存在，抛出异常;提示Topology的名称已存在
			thrownewRuntimeException("Topologywithname`"+name+"` alreadyexistsoncluster");
	}
		//STONE_NOTE调用submitJar方法，提交jar文件 
         submitJar(client,conf); 
         LOG.info("Submittingtopology"+name+"indistributedmodewithconf"+serConf); 
         //STONE_NOTE否则的话，调用分布式集群模式
        if(opts!=null){
		//STONE_NOTE新的提交方式，携带opts参数 提交Topology任务 
        client.getClient().submitTopologyWithOpts(name,path,serConf,topology,opts); 
        }
        else{
			//thisisforbackwardscompatibility 
            //STONE_NOTE这个是为了兼容之前的版本 默认将opts设置为ACTIVE 
            client.getClient().submitTopology(name,path,serConf,topology);
		 }
     finally{
			client.close(); 
     		}
}
	LOG.info("Finishedsubmittingtopology:"+name); }
        catch(InvalidTopologyExceptione){
		LOG.warn("Topologysubmissionexception",e);
		throw e; 
        }catch(AlreadyAliveExceptione){
		LOG.warn("Topologyalreadyaliveexception",e);
		throw e; }
        catch(TopologyAssignExceptione){
		LOG.warn("Failedtoassign"+e.get_msg(),e);
		throw new RuntimeException(e); }
        catch(TExceptione){
		LOG.warn("Failedtoassign",e);
		throw new RuntimeException(e); }
}