storm学习

tuple 是bolt之间传递的变量

spout 的构成

ispout spout 功能

icomponent bolt 和 spout 公用配置和输出

spout中变量：

private SpoutOutputCollector outputCollector; 传送tuple

    
    

     
     

      
      

       
       
public void declareOutputFields(OutputFieldsDeclarer declarer) 
        
        

         
         

          
          

           
           
{declarer.declare(new Fields("commit"));} 定义输出的field
           
           

            
            

             
             

              
              

               
               
@Override
public void nextTuple() {

               
               
  for (String commit : commits) {
    outputCollector.emit(new Values(commit));

               
               
}} 
              
              
             
             
            
            
读取和处理tuple
           
           

            
            

             
             

              
              

               
               
public void open(Map configMap,

               
               
                 TopologyContext context,

               
               
                 SpoutOutputCollector outputCollector) {
  this.outputCollector = outputCollector;

               
               
  try {
    commits = IOUtils.readLines(

               
               
       ClassLoader.getSystemResourceAsStream("changelog.txt"),

              
              
             
             
             
             

              
              

               
               
Implementing a GitHub commit count dashboard in Storm
               
               
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {

               
               
  declarer.declare(new Fields("commit"));｝
               
               


               
               

                
                

                 
                 

                  
                  

                   
                   
public void open(Map configMap,

                   
                   
                 TopologyContext context,

                   
                   
                 SpoutOutputCollector outputCollector) {
  this.outputCollector = outputCollector;

                  
                  
                 
                 
                
                

              
              
             
             
             
             

              
              

               
               
       Charset.defaultCharset().name()
    );

               
               
  } catch (IOException e) {
    throw new RuntimeException(e);

               
               
}} 
              
              
             
             
            
            
读取之前的初始化
           
           


           
           


           
           


           
           
bolt构成中， IbasicBolt 是IRichBolt功能的子集
           
           


           
           

            
            

             
             

              
              

               
               
 @Override
  public void declareOutputFields(OutputFieldsDeclarer declarer) {

               
               
    declarer.declare(new Fields("email"));  声明输出

              
              
             
             
             
             

              
              

               
               
@Override
public void execute(Tuple tuple,

              
              
             
             
             
             

              
              

               
               
                  BasicOutputCollector outputCollector) {
String commit = tuple.getStringByField("commit");

               
               
String[] parts = commit.split(" ");
outputCollector.emit(new Values(parts[1]));

              
              
             
             
             
             

              
              

               
               
} 
              
              
             
             
            
            
ticktuple 配置一段时间后发射一次。
           
           
配置：
           
           
            
            

             
             

              
              

               
               
@Override
public Map<String, Object> getComponentConfiguration() {

               
               
  Config conf = new Config();
  conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 60);
  return conf;

               
               
} 
              
              
             
             
            
            

           
           
例如：
          
          
         
         
        
        
      
      
     
     
    
    

@Override
public void execute(Tuple tuple,

                    BasicOutputCollector outputCollector) {
  if (isTickTuple(tuple)) {

    emitHeatmap(outputCollector);
  } else {

If we got atick tuple,interpret thatas a signal tosee whetherthere are anyheat mapsthat can beemitted.

For all timeintervalswe’recurrentlykeepingtrack of:if they ’veelapsed,removethemfrom thein-memorydatastructureand emitthem.

    Long time = tuple.getLongByField("time");
    LatLng geocode = (LatLng) tuple.getValueByField("geocode");

    Long timeInterval = selectTimeInterval(time);
    List<LatLng> checkins = getCheckinsForInterval(timeInterval);
    checkins.add(geocode);

}

private void emitHeatmap(BasicOutputCollector outputCollector) {
  Long now = System.currentTimeMillis();
  Long emitUpToTimeInterval = selectTimeInterval(now);
  Set<Long> timeIntervalsAvailable = heatmaps.keySet();

}

for (Long timeInterval : timeIntervalsAvailable) {
  if (timeInterval <= emitUpToTimeInterval) {

List<LatLng> hotzones = heatmaps.remove(timeInterval);
outputCollector.emit(new Values(timeInterval, hotzones));

}}

private Long selectTimeInterval(Long time) {
  return time / (15 * 1000);

} 

An executor is a thread of execution within aJVM. 

A task isan instance of a spout or bolt running within that thread of execution. 

Each JVM can berunning one or moreexecutors (threads)of execution. 

A worker node has a JVM thatexecutes the logic in the spoutsand bolts for a topology. 

 builder.setBolt("geocode-lookup", new GeocodeLookup(), 8).setNumTasks(64)

Now we have 64 tasks (instances) of GeocodeLookup running across eight executors(threads).