关于storm中topology worker excutor task 在集群上是怎么分配的一次测试记录

最新推荐文章于 2022-06-29 10:54:36 发布

qq_34626712

最新推荐文章于 2022-06-29 10:54:36 发布

阅读量229

点赞数 1

分类专栏：大数据storm

本文链接：https://blog.csdn.net/qq_34626712/article/details/90728786

版权

大数据storm 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

关于storm什么官网上那一套我就不记录了主要通过测试验证自己的猜想（集群上测试的应该不会有问题）先上代码后解释

1.首先定义个spout

import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;

import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;
import java.util.Map;
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * @author zhangkun
 * @create 2019-05-30 16:38
 */
public class LogSpout extends BaseRichSpout {

    private SpoutOutputCollector collector;
    private Random random;
    /**
     * 自己写消息id 这个是属于普通变量不解释  没执行一次自增一次
     */
    private int id = 0;

    /**
     * 因为是静态的所有在同一个jvm实例中只有一份
     * 如果有两个excutor（线程）执行的时候有可能会出现并发的情况
     * 所以使用juc中atomic 他通过cas实现
     */
    private static AtomicInteger staticId = new AtomicInteger(0);

    /**
     *记录taskid
     */
    private int thisTaskId;

    public LogSpout(){
        System.out.println("==构造函数执行实例化LogSpout==========");
    }

    @Override
    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        this.thisTaskId = context.getThisTaskId();
        this.random = new Random();
        this.collector = collector;
    }

    @Override
    public void nextTuple() {
        //这个地方是为了控制nextTupl方法的执行速率
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        int messageId = this.id++;
        int staticLocalId = staticId.incrementAndGet();
        System.out.println("messageid:" + messageId
                + "======processId:" + getProcessID()
                + "======thread:" + Thread.currentThread().getName()
                + "======taskId:" + thisTaskId +
                "========staticId:" +staticLocalId
                + "===========hashCode:" + hashCode());

        //定义数组，作为数据来源
        String[] sentences = new String[]{ "my storm word count"
                , "hello my storm", "hello storm hello world"
                ,"i love you","you love her","how do you do"};

        String line = sentences[random.nextInt(sentences.length)];
        this.collector.emit(new Values(line), messageId);
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("sentence"));
    }

    /**
     * 获取当前进程id
     * @return
     */
    public static final int getProcessID() {
        RuntimeMXBean runtimeMXBean = ManagementFactory.getRuntimeMXBean();
        return Integer.valueOf(runtimeMXBean.getName().split("@")[0])
                .intValue();
    }
}

2.再来一个Blot(没啥用可以直接忽略不用看这里为了代码完整还是写上了)

import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.BasicOutputCollector;
import org.apache.storm.topology.FailedException;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseBasicBolt;
import org.apache.storm.tuple.Tuple;

import java.util.Map;
import java.util.Random;

/**
 * @author zhangkun
 * @create 2019-05-30 16:50
 */
public class SplitBolt extends BaseBasicBolt {

    private Random random;

    @Override
    public void prepare(Map stormConf, TopologyContext context) {
        super.prepare(stormConf, context);
        random = new Random();
    }

    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        String sentence = input.getStringByField("sentence");
        try {
           int x = 1 / random.nextInt(5);
        }catch (Exception e){
            throw new FailedException(e);
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
    }
}

4 最后一个Topolog组合

/**
 * @author zhangkun
 * @create 2019-05-30 17:01
 */
public class AckTopology {
    public static void main(String[] args) {
        //组装各个组件
        TopologyBuilder topologyBuilder = new TopologyBuilder();
        topologyBuilder.setSpout("logSpout", new LogSpout(),4).setNumTasks(8);
        topologyBuilder.setBolt("splitBolt", new SplitBolt()).localOrShuffleGrouping("logSpout");

        Config config = new Config();
        if(args != null && args.length > 0){
            //传递了运行参数： 集群方式运行
            config.setNumWorkers(2);
            try {
                StormSubmitter.submitTopology(args[0], config, topologyBuilder.createTopology());
            } catch (Exception e) {
                e.printStackTrace();
            }
        }else{
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("ackTopology", config, topologyBuilder.createTopology());
        }
    }
}

5 解释

首先看4中的topology

只需要注意这个地方写了2个worker 给spout分配了4个excutor 和4个task

集群中的分配结果就是一共有两个woker(我测试的时候两个worker分配在不同的机器上) 每个worker中有一个2个excutor 每个excutor中有2个task

所以连个机器上有两份日志分别是两个worker的

机器1（worker1）

messageid:0======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:1===========hashCode:1296109737
messageid:0======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:2===========hashCode:1858461299
messageid:0======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:3===========hashCode:392911204
messageid:0======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:4===========hashCode:2090482681
messageid:1======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:5===========hashCode:1296109737
messageid:1======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:6===========hashCode:1858461299
messageid:1======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:7===========hashCode:392911204
messageid:1======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:8===========hashCode:2090482681
messageid:2======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:9===========hashCode:1296109737
messageid:2======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:10===========hashCode:1858461299
messageid:2======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:11===========hashCode:392911204
messageid:2======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:12===========hashCode:2090482681
messageid:3======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:13===========hashCode:1296109737
messageid:3======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:14===========hashCode:1858461299
messageid:3======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:15===========hashCode:392911204
messageid:3======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:16===========hashCode:2090482681
messageid:4======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:17===========hashCode:1296109737
messageid:4======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:18===========hashCode:1858461299
messageid:4======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:19===========hashCode:392911204
messageid:4======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:20===========hashCode:2090482681
messageid:5======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:21===========hashCode:1296109737
messageid:5======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:22===========hashCode:1858461299
messageid:5======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:23===========hashCode:392911204
messageid:5======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:24===========hashCode:2090482681
messageid:6======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:25===========hashCode:1296109737
messageid:6======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:26===========hashCode:1858461299
messageid:6======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:27===========hashCode:392911204
messageid:6======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:28===========hashCode:2090482681
messageid:7======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:29===========hashCode:1296109737
messageid:7======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:30===========hashCode:1858461299
messageid:7======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:31===========hashCode:392911204
messageid:7======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:32===========hashCode:2090482681
messageid:8======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:33===========hashCode:1296109737
messageid:8======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:34===========hashCode:1858461299
messageid:8======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:35===========hashCode:392911204
messageid:8======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:36===========hashCode:2090482681
messageid:9======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:37===========hashCode:1296109737
messageid:9======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:38===========hashCode:1858461299
messageid:9======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:39===========hashCode:392911204
messageid:9======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:40===========hashCode:2090482681
messageid:10======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:5========staticId:41===========hashCode:1296109737
messageid:10======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:9========staticId:42===========hashCode:1858461299
messageid:10======processId:6721======thread:Thread-9-logSpout-executor[5 6]======taskId:6========staticId:43===========hashCode:392911204
messageid:10======processId:6721======thread:Thread-11-logSpout-executor[9 10]======taskId:10========staticId:44===========hashCode:2090482681

机器二上的(worker2)

messageid:0======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:1===========hashCode:1856491569
messageid:0======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:2===========hashCode:1074445762
messageid:0======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:3===========hashCode:1812309237
messageid:0======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:4===========hashCode:1790029150
messageid:1======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:5===========hashCode:1074445762
messageid:1======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:6===========hashCode:1856491569
messageid:1======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:7===========hashCode:1812309237
messageid:1======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:8===========hashCode:1790029150
messageid:2======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:9===========hashCode:1074445762
messageid:2======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:10===========hashCode:1856491569
messageid:2======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:11===========hashCode:1812309237
messageid:2======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:12===========hashCode:1790029150
messageid:3======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:13===========hashCode:1074445762
messageid:3======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:14===========hashCode:1856491569
messageid:3======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:15===========hashCode:1812309237
messageid:3======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:16===========hashCode:1790029150
messageid:4======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:17===========hashCode:1856491569
messageid:4======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:18===========hashCode:1074445762
messageid:4======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:19===========hashCode:1790029150
messageid:4======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:20===========hashCode:1812309237
messageid:5======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:21===========hashCode:1856491569
messageid:5======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:22===========hashCode:1074445762
messageid:5======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:23===========hashCode:1790029150
messageid:5======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:24===========hashCode:1812309237
messageid:6======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:25===========hashCode:1856491569
messageid:6======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:26===========hashCode:1074445762
messageid:6======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:27===========hashCode:1790029150
messageid:6======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:28===========hashCode:1812309237
messageid:7======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:29===========hashCode:1856491569
messageid:7======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:30===========hashCode:1074445762
messageid:7======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:31===========hashCode:1790029150
messageid:7======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:32===========hashCode:1812309237
messageid:8======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:33===========hashCode:1856491569
messageid:8======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:34===========hashCode:1074445762
messageid:8======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:35===========hashCode:1790029150
messageid:8======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:36===========hashCode:1812309237
messageid:9======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:37===========hashCode:1074445762
messageid:9======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:38===========hashCode:1856491569
messageid:9======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:39===========hashCode:1812309237
messageid:9======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:40===========hashCode:1790029150
messageid:10======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:4========staticId:41===========hashCode:1074445762
messageid:10======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:8========staticId:42===========hashCode:1856491569
messageid:10======processId:6167======thread:Thread-7-logSpout-executor[3 4]======taskId:3========staticId:43===========hashCode:1812309237
messageid:10======processId:6167======thread:Thread-9-logSpout-executor[7 8]======taskId:7========staticId:44===========hashCode:1790029150

两个文件大家要结合着看一块看这些打印信息都在spout中打印的变量也都是在spout中定义的

worker就是一个进程所以processId一共有两个每个worker是一个进程两个worker的jvm是隔离的

excutor 一共定义了4个对应的是4个线程

worker1

Thread-9-logSpout-executor[5 6]

Thread-11-logSpout-executor[9 10]

worker2

Thread-9-logSpout-executor[7 8]

Thread-7-logSpout-executor[3 4]

首先messageid 每个数字出现了8次说明一共有8个spout实例那就是每个task中有一个spout（private int id = 0;）

这个东西因为是静态的所以每个worker（进程）中共享的一份一共有两个work 所以staticid有两个所以同样的数字会出现两次

最后说明一句就是同一个excutor中无论有多少个task 这些task都是串行执行的因为一个excutor就是一个线程只是不断的切换task去执行因为线程是操作系统能够进行运算调度的最小单位