Storm Trident示例partitionAggregate

最新推荐文章于 2024-03-18 21:00:00 发布

nickta

最新推荐文章于 2024-03-18 21:00:00 发布

阅读量404

点赞数

分类专栏： storm 文章标签： partitionAggregate storm repartition

本文链接：https://blog.csdn.net/nickta/article/details/79671064

版权

storm 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

partitionAggregate是针对于每个partition，而不是每个batch,对每个partition当中的tuple做聚合

省略部分代码，省略部分可参考：https://blog.csdn.net/nickta/article/details/79666918

FixedBatchSpout spout = new FixedBatchSpout(new Fields("user", "score"), 3, 
        		new Values("nickt1", 4),
                new Values("nickt2", 7), 
                new Values("nickt1", 8),
                new Values("nickt4", 9), 
                new Values("nickt4", 7),
                new Values("nickt1", 11),
                new Values("nickt7", 5)
                );
        spout.setCycle(false);
        TridentTopology topology = new TridentTopology();
        topology.newStream("spout1", spout)
        		.partitionBy(new Fields("user"))
        		.partitionAggregate(new Fields("score", "user"), new Sum(), new Fields("sum"))
        		.parallelismHint(5)
        		.each(new Fields("sum"),new Debug("print:"));

以上代码

batch1的内容为：

nickt1 4
nickt2 7

nickt1 8

batch2的内容为:

nickt4 9

nickt4 7

nickt1 11

batch3的内容为：

nickt7 5

因为是按partitionBy(new Fields("user"))进行repartition, 以batch1内容为例：

值为nickt1的tuples会被分配到目标partition当中去，

值为nickt2的tuple会被分配到另外一个目标partition当中去（也有可能分配到同一个，因为是按Target Partition = hash(fields) % (number of target partition)算法）

我们于是可以推测：输出结果当中，就包括有nickt1的score相加为12, 另一个为7。

batch2，batch3的结果，可自行分析。

实际输出结果如下，与推测相同：

<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [7]
<Fri Mar 23 18:09:53 CST 2018[partition0-Thread-60-b-0-executor[33 33]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition3-Thread-114-b-0-executor[36 36]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition1-Thread-132-b-0-executor[34 34]]> DEBUG(print:): [12]
<Fri Mar 23 18:09:53 CST 2018[partition4-Thread-136-b-0-executor[37 37]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition3-Thread-114-b-0-executor[36 36]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition1-Thread-132-b-0-executor[34 34]]> DEBUG(print:): [11]
<Fri Mar 23 18:09:53 CST 2018[partition4-Thread-136-b-0-executor[37 37]]> DEBUG(print:): [16]
<Fri Mar 23 18:09:53 CST 2018[partition0-Thread-60-b-0-executor[33 33]]> DEBUG(print:): [0]
<Fri Mar 23 18:09:53 CST 2018[partition2-Thread-53-b-0-executor[35 35]]> DEBUG(print:): [5]