标签: storm
647人阅读 评论(0) 收藏 举报

1 storm并行的基本概念

  • storm集群中的一个机器可以运行一个或者多个worker,对应于一个或者多个topologies.
  • 1个worker进程运行1个或多个excutor线程。每个worker从属于一个topology.
  • executor是单线程。每1个executor运行着相同的组件(spout或bolt)的1个或多个task。
  • 1个task执行着实际的数据处理。
What Description Configuration option How to set in your code (examples)
#worker processes How many worker processes to createfor the topologyacross machines in the cluster. Config#TOPOLOGY_WORKERS Config#setNumWorkers
#executors (threads) How many executors to spawnper component. ? TopologyBuilder#setSpout() and TopologyBuilder#setBolt()

Note that as of Storm 0.8 theparallelism_hint parameter now specifies the initial number of executors (not tasks!) for that bolt.
#tasks How many tasks to create per component. Config#TOPOLOGY_TASKS ComponentConfigurationDeclarer#setNumTasks()

Here is an example code snippet to show these settings in practice:

Configuring the parallelism of a Storm bolt
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(170, 170, 170) !important;">1</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(170, 170, 170) !important;">2</span>
<span class="line-number" style="margin: 0px; padding: 0px; border: 0px; font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; vertical-align: baseline; color: rgb(170, 170, 170) !important;">3</span>
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)

In the above code we configured Storm to run the bolt GreenBolt with an initial number of two executors and four associated tasks. Storm will run two tasks per executor (thread). If you do not explicitly configure the number of tasks, Storm will run by default one task per executor.


2  是否需要提高workers数目

(1)  最好 一台机器上的一个topology只使用一个worker,主要原因时减少了worker之间的数据传输

How many Workers should I use?

The total number of workers is set by the supervisors – there’s some number of JVM slots each supervisor will superintend. The thing you set on the topology is how many worker slots it will try to claim.

There’s no great reason to use more than one worker per topology per machine.

With one topology running on three 8-core nodes, and parallelism hint 24, each bolt gets 8 executors per machine, i.e. one for each core. There are three big benefits to running three workers (with 8 assigned executors each) compare to running say 24 workers (one assigned executor each).

First, data that is repartitioned (shuffles or group-bys) to executors in the same worker will not have to hit the transfer buffer. Instead, tuples are deposited directly from send to receive buffer. That’s a big win. By contrast, if the destination executor were on the same machine in a different worker, it would have to go send -> worker transfer -> local socket -> worker recv -> exec recv buffer. It doesn’t hit the network card, but it’s not as big a win as when executors are in the same worker.

Second, you’re typically better off with three aggregators having very large backing cache than having twenty-four aggregators having small backing caches. This reduces the effect of skew, and improves LRU efficiency.

Lastly, fewer workers reduces control flow chatter.


Increasing the number of workers (responsible for running one or more executors for one or more components) might also gives you a performance benefit, but this also relative as I found from this discussion wherenathanmarz says

Having more workers might have better performance, depending on where your bottleneck is. Each worker has a single thread that passes tuples on to the 0mq connections for transfer to other workers, so if you're bottlenecked on CPU and each worker is dealing with lots of tuples, more workers will probably net you better throughput.

So basically there is no definite answer to this, you should try different configuration based on your environment and design.

3 executor的数目


executor初始数目=spout数目+bolt数目+acker数目 (这些加起来也就是task数目。)


4   是否需要提高TASK数目


Disclaimer: I wrote the article you referenced in your question above.

However I'm a bit confused by the concept of "task". Is a task an running instance of the component(spout or bolt) ? A executor having multiple tasks actually is saying the same component is executed for multiple times by the executor, am I correct ?

Yes, and yes.

Moreover in a general parallelism sense, Storm will spawn a dedicated thread(executor) for a spout or bolt, but what is contributed to the parallelism by an executor(thread) having multiple tasks ?

Running more than one task per executor does not increase the level of parallelism -- an executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.

As I wrote in the article please note that:

  • The number of executor threads can be changed after the topology has been started (seestorm rebalance command).
  • The number of tasks of a topology is static.

And by definition there is the invariant of #executors <= #tasks.

So one reason for having 2+ tasks per executor thread is to give you the flexibility to expand/scale up the topology through thestorm rebalance command in the future without taking the topology offline. For instance, imagine you start out with a Storm cluster of 15 machines but already know that next week another 10 boxes will be added. Here you could opt for running the topology at the anticipated parallelism level of 25 machines already on the 15 initial boxes (which is of course slower than 25 boxes). Once the additional 10 boxes are integrated you can thenstorm rebalance the topology to make full use of all 25 boxes without any downtime.

Another reason to run 2+ tasks per executor is for (primarily functional) testing. For instance, if your dev machine or CI server is only powerful enough to run, say, 2 executors alongside all the other stuff running on the machine, you can still run 30 tasks (here: 15 per executor) to see whether code such as your custom Storm grouping is working as expected.

In practice we normally we run 1 task per executor.

PS: Note that Storm will actually spawn a few more threads behind the scenes. For instance, each executor has its own "send thread" that is responsible for handling outgoing tuples. There are also "system-level" background threads for e.g. acking tuples that run alongside "your" threads. IIRC the Storm UI counts those acking threads in addition to "your" threads.




  1. Worker进程
  2. Executors (线程)
  3. Tasks




  • Storm集群中的其中1台机器可能运行着属于多个拓扑(可能为1个)的多个worker进程(可能为1个)。每个worker进程运行着特定的某个拓扑的executors。
  • 1个或多个excutor可能运行于1个单独的worker进程,每1个executor从属于1个被worker process生成的线程中。每1个executor运行着相同的组件(spout或bolt)的1个或多个task。
  • 1个task执行着实际的数据处理。



1 个task执行着实际的数据处理,你用代码实现的每一个spout或bolt就相当于分布于整个集群中的许多个task。在1个拓扑的生命周期中,1个组 件的task的数量总是一样的,但是1个组件的executor(线程)的数量可以随着时间而改变。这意味着下面的条件总是成立:thread的数量 <= task的数量。默认情况下,task的数量与executor的数量一样,例如,Storm会在每1个线程运行1个task。


注意Storm的术语"并发度(parallelism)"是特别用来描述所谓的parallelism hint的,这代表1个组件的初始的executor(线程)的数量。在此文档中我们使用术语"并发度"的一般意义来描述你不但可以配置executor的数量,还可以配置worker进程的数量,还可以是1个拓扑的task的数量。在用到并发度的狭义的定义时我们会特别提出。

下面的小节给出了一些不同的配置选项,还有你如何在代码中设置它们。有多种方法可以进行设置,表格列举了其中几种。Storm目前有下列的配置优先级:defaults.yaml < storm.yaml < 特定拓扑的配置 < 内部特定组件的配置 < 外部特定组件的配置。



  • 描述: 1个拓扑有多少个分布于集群中的机器的worker进程
  • 如何在代码中设置 (例子):
    • Config#setNumWorkers

executor的数量 (线程)

  • 描述: 每个组件会有多少个executor
  • 配置选项: ?
  • 如何在代码中设置 (例子):
    • TopologyBuilder#setSpout()
    • TopologyBuilder#setBolt()
    • 注意在Storm 0.8以后parallelism_hint参数指定的是bolt的初始的executor的数量。


  • 描述: 每个组件有多少task
  • 如何在代码中设置 (例子):
    • ComponentConfigurationDeclarer#setNumTasks()


topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)


在上面的代码中我们配置了Storm运行GreenBolt指定了初始有2个executor和4个关连的task。Storm会在每个 executor(线程)2个task。如果你不想显式的配置task的数量,Storm会默认的为每个executor运行1个task。


下 面的图表展示了1个简单拓扑在实际操作中看起来是怎样的。这个拓扑包含了3个组件:1个spout叫做BlueSpout,2个bolt分别叫 GreenBolt和YellowBolt。BlueSpout发送它的输出到GreenBolt,GreenBolt又把它的输出发到 YellowBolt。


3个组件的并发度加起来是10,就是说拓扑一共有10个executor,一共有2个worker,每个worker产生10 / 2 = 5条线程。



Config conf = new Config();

conf.setNumWorkers(2); // 使用2个worker进程

topologyBuilder.setSpout(“blue-spout”, new BlueSpout(), 2); // parallelism hint为2

topologyBuilder.setBolt(“green-bolt”, new GreenBolt(), 2) .setNumTasks(4) .shuffleGrouping(“blue-spout”);

topologyBuilder.setBolt(“yellow-bolt”, new YellowBolt(), 6) .shuffleGrouping(“green-bolt”);

StormSubmitter.submitTopology( “mytopology”, conf, topologyBuilder.createTopology() );

And of course Storm comes with additional configuration settings to control the parallelism of a topology, including:


  • TOPOLOGY_MAX_TASK_PARALLELISM: 这个设置指定了1个单独的组件的executor的数量的上限。当在测试阶段使用本地模式运行1个拓扑时,用来限制生成的线程的数量。你可以像下面这样来使用:
  • Config#setMaxTaskParallelism().




  1. 使用Storm的web UI来rebalance。
  2. 像下面描述的那样,使用命令行工具来做:

# 重新配置拓扑 “mytopology” 使用5个worker进程。

# spout “blue-spout” 使用3个executor

# bolt “yellow-bolt” 使用10个executor

$ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10




工作进程(Worker Process) Worker是Spout/Bolt中运行具体处理逻辑的进程。拓扑跨一个或多个Worker进程执行。每个Worker进程是一个物理的JVM和拓扑执行所有任务的...
  • lzm1340458776
  • lzm1340458776
  • 2015年04月26日 10:15
  • 2268


本文可作为 >一书1.4节的读书笔记 在Storm中,一个task就可以理解为在集群中某个节点上运行的一个spout或者bolt实例。 记住一个task是一个实例。 实例明白吧 Class Pe...
  • dlf123321
  • dlf123321
  • 2016年07月02日 15:45
  • 1324


1. storm并行的基本概念 storm集群中的一个supervisor节点可以运行一个或者多个worker,对应于一个或者多个topologies。1个worker进程运行1个或多个exe...
  • victory0508
  • victory0508
  • 2015年04月02日 21:46
  • 1029

Storm 使用经验与性能优化(一)

Storm 内部剖析 源码讲解 使用经验 性能优化
  • u011317245
  • u011317245
  • 2016年09月26日 15:52
  • 611

storm 并发度

  • fjr_huoniao
  • fjr_huoniao
  • 2016年09月08日 20:47
  • 451

Storm_Storm 动态调优,并发度调整

Storm 动态平衡: Storm 里面有个动态调优的功能,可以通过Storm rebalance 进行 worker, executor 等数目的调整。 注意: 虽然此功能较好,但是可能会丢失数...
  • u010003835
  • u010003835
  • 2016年12月13日 11:19
  • 1091


典型的问题是word总数和word个数(去重)并且在高并发情况下完成 一个topology可以包含一个或多个worker(并行的跑在不同的机器上),所以worker process就是执行一个top...
  • zengxiaosen
  • zengxiaosen
  • 2016年10月06日 18:30
  • 898

Storm Topology的并发度

一个Topology可以包含一个或多个worker(并行的跑在不同的machine上), 所以worker process就是执行一个topology的子集, 并且worker只能对应于一个topol...
  • xiaolang85
  • xiaolang85
  • 2014年07月25日 16:55
  • 1226


How to change the parallelism of a running topology, 动态的改变并发度 Storm支持在不restart topology的情况下, 动态的改...
  • scl1991
  • scl1991
  • 2016年07月15日 16:30
  • 828


  • gaisonlee
  • gaisonlee
  • 2010年11月02日 11:24
  • 1106
    访问量: 80万+
    积分: 6432
    排名: 4590