Understanding the Parallelism of a Storm Topology

转载 2015年07月07日 14:55:00

In the past few days I have been test-driving Twitter’sStorm project, which is a distributed real-time data processing platform. One of my findings so far has been that the quality of Storm’s documentation and example code is pretty good – it is very easy to get up and running with Storm. Big props to the Storm developers! At the same time, I found the sections on how a Storm topology runs in a cluster not perfectly clear, and learned that the recent releases of Storm changed some of its behavior in a way that is not yet fully reflected in the Storm wiki and in the API docs.

In this article I want to share my own understanding of the parallelism of a Storm topology after reading the documentation and writing some first prototype code. More specifically, I describe the relationships of worker processes, executors (threads) and tasks, and how you can configure them according to your needs. This article is based on Storm release 0.8.1, the latest version as of October 2012.

Update 2012-11-05: This blog post has been merged into Storm’s documentation.

What is Storm?

For those readers unfamiliar with Storm here is a brief description taken from its homepage:

Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

What makes a running topology: worker processes, executors and tasks

Storm distinguishes between the following three main entities that are used to actually run a topology in a Storm cluster:

  • Worker processes
  • Executors (threads)
  • Tasks

Here is a simple illustration of their relationships:

Figure 1: The relationships of worker processes, executors (threads) and tasks in Storm

worker process executes a subset of a topology, and runs in its own JVM. A worker process belongs to a specific topology and may run one or more executors for one or more components (spouts or bolts) of this topology. A running topology consists of many such processes running on many machines within a Storm cluster.

An executor is a thread that is spawned by a worker process and runs within the worker’s JVM. An executor may run one or more tasks for the same component (spout or bolt). An executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.

task performs the actual data processing and is run within its parent executor’s thread of execution. Each spout or bolt that you implement in your code executes as many tasks across the cluster. The number of tasks for a component is always the same throughout the lifetime of a topology, but the number of executors (threads) for a component can change over time. This means that the following condition holds true: #threads <= #tasks. By default, the number of tasks is set to be the same as the number of executors, i.e. Storm will run one task per thread (which is usually what you want anyways).

Also be aware that:

  • The number of executor threads can be changed after the topology has been started (see storm rebalance command below).
  • The number of tasks of a topology is static.

See Understanding the Internal Message Buffers of Storm for another view on the various threads that are running within the lifetime of a worker process and its associated executors and tasks.

Configuring the parallelism of a topology

Note that in Storm’s terminology “parallelism” is specifically used to describe the so-calledparallelism hint, which means the initial number of executors (threads) of a component. In this article though I use the term “parallelism” in a more general sense to describe how you can configure not only the number of executors but also the number of worker processes and the number of tasks of a Storm topology. I will specifically call out when “parallelism” is used in the narrow definition of Storm.

The following table gives an overview of the various configuration options and how to set them in your code. There is more than one way of setting these options though, and the table lists only some of them. Storm currently has the following order of precedence for configuration settings: external component-specific configuration > internal component-specific configuration > topology-specific configuration > storm.yaml > defaults.yaml. Please take a look at the Storm documentation for more details.

What Description Configuration option How to set in your code (examples)
#worker processes How many worker processes to createfor the topologyacross machines in the cluster. Config#TOPOLOGY_WORKERS Config#setNumWorkers
#executors (threads) How many executors to spawnper component. ? TopologyBuilder#setSpout() and TopologyBuilder#setBolt()

Note that as of Storm 0.8 theparallelism_hint parameter now specifies the initial number of executors (not tasks!) for that bolt.
#tasks How many tasks to create per component. Config#TOPOLOGY_TASKS ComponentConfigurationDeclarer#setNumTasks()

Here is an example code snippet to show these settings in practice:

Configuring the parallelism of a Storm bolt
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)

In the above code we configured Storm to run the bolt GreenBolt with an initial number of two executors and four associated tasks. Storm will run two tasks per executor (thread). If you do not explicitly configure the number of tasks, Storm will run by default one task per executor.

Configuring parallelism on multi-tenant Storm clusters

Storm 0.8.2 introduced the Isolation Scheduler that makes it easy and safe to share a cluster among many topologies, i.e. it solves the multi-tenancy problem — avoiding resource contention between topologies — by providing full isolation between topologies.

When you use the isolation scheduler Nathan recommends you set num workers to a multiple of number of machines. And parallelism hint to a multiple of the number of workers. If you do call setNumTasks() (which most people don’t), you should set that to a multiple of the parallelism hint. If you do this, then what happens is your workload is uniform distributed. Each machine and jvm process will have the same number of threads, and roughly the same amount of work.

Jason Jackson on storm-user grokbase.com/t/gg/storm-user/…

Example of a running topology

The following illustration shows how a simple topology would look like in operation. The topology consists of three components: one spout called BlueSpout and two bolts called GreenBolt and YellowBolt. The components are linked such that BlueSpout sends its output to GreenBolt, which in turns sends its own output to YellowBolt.

Figure 2: Example of a running topology

The GreenBolt was configured as per the code snippet above whereas BlueSpout and YellowBolt only set the parallelism hint (number of executors). Here is the relevant code:

Configuring the parallelism of a simple Storm topology
Config conf = new Config();
conf.setNumWorkers(2); // use two worker processes

topologyBuilder.setSpout("blue-spout", new BlueSpout(), 2); // parallelism hint

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)

topologyBuilder.setBolt("yellow-bolt", new YellowBolt(), 6)


And of course Storm comes with additional configuration settings to control the parallelism of a topology, including:

  • TOPOLOGY_MAX_TASK_PARALLELISM: This setting puts a ceiling on the number of executors that can be spawned for a single component. It is typically used during testing to limit the number of threads spawned when running a topology in local mode. You can set this option via e.g. Config#setMaxTaskParallelism().

Update Oct 18: Nathan Marz informed me that TOPOLOGY_OPTIMIZE will be removed in a future release. I have therefore removed its entry from the configuration list above.

How to change the parallelism of a running topology

A nifty feature of Storm is that you can increase or decrease the number of worker processes and/or executors without being required to restart the cluster or the topology. The act of doing so is called rebalancing.

You have two options to rebalance a topology:

  1. Use the Storm web UI to rebalance the topology.
  2. Use the CLI tool storm rebalance as described below.

Here is an example of using the CLI tool:

 # Reconfigure the topology "mytopology" to use 5 worker processes,
 # the spout "blue-spout" to use 3 executors and
 # the bolt "yellow-bolt" to use 10 executors.

 $ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10

References for this article

To compile this article (and to write my related test code) I used information primarily from the following sources:


My personal impression is that Storm is a very promising tool. On the one hand I like its clean and elegant design, and on the other hand I loved to find out that a young open source tool can still have an excellent documentation. In this article I tried to summarize my own understanding of the parallelism of topologies, which may or may not be 100% correct – feel free to let me know if there are any mistakes in the description above!


Understanding the parallelism of a Storm topology

In the past few days I have been test-driving Twitter’s Storm project, which is a distributed real...

Understanding the Parallelism of a Storm Topology

Understanding the Parallelism of a Storm Topology Oct 16th, 2012 Table of Contents What is Sto...
  • jmppok
  • jmppok
  • 2013年12月10日 15:05
  • 1498

Understanding the Internal Message Buffers of Storm

Understanding the Internal Message Buffers of Storm Jun 21st, 2013 Table of Contents Internal ...

sql server max degree of parallelism 选项

max degree of parallelism 选项 SQL Server 2008 R2 其他版本 SQL Server "Denali" ...


Storm中负责Topo分配的工作由nimbus负责,具体代码在nimbus.clj中。 对于一个新Topo的分配来说,主要经历两个阶段: 1. 逻辑分配阶段 这里又会涉及到两个概念execut...


原创文章,如需转载,请注明出处: Storm计算以topology为单位,topology提交到Storm集群中运行后,通过storm rebalance 命令可对topology进行动态调整。比...
  • jmppok
  • jmppok
  • 2013年12月05日 16:34
  • 2651

Storm入门教程 第二章 构建Topology

转载自:量子恒道官方博客 2.1 Storm基本概念 在运行一个Storm任务之前,需要了解一些概念: TopologiesStreamsSpoutsBoltsStream grou...


  • xeseo
  • xeseo
  • 2014年01月13日 13:14
  • 8663

storm rebalance 命令调整topology并行数及问题分析

原创文章,欢迎转载.转载请注明出处: 通过前面的介绍,我们知道Storm可以实现弹性计算,根据需要实时调整Topology的并行度. 1)翻译:Storm Scalable ——Storm弹...
  • jmppok
  • jmppok
  • 2013年12月10日 14:48
  • 7565

Twitter Storm: 在生产集群上运行topology配置

在生产集群上运行topology跟本地模式差不多。下面是步骤: 1)定义topology(如果是java的话, 用TopologyBuilder) 2) 使用StormSubmitter来...
您举报文章:Understanding the Parallelism of a Storm Topology