Storm 动态平衡:
Storm 里面有个动态调优的功能,可以通过Storm rebalance 进行 worker, executor 等数目的调整。
注意: 虽然此功能较好,但是可能会丢失数据。
Storm rebalance Synatax (Storm rebalance 语法):
命令介绍:
通过 storm help rebalance 进行调整:
Syntax: [storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*]
Sometimes you may wish to spread out where the workers for a topology
are running. For example, let's say you have a 10 node cluster running
4 workers per node, and then let's say you add another 10 nodes to
the cluster. You may wish to have Storm spread out the workers for the
running topology so that each node runs 2 workers. One way to do this
is to kill the topology and resubmit it, but Storm provides a "rebalance"
command that provides an easier way to do this.
Rebalance will first deactivate the topology for the duration of the
message timeout (overridable with the -w flag) and then redistribute
the workers evenly around the cluster. The topology will then return to
its previous state of activation (so a deactivated topology will still
be deactivated and an activated topology will go back to being activated).
The rebalance command can also be used to change the parallelism of a running topology.
Use the -n and -e switches to change the number of workers or number of executors of a component
respectively.
注解:
-n : 调整的是 worker 的数量。
-e : 调整的是Bolt, Spout 的 executor 数量, 最多不能超过 预先设定 task的数量, 若设置为超过task 的数量会调整 task 的上限。
示例:
指令:
storm rebalance transfer_base -w 60 -n 6
效果:
指令:
storm rebalance transfer_base -w 60 -e transfer_bolt=110
效果: