Storm 可以在Topology 中使用 代码 对 Storm 进行配置。
通过配置可以进行调控,达到优化的效果
下面以Java代码进行举例,示范下可进行的配置
1.设置一个 Topology 中用的 worker / slot 数量。
Tips:
1. worker 对应的是slots, 即 worker 与 slot (槽,最终的端口)是一对一关系。
2. worker 里面包含了具体的 bolt 与 sbolt
3. worker 要占用端口,每个worker 占用一个端口,因为 worker 之间要进行数据的通信
Topology 主函数中重要的配置
Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(3); //设置worker的数量
StormSubmitter.submitTopology("firstTopo", conf, builder.createTopology());
主要代码:
conf.setNumWorkers(3); //设置worker的数量
效果:
2.指定spout 与 bolt 的数量, bolt 与 spout 在 worker 中工作,设置 bolt 与 spout 实例化的数量 可以提高程序的并行度
Tips: 如果不进行设置,默认为1,单实例
Tips: 设置了Spout/ bolt 数量,相当于设置了 Spout / bolt 生成几个实例
如: builder.setSpout("spout", new RandomSpout(),worknum);
示例代码:
int worknum = 3;
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSpout(),worknum);
builder.setBolt("bolt", new SenqueceBolt(),2*worknum).shuffleGrouping("spout");
效果:
需要 点击 Topology 去查看
这时候根据上面代码的设置, bolts + spouts = 3 + 2*3 = 9
9 != 12 (task / executor )
因为还有ackernum 的存在,也会占用线程(executor) 或者 task (任务)
3.task (任务) 的含义
task 代表了spout 与 bolt . spout 与 bolt ,acter 都称为 task
默认一个task 在 一个线程中 执行
4.设置Acknum=0. 使得总的 spouts + bolts = executor = task
代码: conf.setNumAckers(0);
完整 Spout 设置代码:
package count;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.topology.TopologyBuilder;
//本地模式需要
//import org.apache.storm.utils.Utils;
//import org.apache.storm.LocalCluster;
public class FirstTopo {
public static void main(String[] args) throws Exception {
int worknum = 3;
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSpout(),worknum);
builder.setBolt("bolt", new SenqueceBolt(),2*worknum).shuffleGrouping("spout");
Config conf = new Config();
conf.setDebug(false);
// if (args != null && args.length > 0) {
//
// conf.setNumWorkers(3); //设置worker的数量
//
// StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
// } else {
// //本地模式:本地提交
// LocalCluster cluster = new LocalCluster();
// cluster.submitTopology("firstTopo", conf, builder.createTopology());
// Utils.sleep(100000);
// cluster.killTopology("firstTopo");
// cluster.shutdown();
//集群模式:集群提交
conf.setNumWorkers(worknum); //设置worker的数量
conf.setNumAckers(0);
StormSubmitter.submitTopology("firstTopo", conf, builder.createTopology());
// }
}
}
效果:
5.一个 线程executor 执行多个 任务task
只需要在最后设置task 的数量即可
核心代码:
builder.setSpout("spout", new RandomSpout(),worknum).setNumTasks(worknum*2);
builder.setBolt("bolt", new SenqueceBolt(),2*worknum).shuffleGrouping("spout").setNumTasks(worknum*2);
示例代码:
package count;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.topology.TopologyBuilder;
//本地模式需要
//import org.apache.storm.utils.Utils;
//import org.apache.storm.LocalCluster;
public class FirstTopo {
public static void main(String[] args) throws Exception {
int worknum = 3;
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSpout(),worknum).setNumTasks(worknum*2);
builder.setBolt("bolt", new SenqueceBolt(),2*worknum).shuffleGrouping("spout").setNumTasks(worknum*2);
Config conf = new Config();
conf.setDebug(false);
// if (args != null && args.length > 0) {
//
// conf.setNumWorkers(3); //设置worker的数量
//
// StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
// } else {
// //本地模式:本地提交
// LocalCluster cluster = new LocalCluster();
// cluster.submitTopology("firstTopo", conf, builder.createTopology());
// Utils.sleep(100000);
// cluster.killTopology("firstTopo");
// cluster.shutdown();
//集群模式:集群提交
conf.setNumWorkers(worknum); //设置worker的数量
conf.setNumAckers(0);
StormSubmitter.submitTopology("firstTopo", conf, builder.createTopology());
// }
}
}
执行的效果: