为了在解读分析时有个统一的思路,本文将从启动一个集群开始分析。
(说明:为了测试方便,采用的是本地模式)。
1.参数设置
Config conf = new Config();
//设置Topology中workers的数量
conf.setNumWorkers(4);
//这意味着用于单元测试以防止元组在测试期间意外超时。
conf.put(Config.TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS, true);
//最大超时时间
conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS , 10);
2.创建TopologyBuilder
TopologyBuilder builder = new TopologyBuilder();
3.设置TopologyBuilder
//3.1设置Spout
//3.2设置Bolt(可能多个)
4.启动集群
启动集群
//启动本地集群
LocalCluster cluster = new LocalCluster();
//启动本地集群的细节:
public LocalCluster() {
synchronized (LocalCluster.class) {
//1.判断当前本地集群是否存在实例,如果存在则抛出异常,否则进行后面的处理
if (instance != null) {
throw new RuntimeException("LocalCluster should be single");
}
//2.设置日志
setLogger();
System.setProperty("java.net.preferIPv4Stack", "true");
//3.调用prepareLocalCluster()方法对本地集群的启动环境做准备
this.state = LocalUtils.prepareLocalCluster();
if (this.state == null)
throw new RuntimeException("prepareLocalCluster error");
instance = this;
}
}
4.3.具体的准备如下:
public static LocalClusterMap prepareLocalCluster() {
LocalClusterMap state = new LocalClusterMap();
try {
//1.创建缓存路径集合
List<String> tmpDirs = new ArrayList();
//2.获取缓存路径
String zkDir = getTmpDir();
tmpDirs.add(zkDir);
//3.启动本地Zookeepr
Factory zookeeper = startLocalZookeeper(zkDir);
//4.获取本地的配置文件
Map conf = getLocalConf(zookeeper.getZooKeeperServer().getClientPort());
//获取(计算得到)nimbus的缓存路径
String nimbusDir = getTmpDir();
tmpDirs.add(nimbusDir);
//5.进行深度拷贝得到nimbusConf配置文件
Map nimbusConf = deepCopyMap(conf);
//6.配置nimbus的缓存路径
nimbusConf.put(Config.STORM_LOCAL_DIR, nimbusDir);
//7.创建Nimbus服务
NimbusServer instance = new NimbusServer();
//8.进行深拷贝得到supervisor的配置文件
Map supervisorConf = deepCopyMap(conf);
String supervisorDir = getTmpDir();
tmpDirs.add(supervisorDir);
supervisorConf.put(Config.STORM_LOCAL_DIR, supervisorDir);
//9.创建Supervisor实例
Supervisor supervisor = new Supervisor();
//IContext:该接口需要实现消息插件。消息插件通过Storm配置(storm.messaging.transport)指定参数。 消息传递插件应该有一个默认的构造函数和实现IContext界面。在构建之后,
我们将根据storm配置调用IContext::prepare(storm_conf)来启用上下文。
//10.获取消息插件
IContext context = getLocalContext(supervisorConf);
//11.设置本地集群中的Nimbus、Supervisor等主从节点以及zookeeper、消息插件等
state.setNimbusServer(instance);
state.setNimbus(instance.launcherLocalServer(nimbusConf, new DefaultInimbus()));
state.setZookeeper(zookeeper);
state.setConf(conf);
state.setTmpDir(tmpDirs);
state.setSupervisor(supervisor.mkSupervisor(supervisorConf, context));
return state;
} catch (Exception e) {
LOG.error("prepare cluster error!", e);
state.clean();
}
return null;
}
4.3.3启动本地Zookeeper的具体细节如下:
程序允许给35535个端口用来启动Zookeeper集群,从2000开始分配,
如果端口2000至端口65534全部被占用了,则会跑出无可用的端口来运行zookeeper的异常
private static Factory startLocalZookeeper(String tmpDir) {
for (int i = 2000; i < 65535; i++) {
try {
return Zookeeper.mkInprocessZookeeper(tmpDir, i);
} catch (Exception e) {
LOG.error("fail to launch zookeeper at port: " + i, e);
}
}
throw new RuntimeException("No port is available to launch an inprocess zookeeper.");
}
对于一个系统性的集群框架来说,启动该集群,必须为其配置相应的参数,比如workers数量,缓存地址等,
上述只是本地集群启动时的一小部分。剩下部分将在下一篇中讲述。
欢迎关注下面二维码进行技术交流: