要分析flume的源码,首先要从Flume的启动开始,然后再分析Flume的各个组件服务Source、Channel、Sink的源码,接下来我们就来看下Flume是如何启动的吧
Flume的启动入口
org.apache.flume.node.Application
1、校验命令行
Options options = new Options();
Option option = new Option("n", "name", true, "the name of this agent");
option.setRequired(true);
options.addOption(option);
option = new Option("f", "conf-file", true,
"specify a config file (required if -z missing)");
option.setRequired(false);
options.addOption(option);
option = new Option(null, "no-reload-conf", false,
"do not reload config file if changed");
options.addOption(option);
// Options for Zookeeper
option = new Option("z", "zkConnString", true,
"specify the ZooKeeper connection to use (required if -f missing)");
option.setRequired(false);
options.addOption(option);
option = new Option("p", "zkBasePath", true,
"specify the base path in ZooKeeper for agent configs");
option.setRequired(false);
options.addOption(option);
option = new Option("h", "help", false, "display help text");
options.addOption(option);
CommandLineParser parser = new GnuParser();
CommandLine commandLine = parser.parse(options, args);
if (commandLine.hasOption('h')) {
new HelpFormatter().printHelp("flume-ng agent", options, true);
return;
}
String agentName = commandLine.getOptionValue('n');
boolean reload = !commandLine.hasOption("no-reload-conf");
boolean isZkConfigured = false;
if (commandLine.hasOption('z') || commandLine.hasOption("zkConnString")) {
isZkConfigured = true;
}
这一块都是在根据启动参数进行校验
2、配置文件的加载一次,还是定时加载,取决于no-reload-conf
在校验的时候,有关于no-reload-conf的校验,这块比较重要
boolean reload = !commandLine.hasOption("no-reload-conf");
这个实在检查启动参数中是否有no-reload-conf,启动命令中不带–no-reload-conf这个参数,flume会自动加载配置参数 默认是30秒,带则只加载一次
动态配置使用了PollingPropertiesFileConfigurationProvider,其中有FileWatcherRunnable来具体实现监控配置文件变化,
如果改动时间晚于文件最后修改时间,会触发eventBus.post(),从而触发带@Subscribe注解的handleConfigurationEvent这个方法,进而会stopAllComponents然后startAllComponents,
PollingPropertiesFileConfigurationProvider中使用了线程池executorService.scheduleWithFixedDelay(fileWatcherRunnable, 0, interval,TimeUnit.SECONDS);
fileWatcherRunnable每隔30秒监控变化
3、配置文件的加载
application.handleConfigurationEvent(configurationProvider.getConfiguration());
(1)AbstractConfigurationProvider中的方法getConfiguration,创建channel、source、sink,并加载到配置
1)loadChannels(agentConf, channelComponentMap);
创建channel,并存入channelComponentMap中
2)loadSources(agentConf, channelComponentMap, sourceRunnerMap);
读取配置文件生成 source , 然后创建 sourceRunner, 并注册到 channel
3)loadSinks(agentConf, channelComponentMap, sinkRunnerMap);
创建sink,并存入sinkRunnerMap中,并注册到channel
(2)handleConfigurationEvent这个方法就是在加载配置文件
@Subscribe这个注解是在监听配置文件是否有变化(是否有eventBus.post()),一旦有则触发这个方法
它是先停掉所有组件服务,然后再重新加载
@Subscribe
public void handleConfigurationEvent(MaterializedConfiguration conf) {
try {
lifecycleLock.lockInterruptibly();
stopAllComponents();
startAllComponents(conf);
} catch (InterruptedException e) {
logger.info("Interrupted while trying to handle configuration event");
return;
} finally {
// If interrupted while trying to lock, we don't own the lock, so must not attempt to unlock
if (lifecycleLock.isHeldByCurrentThread()) {
lifecycleLock.unlock();
}
}
}
(3)startAllComponents(conf),启动顺序:Channel、Sink、Source
private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
logger.info("Starting new configuration:{}", materializedConfiguration);
this.materializedConfiguration = materializedConfiguration;
for (Entry<String, Channel> entry :
materializedConfiguration.getChannels().entrySet()) {
try {
logger.info("Starting Channel " + entry.getKey());
supervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e) {
logger.error("Error while starting {}", entry.getValue(), e);
}
}
/*
* Wait for all channels to start.
*/
for (Channel ch : materializedConfiguration.getChannels().values()) {
while (ch.getLifecycleState() != LifecycleState.START
&& !supervisor.isComponentInErrorState(ch)) {
try {
logger.info("Waiting for channel: " + ch.getName() +
" to start. Sleeping for 500 ms");
Thread.sleep(500);
} catch (InterruptedException e) {
logger.error("Interrupted while waiting for channel to start.", e);
Throwables.propagate(e);
}
}
}
for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
try {
logger.info("Starting Sink " + entry.getKey());
supervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e) {
logger.error("Error while starting {}", entry.getValue(), e);
}
}
for (Entry<String, SourceRunner> entry :
materializedConfiguration.getSourceRunners().entrySet()) {
try {
logger.info("Starting Source " + entry.getKey());
supervisor.supervise(entry.getValue(),
new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
} catch (Exception e) {
logger.error("Error while starting {}", entry.getValue(), e);
}
}
this.loadMonitoring();
}
4、启动程序application.start();
配置文件只加载一次,则会主调调用application.handleConfigurationEvent方法,若是定时加载,则是handleConfigurationEvent的注解@Subscribe来监听变化自动调用
这里是如果定时加载加载配置文件,则components中就是有配置,application.start()方法中就会调用supervisor.supervise启动组件
5、lifecycleAware.start()启动组件服务
总之不管是配置文件只加载一次,还是定时加载启动,都会调用lifecycleAware.start()来启动各个组件服务
MonitorRunnable monitorRunnable = new MonitorRunnable();
monitorRunnable.lifecycleAware = lifecycleAware;
monitorRunnable.supervisoree = process;
monitorRunnable.monitorService = monitorService;
supervisedProcesses.put(lifecycleAware, process);
ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(
monitorRunnable, 0, 3, TimeUnit.SECONDS);
这里是在将各个组件服务添加到监控中monitorService,
在monitorRunnable中关键的就是lifecycleAware.start()这个了,它是各个组件启动的关键,各个组件实现了LifecycleAware的start接口,所以这里就是在启动各个组件服务。