【大数据】Flume源码分析之启动过程

要分析flume的源码，首先要从Flume的启动开始，然后再分析Flume的各个组件服务Source、Channel、Sink的源码，接下来我们就来看下Flume是如何启动的吧

Flume的启动入口

org.apache.flume.node.Application

1、校验命令行

Options options = new Options();

      Option option = new Option("n", "name", true, "the name of this agent");
      option.setRequired(true);
      options.addOption(option);

      option = new Option("f", "conf-file", true,
          "specify a config file (required if -z missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option(null, "no-reload-conf", false,
          "do not reload config file if changed");
      options.addOption(option);

      // Options for Zookeeper
      option = new Option("z", "zkConnString", true,
          "specify the ZooKeeper connection to use (required if -f missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("p", "zkBasePath", true,
          "specify the base path in ZooKeeper for agent configs");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("h", "help", false, "display help text");
      options.addOption(option);

      CommandLineParser parser = new GnuParser();
      CommandLine commandLine = parser.parse(options, args);

      if (commandLine.hasOption('h')) {
        new HelpFormatter().printHelp("flume-ng agent", options, true);
        return;
      }

      String agentName = commandLine.getOptionValue('n');
      boolean reload = !commandLine.hasOption("no-reload-conf");

      boolean isZkConfigured = false;
      if (commandLine.hasOption('z') || commandLine.hasOption("zkConnString")) {
        isZkConfigured = true;
      }

这一块都是在根据启动参数进行校验

2、配置文件的加载一次，还是定时加载，取决于no-reload-conf

在校验的时候，有关于no-reload-conf的校验，这块比较重要

boolean reload = !commandLine.hasOption("no-reload-conf");

这个实在检查启动参数中是否有no-reload-conf，启动命令中不带–no-reload-conf这个参数，flume会自动加载配置参数 默认是30秒，带则只加载一次

动态配置使用了PollingPropertiesFileConfigurationProvider，其中有FileWatcherRunnable来具体实现监控配置文件变化，

如果改动时间晚于文件最后修改时间，会触发eventBus.post()，从而触发带@Subscribe注解的handleConfigurationEvent这个方法，进而会stopAllComponents然后startAllComponents，

PollingPropertiesFileConfigurationProvider中使用了线程池executorService.scheduleWithFixedDelay(fileWatcherRunnable, 0, interval,TimeUnit.SECONDS);

fileWatcherRunnable每隔30秒监控变化

3、配置文件的加载

application.handleConfigurationEvent(configurationProvider.getConfiguration());

（1）AbstractConfigurationProvider中的方法getConfiguration，创建channel、source、sink，并加载到配置

1）loadChannels(agentConf, channelComponentMap);

创建channel，并存入channelComponentMap中

2）loadSources(agentConf, channelComponentMap, sourceRunnerMap);

读取配置文件生成 source , 然后创建 sourceRunner, 并注册到 channel

3）loadSinks(agentConf, channelComponentMap, sinkRunnerMap);

创建sink，并存入sinkRunnerMap中，并注册到channel

（2）handleConfigurationEvent这个方法就是在加载配置文件
@Subscribe这个注解是在监听配置文件是否有变化（是否有eventBus.post()），一旦有则触发这个方法
它是先停掉所有组件服务，然后再重新加载

 @Subscribe
  public void handleConfigurationEvent(MaterializedConfiguration conf) {
    try {
      lifecycleLock.lockInterruptibly();
      stopAllComponents();
      startAllComponents(conf);
    } catch (InterruptedException e) {
      logger.info("Interrupted while trying to handle configuration event");
      return;
    } finally {
      // If interrupted while trying to lock, we don't own the lock, so must not attempt to unlock
      if (lifecycleLock.isHeldByCurrentThread()) {
        lifecycleLock.unlock();
      }
    }
  }

（3）startAllComponents(conf)，启动顺序：Channel、Sink、Source

private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
    logger.info("Starting new configuration:{}", materializedConfiguration);

    this.materializedConfiguration = materializedConfiguration;

    for (Entry<String, Channel> entry :
        materializedConfiguration.getChannels().entrySet()) {
      try {
        logger.info("Starting Channel " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    /*
     * Wait for all channels to start.
     */
    for (Channel ch : materializedConfiguration.getChannels().values()) {
      while (ch.getLifecycleState() != LifecycleState.START
          && !supervisor.isComponentInErrorState(ch)) {
        try {
          logger.info("Waiting for channel: " + ch.getName() +
              " to start. Sleeping for 500 ms");
          Thread.sleep(500);
        } catch (InterruptedException e) {
          logger.error("Interrupted while waiting for channel to start.", e);
          Throwables.propagate(e);
        }
      }
    }

    for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
      try {
        logger.info("Starting Sink " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    for (Entry<String, SourceRunner> entry :
         materializedConfiguration.getSourceRunners().entrySet()) {
      try {
        logger.info("Starting Source " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    this.loadMonitoring();
  }

4、启动程序application.start();

配置文件只加载一次，则会主调调用application.handleConfigurationEvent方法，若是定时加载，则是handleConfigurationEvent的注解@Subscribe来监听变化自动调用

这里是如果定时加载加载配置文件，则components中就是有配置，application.start()方法中就会调用supervisor.supervise启动组件

5、lifecycleAware.start()启动组件服务

总之不管是配置文件只加载一次，还是定时加载启动，都会调用lifecycleAware.start()来启动各个组件服务

    MonitorRunnable monitorRunnable = new MonitorRunnable();
    monitorRunnable.lifecycleAware = lifecycleAware;
    monitorRunnable.supervisoree = process;
    monitorRunnable.monitorService = monitorService;

    supervisedProcesses.put(lifecycleAware, process);

    ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(
        monitorRunnable, 0, 3, TimeUnit.SECONDS);

这里是在将各个组件服务添加到监控中monitorService，
在monitorRunnable中关键的就是lifecycleAware.start()这个了，它是各个组件启动的关键，各个组件实现了LifecycleAware的start接口，所以这里就是在启动各个组件服务。