【大数据】Flume源码分析之启动过程

要分析flume的源码,首先要从Flume的启动开始,然后再分析Flume的各个组件服务Source、Channel、Sink的源码,接下来我们就来看下Flume是如何启动的吧

Flume的启动入口

org.apache.flume.node.Application

在这里插入图片描述

1、校验命令行

Options options = new Options();

      Option option = new Option("n", "name", true, "the name of this agent");
      option.setRequired(true);
      options.addOption(option);

      option = new Option("f", "conf-file", true,
          "specify a config file (required if -z missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option(null, "no-reload-conf", false,
          "do not reload config file if changed");
      options.addOption(option);

      // Options for Zookeeper
      option = new Option("z", "zkConnString", true,
          "specify the ZooKeeper connection to use (required if -f missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("p", "zkBasePath", true,
          "specify the base path in ZooKeeper for agent configs");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("h", "help", false, "display help text");
      options.addOption(option);

      CommandLineParser parser = new GnuParser();
      CommandLine commandLine = parser.parse(options, args);

      if (commandLine.hasOption('h')) {
        new HelpFormatter().printHelp("flume-ng agent", options, true);
        return;
      }

      String agentName = commandLine.getOptionValue('n');
      boolean reload = !commandLine.hasOption("no-reload-conf");

      boolean isZkConfigured = false;
      if (commandLine.hasOption('z') || commandLine.hasOption("zkConnString")) {
        isZkConfigured = true;
      }

这一块都是在根据启动参数进行校验

2、配置文件的加载一次,还是定时加载,取决于no-reload-conf

在校验的时候,有关于no-reload-conf的校验,这块比较重要

boolean reload = !commandLine.hasOption("no-reload-conf");

这个实在检查启动参数中是否有no-reload-conf,启动命令中不带–no-reload-conf这个参数,flume会自动加载配置参数 默认是30秒,带则只加载一次
在这里插入图片描述
动态配置使用了PollingPropertiesFileConfigurationProvider,其中有FileWatcherRunnable来具体实现监控配置文件变化,

如果改动时间晚于文件最后修改时间,会触发eventBus.post(),从而触发带@Subscribe注解的handleConfigurationEvent这个方法,进而会stopAllComponents然后startAllComponents,

PollingPropertiesFileConfigurationProvider中使用了线程池executorService.scheduleWithFixedDelay(fileWatcherRunnable, 0, interval,TimeUnit.SECONDS);

fileWatcherRunnable每隔30秒监控变化

3、配置文件的加载

application.handleConfigurationEvent(configurationProvider.getConfiguration());

(1)AbstractConfigurationProvider中的方法getConfiguration,创建channel、source、sink,并加载到配置

在这里插入图片描述
1)loadChannels(agentConf, channelComponentMap);

创建channel,并存入channelComponentMap中

2)loadSources(agentConf, channelComponentMap, sourceRunnerMap);

读取配置文件生成 source , 然后创建 sourceRunner, 并注册到 channel

3)loadSinks(agentConf, channelComponentMap, sinkRunnerMap);

创建sink,并存入sinkRunnerMap中,并注册到channel

(2)handleConfigurationEvent这个方法就是在加载配置文件
@Subscribe这个注解是在监听配置文件是否有变化(是否有eventBus.post()),一旦有则触发这个方法
它是先停掉所有组件服务,然后再重新加载

 @Subscribe
  public void handleConfigurationEvent(MaterializedConfiguration conf) {
    try {
      lifecycleLock.lockInterruptibly();
      stopAllComponents();
      startAllComponents(conf);
    } catch (InterruptedException e) {
      logger.info("Interrupted while trying to handle configuration event");
      return;
    } finally {
      // If interrupted while trying to lock, we don't own the lock, so must not attempt to unlock
      if (lifecycleLock.isHeldByCurrentThread()) {
        lifecycleLock.unlock();
      }
    }
  }

(3)startAllComponents(conf),启动顺序:Channel、Sink、Source

private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
    logger.info("Starting new configuration:{}", materializedConfiguration);

    this.materializedConfiguration = materializedConfiguration;

    for (Entry<String, Channel> entry :
        materializedConfiguration.getChannels().entrySet()) {
      try {
        logger.info("Starting Channel " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    /*
     * Wait for all channels to start.
     */
    for (Channel ch : materializedConfiguration.getChannels().values()) {
      while (ch.getLifecycleState() != LifecycleState.START
          && !supervisor.isComponentInErrorState(ch)) {
        try {
          logger.info("Waiting for channel: " + ch.getName() +
              " to start. Sleeping for 500 ms");
          Thread.sleep(500);
        } catch (InterruptedException e) {
          logger.error("Interrupted while waiting for channel to start.", e);
          Throwables.propagate(e);
        }
      }
    }

    for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
      try {
        logger.info("Starting Sink " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    for (Entry<String, SourceRunner> entry :
         materializedConfiguration.getSourceRunners().entrySet()) {
      try {
        logger.info("Starting Source " + entry.getKey());
        supervisor.supervise(entry.getValue(),
            new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
      } catch (Exception e) {
        logger.error("Error while starting {}", entry.getValue(), e);
      }
    }

    this.loadMonitoring();
  }

4、启动程序application.start();

在这里插入图片描述

配置文件只加载一次,则会主调调用application.handleConfigurationEvent方法,若是定时加载,则是handleConfigurationEvent的注解@Subscribe来监听变化自动调用
在这里插入图片描述

这里是如果定时加载加载配置文件,则components中就是有配置,application.start()方法中就会调用supervisor.supervise启动组件

5、lifecycleAware.start()启动组件服务

总之不管是配置文件只加载一次,还是定时加载启动,都会调用lifecycleAware.start()来启动各个组件服务

    MonitorRunnable monitorRunnable = new MonitorRunnable();
    monitorRunnable.lifecycleAware = lifecycleAware;
    monitorRunnable.supervisoree = process;
    monitorRunnable.monitorService = monitorService;

    supervisedProcesses.put(lifecycleAware, process);

    ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(
        monitorRunnable, 0, 3, TimeUnit.SECONDS);

这里是在将各个组件服务添加到监控中monitorService,
在monitorRunnable中关键的就是lifecycleAware.start()这个了,它是各个组件启动的关键,各个组件实现了LifecycleAware的start接口,所以这里就是在启动各个组件服务。

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值