flume 源码解读一

通过查看flume-ng.sh 脚本可以知道 flume 首先启动是哪个类？

//位于 flume-ng-node 包中
org.apache.flume.node.Application

1、 flume 有三大模块： source 、 channel 、 sink 。在三个又有 选择器、拦截器等，来自官网的图例。

 

 

2、如何启动一个agent代理，拿官网的例子使用，我们的重点说明在源码中是如何调用的。

$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template

 我们来看下参数的获取，先来读取main函数

 public static void main(String[] args) {

    try {
      SSLUtil.initGlobalSSLParameters();

      Options options = new Options();

      Option option = new Option("n", "name", true, "the name of this agent");
      option.setRequired(true);
      options.addOption(option);

      option = new Option("f", "conf-file", true,
          "specify a config file (required if -z missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option(null, "no-reload-conf", false,
          "do not reload config file if changed");
      options.addOption(option);

      // Options for Zookeeper
      option = new Option("z", "zkConnString", true,
          "specify the ZooKeeper connection to use (required if -f missing)");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("p", "zkBasePath", true,
          "specify the base path in ZooKeeper for agent configs");
      option.setRequired(false);
      options.addOption(option);

      option = new Option("h", "help", false, "display help text");
      options.addOption(option);

      CommandLineParser parser = new GnuParser();
      CommandLine commandLine = parser.parse(options, args);

      if (commandLine.hasOption('h')) {
        new HelpFormatter().printHelp("flume-ng agent", options, true);
        return;
      }

      String agentName = commandLine.getOptionValue('n');
      boolean reload = !commandLine.hasOption("no-reload-conf");

这里是获取的shell脚本的命令行，并进行参数解读，包含了简写和全写的处理。

//.... 上面有省略代码，我们重点看下这段代码
PropertiesFileConfigurationProvider configurationProvider =
              new PropertiesFileConfigurationProvider(agentName, configurationFile);
          application = new Application();
          application.handleConfigurationEvent(configurationProvider.getConfiguration());

可以了解到 PropertiesFileConfigurationProvider  类的构造函数接收两个参数，一个是agentName 和configurationFile.

configurationFile 这个实例类做了什么呢？

在从命名可以看出是我们配置的文件，例如：

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

获取我们创建的实例文件，我们接着看PropertiesFileConfigurationProvider的构造方法。

public PropertiesFileConfigurationProvider(String agentName, File file) {
    super(agentName);
    this.file = file;
  }

里面只是把配置文件对象传个 PropertiesFileConfigurationProvider 的私有属性 file 。

application = new Application();
          application.handleConfigurationEvent(configurationProvider.getConfiguration());

在段代码中看到 application 实例定义了 handleConfigurationEvent 的方法。记得flume 中基本单位吗？ event。

这段代码做了一下事情

1、读取了配置文件，并序列map键值对。 FlumeConfiguration fconfig = getFlumeConfiguration();

2、获取代理，加载相应的source、channel、sink 类。

3、返回配置好的MaterializedConfiguration 类。

// 这个方法返回了一个MaterializedConfiguration 配置类
public MaterializedConfiguration getConfiguration() {
    MaterializedConfiguration conf = new SimpleMaterializedConfiguration();
    FlumeConfiguration fconfig = getFlumeConfiguration();
    AgentConfiguration agentConf = fconfig.getConfigurationFor(getAgentName());
    if (agentConf != null) {
      Map<String, ChannelComponent> channelComponentMap = Maps.newHashMap();
      Map<String, SourceRunner> sourceRunnerMap = Maps.newHashMap();
      Map<String, SinkRunner> sinkRunnerMap = Maps.newHashMap();
      try {
        //这里确定了agent 的source、channel、sink的关系。
        loadChannels(agentConf, channelComponentMap);
        loadSources(agentConf, channelComponentMap, sourceRunnerMap);
        loadSinks(agentConf, channelComponentMap, sinkRunnerMap);
        Set<String> channelNames = new HashSet<String>(channelComponentMap.keySet());
        for (String channelName : channelNames) {
          ChannelComponent channelComponent = channelComponentMap.get(channelName);
          if (channelComponent.components.isEmpty()) {
            LOGGER.warn(String.format("Channel %s has no components connected" +
                " and has been removed.", channelName));
            channelComponentMap.remove(channelName);
            Map<String, Channel> nameChannelMap =
                channelCache.get(channelComponent.channel.getClass());
            if (nameChannelMap != null) {
              nameChannelMap.remove(channelName);
            }
          } else {
            LOGGER.info(String.format("Channel %s connected to %s",
                channelName, channelComponent.components.toString()));
            conf.addChannel(channelName, channelComponent.channel);
          }
        }
        for (Map.Entry<String, SourceRunner> entry : sourceRunnerMap.entrySet()) {
          conf.addSourceRunner(entry.getKey(), entry.getValue());
        }
        for (Map.Entry<String, SinkRunner> entry : sinkRunnerMap.entrySet()) {
          conf.addSinkRunner(entry.getKey(), entry.getValue());
        }
      } catch (InstantiationException ex) {
        LOGGER.error("Failed to instantiate component", ex);
      } finally {
        channelComponentMap.clear();
        sourceRunnerMap.clear();
        sinkRunnerMap.clear();
      }
    } else {
      LOGGER.warn("No configuration found for this host:{}", getAgentName());
    }
    return conf;
  }

最后看下 application.handleConfigurationEvent 方法做了哪些事情？

 1、为处理的添加lifecycleLock.lockInterruptibly(); 获取锁。

2、停止所有的组件。

3、开启所有的组件。

包含了我们代理设定的。

 @Subscribe
  public void handleConfigurationEvent(MaterializedConfiguration conf) {
    try {
      lifecycleLock.lockInterruptibly();
      stopAllComponents();
      startAllComponents(conf);
    } catch (InterruptedException e) {
      logger.info("Interrupted while trying to handle configuration event");
      return;
    } finally {
      // If interrupted while trying to lock, we don't own the lock, so must not attempt to unlock
      if (lifecycleLock.isHeldByCurrentThread()) {
        lifecycleLock.unlock();
      }
    }
  }

flume 的启动过程完。

遗留问题：

1、flume 是如何自动重启的？

2、flume中的选择器、拦截器是在何时执行的？

下次有时间在慢慢聊。。。