Flume 1.9.0 源码解析 : 一篇文章弄清Flume 程序启动流程

最新推荐文章于 2024-08-02 10:40:37 发布
最新推荐文章于 2024-08-02 10:40:37 发布
阅读量4.6k 收藏 12
点赞数 8
分类专栏： Flume Flume1.9.0源码解析
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zhanglong_4444/article/details/88571032
版权

 

一 .环境准备.

 

 

1.配置文件:  example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

2.执行命令参数:

 

--conf-file /todo/flume/example.conf --name a1

3. 运行主类:   org.apache.flume.node.Application#main

 

这个类主要有三部分.

1.加载参数

2.设置服务

3.启动服务, 设置监控&钩子

 

代码如图:

加载参数

 

接下来设置服务:

 PollingPropertiesFileConfigurationProvider configurationProvider =
              new PollingPropertiesFileConfigurationProvider(agentName, configurationFile, eventBus, 30);

 

 

启动服务& 监控&钩子

 

建立监控&钩子 我就不细说了, 直接说创建服务了

PropertiesFileConfigurationProvider configurationProvider =
    new PropertiesFileConfigurationProvider(agentName, configurationFile);

 

 

PollingPropertiesFileConfigurationProvider 继承了 :PropertiesFileConfigurationProvider 继承了 : AbstractConfigurationProvider

 

在抽象类:AbstractConfigurationProvider

private final String agentName;
private final SourceFactory sourceFactory;
private final SinkFactory sinkFactory;
private final ChannelFactory channelFactory;

分别声明了Source , Sink , Channel 工厂.

在构造方法中进行了初始化

 

Source , Sink , Channel 默认分别实现自己对应的工厂接口.

核心的方法是   create(String name, String type) 方法

其实就是根据 type 实例不同的Source , Sink , Channel 实例.

代码如下:

org.apache.flume.source.DefaultSourceFactory#create

 

 

org.apache.flume.channel.DefaultChannelFactory#create

org.apache.flume.sink.DefaultSinkFactory#create

 

到这里的时候, 服务只是创建了, 在哪里启动的呢???

application.start();

 

 

直接跟进去

org.apache.flume.node.Application#start

 

org.apache.flume.lifecycle.LifecycleSupervisor#supervise

monitorService: 这个是一个线程池用于执行程序,调用 start 方法:  ScheduledThreadPoolExecutor ( 在类初始化的时候创建. )

 

org.apache.flume.lifecycle.LifecycleSupervisor#run
 

 

其实就是调用: 

org.apache.flume.node.PollingPropertiesFileConfigurationProvider#start
 

 

会调用org.apache.flume.node.FileWatcherRunnable#run 方法

 

接下来主要看 org.apache.flume.node.AbstractConfigurationProvider#getConfiguration 方法.

这个方法分两步.

1.读取配置文件.

2.生成 source、channel、sink 对应的 runner . 加入到 MaterializedConfiguration 中

 

 

第一步. 读取配置文件.

FlumeConfiguration fconfig = getFlumeConfiguration();

 

org.apache.flume.node.PropertiesFileConfigurationProvider#getFlumeConfiguration

 

第二步: 生成 source、channel、sink 对应的 runner . 加入到 MaterializedConfiguration 中

 

主要是根据配置文件生成 runner 这部分需要细说一下.

这里会生成   Channels, Sources , Sinks 三种 runner 

loadChannels(agentConf, channelComponentMap);
loadSources(agentConf, channelComponentMap, sourceRunnerMap);
loadSinks(agentConf, channelComponentMap, sinkRunnerMap);

分别说明:

1.   loadChannels(agentConf, channelComponentMap);

 

这个方法主要干了四件事件事.

1.将缓存的 channel 加入到一个  ListMultimap<Class<? extends Channel>, String>集合中

2. 创建具有ComponentConfiguration 对象的Channel 实例

3. 创建没有ComponentConfiguration 对象, 但是配置 context 的Channel 实例

4.将缓存中的 channel 与新生成的 channel 做匹配, 去掉配置项中没有的 channel

 

 

private void loadChannels(AgentConfiguration agentConf,
    Map<String, ChannelComponent> channelComponentMap)
        throws InstantiationException {
  LOGGER.info("Creating channels");

  //todo 缓存中的 channel
  ListMultimap<Class<? extends Channel>, String> channelsNotReused =
      ArrayListMultimap.create();
  // assume all channels will not be re-used
  for (Map.Entry<Class<? extends Channel>, Map<String, Channel>> entry :
       channelCache.entrySet()) {
    Class<? extends Channel> channelKlass = entry.getKey();
    Set<String> channelNames = entry.getValue().keySet();
    channelsNotReused.get(channelKlass).addAll(channelNames);
  }

  Set<String> channelNames = agentConf.getChannelSet();
  Map<String, ComponentConfiguration> compMap = agentConf.getChannelConfigMap();


  for (String chName : channelNames) {
    ComponentConfiguration comp = compMap.get(chName);
    if (comp != null) {

      // todo 使用工厂类创建Channel
      Channel channel = getOrCreateChannel(channelsNotReused,
          comp.getComponentName(), comp.getType());

      try {

        //todo 更新配置 , 因为 channelComponentMap 刚开始传进来的时候是空值
        Configurables.configure(channel, comp);
        channelComponentMap.put(comp.getComponentName(),
            new ChannelComponent(channel));
        LOGGER.info("Created channel " + chName);


      } catch (Exception e) {
        String msg = String.format("Channel %s has been removed due to an " +
            "error during configuration", chName);
        LOGGER.error(msg, e);
      }
    }
  }



  //todo 组合没有 ComponentConfiguration配置, 仅仅使用Context的对象.
  for (String chName : channelNames) {
    Context context = agentConf.getChannelContext().get(chName);
    if (context != null) {
      // todo 使用工厂类创建Channel
      Channel channel = getOrCreateChannel(channelsNotReused, chName,
          context.getString(BasicConfigurationConstants.CONFIG_TYPE));

      try {

        // todo 更新配置 , 因为 channelComponentMap 刚开始传进来的时候是空值
        Configurables.configure(channel, context);
        channelComponentMap.put(chName, new ChannelComponent(channel));

        LOGGER.info("Created channel " + chName);
      } catch (Exception e) {
        String msg = String.format("Channel %s has been removed due to an " +
            "error during configuration", chName);
        LOGGER.error(msg, e);
      }
    }
  }

  for (Class<? extends Channel> channelKlass : channelsNotReused.keySet()) {
    Map<String, Channel> channelMap = channelCache.get(channelKlass);
    if (channelMap != null) {
      for (String channelName : channelsNotReused.get(channelKlass)) {
        if (channelMap.remove(channelName) != null) {
          LOGGER.info("Removed {} of type {}", channelName, channelKlass);
        }
      }
      if (channelMap.isEmpty()) {
        //todo 有一些 channel 在配置中没有重新使用, 将会将其从缓存中移除.
        channelCache.remove(channelKlass);

      }
    }
  }
}

 

 

private Channel getOrCreateChannel(
    ListMultimap<Class<? extends Channel>, String> channelsNotReused,
    String name, String type)
    throws FlumeException {

  // todo 根据传入的类型, 获取对应的类
  Class<? extends Channel> channelClass = channelFactory.getClass(type);

  /*
   * Channel has requested a new instance on each re-configuration
   * todo 根据新的配置, 实例化对象.
   */
  //todo 如何类的注解 Disposable 存在, 则直接进行实例化,并返回  只有 jdbc 和 file 模式用到了
  if (channelClass.isAnnotationPresent(Disposable.class)) {
    Channel channel = channelFactory.create(name, type);
    channel.setName(name);
    return channel;
  }

  Map<String, Channel> channelMap = channelCache.get(channelClass);

  //todo 如果缓存中不存在 channel 的话, 那么直接加入缓存.
  if (channelMap == null) {
    channelMap = new HashMap<String, Channel>();
    channelCache.put(channelClass, channelMap);
  }
  //todo 如果channelMap 中的 channel 为 null ,使用工厂类创建.
  Channel channel = channelMap.get(name);
  if (channel == null) {
    channel = channelFactory.create(name, type);
    channel.setName(name);
    channelMap.put(name, channel);
  }
  //todo 如果缓存中已经存在对应的 channel 的话,那么移除它, 后续的方法会更新它 .
  channelsNotReused.get(channelClass).remove(name);
  return channel;
}

 

 

2.loadSources(agentConf, channelComponentMap, sourceRunnerMap);

读取配置文件生成 source , 然后创建 sourceRunner, 并注册到 channel

 

private void loadSources(AgentConfiguration agentConf,
    Map<String, ChannelComponent> channelComponentMap,
    Map<String, SourceRunner> sourceRunnerMap)
    throws InstantiationException {

  Set<String> sourceNames = agentConf.getSourceSet();
  Map<String, ComponentConfiguration> compMap =
      agentConf.getSourceConfigMap();
  /*
   * Components which have a ComponentConfiguration object
   *
   * todo 组合配置ComponentConfiguration 的对象
   */
  for (String sourceName : sourceNames) {
    ComponentConfiguration comp = compMap.get(sourceName);
    if (comp != null) {

      SourceConfiguration config = (SourceConfiguration) comp;

      //todo 使用sourceFactory 直接采用根据类型,采用反射方式  实例化 source
      Source source = sourceFactory.create(comp.getComponentName(),
          comp.getType());

      try {

        //todo 为 source 匹配对应的 channel
        Configurables.configure(source, config);
        Set<String> channelNames = config.getChannels();


        List<Channel> sourceChannels =
                getSourceChannels(channelComponentMap, source, channelNames);

        if (sourceChannels.isEmpty()) {
          String msg = String.format("Source %s is not connected to a " +
              "channel",  sourceName);
          throw new IllegalStateException(msg);
        }


        //todo  创建 selector .
        ChannelSelectorConfiguration selectorConfig =
            config.getSelectorConfiguration();


        ChannelSelector selector = ChannelSelectorFactory.create(
            sourceChannels, selectorConfig);

        ChannelProcessor channelProcessor = new ChannelProcessor(selector);
        Configurables.configure(channelProcessor, config);


        source.setChannelProcessor(channelProcessor);



        //todo 将 source 转换为 SourceRunner  !!!!!!!!!!!!!!!1
        sourceRunnerMap.put(comp.getComponentName(),
            SourceRunner.forSource(source) );

        //todo source 向 channel 反向注册 sourceName
        for (Channel channel : sourceChannels) {
          ChannelComponent channelComponent =
              Preconditions.checkNotNull(channelComponentMap.get(channel.getName()),
                                         String.format("Channel %s", channel.getName()));
          channelComponent.components.add(sourceName);

        }
      } catch (Exception e) {
        String msg = String.format("Source %s has been removed due to an " +
            "error during configuration", sourceName);
        LOGGER.error(msg, e);
      }
    }
  }
  /*
   * Components which DO NOT have a ComponentConfiguration object
   * and use only Context
   *
   * todo 组合没有配置 ComponentConfiguration 但是使用 context  的对象
   */
  Map<String, Context> sourceContexts = agentConf.getSourceContext();
  for (String sourceName : sourceNames) {
    Context context = sourceContexts.get(sourceName);
    if (context != null) {

      //todo 直接采用根据类型,采用反射方式  实例化 source
      Source source =
          sourceFactory.create(sourceName,
                               context.getString(BasicConfigurationConstants.CONFIG_TYPE));


      try {
        Configurables.configure(source, context);
        String[] channelNames = context.getString(
            BasicConfigurationConstants.CONFIG_CHANNELS).split("\\s+");

        //todo 为 source 匹配对应的 channel
        List<Channel> sourceChannels =
                getSourceChannels(channelComponentMap, source, Arrays.asList(channelNames));


        if (sourceChannels.isEmpty()) {
          String msg = String.format("Source %s is not connected to a " +
              "channel",  sourceName);
          throw new IllegalStateException(msg);
        }

        //todo  创建 selector .
        Map<String, String> selectorConfig = context.getSubProperties(
            BasicConfigurationConstants.CONFIG_SOURCE_CHANNELSELECTOR_PREFIX);

        ChannelSelector selector = ChannelSelectorFactory.create(
            sourceChannels, selectorConfig);

        ChannelProcessor channelProcessor = new ChannelProcessor(selector);
        Configurables.configure(channelProcessor, context);
        source.setChannelProcessor(channelProcessor);


        //todo 将 source 转换为 SourceRunner  !!!!!!!!!!!!!!!1
        sourceRunnerMap.put(sourceName,
            SourceRunner.forSource(source));

        //todo source 向 channel 反向注册 sourceName
        for (Channel channel : sourceChannels) {
          ChannelComponent channelComponent =
              Preconditions.checkNotNull(channelComponentMap.get(channel.getName()),
                                         String.format("Channel %s", channel.getName()));
          channelComponent.components.add(sourceName);
        }
      } catch (Exception e) {
        String msg = String.format("Source %s has been removed due to an " +
            "error during configuration", sourceName);
        LOGGER.error(msg, e);
      }
    }
  }
}

 

 

 

3.loadSinks(agentConf, channelComponentMap, sinkRunnerMap);

读取配置文件生成 sink , 并注册到 channel , 然后根据分组情况 sinkRunner, 未设置分组的,单独创建sinkRunner

 

 

private void loadSinks(AgentConfiguration agentConf,
    Map<String, ChannelComponent> channelComponentMap, Map<String, SinkRunner> sinkRunnerMap)
    throws InstantiationException {


  Set<String> sinkNames = agentConf.getSinkSet();
  Map<String, ComponentConfiguration> compMap =
      agentConf.getSinkConfigMap();
  Map<String, Sink> sinks = new HashMap<String, Sink>();
  /*
   * Components which have a ComponentConfiguration object
   * todo 组合配置ComponentConfiguration 的对象
   */
  for (String sinkName : sinkNames) {
    ComponentConfiguration comp = compMap.get(sinkName);
    if (comp != null) {

      //todo 使用SinkFactory 直接采用根据类型,采用反射方式  实例化 Sink
      SinkConfiguration config = (SinkConfiguration) comp;
      Sink sink = sinkFactory.create(comp.getComponentName(), comp.getType());


      try {

        //todo 为 Sink 匹配对应的 channel
        Configurables.configure(sink, config);
        ChannelComponent channelComponent = channelComponentMap.get(config.getChannel());
        if (channelComponent == null) {
          String msg = String.format("Sink %s is not connected to a " +
              "channel",  sinkName);
          throw new IllegalStateException(msg);
        }

        //todo 检查 channel 是否可用 :  sink 的 batch size 要小于 channel 的 transaction capacity
        checkSinkChannelCompatibility(sink, channelComponent.channel);

        sink.setChannel(channelComponent.channel);

        sinks.put(comp.getComponentName(), sink);

        //todo Sink 向 channel 反向注册 SinkName
        channelComponent.components.add(sinkName);
      } catch (Exception e) {
        String msg = String.format("Sink %s has been removed due to an " +
            "error during configuration", sinkName);
        LOGGER.error(msg, e);
      }
    }
  }
  /*
   * Components which DO NOT have a ComponentConfiguration object
   * and use only Context
   * todo 组合没有配置 ComponentConfiguration 但是使用 context  的对象
   */
  Map<String, Context> sinkContexts = agentConf.getSinkContext();
  for (String sinkName : sinkNames) {
    Context context = sinkContexts.get(sinkName);
    if (context != null) {

      //todo 直接采用根据类型,采用反射方式  实例化 Sink
      Sink sink = sinkFactory.create(sinkName, context.getString(
          BasicConfigurationConstants.CONFIG_TYPE));


      try {

        //todo 为 Sink 匹配对应的 channel
        Configurables.configure(sink, context);
        ChannelComponent channelComponent =
            channelComponentMap.get(
                context.getString(BasicConfigurationConstants.CONFIG_CHANNEL));
        if (channelComponent == null) {
          String msg = String.format("Sink %s is not connected to a " +
              "channel",  sinkName);
          throw new IllegalStateException(msg);
        }
        //todo 检查 channel 是否可用 :  sink 的 batch size 要大于 channel 的 transaction capacity
        checkSinkChannelCompatibility(sink, channelComponent.channel);


        sink.setChannel(channelComponent.channel);
        sinks.put(sinkName, sink);
        channelComponent.components.add(sinkName);

      } catch (Exception e) {
        String msg = String.format("Sink %s has been removed due to an " +
            "error during configuration", sinkName);
        LOGGER.error(msg, e);
      }
    }
  }

  //todo 对 sink 进行分组
  loadSinkGroups(agentConf, sinks, sinkRunnerMap);
}

 

private void loadSinkGroups(AgentConfiguration agentConf,
    Map<String, Sink> sinks, Map<String, SinkRunner> sinkRunnerMap)
        throws InstantiationException {


  // todo 获取配置中的 group 分组
  Set<String> sinkGroupNames = agentConf.getSinkgroupSet();


  Map<String, ComponentConfiguration> compMap =
      agentConf.getSinkGroupConfigMap();

  Map<String, String> usedSinks = new HashMap<String, String>();

  for (String groupName: sinkGroupNames) {

    ComponentConfiguration comp = compMap.get(groupName);

    if (comp != null) {
      SinkGroupConfiguration groupConf = (SinkGroupConfiguration) comp;
      List<Sink> groupSinks = new ArrayList<Sink>();
      for (String sink : groupConf.getSinks()) {
        Sink s = sinks.remove(sink);
        if (s == null) {
          String sinkUser = usedSinks.get(sink);
          if (sinkUser != null) {
            throw new InstantiationException(String.format(
                "Sink %s of group %s already " +
                    "in use by group %s", sink, groupName, sinkUser));
          } else {
            throw new InstantiationException(String.format(
                "Sink %s of group %s does "
                    + "not exist or is not properly configured", sink,
                    groupName));
          }
        }
        groupSinks.add(s);
        usedSinks.put(sink, groupName);
      }
      try {

        SinkGroup group = new SinkGroup(groupSinks);
        Configurables.configure(group, groupConf);

        //todo 创建 sinkRunner
        sinkRunnerMap.put(comp.getComponentName(),
            new SinkRunner(group.getProcessor()));

      } catch (Exception e) {
        String msg = String.format("SinkGroup %s has been removed due to " +
            "an error during configuration", groupName);
        LOGGER.error(msg, e);
      }
    }
  }

  // add any unassigned sinks to solo collectors
  // todo 对未分组的 sink 进行处理
  for (Entry<String, Sink> entry : sinks.entrySet()) {
    if (!usedSinks.containsValue(entry.getKey())) {
      try {

        SinkProcessor pr = new DefaultSinkProcessor();
        List<Sink> sinkMap = new ArrayList<Sink>();
        sinkMap.add(entry.getValue());
        pr.setSinks(sinkMap);
        Configurables.configure(pr, new Context());

        //todo 创建 SinkRunner
        sinkRunnerMap.put(entry.getKey(), new SinkRunner(pr));
        
      } catch (Exception e) {
        String msg = String.format("SinkGroup %s has been removed due to " +
            "an error during configuration", entry.getKey());
        LOGGER.error(msg, e);
      }
    }
  }


}

再回到方法:  org.apache.flume.node.AbstractConfigurationProvider#getConfiguration

 

将  Channels, Sources , Sinks 具体的Runner 加载进配置. 直接返回配置.

在回到上一级方法: org.apache.flume.node.PollingPropertiesFileConfigurationProvider#run
 

 

eventBus.post(getConfiguration());

 

eventBus放松的信息去哪了呢? 干了啥??????

 

 org.apache.flume.node.Application#handleConfigurationEvent
 

这个方法添加了 @Subscribe 注解.
 

 

直接看开启所有组件服务吧. org.apache.flume.node.Application#startAllComponents

 

private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
  logger.info("Starting new configuration:{}", materializedConfiguration);

  this.materializedConfiguration = materializedConfiguration;


  // todo  启动所有的 channel
  for (Entry<String, Channel> entry :

      materializedConfiguration.getChannels().entrySet()) {
    try {
      logger.info("Starting Channel " + entry.getKey());

      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);

    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }

  /*
   * Wait for all channels to start.
   * todo 等待所有 channels 启动
   */
  for (Channel ch : materializedConfiguration.getChannels().values()) {
    while (ch.getLifecycleState() != LifecycleState.START
        && !supervisor.isComponentInErrorState(ch)) {
      try {
        logger.info("Waiting for channel: " + ch.getName() +
            " to start. Sleeping for 500 ms");
        Thread.sleep(500);
      } catch (InterruptedException e) {
        logger.error("Interrupted while waiting for channel to start.", e);
        Throwables.propagate(e);
      }
    }
  }

  // todo  启动所有的 sink
  for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
    try {
      logger.info("Starting Sink " + entry.getKey());
      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }

  // todo  启动所有的 source
  for (Entry<String, SourceRunner> entry :
       materializedConfiguration.getSourceRunners().entrySet()) {
    try {
      logger.info("Starting Source " + entry.getKey());
      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }
  
  
  //todo 加载监控

  this.loadMonitoring();
}

 

那么 Channels, Sources , Sinks 是怎么启动的呢?

其实就是用一个线程池,来启动 Channels Sources , Sinks 的Runner 实例.

 

public synchronized void supervise(LifecycleAware lifecycleAware,
    SupervisorPolicy policy, LifecycleState desiredState) {
  if (this.monitorService.isShutdown()
      || this.monitorService.isTerminated()
      || this.monitorService.isTerminating()) {
    throw new FlumeException("Supervise called on " + lifecycleAware + " " +
        "after shutdown has been initiated. " + lifecycleAware + " will not" +
        " be started");
  }

  Preconditions.checkState(!supervisedProcesses.containsKey(lifecycleAware),
      "Refusing to supervise " + lifecycleAware + " more than once");

  if (logger.isDebugEnabled()) {
    logger.debug("Supervising service:{} policy:{} desiredState:{}",
        new Object[] { lifecycleAware, policy, desiredState });
  }

  Supervisoree process = new Supervisoree();
  process.status = new Status();

  process.policy = policy;
  process.status.desiredState = desiredState;
  process.status.error = false;

  MonitorRunnable monitorRunnable = new MonitorRunnable();

  //todo lifecycleAware 这个就是  PollingPropertiesFileConfigurationProvider 最终要调用里面的 start 方法!!!
  monitorRunnable.lifecycleAware = lifecycleAware;



  monitorRunnable.supervisoree = process;
  monitorRunnable.monitorService = monitorService;

  supervisedProcesses.put(lifecycleAware, process);

  // todo      交由线程池 启动服务.
  // todo      monitorService 是一个线程池,在对象创建的时候初始化
  ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(
      monitorRunnable, 0, 3, TimeUnit.SECONDS);

  monitorFutures.put(lifecycleAware, future);
}

 

最后我们分别看一下 Sources , Sinks  的Runner 实例是啥样的吧.

 

  

创建 sourceRunner  : org.apache.flume.SourceRunner#forSource

/**
 * Static factory method to instantiate a source runner implementation that
 * corresponds to the type of {@link Source} specified.
 *
 * @param source The source to run
 * @return A runner that can run the specified source
 * @throws IllegalArgumentException if the specified source does not implement
 * a supported derived interface of {@link SourceRunner}.
 */
public static SourceRunner forSource(Source source) {
  SourceRunner runner = null;

  if (source instanceof PollableSource) {
    runner = new PollableSourceRunner();
    ((PollableSourceRunner) runner).setSource((PollableSource) source);
  } else if (source instanceof EventDrivenSource) {
    runner = new EventDrivenSourceRunner();
    ((EventDrivenSourceRunner) runner).setSource((EventDrivenSource) source);
  } else {
    throw new IllegalArgumentException("No known runner type for source "
        + source);
  }

  return runner;
}

 

Source 分两种: 

EventDrivenSource  和  PollableSource

 

PollableSource:   持续运行的那种. 比如 TaildirSource 要持续监控指定的文件 , 会在 start 方法里面创建线程. 

具体关注:

org.apache.flume.source.PollableSourceRunner.start

 

 org.apache.flume.source.PollingRunner

 


 

 

EventDrivenSource:  针对事件 , 且只执行一次.

 

 

创建 SinkRunner  : org.apache.flume.SinkRunner

 

 

 

总结来说, Flume 其实就是一个生产者,消费者模式的一个变形.

 

source : 生产者 

sink : 消费者

channel : 缓存.
 

 

目前先写到这里, 有什么不正确的地方,欢迎大家指正, 不胜感激..............

 

 

 

 

 

 

 

 

张伯毅
关注
专栏目录
Apache Flume1.9.0源码编译
zhengzaifeidelushang的博客
04-13 489
Apache Flume1.9.0源码编译一、注释flume-ng-morphline-solr-sink模块二、在Apahce Flume的pom.xml文件中增加阿里云和CDH的repository三、命令行打包Apache Flume1.9.0四、编译过程可能遇到的报错解决方法错误1:Cannot resolve ua_parser:ua-parser:1.3.0错误2:group id '656741719' is too big ( > 2097151 ).错误3：Could not tra
flume 1.9 源码阅读(三)
Cynric的专栏
11-11 479
Flume-NG Configuration 上一节，看完了flume启动脚本，其中里面解析了各种用户传入的参数，并将其作为参数传给了 org.apache.flume.node.Application。 本节从 Application 入口类了解一下 flume-ng 如何启动。以及 flume-ng-configuration 模块是如何解析配置文件的。 org.apache.flume.no...
Flume1.9.0配置与简单使用
最新发布
taogumo的专栏
08-02 276
将apache-flume-1.9.0-bin.tar.gz软件包上传至/usr/local/soft文件夹下。#分别指定agent的sources，sinks, channels的名称 名称可以自定义。客户端(Source)：收集信息。-c:配置文件路径conf。-n:agent名称。
flume 1.9 源码阅读(一)
Cynric的专栏
10-29 812
Flume源码阅读Flume NG和 Flume OG 相比Flume 源码目录结构Flume 结构数据流模型 参考: https://www.jianshu.com/p/0187459831af http://flume.apache.org/FlumeUserGuide.html Flume NG Flume 是一个分布式的、可靠的、高效的日志采集、聚合、转移系统。他可以从多个不同的Sourc...
Flume-NG启动过程源码分析（一）(原创)
weixin_30768661的博客
04-16 130
　　从bin/flume 这个shell脚本可以看到Flume的起始于org.apache.flume.node.Application类，这是flume的main函数所在。 　　main方法首先会先解析shell命令，如果指定的配置文件不存在就甩出异常。 　　根据命令中含有"no-reload-conf"参数，决定采用那种加载配置文件方式：一、没有此参数，会动态加载配置文件，默认每30秒加载...
flume源码
大学之道
08-01 844
http://www.cnblogs.com/lxf20061900/p/3667824.html
Flume 1.9.0用户指南
张伯毅的专栏
03-14 9245
概述 系统要求 架构 数据流模型 复杂的流程 可靠性 可恢复性 设置 设置 agent 配置单个组件 将各个部分连接在一起 启动 agent 一个简单的例子 在配置文件中使用环境变量 记录原始数据 基于Zookeeper的配置 plugins.d目录 插件的目录布局 数据摄取 RPC 网络流 设置多 agent 流程 多路复用流程 配置 定义...
flume源码阅读-总体介绍
Norman_Li的技术点滴
04-02 716
flume源码阅读之总体介绍 Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。 Flume官网地址: Flume ， 源代码地址:source为什么读flume源代码 为什么要去读flume的源码，最近公司的
apache-flume-1.9.0-bin.tar.gz
07-28
描述中的 "编译好的flume1.9.0，下载安装即可使用" 提醒我们，该文件包含的 Flume 实例是已经准备好运行的，只需遵循安装步骤，就可以在你的系统上快速部署和启动 Flume 服务。 在标签中，"flume" 和 "flume1.9.0" ...
Flume1.9.0配置.docx
09-14
Flume 是一个分布式、可靠、可扩展的日志聚合系统，可以从各种数据源中收集数据，并将其传输到目标系统中，本文将对 Flume 1.9.0 的配置进行详细的介绍。 一、Flume 的安装和环境配置 在安装 Flume 之前，需要安装...
apache-flume-1.9.0-bin.tar.zip
10-07
4. **配置与拓扑**：Flume通过配置文件定义数据流的拓扑结构，用户可以创建多个源头、通道和接收器的组合，形成复杂的流处理链路。这些配置可以动态调整，以适应变化的数据处理需求。 5. **容错与高可用性**：Flume...
flume1.7.0源码
05-19
flume源码
flume1.9.0用户中文指南.pdf
08-14
Apache Flume 1.9.0用户中文指南.pdf。flume是一个分布式，可靠且可用的系统，用于有效地从许多不同的数据源收集、聚合和移动大量日志数据到集中式数据存储。 官网翻译版
Flume1.9.0配置
m0_64900521的博客
12-10 2098
一、把安装包拉到/opt/software/下面 把apache-flume-1.9.0-bin.tar.gz的安装包拉到/opt/software/下面 二、解压 cd /usr/local/soft tar -zxvf apache-flume-1.9.0-bin.tar.gz-C /opt/module/ 三、添加系统环境变量(需配置Java JDK系统参数) 切换到cd /opt/module/下面 vi /etc/profile exp...
Flume源码分析(Flume Source Code Analysis)
程序员光剑
08-31 1600
作者：禅与计算机程序设计艺术 1.简介 Apache Flume是一个开源、分布式、高可用的海量日志采集、聚合和传输系统。它最初起源于Yahoo！公司，由开发者在2007年创建，主要用于收集网站访问日志并将其传送到HDFS上。目前Flume已经成为Apache顶级项目，并已经成为大数据处理中必不可少的组
flume【源码分析】分析Flume的拦截器
weixin_34023863的博客
12-29 167
flume-拦截器 　　有的时候希望通过Flume将读取的文件再细分存储，比如讲source的数据按照业务类型分开存储，具体一点比如类似：将source中web、wap、media等的内容分开存储；比如丢弃或修改一些数据。这时可以考虑使用拦截器Interceptor。 　　flume通过拦截器实现修改和丢弃事件的功能。拦截器通过定义类继承org.apache.flume.intercept...
Flume 1.7 源码分析（二）整体架构
Lnho的专栏
02-27 4160
Flume 1.7 源码分析（一）源码编译 Flume 1.7 源码分析（二）整体架构 3 整体架构Flume有三大组件：Source、Channel、Sink。 Source就是数据来源，例如Web Server产生日志后，可使用ExecSource执行tail -F命令后不断监听日志文件新生成的数据，然后传给Channel。 Channel就是一个缓存队列，由于读取数据和写入数据的速
16_Flume之源码分析
weixin_40727028的博客
04-16 390
随后，这个轮询线程会通过 Guava 的 EventBus 机制通知 Application 类配置发生了更新，从而触发 Application#handleConfigurationEvent 方法，重新加载所有的组件。组件的start方法是在 MonitorRunnable 的run方法中调用的，MonitorRunnable 是一个线程，run方法中会获取当前组件的状态(默认STOP)以及预期的状态(START)，如果不相等则通过预期的状态执行相应的逻辑（ETCD直呼内行~~~）
flume源码分析
亚瑟的守护的专栏
10-08 2576
1、LifecycleAware public interface LifecycleAware {    public void start();    public void stop();    public LifecycleState getLifecycleState();  } 所有核心组件，全都实现这个接口。通过start，stop启动关闭，通过
评论 11
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值