源码分析Flume启动过程

对Flume-NG的agent启动过程进行详细的分析。 

启动过程

flume的main函数在Application.java中，在flume-ng的shell启动脚本中会用java来起flume：

$EXEC $JAVA_HOME/bin/java $JAVA_OPTS $FLUME_JAVA_OPTS "${arr_java_props[@]}" -cp "$FLUME_CLASSPATH" -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH "$FLUME_APPLICATION_CLASS" $*
 
 
1

在main函数中，会检查一系列参数，最重要的是no-reload-conf，根据reload的不同，判断是否动态加载配置文件，然后start：

List<LifecycleAware> components = Lists.newArrayList();

        if (reload) {
          EventBus eventBus = new EventBus(agentName + "-event-bus");
          PollingPropertiesFileConfigurationProvider configurationProvider =
            new PollingPropertiesFileConfigurationProvider(
              agentName, configurationFile, eventBus, 30);
          components.add(configurationProvider);
          application = new Application(components);
          eventBus.register(application);
        } else {
          PropertiesFileConfigurationProvider configurationProvider =
            new PropertiesFileConfigurationProvider(
              agentName, configurationFile);
          application = new Application();
          application.handleConfigurationEvent(configurationProvider
            .getConfiguration());
        }
        application.start();
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

这里面有很多需要讲解的：

configurationProvider.getConfiguration()会根据配置文件完成source、channel、sink的初始化。

1.1 LifecycleAware接口

实现这个接口的类是有定义好的一系列状态转化的生命周期的：

@InterfaceAudience.Public
@InterfaceStability.Stable
public interface LifecycleAware {

  /**
   * <p>
   * Starts a service or component.
   * </p>
   * <p>
   * Implementations should determine the result of any start logic and effect
   * the return value of {@link #getLifecycleState()} accordingly.
   * </p>
   *
   * @throws LifecycleException
   * @throws InterruptedException
   */
  public void start();

  /**
   * <p>
   * Stops a service or component.
   * </p>
   * <p>
   * Implementations should determine the result of any stop logic and effect
   * the return value of {@link #getLifecycleState()} accordingly.
   * </p>
   *
   * @throws LifecycleException
   * @throws InterruptedException
   */
  public void stop();

  /**
   * <p>
   * Return the current state of the service or component.
   * </p>
   */
  public LifecycleState getLifecycleState();

}
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

比如给的样例：

 * Example usage
 * </p>
 * <code>
 *  public class MyService implements LifecycleAware {
 *
 *    private LifecycleState lifecycleState;
 *
 *    public MyService() {
 *      lifecycleState = LifecycleState.IDLE;
 *    }
 *
 *    @Override
 *    public void start(Context context) throws LifecycleException,
 *      InterruptedException {
 *
 *      ...your code does something.
 *
 *      lifecycleState = LifecycleState.START;
 *    }
 *
 *    @Override
 *    public void stop(Context context) throws LifecycleException,
 *      InterruptedException {
 *
 *      try {
 *        ...you stop services here.
 *      } catch (SomethingException) {
 *        lifecycleState = LifecycleState.ERROR;
 *      }
 *
 *      lifecycleState = LifecycleState.STOP;
 *    }
 *
 *    @Override
 *    public LifecycleState getLifecycleState() {
 *      return lifecycleState;
 *    }
 *
 *  }
 * </code>
 */
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

每一个flume node，每一个flume component(source, channel, sink)都实现了这个接口。

1.2 EventBus事件监听和发布订阅模式

回到上面，再往前判断是否设置了no-reload-conf，如果设置了，会新建一个EventBus，它是Guava(Google写的java库)的事件处理机制，是设计模式中的观察者模式（生产/消费者编程模型）的优雅实现。

使用Guava之后, 如果要订阅消息, 就不用再继承指定的接口, 只需要在指定的方法上加上@Subscribe注解即可。

我们看flume是如何用的：

PollingPropertiesFileConfigurationProvider configurationProvider =
            new PollingPropertiesFileConfigurationProvider(
              agentName, configurationFile, eventBus, 30);
 
 
1
2
3

将eventBus传给PollingPropertiesFileConfigurationProvider，一方面它继承了PropertiesFileConfigurationProvider类，说明它是配置文件的提供者，另一方面，它实现了LifecycleAware接口，说明它是有生命周期的。

那么它在生命周期做了什么？

  @Override
  public void start() {
    LOGGER.info("Configuration provider starting");

    Preconditions.checkState(file != null,
        "The parameter file must not be null");

    executorService = Executors.newSingleThreadScheduledExecutor(
            new ThreadFactoryBuilder().setNameFormat("conf-file-poller-%d")
                .build());

    FileWatcherRunnable fileWatcherRunnable =
        new FileWatcherRunnable(file, counterGroup);

    executorService.scheduleWithFixedDelay(fileWatcherRunnable, 0, interval,
        TimeUnit.SECONDS);

    lifecycleState = LifecycleState.START;

    LOGGER.debug("Configuration provider started");
  }

  @Override
  public void stop() {
    LOGGER.info("Configuration provider stopping");

    executorService.shutdown();
    try{
      while(!executorService.awaitTermination(500, TimeUnit.MILLISECONDS)) {
        LOGGER.debug("Waiting for file watcher to terminate");
      }
    } catch (InterruptedException e) {
      LOGGER.debug("Interrupted while waiting for file watcher to terminate");
      Thread.currentThread().interrupt();
    }
    lifecycleState = LifecycleState.STOP;
    LOGGER.debug("Configuration provider stopped");
  }
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

可以看到，在start时候，它起了一个周期调用线程executorService，这个周期调用线程又回每隔30s调用fileWatcherRunnable这个配置文件监控线程，在FileWatcherRunnable这里面，会去监听flume配置文件的变化，如果修改时间发生变化，eventBus会说我感兴趣的事件发生了！即eventBus.post(getConfiguration())

    @Override
    public void run() {
      LOGGER.debug("Checking file:{} for changes", file);

      counterGroup.incrementAndGet("file.checks");

      long lastModified = file.lastModified();

      if (lastModified > lastChange) {
        LOGGER.info("Reloading configuration file:{}", file);

        counterGroup.incrementAndGet("file.loads");

        lastChange = lastModified;

        try {
          eventBus.post(getConfiguration());
        } catch (Exception e) {
          LOGGER.error("Failed to load configuration data. Exception follows.",
              e);
        } catch (NoClassDefFoundError e) {
          LOGGER.error("Failed to start agent because dependencies were not " +
              "found in classpath. Error follows.", e);
        } catch (Throwable t) {
          // caught because the caller does not handle or log Throwables
          LOGGER.error("Unhandled error", t);
        }
      }
    }
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

之后对该事件感兴趣的listener就会进行事件处理，这里flume本身的Application对配置文件的变化感兴趣：

eventBus.register(application);
# 相当于 
eventBus.register(listener);
 
 
1
2
3

在application中，用注解@Subscribe标明的方法就告诉了我们，事件发生后，如何处理：

  @Subscribe
  public synchronized void handleConfigurationEvent(MaterializedConfiguration conf) {
    stopAllComponents();
    startAllComponents(conf);
  }
 
 
1
2
3
4
5

到这里为止，讲清楚了如果启动flume时候配置了no-reload-con参数，flume就会动态加载配置文件，默认每30秒检查一次配置文件，如果有修改，会重启所有的components；如果没有配置该参数，则只会启动一次。

1.3 Application的start前与后

Application的start()和handleConfigurationEvent(MaterializedConfiguration conf)，handleConfigurationEvent方法是在启动时或者需要动态读取配置文件而配置文件发生变化时，会通过eventBus调用此方法。

，该方法会先关闭所有组件再启动所有组件，因此，flume 所谓的动态加载并不是真正的动态，只能算自动重启吧，代码如下（org.apache.flume.node.Application）：

[java]  view plain  copy

1. @Subscribe  
2. public synchronized void handleConfigurationEvent(MaterializedConfiguration conf) {  
3.     stopAllComponents();  
4.     startAllComponents(conf);  
5. }  

进入stopallComponents方法，该方法是关闭所有的组件：

[java]  view plain  copy

1. private void stopAllComponents() {  
2.     if (this.materializedConfiguration != null) {  
3.         logger.info("Shutting down configuration: {}", this.materializedConfiguration);  
4.         for (Entry<String, SourceRunner> entry : this.materializedConfiguration.getSourceRunners().entrySet()) {  
5.             try {  
6.                 logger.info("Stopping Source " + entry.getKey());  
7.                 supervisor.unsupervise(entry.getValue());  
8.             } catch (Exception e) {  
9.                 logger.error("Error while stopping {}", entry.getValue(), e);  
10.             }  
11.         }  
12.   
13.         for (Entry<String, SinkRunner> entry : this.materializedConfiguration.getSinkRunners().entrySet()) {  
14.             try {  
15.                 logger.info("Stopping Sink " + entry.getKey());  
16.                 supervisor.unsupervise(entry.getValue());  
17.             } catch (Exception e) {  
18.                 logger.error("Error while stopping {}", entry.getValue(), e);  
19.             }  
20.         }  
21.   
22.         for (Entry<String, Channel> entry : this.materializedConfiguration.getChannels().entrySet()) {  
23.             try {  
24.                 logger.info("Stopping Channel " + entry.getKey());  
25.                 supervisor.unsupervise(entry.getValue());  
26.             } catch (Exception e) {  
27.                 logger.error("Error while stopping {}", entry.getValue(), e);  
28.             }  
29.         }  
30.     }  
31.     if (monitorServer != null) {  
32.         monitorServer.stop();  
33.     }  
34. }  

可以看出，flume关闭组件的顺序为source->sink->channel。

另外，这些组件都调用了supervisor.unsupervise(entry.getValue());这个方法来关闭组件，进入unsupervise方法：

[java]  view plain  copy

1. public synchronized void unsupervise(LifecycleAware lifecycleAware) {  
2.   
3.     Preconditions.checkState(supervisedProcesses.containsKey(lifecycleAware),  
4.             "Unaware of " + lifecycleAware + " - can not unsupervise");  
5.   
6.     logger.debug("Unsupervising service:{}", lifecycleAware);  
7.   
8.     synchronized (lifecycleAware) {  
9.         Supervisoree supervisoree = supervisedProcesses.get(lifecycleAware);  
10.         supervisoree.status.discard = true;  
11.         this.setDesiredState(lifecycleAware, LifecycleState.STOP);  
12.         logger.info("Stopping component: {}", lifecycleAware);  
13.         lifecycleAware.stop();  
14.     }  
15.     supervisedProcesses.remove(lifecycleAware);  
16.     // We need to do this because a reconfiguration simply unsupervises old  
17.     // components and supervises new ones.  
18.     monitorFutures.get(lifecycleAware).cancel(false);  
19.     // purges are expensive, so it is done only once every 2 hours.  
20.     needToPurge = true;  
21.     monitorFutures.remove(lifecycleAware);  
22. }  

这些方法主要是将组件以及监控等从内存中移除。lifecycleAware.stop()方法执行具体的lifecycleAware的stop，LifecycleAware是一个顶级接口，定义了组件的开始，结束以及当前状态，flume中重要组件如source，sink，channel都实现了这个接口：

[java]  view plain  copy

1. public interface LifecycleAware {  
2.   public void start();  
3.   public void stop();  
4.   public LifecycleState getLifecycleState();  
5. }  

通过该接口实现了多态，不同组件执行自己的start，stop方法。组件的停止分析到此处，下面分析另一个方法：startAllComponents方法：

[java]  view plain  copy

1. private void startAllComponents(MaterializedConfiguration materializedConfiguration) {  
2.     logger.info("Starting new configuration:{}", materializedConfiguration);  
3.     //使用读取的配置文件初始化materializedConfiguration对象  
4.     this.materializedConfiguration = materializedConfiguration;  
5.     //先启动channel，等待启动完毕，然后启动sink，最后启动source  
6.       
7.     //从materializedConfiguration中读取channel信息，  
8.     for (Entry<String, Channel> entry : materializedConfiguration.getChannels().entrySet()) {  
9.         try {  
10.             logger.info("Starting Channel " + entry.getKey());  
11.             supervisor.supervise(entry.getValue(), new SupervisorPolicy.AlwaysRestartPolicy(),  
12.                     LifecycleState.START);  
13.         } catch (Exception e) {  
14.             logger.error("Error while starting {}", entry.getValue(), e);  
15.         }  
16.     }  
17.   
18.     /* 
19.      * Wait for all channels to start. 
20.      */  
21.     for (Channel ch : materializedConfiguration.getChannels().values()) {  
22.         while (ch.getLifecycleState() != LifecycleState.START && !supervisor.isComponentInErrorState(ch)) {  
23.             try {  
24.                 logger.info("Waiting for channel: " + ch.getName() + " to start. Sleeping for 500 ms");  
25.                 Thread.sleep(500);  
26.             } catch (InterruptedException e) {  
27.                 logger.error("Interrupted while waiting for channel to start.", e);  
28.                 Throwables.propagate(e);  
29.             }  
30.         }  
31.     }  
32.   
33.     for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {  
34.         try {  
35.             logger.info("Starting Sink " + entry.getKey());  
36.             supervisor.supervise(entry.getValue(), new SupervisorPolicy.AlwaysRestartPolicy(),  
37.                     LifecycleState.START);  
38.         } catch (Exception e) {  
39.             logger.error("Error while starting {}", entry.getValue(), e);  
40.         }  
41.     }  
42.   
43.     for (Entry<String, SourceRunner> entry : materializedConfiguration.getSourceRunners().entrySet()) {  
44.         try {  
45.             logger.info("Starting Source " + entry.getKey());  
46.             supervisor.supervise(entry.getValue(), new SupervisorPolicy.AlwaysRestartPolicy(),  
47.                     LifecycleState.START);  
48.         } catch (Exception e) {  
49.             logger.error("Error while starting {}", entry.getValue(), e);  
50.         }  
51.     }  
52.   
53.     this.loadMonitoring();  
54. }  

可见，与关闭顺序不同，启动组件先启动channel，等待启动完毕，然后启动sink，最后启动source。三个组件的启动都是调用了supervisor.supervise这个方法：

[java]  view plain  copy

1. //supervise方法用于监控对应的组件  
2. public synchronized void supervise(LifecycleAware lifecycleAware, SupervisorPolicy policy,  
3.         LifecycleState desiredState) {  
4.     if (this.monitorService.isShutdown() || this.monitorService.isTerminated()  
5.             || this.monitorService.isTerminating()) {  
6.         throw new FlumeException("Supervise called on " + lifecycleAware + " "  
7.                 + "after shutdown has been initiated. " + lifecycleAware + " will not" + " be started");  
8.     }  
9.     //判断这个组件是不是已经被监控起来，如果已经监控则不再添加到监控map中  
10.     Preconditions.checkState(!supervisedProcesses.containsKey(lifecycleAware),  
11.             "Refusing to supervise " + lifecycleAware + " more than once");  
12.   
13.     if (logger.isDebugEnabled()) {  
14.         logger.debug("Supervising service:{} policy:{} desiredState:{}",  
15.                 new Object[] { lifecycleAware, policy, desiredState });  
16.     }  
17.     //记录状态信息  
18.     Supervisoree process = new Supervisoree();  
19.     process.status = new Status();  
20.   
21.     process.policy = policy;  
22.     process.status.desiredState = desiredState;  
23.     process.status.error = false;  
24.     //MonitorRunnable是一个线程，每过一段时间去检查组件的状态，如果组件状态有误，则改正过来  
25.     //比如本应该start状态，但是组件挂了，则把组件启动起来  
26.     MonitorRunnable monitorRunnable = new MonitorRunnable();  
27.     monitorRunnable.lifecycleAware = lifecycleAware;//监控的对象  
28.     monitorRunnable.supervisoree = process;//监控状态  
29.     monitorRunnable.monitorService = monitorService;//监控的线程池  
30.     //放入当前持有的监控map中  
31.     supervisedProcesses.put(lifecycleAware, process);  
32.     //将持有监控对象，对象状态的monitorrunnable对象吊起来，并且每隔三秒区监控  
33.     ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(monitorRunnable, 0, 3, TimeUnit.SECONDS);  
34.     //存放每个LifecycleAware组件和调度对应关系记录起来  
35.     monitorFutures.put(lifecycleAware, future);  
36. }  

注意到上面ScheduledFuture<?> future = monitorService.scheduleWithFixedDelay(monitorRunnable, 0, 3, TimeUnit.SECONDS);这段代码启动了一个3s执行的定时任务，每三秒区执行monitorRunnable这个线程，查看该线程的run：

[java]  view plain  copy

1. @Override  
2. public void run() {  
3.     logger.debug("checking process:{} supervisoree:{}", lifecycleAware, supervisoree);  
4.   
5.     long now = System.currentTimeMillis();  
6.   
7.     try {  
8.         if (supervisoree.status.firstSeen == null) {  
9.             logger.debug("first time seeing {}", lifecycleAware);  
10.             // 第一次开始运行时，设置firstSeen为当前的时间System.currentTimeMillis()  
11.             supervisoree.status.firstSeen = now;  
12.         }  
13.   
14.         supervisoree.status.lastSeen = now;  
15.         synchronized (lifecycleAware) {  
16.             //如果是discard或者error，就丢弃   
17.             if (supervisoree.status.discard) {  
18.                 // Unsupervise has already been called on this.  
19.                 logger.info("Component has already been stopped {}", lifecycleAware);  
20.                 return;  
21.             } else if (supervisoree.status.error) {  
22.                 logger.info(  
23.                         "Component {} is in error state, and Flume will not" + "attempt to change its state",  
24.                         lifecycleAware);  
25.                 return;  
26.             }  
27.               
28.             supervisoree.status.lastSeenState = lifecycleAware.getLifecycleState();  
29.             //如果状态不是理想的状态，比如理想的状态应该是start，但是现在的状态时stop，那么把组件启动  
30.             //状态只有两种：start和stop  
31.             //否则什么都不做  
32.             if (!lifecycleAware.getLifecycleState().equals(supervisoree.status.desiredState)) {  
33.                   
34.                 logger.debug("Want to transition {} from {} to {} (failures:{})",  
35.                         new Object[] { lifecycleAware, supervisoree.status.lastSeenState,  
36.                                 supervisoree.status.desiredState, supervisoree.status.failures });  
37.   
38.                 switch (supervisoree.status.desiredState) {  
39.                   
40.                 //本该start状态，但是当前非start状态，则调用该组件的start方法将其启动  
41.                 case START:  
42.                     try {  
43.                         lifecycleAware.start();  
44.                     } catch (Throwable e) {  
45.                         logger.error("Unable to start " + lifecycleAware + " - Exception follows.", e);  
46.                         if (e instanceof Error) {  
47.                             // This component can never recover, shut it  
48.                             // down.  
49.                             supervisoree.status.desiredState = LifecycleState.STOP;  
50.                             try {  
51.                                 lifecycleAware.stop();  
52.                                 logger.warn(  
53.                                         "Component {} stopped, since it could not be"  
54.                                                 + "successfully started due to missing dependencies",  
55.                                         lifecycleAware);  
56.                             } catch (Throwable e1) {  
57.                                 logger.error("Unsuccessful attempt to "  
58.                                         + "shutdown component: {} due to missing dependencies."  
59.                                         + " Please shutdown the agent"  
60.                                         + "or disable this component, or the agent will be"  
61.                                         + "in an undefined state.", e1);  
62.                                 supervisoree.status.error = true;  
63.                                 if (e1 instanceof Error) {  
64.                                     throw (Error) e1;  
65.                                 }  
66.                                 // Set the state to stop, so that the  
67.                                 // conf poller can  
68.                                 // proceed.  
69.                             }  
70.                         }  
71.                         supervisoree.status.failures++;  
72.                     }  
73.                     break;  
74.                 case STOP:  
75.                     //本该stop状态，但是当前非stop状态，则调用该组件的stop方法将其停止  
76.                     try {  
77.                         lifecycleAware.stop();  
78.                     } catch (Throwable e) {  
79.                         logger.error("Unable to stop " + lifecycleAware + " - Exception follows.", e);  
80.                         if (e instanceof Error) {  
81.                             throw (Error) e;  
82.                         }  
83.                         supervisoree.status.failures++;  
84.                     }  
85.                     break;  
86.                 default:  
87.                     logger.warn("I refuse to acknowledge {} as a desired state",  
88.                             supervisoree.status.desiredState);  
89.                 }  
90.   
91.                 if (!supervisoree.policy.isValid(lifecycleAware, supervisoree.status)) {  
92.                     logger.error("Policy {} of {} has been violated - supervisor should exit!",  
93.                             supervisoree.policy, lifecycleAware);  
94.                 }  
95.             }  
96.         }  
97.     } catch (Throwable t) {  
98.         logger.error("Unexpected error", t);  
99.     }  
100.     logger.debug("Status check complete");  
101. }  

可见，该线程主要监控各个组件的执行状态，状态出错则纠正，或重启组件。另外有个线程定期清空缓存里不需要的调度任务：

[java]  view plain  copy

1. private class Purger implements Runnable {  
2.   
3.     @Override  
4.     public void run() {  
5.         if (needToPurge) {  
6.             //从工作队列中删除已经cancel的java.util.concurrent.Future对象（释放队列空间）  
7.             //ScheduledFuture的cancel执行后，ScheduledFuture.purge会移除被cancel的任务  
8.             monitorService.purge();  
9.             needToPurge = false;  
10.         }  
11.     }  
12. }  

以上分析的组件启动，关闭，状态监控位于org.apache.flume.lifecycle包的LifecycleSupervisor类。到此，flume组件的关闭，启动，监控分析完毕

Flume 提供了 SourceRunner 用来启动 Source 的流转:

PollableSourceRunner对应PollableSource

EventDrivenSourceRunner对应普通Source