Flume Source&SourceRunner 介绍

Flume Source&SourceRunner 介绍

Source 作为Flume 的数据接收源 source 有两个子类 PollableSource 和 EventDrivenSource

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Xf1LU4dW-1571214286033)(https://i.ibb.co/cb2hSxX/TIM-20191016154243.png)]

但是 Source 的启动主要依靠的是 SourceRunner 启动的，同样SourceRunner 分为 PollableSourceRunner 和 EventDrivenSourceRunner

SourceRunner

根据source获取到SourceRunner

public static SourceRunner forSource(Source source) {
    SourceRunner runner = null;

    if (source instanceof PollableSource) {
        runner = new PollableSourceRunner();
        ((PollableSourceRunner) runner).setSource((PollableSource) source);
    } else if (source instanceof EventDrivenSource) {
        runner = new EventDrivenSourceRunner();
        ((EventDrivenSourceRunner) runner).setSource((EventDrivenSource) source);
    } else {
        throw new IllegalArgumentException("No known runner type for source "
            + source);
    }

    return runner;
}

PollableSource & PollableSourceRunner 轮询机制

PollableSourceRunner

  @Override
  public void start() {
    PollableSource source = (PollableSource) getSource();
    ChannelProcessor cp = source.getChannelProcessor();
    cp.initialize();
    source.start();

    runner = new PollingRunner();

    runner.source = source;
    runner.counterGroup = counterGroup;
    runner.shouldStop = shouldStop;

    // 轮询模式启动了一个线程去执行
    runnerThread = new Thread(runner);
    runnerThread.setName(getClass().getSimpleName() + "-" + 
        source.getClass().getSimpleName() + "-" + source.getName());
    runnerThread.start();

    lifecycleState = LifecycleState.START;
  }

public static class PollingRunner implements Runnable {

    private PollableSource source;
    private AtomicBoolean shouldStop;
    private CounterGroup counterGroup;

    @Override
    public void run() {
      logger.debug("Polling runner starting. Source:{}", source);

      // 只要为被关闭就死循环的执行  
      while (!shouldStop.get()) {
        counterGroup.incrementAndGet("runner.polls");

        try {
          // source#process 应该是核心的业务逻辑
          if (source.process().equals(PollableSource.Status.BACKOFF)) {
            counterGroup.incrementAndGet("runner.backoffs");

            Thread.sleep(Math.min(
                counterGroup.incrementAndGet("runner.backoffs.consecutive")
                * backoffSleepIncrement, maxBackoffSleep));
          } else {
            counterGroup.set("runner.backoffs.consecutive", 0L);
          }
        } catch (InterruptedException e) {
          logger.info("Source runner interrupted. Exiting");
          counterGroup.incrementAndGet("runner.interruptions");
        } catch (EventDeliveryException e) {
          logger.error("Unable to deliver event. Exception follows.", e);
          counterGroup.incrementAndGet("runner.deliveryErrors");
        } catch (Exception e) {
          counterGroup.incrementAndGet("runner.errors");
          logger.error("Unhandled exception, logging and sleeping for " +
              maxBackoffSleep + "ms", e);
          try {
            Thread.sleep(maxBackoffSleep);
          } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
          }
        }
      }

      logger.debug("Polling runner exiting. Metrics:{}", counterGroup);
    }

  }

public interface PollableSource extends Source {
  /**
   * <p>
   * Attempt to pull an item from the source, sending it to the channel.
   * </p>
   * <p>
   * When driven by an {@link EventDrivenSourceRunner} process is guaranteed
   * to be called only by a single thread at a time, with no concurrency.
   * Any other mechanism driving a pollable source must follow the same
   * semantics.
   * </p>
   * @return {@code READY} if one or more events were created from the source.
   * {@code BACKOFF} if no events could be created from the source.
   * @throws EventDeliveryException If there was a failure in delivering to
   * the attached channel, or if a failure occurred in acquiring data from
   * the source.
   */
   /**
    * <p>
    *尝试从源中提取项目，然后将其发送到频道。
    * </ p>
    * <p>
    *由{@link EventDrivenSourceRunner}进程驱动时，可以保证
    *一次只能由单个线程调用，没有并发。
    *任何其他驱动可轮询源的机制都必须遵循相同的原则
    *语义。
    * </ p>
    * @return {@code READY}（如果源中创建了一个或多个事件）。
    * {@code BACKOFF}如果无法从源创建任何事件。
    * @throws EventDeliveryException如果传递到
    *连接的通道，或者从以下通道获取数据失败
    * 来源。
    **/
  public Status process() throws EventDeliveryException;

  public static enum Status {
    READY, BACKOFF
  }

}

EventDrivenSource & EventDrivenSourceRunner 事件驱动

EventDrivenSourceRunner

  @Override
  public void start() {
    Source source = getSource();
    ChannelProcessor cp = source.getChannelProcessor();
    cp.initialize();
    source.start();
    lifecycleState = LifecycleState.START;
  }

这个SourceRunner就比较简单，直接吧内部的source启动就完事了

因为sourceRunner内部没有启动额外线程监听事件，所以EventDrivenSource通常需要在start方法内部启动额外线程去执行操作

ExecSource

  @Override
  public void start() {
    logger.info("Exec source starting with command:{}", command);

    executor = Executors.newSingleThreadExecutor();
    //使用线程池提交了一个新的runnable 在start内部启动新的线程执行监听操作
    runner = new ExecRunnable(shell, command, getChannelProcessor(), sourceCounter,
        restart, restartThrottle, logStderr, bufferCount, batchTimeout, charset);

    runnerFuture = executor.submit(runner);

    sourceCounter.start();
    super.start();

    logger.debug("Exec source started");
  }

    public void run() {
        do {
        ...
        try {
            ...
            synchronized (eventList) {
                if(!eventList.isEmpty()) {
                flushEventBatch(eventList);
                }
            }
        } catch (Exception e) {
            ...
        } finally {
            ...
        }
        if(restart) {
            ...;
        }
        // 当执行stop 方法时 restart 设置为 false 也是一个死循环的处理
        } while(restart);
    }