Flume 源码学习（二）Channel组件介绍

Flume 源码学习（二）Channel组件介绍

---

Channel是Flume中第二个组件，是日志从source传输到sink的通道。根据Flume文档，channel有两个重要衡量指标：

• Reliability：可依赖性。即channel需要接收到的event被下个agent接收或被最终的sink接收。Flume使用事务来保证Reliability。支持可以持久化的基于文件系统的FileChannel和不能保证持久化的MemoryChannel
• Recoverability：可恢复性。如果发生Failure，是否能够恢复数据；Flume支持可恢复的FileChannel和不支持恢复但传输速度快得MemoryChannel

---

类图

首先还是来看Channel相关类的类图：

两个望文生义的类的说明：

接口名	作用
BasicChannelSemantics	基本的channel语义的实现，包含了Threadlocal保存的Transaction实例。通俗的讲就是采用ThreadLocal来保存和当前Channel关联的Transaction的抽象父类
SpillableMemoryChannel	集成了FileChannel和MemoryChannel功能，首先用MemoryChannel,在这个channel达到capacity再改用FileChannel继续

MemoryChannel

我们首先来看MemoryChannel的实现，MemoryChannel有几个重要参数可以配置：

• capacity：Channel的最大容积
• transactionCapacity：一个事务最多可以包含多少个events
• byteCapacity：Channel的Queue的最大的byte 容积
• byteCapacityBufferPercent：定义了byteCapacity和评估的event size之间缓冲的百分比，概要的说就是如果MemoryChannel中等待写出的EventList的byte size最多只能达到(1-byteCapacityBufferPercent/100)*100%，剩下的是缓冲buffer（如果没有buffer，很可能出现内存溢出）
• keep-alive：定义了允许在队列中等待的最大秒数

根据上面的类图，我们知道MemoryChannel继承了BasicChannelSemantics抽象类，而后者
有一个抽象方法createTransaction()需要子类自己实现。所以，MemoryChannel中第一个重要的功能就是基于内存的Transaction的实现。MemoryChannel中是MemoryTransaction这一个内部类，代码如下：

private class MemoryTransaction extends BasicTransactionSemantics {
    private LinkedBlockingDeque<Event> takeList;//从channel中take，准备推送到sink
    private LinkedBlockingDeque<Event> putList;//从source put到channel的队列
    private final ChannelCounter channelCounter;//counter
    private int putByteCounter = 0;//put的字节counter
    private int takeByteCounter = 0;//take的字节counter

    public MemoryTransaction(int transCapacity, ChannelCounter counter) {
      putList = new LinkedBlockingDeque<Event>(transCapacity);
      takeList = new LinkedBlockingDeque<Event>(transCapacity);

      channelCounter = counter;
    }
    //往putList增加Event，如果没有空间抛错
    @Override
    protected void doPut(Event event) throws InterruptedException {
      channelCounter.incrementEventPutAttemptCount();
      int eventByteSize = (int)Math.ceil(estimateEventSize(event)/byteCapacitySlotSize);

      if (!putList.offer(event)) {
        throw new ChannelException(
          "Put queue for MemoryTransaction of capacity " +
            putList.size() + " full, consider committing more frequently, " +
            "increasing capacity or increasing thread count");
      }
      putByteCounter += eventByteSize;
    }
    //从外部类MemoryChannel的等待Take的Event List中take一个过来放到事务的takeList里面，作为本次事务需要提交的Event
    @Override
    protected Event doTake() throws InterruptedException {
      channelCounter.incrementEventTakeAttemptCount();
      if(takeList.remainingCapacity() == 0) {//首先判断takeList是否还有空间
        throw new ChannelException("Take list for MemoryTransaction, capacity " +
            takeList.size() + " full, consider committing more frequently, " +
            "increasing capacity, or increasing thread count");
      }
      if(!queueStored.tryAcquire(keepAlive, TimeUnit.SECONDS)) {//通过queueStored这Semaphore信号来判断外部类MemoryChannel是否有queued Event等待被take，用信号量而不是直接判断queueStored（LinkedBlockingDeque<Event>）的poll()主要是后者会阻塞queueStored上的所有操作，尤其是包括为空得情况下put操作。可以提高throughput。
        return null;//没有获取到，意味着可能queue当前为空。
      }
      Event event;
      synchronized(queueLock) {
        event = queue.poll();//queueStored不为空。
      }
      Preconditions.checkNotNull(event, "Queue.poll returned NULL despite semaphore " +
          "signalling existence of entry");
      takeList.put(event);
      //计算taken的byte
      int eventByteSize = (int)Math.ceil(estimateEventSize(event)/byteCapacitySlotSize);
      takeByteCounter += eventByteSize;

      return event;
    }
    //实际事务的提交
    @Override
    protected void doCommit() throws InterruptedException {
      int remainingChange = takeList.size() - putList.size();
      if(remainingChange < 0) {//take size < putsize，sink的消费速度慢于source的产生速度
        if(!bytesRemaining.tryAcquire(putByteCounter, keepAlive,
          TimeUnit.SECONDS)) {//判断是否有足够空间接收putList中events所占的空间
          throw new ChannelException("Cannot commit transaction. Byte capacity " +
            "allocated to store event body " + byteCapacity * byteCapacitySlotSize +
            "reached. Please increase heap space/byte capacity allocated to " +
            "the channel as the sinks may not be keeping up with the sources");
        }
        if(!queueRemaining.tryAcquire(-remainingChange, keepAlive, TimeUnit.SECONDS)) {//判断queue是否还有空间接收（因为生产速度快于消费速度）
          bytesRemaining.release(putByteCounter);
          throw new ChannelFullException("Space for commit to queue couldn't be acquired." +
              " Sinks are likely not keeping up with sources, or the buffer size is too tight");
        }
      }
      int puts = putList.size();//事务期间生产的event
      int takes = takeList.size();//事务期间等待消费的event
      synchronized(queueLock) {
        if(puts > 0 ) {
          while(!putList.isEmpty()) {
            if(!queue.offer(putList.removeFirst())) {//将putList中新生产的Events保存到queue
              throw new RuntimeException("Queue add failed, this shouldn't be able to happen");
            }
          }
        }
        putList.clear();//重置事务的putList，下同
        takeList.clear();
      }
      bytesRemaining.release(takeByteCounter);
      takeByteCounter = 0;
      putByteCounter = 0;

      queueStored.release(puts);//从queueStored释放puts个信号量
      if(remainingChange > 0) {
        queueRemaining.release(remainingChange);//queueRemaining释放remainingChange个信号量
      }
      if (puts > 0) {
        channelCounter.addToEventPutSuccessCount(puts);
      }
      if (takes > 0) {
        channelCounter.addToEventTakeSuccessCount(takes);
      }

      channelCounter.setChannelSize(queue.size());
    }

    @Override
    protected void doRollback() {
      int takes = takeList.size();
      synchronized(queueLock) {
        Preconditions.checkState(queue.remainingCapacity() >= takeList.size(), "Not enough space in memory channel " +
            "queue to rollback takes. This should never happen, please report");
        while(!takeList.isEmpty()) {
          queue.addFirst(takeList.removeLast());//把takelist中的events原封不动的放回queue
        }
        putList.clear();
      }
      bytesRemaining.release(putByteCounter);
      putByteCounter = 0;
      takeByteCounter = 0;

      queueStored.release(takes);
      channelCounter.setChannelSize(queue.size());
    }

  }

通过上面的代码我们可以看到MemoryChanne的createTransaction内部使用的是基于内存的MemoryTransaction，没有实现持久化。在生产和消费的过程中，非常关注byteCapacity，防止内存溢出。

---

FileChannel

接下来来看FileChannel的实现TBD