hbase 原代码分析 (11) WAL 写日志过程



在put和delete过程的时候都会有WAL。这一章主要分析这个过程。

重点:
1) WAL在创建region的时候创建。
2)    WAL 存储在HDFS,主要类FSHLog
3) 在put 和delete写入内存之前都会写入WAL,主要是防止机器崩溃时能恢复。
4)用户在协处理器里可以自己调用WALEdit。

写入过程
调用disrupter框架。


1)创建HRegion.java 类 ,具体创建过程可以参考第10节。
  1. public static HRegion createHRegion(final HRegionInfo info, final Path rootDir,
  2. final Path tableDir, final Configuration conf, final HTableDescriptor hTableDescriptor,
  3. final WAL wal, final boolean initialize, final boolean ignoreWAL)
  4. throws IOException {
  5. HRegionFileSystem.createRegionOnFileSystem(conf, fs, tableDir, info);
  6. WAL effectiveWAL = wal;
  7. if (wal == null && !ignoreWAL) {
  8. Configuration confForWAL = new Configuration(conf);
  9. confForWAL.set(HConstants.HBASE_DIR, rootDir.toString());
  10. effectiveWAL = (new WALFactory(confForWAL,
  11. Collections.<WALActionsListener>singletonList(new MetricsWAL()),
  12. "hregion-" + RandomStringUtils.randomNumeric(8))).
  13. getWAL(info.getEncodedNameAsBytes(), info.getTable().getNamespace());
  14. }
  15. }
在工程类里面最重要的。这里的defaultProvider(DefaultWALProvider.class),
  1. provider = getProvider(WAL_PROVIDER, DEFAULT_WAL_PROVIDER, listeners, null);

这个版本1.2.3和1.3.1 有点不同,这里以1.3.1为准
  1. @Override
  2. public WAL getWAL(final byte[] identifier, byte[] namespace) throws IOException {
  3. if (log == null) {
  4. // only lock when need to create wal, and need to lock since
  5. // creating hlog on fs is time consuming
  6. synchronized (walCreateLock) {
  7. if (log == null) {
  8. log = new FSHLog(FileSystem.get(conf), FSUtils.getRootDir(conf),
  9. getWALDirectoryName(factory.factoryId), HConstants.HREGION_OLDLOGDIR_NAME, conf,
  10. listeners, true, logPrefix,
  11. META_WAL_PROVIDER_ID.equals(providerId) ? META_WAL_PROVIDER_ID : null);
  12. }
  13. }
  14. }
  15. return log;
  16. }

所以Log 的目录在:有点混淆。
hdfs = root/WALs/WALFactory.encode/oldWALs/WALFactory/wal.
root=hbase
后缀:.meta 或者为null

这样HLog的文件位置,名字 都有了,并且写入Region中。

这个FSHLog里面定义很多东西,最重要的。

  1. public FSHLog(final FileSystem fs, final Path rootDir, final String logDir,
  2. final String archiveDir, final Configuration conf,
  3. final List<WALActionsListener> listeners,
  4. final boolean failIfWALExists, final String prefix, final String suffix)
  5. throws IOException {
  6. //HLog文件写流  
  7. // rollWriter sets this.hdfs_out if it can.
  8. rollWriter();
  9. this.disruptor =
  10. new Disruptor<RingBufferTruck>(RingBufferTruck.EVENT_FACTORY, preallocatedEventCount,
  11. this.appendExecutor, ProducerType.MULTI, new BlockingWaitStrategy());
  12. this.disruptor.getRingBuffer().next();
  13. this.ringBufferEventHandler =
  14. new RingBufferEventHandler(conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
  15. maxHandlersCount);
  16. this.disruptor.handleExceptionsWith(new RingBufferExceptionHandler());
  17. this.disruptor.handleEventsWith(new RingBufferEventHandler [] {this.ringBufferEventHandler});
  18. // Presize our map of SyncFutures by handler objects.
  19. this.syncFuturesByHandler = new ConcurrentHashMap<Thread, SyncFuture>(maxHandlersCount);
  20. this.disruptor.start();
  21. }
这个FSHLog 里面最要的几个东西
rollWriter()定义了一个FSDataOutputStream这个可以往hlog写文件。 rollWriter 里实现了ProtobufLogWriter 类,在该类的init方法里定义了write 与文件的关系。
FSDataOutputStream nextHdfsOut = null;
if (nextWriter instanceof ProtobufLogWriter) {
  nextHdfsOut = ((ProtobufLogWriter)nextWriter).getStream();
disruptor 这个是一个队列工具。采用生产者与消费者模式。当有时间append时,会调用RingBufferEventHandler的onEven方法。
RingBufferEventHandler new里面定义很多同步的runner 这个主要是写write过程。
  1. RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
  2. this.syncFutures = new SyncFuture[maxHandlersCount];
  3. this.syncRunners = new SyncRunner[syncRunnerCount];
  4. for (int i = 0; i < syncRunnerCount; i++) {
  5. this.syncRunners[i] = new SyncRunner("sync." + i, maxHandlersCount);
  6. }
  7. }


2) 记录数据。
这个主要是put过程了。具体put过程可以参考第七章
主要是HRegin类,因为每一个region都有一个WAL。WALEdit 是处理Wal的工具。
put过程主要哪些地方要写WAL。
1)rowProcessor 的钩子,用户可以定义
2)preBatchMutate
3)写入store 内存时
4)MVCC
5)结束的钩子
这里最重要的写入内存之前
  1. @Override
  2. public void processRowsWithLocks(RowProcessor<?,?> processor, long timeout,
  3. long nonceGroup, long nonce) throws IOException {
  4. //new
  5. WALEdit walEdit = new WALEdit();、
  6. // 1. Run pre-process hook
  7. processor.preProcess(this, walEdit);
  8. MultiVersionConcurrencyControl.WriteEntry writeEntry = null;
  9. WALKey walKey = null;
  10. // 5. Call the preBatchMutate hook
  11. processor.preBatchMutate(this, walEdit);
  12. long txid = 0;
  13. // 6. Append no sync
  14. if (!walEdit.isEmpty()) {
  15. // we use HLogKey here instead of WALKey directly to support legacy coprocessors.
  16. walKey = new HLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
  17. this.htableDescriptor.getTableName(), WALKey.NO_SEQUENCE_ID, now,
  18. processor.getClusterIds(), nonceGroup, nonce, mvcc);
  19. txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(),
  20. walKey, walEdit, true);
  21. }
  22. if(walKey == null){
  23. // since we use wal sequence Id as mvcc, for SKIP_WAL changes we need a "faked" WALEdit
  24. // to get a sequence id assigned which is done by FSWALEntry#stampRegionSequenceId
  25. walKey = this.appendEmptyEdit(this.wal);
  26. }
  27. // 7. Start mvcc transaction
  28. writeEntry = walKey.getWriteEntry();
  29. mvccNum = walKey.getSequenceId();
  30. // 8. Apply to memstore
  31.      Store store = getStore(cell);
  32. addedSize += store.add(cell);
  33. }
  34. // 11. Sync edit log
  35. if (txid != 0) {
  36. syncOrDefer(txid, getEffectiveDurability(processor.useDurability()));
  37. }
  38. walSyncSuccessful = true;
  39. } finally {
  40. if (!mutations.isEmpty() && !walSyncSuccessful) {
  41. if (writeEntry != null) {
  42. mvcc.complete(writeEntry);
  43. writeEntry = null;
  44. }
  45. }
  46. // 13. Roll mvcc forward
  47. if (writeEntry != null) {
  48. mvcc.completeAndWait(writeEntry);
  49. }
  50. }
  51. // 14. Run post-process hook
  52. processor.postProcess(this, walEdit, walSyncSuccessful);
  53. }
  54. }

这个里面都简化了。留下来与Wal 有关的。里面最重要的已经标红。
第一个。
  1. walKey = new HLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
  2. this.htableDescriptor.getTableName(), WALKey.NO_SEQUENCE_ID, now,
  3. processor.getClusterIds(), nonceGroup, nonce, mvcc);
  4. //txid 好比快递号,实时查询状态。
  5. txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(),
  6. walKey, walEdit, true);
第二个
syncOrDefer(txid, getEffectiveDurability(processor.useDurability()));
FSHlog类
  1. @Override
  2. public long append(final HTableDescriptor htd, final HRegionInfo hri, final WALKey key,
  3. final WALEdit edits, final boolean inMemstore) throws IOException {
  4. if (this.closed) throw new IOException("Cannot append; log is closed");
  5. // Make a trace scope for the append. It is closed on other side of the ring buffer by the
  6. // single consuming thread. Don't have to worry about it.
  7. TraceScope scope = Trace.startSpan("FSHLog.append");
  8. FSWALEntry entry = null;
  9. long sequence = this.disruptor.getRingBuffer().next();
  10. try {
  11.         //获得车厢。
  12. RingBufferTruck truck = this.disruptor.getRingBuffer().get(sequence);
  13.         //定义实体
  14. entry = new FSWALEntry(sequence, key, edits, htd, hri, inMemstore);
  15.         //放入车厢
  16. truck.loadPayload(entry, scope.detach());
  17. } finally {
  18.     //告诉车可以走了。
  19. this.disruptor.getRingBuffer().publish(sequence);
  20. }
  21. return sequence;
  22. }
  1. @Override
  2.     //这是一个同步方法,如果车厢已经空了,说明已经结束。否则等待。
  3. public void sync(long txid) throws IOException {
  4. if (this.highestSyncedSequence.get() >= txid){
  5. // Already sync'd.
  6.     //车厢已经空了。
  7. return;
  8. }
  9. TraceScope scope = Trace.startSpan("FSHLog.sync");
  10. try {
    1. //这里会等待空。
  11. scope = Trace.continueSpan(publishSyncThenBlockOnCompletion(scope.detach()));
  12. } finally {
  13. assert scope == NullScope.INSTANCE || !scope.isDetached();
  14. scope.close();
  15. }
  16. }

这个RingBufferEventHandler类的onEven 方法:不解释了,太晚了。这个主要是将下面车厢里的东西拿出来,也就是收货了,然后写到write里面。最后调用SysRunner里的run(这个方法无线循环)方法。run里的write.syn() .写到HDFS。
  1. @Override
    // We can set endOfBatch in the below method if at end of our this.syncFutures array
    public void onEvent(final RingBufferTruck truck, final long sequence, boolean endOfBatch)
    throws Exception {
    
      try {
        if (truck.hasSyncFuturePayload()) {
          this.syncFutures[this.syncFuturesCount++] = truck.unloadSyncFuturePayload();
          // Force flush of syncs if we are carrying a full complement of syncFutures.
          if (this.syncFuturesCount == this.syncFutures.length) endOfBatch = true;
        } else if (truck.hasFSWALEntryPayload()) {
          TraceScope scope = Trace.continueSpan(truck.unloadSpanPayload());
          try {
            FSWALEntry entry = truck.unloadFSWALEntryPayload();
            if (this.exception != null) {
              entry.stampRegionSequenceId();
              // Return to keep processing events coming off the ringbuffer
              return;
            }
            append(entry);
          } catch (Exception e) {
            // Failed append. Record the exception.
            this.exception = e;
            // invoking cleanupOutstandingSyncsOnException when append failed with exception,
            // it will cleanup existing sync requests recorded in syncFutures but not offered to SyncRunner yet,
            // so there won't be any sync future left over if no further truck published to disruptor.
            cleanupOutstandingSyncsOnException(sequence,
                this.exception instanceof DamagedWALException ? this.exception
                    : new DamagedWALException("On sync", this.exception));
            // Return to keep processing events coming off the ringbuffer
            return;
          } finally {
            assert scope == NullScope.INSTANCE || !scope.isDetached();
            scope.close(); // append scope is complete
          }
        } else {
          // What is this if not an append or sync. Fail all up to this!!!
          cleanupOutstandingSyncsOnException(sequence,
            new IllegalStateException("Neither append nor sync"));
          // Return to keep processing.
          return;
        }
    
        // TODO: Check size and if big go ahead and call a sync if we have enough data.
        // This is a sync. If existing exception, fall through. Else look to see if batch.
        if (this.exception == null) {
          // If not a batch, return to consume more events from the ring buffer before proceeding;
          // we want to get up a batch of syncs and appends before we go do a filesystem sync.
          if (!endOfBatch || this.syncFuturesCount <= 0) return;
          // syncRunnerIndex is bound to the range [0, Integer.MAX_INT - 1] as follows:
          //   * The maximum value possible for syncRunners.length is Integer.MAX_INT
          //   * syncRunnerIndex starts at 0 and is incremented only here
          //   * after the increment, the value is bounded by the '%' operator to [0, syncRunners.length),
          //     presuming the value was positive prior to the '%' operator.
          //   * after being bound to [0, Integer.MAX_INT - 1], the new value is stored in syncRunnerIndex
          //     ensuring that it can't grow without bound and overflow.
          //   * note that the value after the increment must be positive, because the most it could have
          //     been prior was Integer.MAX_INT - 1 and we only increment by 1.
          this.syncRunnerIndex = (this.syncRunnerIndex + 1) % this.syncRunners.length;
          try {
            // Below expects that the offer 'transfers' responsibility for the outstanding syncs to
            // the syncRunner. We should never get an exception in here.
            this.syncRunners[this.syncRunnerIndex].offer(sequence, this.syncFutures,
              this.syncFuturesCount);
          } catch (Exception e) {
            // Should NEVER get here.
            requestLogRoll();
            this.exception = new DamagedWALException("Failed offering sync", e);
          }
        }
        // We may have picked up an exception above trying to offer sync
        if (this.exception != null) {
          cleanupOutstandingSyncsOnException(sequence,
            this.exception instanceof DamagedWALException?
              this.exception:
              new DamagedWALException("On sync", this.exception));
        }
        attainSafePoint(sequence);
        this.syncFuturesCount = 0;
      } catch (Throwable t) {
        LOG.error("UNEXPECTED!!! syncFutures.length=" + this.syncFutures.length, t);
      }
    }
    

到此WAL写日志过程全部解释清楚。里面还有一个回滚。不过FSHlog有个度数据流。所以就可以搞定了。


如果有什么没说清楚的,可以留言。欢迎交流。

在put和delete过程的时候都会有WAL。这一章主要分析这个过程。

重点:
1) WAL在创建region的时候创建。
2)    WAL 存储在HDFS,主要类FSHLog
3) 在put 和delete写入内存之前都会写入WAL,主要是防止机器崩溃时能恢复。
4)用户在协处理器里可以自己调用WALEdit。


1)创建HRegion.java 类 ,具体创建过程可以参考第10节。
  1. public static HRegion createHRegion(final HRegionInfo info, final Path rootDir,
  2. final Path tableDir, final Configuration conf, final HTableDescriptor hTableDescriptor,
  3. final WAL wal, final boolean initialize, final boolean ignoreWAL)
  4. throws IOException {
  5. HRegionFileSystem.createRegionOnFileSystem(conf, fs, tableDir, info);
  6. WAL effectiveWAL = wal;
  7. if (wal == null && !ignoreWAL) {
  8. Configuration confForWAL = new Configuration(conf);
  9. confForWAL.set(HConstants.HBASE_DIR, rootDir.toString());
  10. effectiveWAL = (new WALFactory(confForWAL,
  11. Collections.<WALActionsListener>singletonList(new MetricsWAL()),
  12. "hregion-" + RandomStringUtils.randomNumeric(8))).
  13. getWAL(info.getEncodedNameAsBytes(), info.getTable().getNamespace());
  14. }
  15. }
在工程类里面最重要的。这里的defaultProvider(DefaultWALProvider.class),
  1. provider = getProvider(WAL_PROVIDER, DEFAULT_WAL_PROVIDER, listeners, null);

这个版本1.2.3和1.3.1 有点不同,这里以1.3.1为准
  1. @Override
  2. public WAL getWAL(final byte[] identifier, byte[] namespace) throws IOException {
  3. if (log == null) {
  4. // only lock when need to create wal, and need to lock since
  5. // creating hlog on fs is time consuming
  6. synchronized (walCreateLock) {
  7. if (log == null) {
  8. log = new FSHLog(FileSystem.get(conf), FSUtils.getRootDir(conf),
  9. getWALDirectoryName(factory.factoryId), HConstants.HREGION_OLDLOGDIR_NAME, conf,
  10. listeners, true, logPrefix,
  11. META_WAL_PROVIDER_ID.equals(providerId) ? META_WAL_PROVIDER_ID : null);
  12. }
  13. }
  14. }
  15. return log;
  16. }

所以Log 的目录在:有点混淆。
hdfs = root/WALs/WALFactory.encode/oldWALs/WALFactory/wal.
root=hbase
后缀:.meta 或者为null

这样HLog的文件位置,名字 都有了,并且写入Region中。

这个FSHLog里面定义很多东西,最重要的。

  1. public FSHLog(final FileSystem fs, final Path rootDir, final String logDir,
  2. final String archiveDir, final Configuration conf,
  3. final List<WALActionsListener> listeners,
  4. final boolean failIfWALExists, final String prefix, final String suffix)
  5. throws IOException {
  6. //HLog文件写流  
  7. // rollWriter sets this.hdfs_out if it can.
  8. rollWriter();
  9. this.disruptor =
  10. new Disruptor<RingBufferTruck>(RingBufferTruck.EVENT_FACTORY, preallocatedEventCount,
  11. this.appendExecutor, ProducerType.MULTI, new BlockingWaitStrategy());
  12. this.disruptor.getRingBuffer().next();
  13. this.ringBufferEventHandler =
  14. new RingBufferEventHandler(conf.getInt("hbase.regionserver.hlog.syncer.count", 5),
  15. maxHandlersCount);
  16. this.disruptor.handleExceptionsWith(new RingBufferExceptionHandler());
  17. this.disruptor.handleEventsWith(new RingBufferEventHandler [] {this.ringBufferEventHandler});
  18. // Presize our map of SyncFutures by handler objects.
  19. this.syncFuturesByHandler = new ConcurrentHashMap<Thread, SyncFuture>(maxHandlersCount);
  20. this.disruptor.start();
  21. }
这个FSHLog 里面最要的几个东西
rollWriter()定义了一个FSDataOutputStream这个可以往hlog写文件。 rollWriter 里实现了ProtobufLogWriter 类,在该类的init方法里定义了write 与文件的关系。
FSDataOutputStream nextHdfsOut = null;
if (nextWriter instanceof ProtobufLogWriter) {
  nextHdfsOut = ((ProtobufLogWriter)nextWriter).getStream();
disruptor 这个是一个队列工具。采用生产者与消费者模式。当有时间append时,会调用RingBufferEventHandler的onEven方法。
RingBufferEventHandler new里面定义很多同步的runner 这个主要是写write过程。
  1. RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
  2. this.syncFutures = new SyncFuture[maxHandlersCount];
  3. this.syncRunners = new SyncRunner[syncRunnerCount];
  4. for (int i = 0; i < syncRunnerCount; i++) {
  5. this.syncRunners[i] = new SyncRunner("sync." + i, maxHandlersCount);
  6. }
  7. }


2) 记录数据。
这个主要是put过程了。具体put过程可以参考第七章
主要是HRegin类,因为每一个region都有一个WAL。WALEdit 是处理Wal的工具。
put过程主要哪些地方要写WAL。
1)rowProcessor 的钩子,用户可以定义
2)preBatchMutate
3)写入store 内存时
4)MVCC
5)结束的钩子
这里最重要的写入内存之前
  1. @Override
  2. public void processRowsWithLocks(RowProcessor<?,?> processor, long timeout,
  3. long nonceGroup, long nonce) throws IOException {
  4. //new
  5. WALEdit walEdit = new WALEdit();、
  6. // 1. Run pre-process hook
  7. processor.preProcess(this, walEdit);
  8. MultiVersionConcurrencyControl.WriteEntry writeEntry = null;
  9. WALKey walKey = null;
  10. // 5. Call the preBatchMutate hook
  11. processor.preBatchMutate(this, walEdit);
  12. long txid = 0;
  13. // 6. Append no sync
  14. if (!walEdit.isEmpty()) {
  15. // we use HLogKey here instead of WALKey directly to support legacy coprocessors.
  16. walKey = new HLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
  17. this.htableDescriptor.getTableName(), WALKey.NO_SEQUENCE_ID, now,
  18. processor.getClusterIds(), nonceGroup, nonce, mvcc);
  19. txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(),
  20. walKey, walEdit, true);
  21. }
  22. if(walKey == null){
  23. // since we use wal sequence Id as mvcc, for SKIP_WAL changes we need a "faked" WALEdit
  24. // to get a sequence id assigned which is done by FSWALEntry#stampRegionSequenceId
  25. walKey = this.appendEmptyEdit(this.wal);
  26. }
  27. // 7. Start mvcc transaction
  28. writeEntry = walKey.getWriteEntry();
  29. mvccNum = walKey.getSequenceId();
  30. // 8. Apply to memstore
  31.      Store store = getStore(cell);
  32. addedSize += store.add(cell);
  33. }
  34. // 11. Sync edit log
  35. if (txid != 0) {
  36. syncOrDefer(txid, getEffectiveDurability(processor.useDurability()));
  37. }
  38. walSyncSuccessful = true;
  39. } finally {
  40. if (!mutations.isEmpty() && !walSyncSuccessful) {
  41. if (writeEntry != null) {
  42. mvcc.complete(writeEntry);
  43. writeEntry = null;
  44. }
  45. }
  46. // 13. Roll mvcc forward
  47. if (writeEntry != null) {
  48. mvcc.completeAndWait(writeEntry);
  49. }
  50. }
  51. // 14. Run post-process hook
  52. processor.postProcess(this, walEdit, walSyncSuccessful);
  53. }
  54. }

这个里面都简化了。留下来与Wal 有关的。里面最重要的已经标红。
第一个。
  1. walKey = new HLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
  2. this.htableDescriptor.getTableName(), WALKey.NO_SEQUENCE_ID, now,
  3. processor.getClusterIds(), nonceGroup, nonce, mvcc);
  4. //txid 好比快递号,实时查询状态。
  5. txid = this.wal.append(this.htableDescriptor, this.getRegionInfo(),
  6. walKey, walEdit, true);
第二个
syncOrDefer(txid, getEffectiveDurability(processor.useDurability()));
FSHlog类
  1. @Override
  2. public long append(final HTableDescriptor htd, final HRegionInfo hri, final WALKey key,
  3. final WALEdit edits, final boolean inMemstore) throws IOException {
  4. if (this.closed) throw new IOException("Cannot append; log is closed");
  5. // Make a trace scope for the append. It is closed on other side of the ring buffer by the
  6. // single consuming thread. Don't have to worry about it.
  7. TraceScope scope = Trace.startSpan("FSHLog.append");
  8. FSWALEntry entry = null;
  9. long sequence = this.disruptor.getRingBuffer().next();
  10. try {
  11.         //获得车厢。
  12. RingBufferTruck truck = this.disruptor.getRingBuffer().get(sequence);
  13.         //定义实体
  14. entry = new FSWALEntry(sequence, key, edits, htd, hri, inMemstore);
  15.         //放入车厢
  16. truck.loadPayload(entry, scope.detach());
  17. } finally {
  18.     //告诉车可以走了。
  19. this.disruptor.getRingBuffer().publish(sequence);
  20. }
  21. return sequence;
  22. }
  1. @Override
  2.     //这是一个同步方法,如果车厢已经空了,说明已经结束。否则等待。
  3. public void sync(long txid) throws IOException {
  4. if (this.highestSyncedSequence.get() >= txid){
  5. // Already sync'd.
  6.     //车厢已经空了。
  7. return;
  8. }
  9. TraceScope scope = Trace.startSpan("FSHLog.sync");
  10. try {
    1. //这里会等待空。
  11. scope = Trace.continueSpan(publishSyncThenBlockOnCompletion(scope.detach()));
  12. } finally {
  13. assert scope == NullScope.INSTANCE || !scope.isDetached();
  14. scope.close();
  15. }
  16. }

这个RingBufferEventHandler类的onEven 方法:不解释了,太晚了。这个主要是将下面车厢里的东西拿出来,也就是收货了,然后写到write里面。最后调用SysRunner里的run(这个方法无线循环)方法。run里的write.syn() .写到HDFS。
  1. @Override
    // We can set endOfBatch in the below method if at end of our this.syncFutures array
    public void onEvent(final RingBufferTruck truck, final long sequence, boolean endOfBatch)
    throws Exception {
    
      try {
        if (truck.hasSyncFuturePayload()) {
          this.syncFutures[this.syncFuturesCount++] = truck.unloadSyncFuturePayload();
          // Force flush of syncs if we are carrying a full complement of syncFutures.
          if (this.syncFuturesCount == this.syncFutures.length) endOfBatch = true;
        } else if (truck.hasFSWALEntryPayload()) {
          TraceScope scope = Trace.continueSpan(truck.unloadSpanPayload());
          try {
            FSWALEntry entry = truck.unloadFSWALEntryPayload();
            if (this.exception != null) {
              entry.stampRegionSequenceId();
              // Return to keep processing events coming off the ringbuffer
              return;
            }
            append(entry);
          } catch (Exception e) {
            // Failed append. Record the exception.
            this.exception = e;
            // invoking cleanupOutstandingSyncsOnException when append failed with exception,
            // it will cleanup existing sync requests recorded in syncFutures but not offered to SyncRunner yet,
            // so there won't be any sync future left over if no further truck published to disruptor.
            cleanupOutstandingSyncsOnException(sequence,
                this.exception instanceof DamagedWALException ? this.exception
                    : new DamagedWALException("On sync", this.exception));
            // Return to keep processing events coming off the ringbuffer
            return;
          } finally {
            assert scope == NullScope.INSTANCE || !scope.isDetached();
            scope.close(); // append scope is complete
          }
        } else {
          // What is this if not an append or sync. Fail all up to this!!!
          cleanupOutstandingSyncsOnException(sequence,
            new IllegalStateException("Neither append nor sync"));
          // Return to keep processing.
          return;
        }
    
        // TODO: Check size and if big go ahead and call a sync if we have enough data.
        // This is a sync. If existing exception, fall through. Else look to see if batch.
        if (this.exception == null) {
          // If not a batch, return to consume more events from the ring buffer before proceeding;
          // we want to get up a batch of syncs and appends before we go do a filesystem sync.
          if (!endOfBatch || this.syncFuturesCount <= 0) return;
          // syncRunnerIndex is bound to the range [0, Integer.MAX_INT - 1] as follows:
          //   * The maximum value possible for syncRunners.length is Integer.MAX_INT
          //   * syncRunnerIndex starts at 0 and is incremented only here
          //   * after the increment, the value is bounded by the '%' operator to [0, syncRunners.length),
          //     presuming the value was positive prior to the '%' operator.
          //   * after being bound to [0, Integer.MAX_INT - 1], the new value is stored in syncRunnerIndex
          //     ensuring that it can't grow without bound and overflow.
          //   * note that the value after the increment must be positive, because the most it could have
          //     been prior was Integer.MAX_INT - 1 and we only increment by 1.
          this.syncRunnerIndex = (this.syncRunnerIndex + 1) % this.syncRunners.length;
          try {
            // Below expects that the offer 'transfers' responsibility for the outstanding syncs to
            // the syncRunner. We should never get an exception in here.
            this.syncRunners[this.syncRunnerIndex].offer(sequence, this.syncFutures,
              this.syncFuturesCount);
          } catch (Exception e) {
            // Should NEVER get here.
            requestLogRoll();
            this.exception = new DamagedWALException("Failed offering sync", e);
          }
        }
        // We may have picked up an exception above trying to offer sync
        if (this.exception != null) {
          cleanupOutstandingSyncsOnException(sequence,
            this.exception instanceof DamagedWALException?
              this.exception:
              new DamagedWALException("On sync", this.exception));
        }
        attainSafePoint(sequence);
        this.syncFuturesCount = 0;
      } catch (Throwable t) {
        LOG.error("UNEXPECTED!!! syncFutures.length=" + this.syncFutures.length, t);
      }
    }
    

到此WAL写日志过程全部解释清楚。里面还有一个回滚。不过FSHlog有个度数据流。所以就可以搞定了。


如果有什么没说清楚的,可以留言。欢迎交流。

展开阅读全文

没有更多推荐了,返回首页