Alluxio基于冷热数据分离的元数据管理策略

前言

---

上篇文章末尾，笔者聊到了一种叫做分层元数据管理模式。它主张的思想是将元数据进行分级对待，比如Cache+Persist层2种，cache拿来用于热点数据的访问，而persist层即持久层则存储那些冷的访问不频繁的数据，以此达到元数据的强扩展性和一个较好的访问性能。当今存储系统Alluxio就是使用了这种分层级对待的元数据管理模式。本文我们就来简单聊聊Alluxio的tier layer的元数据管理。

Alluxio内部元数据管理架构

---

相比较于将元数据全部load到memory然后以此提高快速访问能力的元数据管理方式，Alluxio在这点上做了优化改进，只cache那些active的数据，这是其内部元数据管理的一大特点。对于那些近期没有访问过的冷数据，则保存在本地的rocksdb内。

在Alluxio中，有专门的定义来定义上述元数据的存储，在内存中cache active数据的存储层，我们叫做cache store，底层rocksdb层则叫做baking store。

Alluxio就是基于上面提到的2层store做数据数据然后对外提供数据访问能力，架构图如下所示：

本文笔者这里想重点聊的点在于Cache store如何和上面Rocks store(Baking store)进行数据交互的。

Alluxio的支持异步写出功能的自定义Cache实现

---

在Cache store层，它需要做以下2件事情来保证元数据的正常更新：

• 及时将那些访问频率降低的热点数据移除并写出到baking store里去。
• 有新的数据访问来时，将这些数据从baking store读出来并加载到cache里去。

在上面两点中，毫无疑问，第一点是Alluxio具体要实现。那么Alluxio采用的是什么办法呢？用现有成熟Cache工具，guava cache？Guava cache自带expireAfterAccess能很好的满足上述的使用场景。

不过最终Alluxio并没有使用Guava cahce的方案。这点笔者认为主要的一点在于guava cahce不支持异步的entry过期写出功能。Gauva cache在更新过期entry时并没有开启额外线程的方式来做过期entry的处理，而是放在了后面的每次的cache访问操作里顺带做了。那么这里其实会有一个隐患：当cache很久没有被访问过了，然后下一次cache访问发生在已经超过大部分entry的过期时间之后，那么这时候可能会触发大量的cache更新重新加载的行为。此时Guava Cache本身将会消耗掉很多的CPU来做这样的事情，这也势必会影响Cache对外提供数据访问的能力。另外一点，Gauva Cache的entry更新是要带锁的，如果Cache entry更新的缓慢是会block住其它想要访问此entry的thread的。

结论是说，如果我们想要Cache entry能够被及时的移除以及更新，可以自己实现一个thread来触发更新的行为。下面是Guava cache的Git文档对这块的一个说明解释，里面也提到了为什么Guava Cahce为什么不在内部实现启动线程来做cache过期更新的原因：

When Does Cleanup Happen?
Caches built with CacheBuilder do not perform cleanup and evict values "automatically," or instantly after a value expires, or anything of the sort. Instead, it performs small amounts of maintenance during write operations, or during occasional read operations if writes are rare.

The reason for this is as follows: if we wanted to perform Cache maintenance continuously, we would need to create a thread, and its operations would be competing with user operations for shared locks. Additionally, some environments restrict the creation of threads, which would make CacheBuilder unusable in that environment.

Instead, we put the choice in your hands. If your cache is high-throughput, then you don't have to worry about performing cache maintenance to clean up expired entries and the like. If your cache does writes only rarely and you don't want cleanup to block cache reads, you may wish to create your own maintenance thread that calls Cache.cleanUp() at regular intervals.

If you want to schedule regular cache maintenance for a cache which only rarely has writes, just schedule the maintenance using ScheduledExecutorService.

OK，下面我们就来看看Alluxio内部实现的带异步写出outdated entry功能的cache实现。这里我们对着其代码实现做具体阐述。

首先是上面架构图中的CachingInodeStore的定义：

public final class CachingInodeStore implements InodeStore, Closeable {
   
  private static final Logger LOG = LoggerFactory.getLogger(CachingInodeStore.class);
  // Backing store用户数据写出持久化的store
  private final InodeStore mBackingStore;
  private final InodeLockManager mLockManager;

  // Cache recently-accessed inodes.
  @VisibleForTesting
  final InodeCache mInodeCache;

  // Cache recently-accessed inode tree edges.
  @VisibleForTesting
  final EdgeCache mEdgeCache;

  @VisibleForTesting
  final ListingCache mListingCache;

  // Starts true, but becomes permanently false if we ever need to spill metadata to the backing
  // store. When true, we can optimize lookups for non-existent inodes because we don't need to
  // check the backing store. We can also optimize getChildren by skipping the range query on the
  // backing store.
  private volatile boolean mBackingStoreEmpty;
  ...
  
  public CachingInodeStore(InodeStore backingStore, InodeLockManager lockManager) {
   
    mBackingStore = backingStore;
    mLockManager = lockManager;
    AlluxioConfiguration conf = ServerConfiguration.global();
    int maxSize = conf.getInt(PropertyKey.MASTER_METASTORE_INODE_CACHE_MAX_SIZE);
    Preconditions.checkState(maxSize > 0,
        "Maximum cache size %s must be positive, but is set to %s",
        PropertyKey.MASTER_METASTORE_INODE_CACHE_MAX_SIZE.getName(), maxSize);
    float highWaterMarkRatio = ConfigurationUtils.checkRatio(conf,
        PropertyKey.MASTER_METASTORE_INODE_CACHE_HIGH_WATER_MARK_RATIO);
    // 最高水位的计算
    int highWaterMark = Math.r