缓冲大小 设置_Caffeine 详解 —— Caffeine 的读缓冲设计和 maintenance 过程

读缓冲设计

以 BoundedLocalCache 为例：

readBuffer 定义

readBuffer = evicts() || collectKeys() || collectValues() || expiresAfterAccess()
        ? new BoundedBuffer<>()
        : Buffer.disabled();

readBuffer 的类型是 BoundedBuffer，它的实现是一个 Striped Ring 的 buffer。

首先考虑到 readBuffer 的特点是多生产者-单消费者（MPSC），所以只需要考虑写入端的并发问题。

生产者并行（可能存在竞争）读取计数，检查是否有可用的容量，如果可用，则尝试一下 CAS 写入计数的操作。如果增量成功，则生产者会懒发布这个元素。由于 CAS 失败或缓冲区已满而失败时，生产方不会重试或阻塞。

消费者读取计数并获取可用元素，然后清除元素的并懒设置读取计数。

缓冲区分成很多条（这就是 Striped 的含义）。如果检测到竞争，则重新哈希并动态添加新缓冲区来进一步提高并发性，直到一个内部最大值。当重新哈希发现了可用的缓冲区时，生产者可以重试添加元素以确定是否找到了满足的缓冲区，或者是否有必要调整大小。

具体代码不再列举，一些关键参数：

• 每条 ring buffer 允许 16 个元素（每个元素一个 4 字节的引用）
• 最大允许 4 * ceilingNextPowerOfTwo(CPU 数) 条 ring buffer（ceilingNextPowerOfTwo 表示向上取 2 的整数幂）

maintenance 过程

@GuardedBy("evictionLock")
  void maintenance(@Nullable Runnable task) {
    lazySetDrainStatus(PROCESSING_TO_IDLE);

    try {
      drainReadBuffer();

      drainWriteBuffer();
      if (task != null) {
        task.run();
      }

      drainKeyReferences();
      drainValueReferences();

      expireEntries();
      evictEntries();

      climb();
    } finally {
      if ((drainStatus() != PROCESSING_TO_IDLE) || !casDrainStatus(PROCESSING_TO_IDLE, IDLE)) {
        lazySetDrainStatus(REQUIRED);
      }
    }
  }

1. 设置状态位为 PROCESSING_TO_IDLE
2. 清空读缓存
3. 清空写缓存
4. 一般只有 afterWrite 的情况有正在执行的 task（比如添加缓存项时发现已达到最大上限，此时 task 就是正在进行的添加缓存的操作），如果有则执行 task
5. 清空 key 引用和 value 引用队列
6. 处理过期项
7. 淘汰项
8. 调整窗口比例（climbing hill 算法）
9. 尝试将状态从 PROCESSING_TO_IDLE 改成 IDLE，否则记为 REQUIRED

这里有一个小设计：BLCHeader.DrainStatusRef<K, V> 包含一个 volatile 的 drainStatus 状态位 + 15 个 long 的填充位。

注：lazySetDrainStatus 本质调用的是 unsafe 的 putOrderedInt 方法，可以 lazy set 一个 volatile 的变量

清空读缓冲

在上文的读缓冲设计已经介绍过。清空就是将所有的 readBuffer 使用 accessPolicy 清空。

accessPolicy = (evicts() || expiresAfterAccess()) ? this::onAccess : e -> {};

  /** Updates the node's location in the page replacement policy. */
  @GuardedBy("evictionLock")
  void onAccess(Node<K, V> node) {
    if (evicts()) {
      K key = node.getKey();
      if (key == null) {
        return;
      }
      frequencySketch().increment(key);
      if (node.inWindow()) {
        reorder(accessOrderWindowDeque(), node);
      } else if (node.inMainProbation()) {
        reorderProbation(node);
      } else {
        reorder(accessOrderProtectedDeque(), node);
      }
      setHitsInSample(hitsInSample() + 1);
    } else if (expiresAfterAccess()) {
      reorder(accessOrderWindowDeque(), node);
    }
    if (expiresVariable()) {
      timerWheel().reschedule(node);
    }
  }

这个 onAccess 主要是统计 tinyLFU 的计数和将节点在队列中重排序，以及更新统计信息。

清空写缓存

写缓存的实现在 BoundedLocalCache 类的生成代码里，一般是 MpscGrowableArrayQueue：

/** The initial capacity of the write buffer. */
    static final int WRITE_BUFFER_MIN = 4;
    /** The maximum capacity of the write buffer. */
    static final int WRITE_BUFFER_MAX = 128 * ceilingPowerOfTwo(NCPU);

    this.writeBuffer = new MpscGrowableArrayQueue<>(WRITE_BUFFER_MIN, WRITE_BUFFER_MAX);

它的实现来自 https://github.com/JCTools/JCTools 的 2.0 版本。

清空 key 引用和 value 引用队列

@GuardedBy("evictionLock")
  void drainKeyReferences() {
    if (!collectKeys()) {
      return;
    }
    Reference<? extends K> keyRef;
    while ((keyRef = keyReferenceQueue().poll()) != null) {
      Node<K, V> node = data.get(keyRef);
      if (node != null) {
        evictEntry(node, RemovalCause.COLLECTED, 0L);
      }
    }
  }

  /** Drains the weak / soft value references queue. */
  @GuardedBy("evictionLock")
  void drainValueReferences() {
    if (!collectValues()) {
      return;
    }
    Reference<? extends V> valueRef;
    while ((valueRef = valueReferenceQueue().poll()) != null) {
      @SuppressWarnings("unchecked")
      InternalReference<V> ref = (InternalReference<V>) valueRef;
      Node<K, V> node = data.get(ref.getKeyReference());
      if ((node != null) && (valueRef == node.getValueReference())) {
        evictEntry(node, RemovalCause.COLLECTED, 0L);
      }
    }
  }

evictEntry 在下文淘汰项中详细讲。

处理过期项

/** Expires entries that have expired by access, write, or variable. */
  @GuardedBy("evictionLock")
  void expireEntries() {
    long now = expirationTicker().read();
    expireAfterAccessEntries(now);
    expireAfterWriteEntries(now);
    expireVariableEntries(now);

    if (pacer() != null) {
      long delay = getExpirationDelay(now);
      if (delay != Long.MAX_VALUE) {
        pacer().schedule(executor, drainBuffersTask, now, delay);
      }
    }
  }

  /** Expires entries in the access-order queue. */
  @GuardedBy("evictionLock")
  void expireAfterAccessEntries(long now) {
    if (!expiresAfterAccess()) {
      return;
    }

    expireAfterAccessEntries(accessOrderWindowDeque(), now);
    if (evicts()) {
      expireAfterAccessEntries(accessOrderProbationDeque(), now);
      expireAfterAccessEntries(accessOrderProtectedDeque(), now);
    }
  }

  /** Expires entries in an access-order queue. */
  @GuardedBy("evictionLock")
  void expireAfterAccessEntries(AccessOrderDeque<Node<K, V>> accessOrderDeque, long now) {
    long duration = expiresAfterAccessNanos();
    for (;;) {
      Node<K, V> node = accessOrderDeque.peekFirst();
      if ((node == null) || ((now - node.getAccessTime()) < duration)) {
        return;
      }
      evictEntry(node, RemovalCause.EXPIRED, now);
    }
  }

  /** Expires entries on the write-order queue. */
  @GuardedBy("evictionLock")
  void expireAfterWriteEntries(long now) {
    if (!expiresAfterWrite()) {
      return;
    }
    long duration = expiresAfterWriteNanos();
    for (;;) {
      final Node<K, V> node = writeOrderDeque().peekFirst();
      if ((node == null) || ((now - node.getWriteTime()) < duration)) {
        break;
      }
      evictEntry(node, RemovalCause.EXPIRED, now);
    }
  }

  /** Expires entries in the timer wheel. */
  @GuardedBy("evictionLock")
  void expireVariableEntries(long now) {
    if (expiresVariable()) {
      timerWheel().advance(now);
    }
  }

就是将各个队列从头开始拿，如果发现过期则调用 evictEntry 处决。

关于时间轮的逻辑这里暂不展开。

淘汰项

主流程

/** Evicts entries if the cache exceeds the maximum. */
  @GuardedBy("evictionLock")
  void evictEntries() {
    if (!evicts()) {
      return;
    }
    int candidates = evictFromWindow();
    evictFromMain(candidates);
  }

1. 判断是否需要淘汰（没有最大值约束等情况可以不用淘汰）
2. 从窗口淘汰
3. 从主存淘汰

从窗口淘汰

/**
   * Evicts entries from the window space into the main space while the window size exceeds a
   * maximum.
   *
   * @return the number of candidate entries evicted from the window space
   */
  @GuardedBy("evictionLock")
  int evictFromWindow() {
    int candidates = 0;
    Node<K, V> node = accessOrderWindowDeque().peek();
    while (windowWeightedSize() > windowMaximum()) {
      // The pending operations will adjust the size to reflect the correct weight
      if (node == null) {
        break;
      }

      Node<K, V> next = node.getNextInAccessOrder();
      if (node.getWeight() != 0) {
        node.makeMainProbation();
        accessOrderWindowDeque().remove(node);
        accessOrderProbationDeque().add(node);
        candidates++;

        setWindowWeightedSize(windowWeightedSize() - node.getPolicyWeight());
      }
      node = next;
    }

    return candidates;
  }

按获取顺序（access order），从 window 队列转移至 probation 队列，直到窗口区大小降到 windowMaximum 以下

从主存淘汰

/**
   * Evicts entries from the main space if the cache exceeds the maximum capacity. The main space
   * determines whether admitting an entry (coming from the window space) is preferable to retaining
   * the eviction policy's victim. This is decision is made using a frequency filter so that the
   * least frequently used entry is removed.
   *
   * The window space candidates were previously placed in the MRU position and the eviction
   * policy's victim is at the LRU position. The two ends of the queue are evaluated while an
   * eviction is required. The number of remaining candidates is provided and decremented on
   * eviction, so that when there are no more candidates the victim is evicted.
   *
   * @param candidates the number of candidate entries evicted from the window space
   */
  @GuardedBy("evictionLock")
  void evictFromMain(int candidates) {
    int victimQueue = PROBATION;
    Node<K, V> victim = accessOrderProbationDeque().peekFirst();
    Node<K, V> candidate = accessOrderProbationDeque().peekLast();
    while (weightedSize() > maximum()) {
      // Stop trying to evict candidates and always prefer the victim
      if (candidates == 0) {
        candidate = null;
      }

      // Try evicting from the protected and window queues
      if ((candidate == null) && (victim == null)) {
        if (victimQueue == PROBATION) {
          victim = accessOrderProtectedDeque().peekFirst();
          victimQueue = PROTECTED;
          continue;
        } else if (victimQueue == PROTECTED) {
          victim = accessOrderWindowDeque().peekFirst();
          victimQueue = WINDOW;
          continue;
        }

        // The pending operations will adjust the size to reflect the correct weight
        break;
      }

      // Skip over entries with zero weight
      if ((victim != null) && (victim.getPolicyWeight() == 0)) {
        victim = victim.getNextInAccessOrder();
        continue;
      } else if ((candidate != null) && (candidate.getPolicyWeight() == 0)) {
        candidate = candidate.getPreviousInAccessOrder();
        candidates--;
        continue;
      }

      // Evict immediately if only one of the entries is present
      if (victim == null) {
        @SuppressWarnings("NullAway")
        Node<K, V> previous = candidate.getPreviousInAccessOrder();
        Node<K, V> evict = candidate;
        candidate = previous;
        candidates--;
        evictEntry(evict, RemovalCause.SIZE, 0L);
        continue;
      } else if (candidate == null) {
        Node<K, V> evict = victim;
        victim = victim.getNextInAccessOrder();
        evictEntry(evict, RemovalCause.SIZE, 0L);
        continue;
      }

      // Evict immediately if an entry was collected
      K victimKey = victim.getKey();
      K candidateKey = candidate.getKey();
      if (victimKey == null) {
        @NonNull Node<K, V> evict = victim;
        victim = victim.getNextInAccessOrder();
        evictEntry(evict, RemovalCause.COLLECTED, 0L);
        continue;
      } else if (candidateKey == null) {
        candidates--;
        @NonNull Node<K, V> evict = candidate;
        candidate = candidate.getPreviousInAccessOrder();
        evictEntry(evict, RemovalCause.COLLECTED, 0L);
        continue;
      }

      // Evict immediately if the candidate's weight exceeds the maximum
      if (candidate.getPolicyWeight() > maximum()) {
        candidates--;
        Node<K, V> evict = candidate;
        candidate = candidate.getPreviousInAccessOrder();
        evictEntry(evict, RemovalCause.SIZE, 0L);
        continue;
      }

      // Evict the entry with the lowest frequency
      candidates--;
      if (admit(candidateKey, victimKey)) {
        Node<K, V> evict = victim;
        victim = victim.getNextInAccessOrder();
        evictEntry(evict, RemovalCause.SIZE, 0L);
        candidate = candidate.getPreviousInAccessOrder();
      } else {
        Node<K, V> evict = candidate;
        candidate = candidate.getPreviousInAccessOrder();
        evictEntry(evict, RemovalCause.SIZE, 0L);
      }
    }
  }

1. 从缓刑队列（probation）队头取出 victim，从队尾取出刚刚放入的 candidate
2. 只要总大小超过 maximum，循环执行后续的操作
3. 如果 candidate 已经取完了，也就是 candidates = 0，candidate 置为 null
4. 如果 victim 为空和 candidate 都为空，也就是 probation 队列空了，从 protected 队列取 victim；也空了就从 window 队列取 victim。都取完了说明参数配置有问题（比如 maximum 为负数）
5. 不管是受害者还是候选人，weight 为 0 的直接跳过
6. 受害者还是候选人有一方为 null 的时候，那么毫无疑问是将不为 null 的一方过期掉
7. （此时受害人和候选人都不为 null）如果候选人的权重大于最大值，直接淘汰（否则他会淘汰掉所有的项再把自己淘汰掉）
8. 根据 tinyLFU 计算两者频率，判断应该淘汰谁（具体过程在 admit 方法里），无论淘汰谁，都要取前一项作为新的 candidate。

evictEntry

尝试挽救，救不了则执行删除流程，缩减 size，从各种队列清空，记录统计信息，通知监听者

/**
   * Attempts to evict the entry based on the given removal cause. A removal due to expiration or
   * size may be ignored if the entry was updated and is no longer eligible for eviction.
   *
   * @param node the entry to evict
   * @param cause the reason to evict
   * @param now the current time, used only if expiring
   * @return if the entry was evicted
   */
  @GuardedBy("evictionLock")
  @SuppressWarnings({"PMD.CollapsibleIfStatements", "GuardedByChecker"})
  boolean evictEntry(Node<K, V> node, RemovalCause cause, long now) {
    K key = node.getKey();
    @SuppressWarnings("unchecked")
    V[] value = (V[]) new Object[1];
    boolean[] removed = new boolean[1];
    boolean[] resurrect = new boolean[1];
    RemovalCause[] actualCause = new RemovalCause[1];

    data.computeIfPresent(node.getKeyReference(), (k, n) -> {
      if (n != node) {
        return n;
      }
      synchronized (n) {
        value[0] = n.getValue();

        actualCause[0] = (key == null) || (value[0] == null) ? RemovalCause.COLLECTED : cause;
        if (actualCause[0] == RemovalCause.EXPIRED) {
          boolean expired = false;
          if (expiresAfterAccess()) {
            expired |= ((now - n.getAccessTime()) >= expiresAfterAccessNanos());
          }
          if (expiresAfterWrite()) {
            expired |= ((now - n.getWriteTime()) >= expiresAfterWriteNanos());
          }
          if (expiresVariable()) {
            expired |= (n.getVariableTime() <= now);
          }
          if (!expired) {
            resurrect[0] = true;
            return n;
          }
        } else if (actualCause[0] == RemovalCause.SIZE) {
          int weight = node.getWeight();
          if (weight == 0) {
            resurrect[0] = true;
            return n;
          }
        }

        if (key != null) {
          writer.delete(key, value[0], actualCause[0]);
        }
        makeDead(n);
      }
      removed[0] = true;
      return null;
    });

    // The entry is no longer eligible for eviction
    if (resurrect[0]) {
      return false;
    }

    // If the eviction fails due to a concurrent removal of the victim, that removal may cancel out
    // the addition that triggered this eviction. The victim is eagerly unlinked before the removal
    // task so that if an eviction is still required then a new victim will be chosen for removal.
    if (node.inWindow() && (evicts() || expiresAfterAccess())) {
      accessOrderWindowDeque().remove(node);
    } else if (evicts()) {
      if (node.inMainProbation()) {
        accessOrderProbationDeque().remove(node);
      } else {
        accessOrderProtectedDeque().remove(node);
      }
    }
    if (expiresAfterWrite()) {
      writeOrderDeque().remove(node);
    } else if (expiresVariable()) {
      timerWheel().deschedule(node);
    }

    if (removed[0]) {
      statsCounter().recordEviction(node.getWeight(), actualCause[0]);
      if (hasRemovalListener()) {
        // Notify the listener only if the entry was evicted. This must be performed as the last
        // step during eviction to safe guard against the executor rejecting the notification task.
        notifyRemoval(key, value[0], actualCause[0]);
      }
    } else {
      // Eagerly decrement the size to potentially avoid an additional eviction, rather than wait
      // for the removal task to do it on the next maintenance cycle.
      makeDead(node);
    }

    return true;
  }

1. 尝试挽救：如果因为过期时间则再次检查过期时间；如果因为大小，则再次检查权重是否为 0；满足挽救条件则保留
2. （挽救不了）执行 writer 的 delete 方法，然后使用 makeDead 方法缩减 size 并清掉 node 信息（引用置空或特定空值）
3. 从窗口、缓刑或保护队列里删除该节点
4. 有 expiresAfterWrite 配置则从写顺序队列删除；有 expiresAfter 配置则从时间轮队列中移除 node
5. 记录统计信息，通知 removalListener

makeDead

/**
   * Atomically transitions the node to the <tt>dead</tt> state and decrements the
   * <tt>weightedSize</tt>.
   *
   * @param node the entry in the page replacement policy
   */
  @GuardedBy("evictionLock")
  void makeDead(Node<K, V> node) {
    synchronized (node) {
      if (node.isDead()) {
        return;
      }
      if (evicts()) {
        // The node's policy weight may be out of sync due to a pending update waiting to be
        // processed. At this point the node's weight is finalized, so the weight can be safely
        // taken from the node's perspective and the sizes will be adjusted correctly.
        if (node.inWindow()) {
          setWindowWeightedSize(windowWeightedSize() - node.getWeight());
        } else if (node.inMainProtected()) {
          setMainProtectedWeightedSize(mainProtectedWeightedSize() - node.getWeight());
        }
        setWeightedSize(weightedSize() - node.getWeight());
      }
      node.die();
    }
  }

node.die() 根据 key 和 value 引用类型（强弱软）的不同使用不同的方式将字段置空。

调整窗口比例

@GuardedBy("evictionLock")
  void climb() {
    if (!evicts()) {
      return;
    }

    determineAdjustment();
    demoteFromMainProtected();
    long amount = adjustment();
    if (amount == 0) {
      return;
    } else if (amount > 0) {
      increaseWindow();
    } else {
      decreaseWindow();
    }
  }

根据调整结果评估是继续放大还是缩小窗口。