rbd 删除慢问题分析

近日在测试环境中遇到了没有写数据rbd删除满的情况，通过对比测试当rbd有如下的feathers删除会变慢

rbd image 'no_lock_rbd':
    size 1024 MB in 256 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.1159643c9869
    format: 2
    features: layering, deep-flatten
    flags: 
    create_timestamp: Wed Jul 24 23:59:23 2019

相比默认创建少了fast-diff,object-map,exclusive-lock属性

通过分析删除逻辑的源码

TrimRequest.cc

int send() override {
    I &image_ctx = this->m_image_ctx;
    assert(image_ctx.owner_lock.is_locked());
    assert(image_ctx.exclusive_lock == nullptr ||
           image_ctx.exclusive_lock->is_lock_owner());

    {
      RWLock::RLocker snap_locker(image_ctx.snap_lock);
      if (image_ctx.object_map != nullptr &&
          !image_ctx.object_map->object_may_exist(m_object_no)) {
        return 1;
      }
    }

    string oid = image_ctx.get_object_name(m_object_no);
    ldout(image_ctx.cct, 10) << "removing " << oid << dendl;

    librados::AioCompletion *rados_completion =
      util::create_rados_callback(this);
    int r = image_ctx.data_ctx.aio_remove(oid, rados_completion);
    assert(r == 0);
    rados_completion->release();
    return 0;
  }

bool ObjectMap<I>::object_may_exist(uint64_t object_no) const
{
  assert(m_image_ctx.snap_lock.is_locked());

  // Fall back to default logic if object map is disabled or invalid
  if (!m_image_ctx.test_features(RBD_FEATURE_OBJECT_MAP,
                                 m_image_ctx.snap_lock)) {
    return true;
  }

  bool flags_set;
  int r = m_image_ctx.test_flags(RBD_FLAG_OBJECT_MAP_INVALID,
                                 m_image_ctx.snap_lock, &flags_set);
  if (r < 0 || flags_set) {
    return true;
  }

  RWLock::RLocker l(m_image_ctx.object_map_lock);
  uint8_t state = (*this)[object_no];
  bool exists = (state != OBJECT_NONEXISTENT);
  ldout(m_image_ctx.cct, 20) << "object_no=" << object_no << " r=" << exists
                 << dendl;
  return exists;
}

所以当关闭object-map属性后，会相rados发送请求，不关闭属性直接放回，造成了删除速度的区别，在关闭object-map的时候删除rbd时，会在磁盘上发现大量的IO。

 

尝试恢复rbd的特性发现时无效的

rbd image 'no_lock_rbd':
    size 1024 MB in 256 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.1159643c9869
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, data-pool
    flags: object map invalid, fast diff invalid
    create_timestamp: Wed Jul 24 23:59:23 2019

从上面的代码看，走到了如下的逻辑里

  bool flags_set;
  int r = m_image_ctx.test_flags(RBD_FLAG_OBJECT_MAP_INVALID,
                                 m_image_ctx.snap_lock, &flags_set);
  if (r < 0 || flags_set) {
    return true;
  }

 

结论：经过分析objectmap应该是记录了object是否写入数据的的情况，目前还没追源码查看，后期看下