rbd基于存储池设置qos和基于image设置qos的区别和原理

设置方式

设置pool qos

rbd config pool set rbddata rbd_qos_read_bps_limit 10

相关参数还有rbd_qos_read_iops_limit,rbd_qos_write_bps_limit,rbd_qos_write_iops_limit

• 设置效果

  [root@k8s-node1 ~]# rados listomapvals -p rbddata rbd_info
  metadata_conf_rbd_qos_read_bps_limit
  value (2 bytes) :
  00000000  31 30                                             |10|
  00000002
  

  可以看到是设置在了rbd_info这个omap属性中。

设置image qos

rbd config image set rbddata/lun1 rbd_qos_read_bps_limit 1

相关参数还有rbd_qos_read_iops_limit,rbd_qos_write_bps_limit,rbd_qos_write_iops_limit

• 设置效果

  [root@k8s-node1 ~]# rados listomapvals -p rbddata rbd_header.64cc674090ca
  metadata_conf_rbd_qos_read_bps_limit
  value (1 bytes) :
  00000000  31                                                |1|
  00000001
  

  可以看到是在rbd的header对象的omap中设置的。

原理

• 先看日志

2023-09-04 16:28:44.430 7f70557fa700 10 []librbd::image::OpenRequest: 0x563ee1faf140 send_refresh
2023-09-04 16:28:44.432 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_mutable_metadata
2023-09-04 16:28:44.433 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_mutable_metadata: r=0
2023-09-04 16:28:44.433 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_parent: legacy=0
2023-09-04 16:28:44.434 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_parent: r=0
2023-09-04 16:28:44.434 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_metadata: start_key=
2023-09-04 16:28:44.435 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_metadata: r=0
2023-09-04 16:28:44.435 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_metadata: start_key=conf_rbd_qos_read_bps_limit
2023-09-04 16:28:44.436 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_metadata: r=0
2023-09-04 16:28:44.436 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_pool_metadata: start_key=
2023-09-04 16:28:44.436 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_pool_metadata: r=0
2023-09-04 16:28:44.436 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 send_v2_get_pool_metadata: start_key=conf_rbd_qos_read_bps_limit
2023-09-04 16:28:44.437 7f70557fa700 10 []librbd::image::RefreshRequest: 0x7f70440065c0 handle_v2_get_pool_metadata: r=0
2023-09-04 16:28:44.437 7f70557fa700 20 []librbd::ImageCtx: apply_metadata
2023-09-04 16:28:44.437 7f70557fa700 20 []librbd::ImageCtx: apply_metadata: rbd_qos_read_bps_limit=1

从日志中可以看出最终rbd_qos_read_bps_limit设置成为了rbd_qos_read_bps_limit=1，说明以image为准。

• 代码

  在open image的时候，会去refresh image，调用栈如下：

  |--RefreshRequest<I>::send_v2_get_metadata()
     |--RefreshRequest<I>::handle_v2_get_metadata
        |--RefreshRequest<I>::send_v2_get_pool_metadata()
           |--RefreshRequest<I>::handle_v2_get_pool_metadata
              |--ImageCtx::apply_metadata
  

  看似是调用获取pool的元数据在后，应该后面覆盖前，其实不然：

    if (!metadata.empty()) {
      m_metadata.insert(metadata.begin(), metadata.end());
      m_last_metadata_key = metadata.rbegin()->first;
      if (boost::starts_with(m_last_metadata_key,
                             ImageCtx::METADATA_CONF_PREFIX)) {
        send_v2_get_pool_metadata();
        return nullptr;
      }
    }
  

  在RefreshRequest::handle_v2_get_pool_metadata中m_metadata.insert行，m_metadata是一个map，如果kv存在，那么再往里插入，也不会成功。

  结论

  	pool设置Qos	pool未设置Qos
  image设置了Qos	image 为准	Image 为准
  image没有设置Qos	Pool 为准	无Qos

  但是对于image创建，如果配置文件中设置了qos，而pool中设置了qos，那么就以pool中为准，这个值会设置到image中。