RBD: Exclusive Lock 原理

排他锁的作用

一种分布式的实现，如果一个LUN配置了排它锁feature，对于一些配置流程：如创建快照、clone等，IO流程：如写IO。客户端会去获取排他锁，防止同时有两个客户端对同一个LUN进行操作。

锁的实现逻辑

持久化位置

[root@VM-16-7-centos ~]# rados getxattr -p pobj rbd_header.bb145d353930f lock.rbd_lock > 3.txt
[root@VM-16-7-centos ~]# ceph-dencoder import 3.txt type rados::cls::lock::lock_info_t decode dump_json
{
    "lock_type": 1,
    "tag": "",
    "lockers": [
        {
            "id": {
                "locker": "client.766298",
                "cookie": "auto 18446462598732840963"
            },
            "info": {
                "expiration": "0.000000",
                "addr": "172.27.16.7:0/2340907161",
                "description": ""
            }
        }
    ]
}

为了实现排它锁，客户端会写rados对象rbd_header的的xattr属性，从而记录一条锁记录，lock.rbd_lock。

锁的策略

StandardPolicy
AutomaticPolicy
//初始化位置
if (cct->_conf.get_val<bool>("rbd_auto_exclusive_lock_until_manual_request")) {
	exclusive_lock_policy = new exclusive_lock::AutomaticPolicy(this);
} else {
	exclusive_lock_policy = new exclusive_lock::StandardPolicy(this);
}

排他锁分为StandardPolicy和AutomaticPolicy

• StandardPolicy：当一个客户端加锁失败，且检测到了当前锁的owner还活着，那么就会发送一个notify请求给lock的owner，如果当前owner采用该策略，那么他不会释放锁，如果一个客户端想要打破锁，那么需要主动去调用函数rbd_lock_break。

• AutomaticPolicy：当一个客户端加锁失败，且检测到了当前锁的owner还活着，那么就会发送一个notify请求给lock的owner，如果当前owner采用的加锁策略使用的是该策略，那么就会主动释放锁，并发送notify给其他客户端，让其他客户端去抢锁。

锁初始化

|--RefreshRequest<I>::send_v2_init_exclusive_lock
	 |--m_exclusive_lock->init(m_features, ctx);

init 实现

void ExclusiveLock<I>::init(uint64_t features, Context *on_init) {
ceph_assert(m_image_ctx.owner_lock.is_locked());
ldout(m_image_ctx.cct, 10) << dendl;

{
	Mutex::Locker locker(ML<I>::m_lock);
	ML<I>::set_state_initializing();
}

	m_image_ctx.io_work_queue->block_writes(new C_InitComplete(this, features, on_init));
}

void ExclusiveLock<I>::handle_init_complete(uint64_t features) {
	ldout(m_image_ctx.cct, 10) << ": features=" << features << dendl;
	{
		RWLock::RLocker owner_locker(m_image_ctx.owner_lock);
		if (m_image_ctx.clone_copy_on_read ||
		(features & RBD_FEATURE_JOURNALING) != 0) {
			m_image_ctx.io_work_queue->set_require_lock(io::DIRECTION_BOTH, true);
		} else {
			m_image_ctx.io_work_queue->set_require_lock(io::DIRECTION_WRITE, true);
		}
	}

	Mutex::Locker locker(ML<I>::m_lock);
	ML<I>::set_state_unlocked();
}


void RefreshRequest<I>::apply() {
	//根据逻辑把锁更新到image_ctx中
}

初始化排他锁实在RefreshRequest中去完成的，初始化的过程会block io，最终会把RefreshRequest初始化成功的排它锁类的实例赋值到image_ctx中

对于一个请求是否要加锁的判断逻辑

void ImageRequestWQ<I>::set_require_lock(Direction direction, bool enabled) {
  CephContext *cct = m_image_ctx.cct;
  ldout(cct, 20) << dendl;

  bool wake_up = false;
  {
    RWLock::WLocker locker(m_lock);
    switch (direction) {
    case DIRECTION_READ:
      wake_up = (enabled != m_require_lock_on_read);
      m_require_lock_on_read = enabled;
      break;
    case DIRECTION_WRITE:
      wake_up = (enabled != m_require_lock_on_write);
      m_require_lock_on_write = enabled;
      break;
    case DIRECTION_BOTH:
      wake_up = (enabled != m_require_lock_on_read ||
                 enabled != m_require_lock_on_write);
      m_require_lock_on_read = enabled;
      m_require_lock_on_write = enabled;
      break;
    }
  }

  // wake up the thread pool whenever the state changes so that
  // we can re-request the lock if required
  if (wake_up) {
    this->signal();
  }
}

加锁过程

创建快照加排它锁日志（前提LUN已有其他客户端加了AutomaticPolicy排它锁）

2023-11-03 16:47:39.461 7f83059b6c80 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.461 7f83059b6c80 20 []librbd::Operations: send_acquire_exclusive_lock
2023-11-03 16:47:39.461 7f83059b6c80 10 []librbd::ManagedLock: 0x7f82d0006c20 try_acquire_lock: 
2023-11-03 16:47:39.461 7f83059b6c80 10 []librbd::ManagedLock: 0x7f82d0006c20 send_acquire_lock: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 pre_acquire_lock_handler
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 send_prepare_lock: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 prepare_lock
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_prepare_lock_unlock
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 handle_prepare_lock: r=0
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 send_flush_notifies: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 handle_flush_notifies: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_pre_acquire_lock: r=0
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_get_locker: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 send_get_lockers: 
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 handle_get_lockers: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 handle_get_lockers: retrieved exclusive locker: client.5075054@172.27.16.3:0/1315755207
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 finish: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_get_locker: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_lock: entity=client.4816263, cookie=auto 140199610896496
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_lock: r=-16
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_break_lock: 
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 send_get_watchers: 
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: r=0
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: watcher=[addr=172.27.16.3:0/3462107589, entity=client.4816263]
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: watcher=[addr=172.27.16.3:0/1315755207, entity=client.5075054]
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: lock owner is still alive
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 finish: r=-11
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_break_lock: r=-11
2023-11-03 16:47:39.465 7f82d9ffb700  5 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_break_lock: lock owner is still alive
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: r=-11
2023-11-03 16:47:39.465 7f82d97fa700  5 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: unable to acquire exclusive lock
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 post_acquire_lock_handler: r=-11
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 handle_prepare_lock_complete
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_post_acquire_lock: r=0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::Operations: handle_acquire_exclusive_lock: r=0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::Operations: send_remote_request
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 send_notify
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:39.465 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0006ea0 C_NotifyAck: id=554050815783, handle=140199610896496
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::ExclusiveLock: 0x7f82d0006c20 accept_request=0 (request_type=0)
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0006ea0 finish: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 handle_notify: r=0
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::Operations: handle_remote_request: r=-95
2023-11-03 16:47:39.466 7f82d97fa700  5 []librbd::Operations: snap_create not supported by current lock owner

从日志流程可以看出，AcquireRequest中先去获取锁实例，并且去加锁，且尝试去break lock，但是break时发现owner还活着，因此需要重试，AcquireRequest获取锁总体逻辑：

|--AcquireRequest<I>::send_get_locker
	 |--AcquireRequest<I>::send
	    |--AcquireRequest<I>::send_get_locker
	       |--AcquireRequest<I>::handle_get_locker
|--AcquireRequest<I>::handle_get_locker  //只要过程不报错就回去尝试Locker，否则流程不会再往下走
|--AcquireRequest<I>::send_lock   //发请求去加锁
|--AcquireRequest<I>::handle_lock(int r) 
|--AcquireRequest<I>::send_break_lock
   |--BreakRequest<I>::send_get_watchers
   |--BreakRequest<I>::handle_get_watchers
   |--BreakRequest<I>::send_get_locker
   |--BreakRequest<I>::send_blacklist
   |--BreakRequest<I>::handle_blacklist
   |--BreakRequest<I>::send_break_lock

其中重点看一下break锁的核心逻辑：

void BreakRequest<I>::handle_get_watchers(int r) {
  ldout(m_cct, 10) << "r=" << r << dendl;

  if (r == 0) {
    r = m_watchers_ret_val;
  }
  if (r < 0) {
    lderr(m_cct) << "failed to retrieve watchers: " << cpp_strerror(r)
                 << dendl;
    finish(r);
    return;
  }

  bool found_alive_locker = false;
  for (auto &watcher : m_watchers) {
    ldout(m_cct, 20) << "watcher=["
                     << "addr=" << watcher.addr << ", "
                     << "entity=client." << watcher.watcher_id << "]" << dendl;

    if ((strncmp(m_locker.address.c_str(),
                 watcher.addr, sizeof(watcher.addr)) == 0) &&
        (m_locker.handle == watcher.cookie)) {
      ldout(m_cct, 10) << "lock owner is still alive" << dendl;
      found_alive_locker = true;
    }
  }

  if (!m_force_break_lock && found_alive_locker) {
    finish(-EAGAIN);
    return;
  }

  send_get_locker();
}

28行:

• 如果带了强制打破标记(rbd_lock_break会带改标记)，或者检测到锁的owner已经挂了，那么流程往下走，最终锁owner的客户端会被加入到黑名单，且锁也会被break。

• 如果不带强制打破标记，且发现当前locker的owner还活着，那么就返回重试的错误码。

notify机制

如果AcquireRequest没有成功加锁，在ExclusiveLock<I>::post_acquire_lock_handler中会去notify当前锁的owner释放锁，这里的调用栈需要标注一下，否则容易看混。

|--ManagedLock<I>::handle_acquire_lock
|--ExclusiveLock<I>::post_acquire_lock_handler
|--ImageWatcher<I>::notify_request_lock
	 |--ImageWatcher<I>::notify_lock_owner
   |--ImageWatcher<I>::handle_request_lock //notify
|--PostAcquireRequest<I>::send 
|--PostAcquireRequest<I>::send_refresh
|--RefreshRequest<I>::send

bool ImageWatcher<I>::handle_payload(const RequestLockPayload &payload,
                                     C_NotifyAck *ack_ctx) {
  ldout(m_image_ctx.cct, 10) << this << " exclusive lock requested" << dendl;
  if (payload.client_id == get_client_id()) {
    return true;
  }

  RWLock::RLocker l(m_image_ctx.owner_lock);
  if (m_image_ctx.exclusive_lock != nullptr &&
      m_image_ctx.exclusive_lock->is_lock_owner()) {
    int r = 0;
    bool accept_request = m_image_ctx.exclusive_lock->accept_request(
      exclusive_lock::OPERATION_REQUEST_TYPE_GENERAL, &r);

    if (accept_request) {
      ceph_assert(r == 0);
      Mutex::Locker owner_client_id_locker(m_owner_client_id_lock);
      if (!m_owner_client_id.is_valid()) {
        return true;
      }

      ldout(m_image_ctx.cct, 10) << this << " queuing release of exclusive lock"
                                 << dendl;
      r = m_image_ctx.get_exclusive_lock_policy()->lock_requested(
        payload.force);
    }
    encode(ResponseMessage(r), ack_ctx->out);
  }
  return true;
}

上面函数是owner收到其他客户端释放锁请求notify的处理函数，即客户端发送notify_request_lock后，owner的处理函数。

• policy的实现如果是StandardPolicy，lock_requested函数不会释放锁。

• policy的实现如果是AutomaticPolicy，lock_requested函数会释放锁。

该函数会回调通过encode(ResponseMessage®, ack_ctx->out)发response，发起notify的客户端收到该response后，就会回调ImageWatcher::handle_request_lock，如果返回结果r=0，该函数会起一个定时器，定时去获取锁。这个定时器是有意义的，意义在于假设owner的策略是StandardPolicy，那么此时owner并不会去释放锁，而如果有人为的去调用rbd_lock_break打破了锁，但是这时并不会notify该客户端，那么这个定时器触发的时候，就能够尝试重新去获取锁。

前面提到，owner如果采用AutomaticPolicy策略，lock_requested会释放锁，由于释放锁是异步的，因此不会马上发送notify，等释放锁以后，owner会发notify其他watcher的客户端，收到notify的客户端会调用下面函数：

bool ImageWatcher<I>::handle_payload(const ReleasedLockPayload &payload,
                                     C_NotifyAck *ack_ctx) {
  ldout(m_image_ctx.cct, 10) << this << " exclusive lock released" << dendl;

  bool cancel_async_requests = true;
  if (payload.client_id.is_valid()) {
    Mutex::Locker l(m_owner_client_id_lock);
    if (payload.client_id != m_owner_client_id) {
      ldout(m_image_ctx.cct, 10) << this << " unexpected owner: "
                                 << payload.client_id << " != "
                                 << m_owner_client_id << dendl;
      cancel_async_requests = false;
    } else {
      set_owner_client_id(ClientId());
    }
  }

  RWLock::RLocker owner_locker(m_image_ctx.owner_lock);
  if (cancel_async_requests &&
      (m_image_ctx.exclusive_lock == nullptr ||
       !m_image_ctx.exclusive_lock->is_lock_owner())) {
    schedule_cancel_async_requests();
  }

  // alert the exclusive lock state machine that the lock is available
  if (m_image_ctx.exclusive_lock != nullptr &&
      !m_image_ctx.exclusive_lock->is_lock_owner()) {
    m_task_finisher->cancel(TASK_CODE_REQUEST_LOCK);
    m_image_ctx.exclusive_lock->handle_peer_notification(0);
  }
  return true;
}

该函数最终会调用m_image_ctx.exclusive_lock->handle_peer_notification(0);该函数最终会去重新会去锁。

附录

tcmu 显式加排他锁和主动break lock的例子

• 加锁

  int lock_acquire(ImageCtx *ictx, rbd_lock_mode_t lock_mode)
  {
    ...
  
      if (ictx->get_exclusive_lock_policy()->may_auto_request_lock())
      {
        ictx->set_exclusive_lock_policy(
          new exclusive_lock::StandardPolicy(ictx));
      }
    return 0;
  }
  
  

• break 锁

static int tcmu_rbd_lock_break(struct tcmu_device *dev)
{
	struct tcmu_rbd_state *state = tcmur_dev_get_private(dev);
	rbd_lock_mode_t lock_mode;
	char *owners[1];
	size_t num_owners = 1;
	int ret;

	ret = rbd_lock_get_owners(state->image, &lock_mode, owners,
				  &num_owners);
	if (ret == -ENOENT || (!ret && !num_owners))
		return 0;

	if (ret < 0) {
		tcmu_dev_err(dev, "Could not get lock owners to break lock %d\n",
			     ret);
		return ret;
	}

	if (lock_mode != RBD_LOCK_MODE_EXCLUSIVE) {
		tcmu_dev_err(dev, "Invalid lock type (%d) found\n", lock_mode);
		ret = -EIO;
		goto free_owners;
	}

	tcmu_dev_dbg(dev, "Attempting to break lock from %s.\n", owners[0]);

	ret = rbd_lock_break(state->image, lock_mode, owners[0]);
	if (ret < 0)
		tcmu_dev_err(dev, "Could not break lock from %s. (Err %d)\n",
			     owners[0], ret);
free_owners:
	rbd_lock_get_owners_cleanup(owners, num_owners);
	return ret;
}

创建快照加排它锁总体日志（前提LUN已有其他客户端加了AutomaticPolicy排它锁）

[root@k8s-master ~]# rbd snap create 00000001-fs.meta/lun1@lun1_snap3 --debug-limit=20 --debug-rbd=20
2023-11-03 16:47:39.448 7f83059b6c80  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 ImageRequestWQ: ictx=0x55b9509b27d0
2023-11-03 16:47:39.448 7f83059b6c80 20 []librbd::ImageState: 0x55b950a2c400 open
2023-11-03 16:47:39.448 7f83059b6c80 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_open_unlock
2023-11-03 16:47:39.448 7f83059b6c80 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_detect_header
2023-11-03 16:47:39.450 7f82d9ffb700 10 []librbd::image::OpenRequest: handle_v2_detect_header: r=0
2023-11-03 16:47:39.450 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_get_id
2023-11-03 16:47:39.450 7f82d9ffb700 10 []librbd::image::OpenRequest: handle_v2_get_id: r=0
2023-11-03 16:47:39.450 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_get_initial_metadata
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: handle_v2_get_initial_metadata: r=0
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_get_create_timestamp
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 handle_v2_get_create_timestamp: r=0
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_get_access_modify_timestamp
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 handle_v2_get_access_modify_timestamp: r=0
2023-11-03 16:47:39.453 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_v2_get_data_pool
2023-11-03 16:47:39.454 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 handle_v2_get_data_pool: r=0
2023-11-03 16:47:39.454 7f82d9ffb700 10 []librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.4d296a5b7521b8 format rbd_data.4d296a5b7521b8.%016llx
2023-11-03 16:47:39.454 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_refresh
2023-11-03 16:47:39.454 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_mutable_metadata
2023-11-03 16:47:39.455 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_mutable_metadata: r=0
2023-11-03 16:47:39.455 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_parent: legacy=0
2023-11-03 16:47:39.456 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_parent: r=0
2023-11-03 16:47:39.456 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_metadata: start_key=
2023-11-03 16:47:39.456 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_metadata: r=0
2023-11-03 16:47:39.456 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_pool_metadata: start_key=
2023-11-03 16:47:39.457 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_pool_metadata: r=0
2023-11-03 16:47:39.457 7f82d9ffb700 20 []librbd::ImageCtx: apply_metadata
2023-11-03 16:47:39.457 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_group
2023-11-03 16:47:39.457 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_group: r=0
2023-11-03 16:47:39.457 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_get_snapshots
2023-11-03 16:47:39.458 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_get_snapshots: r=0
2023-11-03 16:47:39.458 7f82d9ffb700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_init_exclusive_lock
2023-11-03 16:47:39.458 7f82d9ffb700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 init
2023-11-03 16:47:39.458 7f82d9ffb700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 block_writes: 0x55b9509b27d0, num=1
2023-11-03 16:47:39.458 7f82d9ffb700 20 []librbd::io::AsyncOperation: 0x7f82d000be18 start_op
2023-11-03 16:47:39.458 7f82d9ffb700 20 []librbd::io::ImageRequest: 0x7f82d9ff6340 send: aio_flush: ictx=0x55b9509b27d0, completion=0x7f82d000bcf0
2023-11-03 16:47:39.458 7f82d9ffb700 20 []librbd::io::AioCompletion: 0x7f82d000bcf0 set_request_count: pending=1
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82d000bf60
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82d000bcf0 complete_request: cb=1, pending=0
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82d000bcf0 finalize: r=0
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 handle_init_complete: features=61
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::ImageRequestWQ: 0x55b950a2d160 set_require_lock: 
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_init_exclusive_lock: r=0
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_v2_apply
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82d000be18 finish_op
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_v2_apply
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::image::RefreshRequest: 0x7f82d0006820 apply
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::image::RefreshRequest: new snapshot id=4 name=lun1_snap1 size=1073741824
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 send_flush_aio
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c40020c8 start_op
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::ImageRequest: 0x7f82d97f5360 send: aio_flush: ictx=0x55b9509b27d0, completion=0x7f82c4001fa0
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4001fa0 set_request_count: pending=1
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4002270
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4001fa0 complete_request: cb=1, pending=0
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4001fa0 finalize: r=0
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::RefreshRequest: 0x7f82d0006820 handle_flush_aio: r=0
2023-11-03 16:47:39.458 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c40020c8 finish_op
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::OpenRequest: handle_refresh: r=0
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_init_cache
2023-11-03 16:47:39.458 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 init: 
2023-11-03 16:47:39.458 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 init: enabling caching...
2023-11-03 16:47:39.458 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 init: Initial cache settings: size=33554432 num_objects=10 max_dirty=25165824 target_dirty=16777216 max_dirty_age=1
2023-11-03 16:47:39.458 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 init:  cache bytes 33554432 -> about 1048 objects
2023-11-03 16:47:39.458 7f82d97fa700  5 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 register_object_dispatch: object_dispatch_layer=1
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 send_register_watch
2023-11-03 16:47:39.458 7f82d97fa700 10 []librbd::Watcher: 0x7f82d0005200 register_watch: 
2023-11-03 16:47:39.461 7f82d9ffb700 10 []librbd::Watcher: 0x7f82d0005200 handle_register_watch: r=0
2023-11-03 16:47:39.461 7f82d9ffb700 10 []librbd::image::OpenRequest: 0x55b950a2fc00 handle_register_watch: r=0
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 handle_open: r=0
2023-11-03 16:47:39.461 7f83059b6c80  5 []librbd::Operations: 0x55b9509c0670 snap_create: snap_name=lun1_snap3
2023-11-03 16:47:39.461 7f83059b6c80 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.461 7f83059b6c80 20 []librbd::Operations: send_acquire_exclusive_lock
2023-11-03 16:47:39.461 7f83059b6c80 10 []librbd::ManagedLock: 0x7f82d0006c20 try_acquire_lock: 
2023-11-03 16:47:39.461 7f83059b6c80 10 []librbd::ManagedLock: 0x7f82d0006c20 send_acquire_lock: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 pre_acquire_lock_handler
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 send_prepare_lock: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 prepare_lock
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_prepare_lock_unlock
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 handle_prepare_lock: r=0
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 send_flush_notifies: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4002250 handle_flush_notifies: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_pre_acquire_lock: r=0
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_get_locker: 
2023-11-03 16:47:39.461 7f82d97fa700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 send_get_lockers: 
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 handle_get_lockers: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 handle_get_lockers: retrieved exclusive locker: client.5075054@172.27.16.3:0/1315755207
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015420 finish: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_get_locker: r=0
2023-11-03 16:47:39.462 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_lock: entity=client.4816263, cookie=auto 140199610896496
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_lock: r=-16
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 send_break_lock: 
2023-11-03 16:47:39.464 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 send_get_watchers: 
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: r=0
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: watcher=[addr=172.27.16.3:0/3462107589, entity=client.4816263]
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: watcher=[addr=172.27.16.3:0/1315755207, entity=client.5075054]
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 handle_get_watchers: lock owner is still alive
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007460 finish: r=-11
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_break_lock: r=-11
2023-11-03 16:47:39.465 7f82d9ffb700  5 []librbd::managed_lock::AcquireRequest: 0x7f82c4015690 handle_break_lock: lock owner is still alive
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: r=-11
2023-11-03 16:47:39.465 7f82d97fa700  5 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: unable to acquire exclusive lock
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 post_acquire_lock_handler: r=-11
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 handle_prepare_lock_complete
2023-11-03 16:47:39.465 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_post_acquire_lock: r=0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::Operations: handle_acquire_exclusive_lock: r=0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::Operations: send_remote_request
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 send_notify
2023-11-03 16:47:39.465 7f82d97fa700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:39.465 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0006ea0 C_NotifyAck: id=554050815783, handle=140199610896496
2023-11-03 16:47:39.465 7f82d9ffb700 20 []librbd::ExclusiveLock: 0x7f82d0006c20 accept_request=0 (request_type=0)
2023-11-03 16:47:39.465 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0006ea0 finish: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 handle_notify: r=0
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::Operations: handle_remote_request: r=-95
2023-11-03 16:47:39.466 7f82d97fa700  5 []librbd::Operations: snap_create not supported by current lock owner
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.466 7f82d97fa700 20 []librbd::Operations: send_acquire_exclusive_lock
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 acquire_lock: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 send_acquire_lock: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 pre_acquire_lock_handler
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4015a30 send_prepare_lock: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 prepare_lock
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_prepare_lock_unlock
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4015a30 handle_prepare_lock: r=0
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4015a30 send_flush_notifies: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c4015a30 handle_flush_notifies: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_pre_acquire_lock: r=0
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 send_get_locker: 
2023-11-03 16:47:39.466 7f82d97fa700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015950 send_get_lockers: 
2023-11-03 16:47:39.466 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015950 handle_get_lockers: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015950 handle_get_lockers: retrieved exclusive locker: client.5075054@172.27.16.3:0/1315755207
2023-11-03 16:47:39.466 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4015950 finish: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 handle_get_locker: r=0
2023-11-03 16:47:39.466 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 send_lock: entity=client.4816263, cookie=auto 140199610896496
2023-11-03 16:47:39.472 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 handle_lock: r=-16
2023-11-03 16:47:39.472 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 send_break_lock: 
2023-11-03 16:47:39.472 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 send_get_watchers: 
2023-11-03 16:47:39.472 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 handle_get_watchers: r=0
2023-11-03 16:47:39.473 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 handle_get_watchers: watcher=[addr=172.27.16.3:0/3462107589, entity=client.4816263]
2023-11-03 16:47:39.473 7f82d9ffb700 20 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 handle_get_watchers: watcher=[addr=172.27.16.3:0/1315755207, entity=client.5075054]
2023-11-03 16:47:39.473 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 handle_get_watchers: lock owner is still alive
2023-11-03 16:47:39.473 7f82d9ffb700 10 []librbd::managed_lock::BreakRequest: 0x7f82d0007500 finish: r=-11
2023-11-03 16:47:39.473 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 handle_break_lock: r=-11
2023-11-03 16:47:39.473 7f82d9ffb700  5 []librbd::managed_lock::AcquireRequest: 0x7f82c4017eb0 handle_break_lock: lock owner is still alive
2023-11-03 16:47:39.473 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: r=-11
2023-11-03 16:47:39.473 7f82d97fa700  5 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: unable to acquire exclusive lock
2023-11-03 16:47:39.473 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 post_acquire_lock_handler: r=-11
2023-11-03 16:47:39.473 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 handle_prepare_lock_complete
2023-11-03 16:47:39.473 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.473 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 notify request lock
2023-11-03 16:47:39.473 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 send_notify
2023-11-03 16:47:39.473 7f82d97fa700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:39.473 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_post_acquire_lock: r=-125
2023-11-03 16:47:39.474 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:39.474 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 C_NotifyAck: id=554050815784, handle=140199610896496
2023-11-03 16:47:39.474 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 exclusive lock requested
2023-11-03 16:47:39.474 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 finish: r=0
2023-11-03 16:47:39.474 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:39.474 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:39.474 7f82d97fa700 20 []librbd::image_watcher::NotifyLockOwner: 0x7f82c40153d0 handle_notify: r=0
2023-11-03 16:47:39.474 7f82d97fa700 15 []librbd::ImageWatcher: 0x7f82d0005200 will retry in 10 seconds
2023-11-03 16:47:39.474 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.474 7f82d97fa700 15 []librbd::ImageWatcher: 0x7f82d0005200 requesting exclusive lock
2023-11-03 16:47:39.480 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 C_NotifyAck: id=554050815785, handle=140199610896496
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 exclusive lock released
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 unexpected owner: [5075054,18446462598732840961] != [0,0]
2023-11-03 16:47:39.480 7f82d9ffb700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =0
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 handle_peer_notification
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::ManagedLock: 0x7f82d0006c20 send_acquire_lock: 
2023-11-03 16:47:39.480 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 finish: r=0
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 pre_acquire_lock_handler
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c40023a0 send_prepare_lock: 
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 prepare_lock
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_prepare_lock_unlock
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c40023a0 handle_prepare_lock: r=0
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c40023a0 send_flush_notifies: 
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::exclusive_lock::PreAcquireRequest: 0x7f82c40023a0 handle_flush_notifies: 
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_pre_acquire_lock: r=0
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4016ae0 send_get_locker: 
2023-11-03 16:47:39.480 7f82d97fa700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4016960 send_get_lockers: 
2023-11-03 16:47:39.481 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4016960 handle_get_lockers: r=0
2023-11-03 16:47:39.481 7f82d9ffb700 20 []librbd::managed_lock::GetLockerRequest: 0x7f82c4016960 handle_get_lockers: no lockers detected
2023-11-03 16:47:39.481 7f82d9ffb700 10 []librbd::managed_lock::GetLockerRequest: 0x7f82c4016960 finish: r=-2
2023-11-03 16:47:39.481 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4016ae0 handle_get_locker: r=-2
2023-11-03 16:47:39.481 7f82d9ffb700 20 []librbd::managed_lock::AcquireRequest: 0x7f82c4016ae0 handle_get_locker: no lockers detected
2023-11-03 16:47:39.481 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4016ae0 send_lock: entity=client.4816263, cookie=auto 140199610896496
2023-11-03 16:47:39.483 7f82d9ffb700 10 []librbd::managed_lock::AcquireRequest: 0x7f82c4016ae0 handle_lock: r=0
2023-11-03 16:47:39.483 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: r=0
2023-11-03 16:47:39.483 7f82d97fa700  5 []librbd::ManagedLock: 0x7f82d0006c20 handle_acquire_lock: successfully acquired exclusive lock
2023-11-03 16:47:39.483 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 post_acquire_lock_handler: r=0
2023-11-03 16:47:39.483 7f82d97fa700 10 []librbd::exclusive_lock::PostAcquireRequest: 0x7f82c4017e10 send_open_object_map: 
2023-11-03 16:47:39.483 7f82d97fa700 20 []librbd::object_map::RefreshRequest: 0x7f82c4016cd0 send: object_count=256
2023-11-03 16:47:39.483 7f82d97fa700 10 []librbd::object_map::RefreshRequest: 0x7f82c4016cd0 send_lock: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:39.483 7f82d97fa700 10 []librbd::object_map::LockRequest: 0x7f82c4016dc0 send_lock: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:39.485 7f82d9ffb700 10 []librbd::object_map::LockRequest: 0x7f82c4016dc0 handle_lock: r=0
2023-11-03 16:47:39.485 7f82d9ffb700 10 []librbd::object_map::RefreshRequest: 0x7f82c4016cd0 handle_lock
2023-11-03 16:47:39.485 7f82d9ffb700 10 []librbd::object_map::RefreshRequest: 0x7f82c4016cd0 send_load: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:39.486 7f82d9ffb700 10 []librbd::object_map::RefreshRequest: 0x7f82c4016cd0 handle_load: r=0
2023-11-03 16:47:39.486 7f82d9ffb700 20 []librbd::object_map::RefreshRequest: refreshed object map: num_objs=256
2023-11-03 16:47:39.486 7f82d9ffb700 10 []librbd::exclusive_lock::PostAcquireRequest: 0x7f82c4017e10 handle_open_object_map: r=0
2023-11-03 16:47:39.486 7f82d9ffb700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 handle_post_acquiring_lock
2023-11-03 16:47:39.486 7f82d9ffb700 10 []librbd::ImageState: 0x55b950a2c400 handle_prepare_lock_complete
2023-11-03 16:47:39.486 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 handle_post_acquired_lock: r=0
2023-11-03 16:47:39.486 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 notify acquired lock
2023-11-03 16:47:39.486 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 current lock owner: [4816263,140199610896496]
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::ImageRequestWQ: 0x55b950a2d160 set_require_lock: 
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 unblock_writes: 0x55b9509b27d0, num=0
2023-11-03 16:47:39.486 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_post_acquire_lock: r=0
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::Operations: handle_acquire_exclusive_lock: r=0
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =1
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::Operations: send_local_request
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =1
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::Operations: 0x55b9509c0670 execute_snap_create: snap_name=lun1_snap3
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 send_suspend_requests
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 send_suspend_aio
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 block_writes: 0x55b9509b27d0, num=1
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 start_op
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::ImageRequest: 0x7f82d97f5270 send: aio_flush: ictx=0x55b9509b27d0, completion=0x7f82c4017930
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 set_request_count: pending=1
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4018fc0
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 flush: 
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4018fc0
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 complete_request: cb=1, pending=0
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 finalize: r=0
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 handle_suspend_aio: r=0
2023-11-03 16:47:39.486 7f82d97fa700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 send_allocate_snap_id
2023-11-03 16:47:39.486 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 finish_op
2023-11-03 16:47:39.487 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:39.487 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 C_NotifyAck: id=554050815786, handle=140199610896496
2023-11-03 16:47:39.487 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 image exclusively locked announcement
2023-11-03 16:47:39.487 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 current lock owner: [4816263,140199610896496]
2023-11-03 16:47:39.487 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007710 finish: r=0
2023-11-03 16:47:39.487 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:39.487 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:40.269 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 handle_allocate_snap_id: r=0, snap_id=6
2023-11-03 16:47:40.269 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 send_create_snap
2023-11-03 16:47:40.269 7f82d9ffb700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =1
2023-11-03 16:47:40.283 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 handle_create_snap: r=0
2023-11-03 16:47:40.283 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 send_create_object_map
2023-11-03 16:47:40.283 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 send_read_map: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:40.283 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 should_complete: state=READ_MAP, r=0
2023-11-03 16:47:40.283 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 send_write_map: snap_oid=rbd_object_map.4d296a5b7521b8.0000000000000006
2023-11-03 16:47:40.288 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 should_complete: state=WRITE_MAP, r=0
2023-11-03 16:47:40.288 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 send_add_snapshot: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:40.296 7f82d9ffb700  5 []librbd::object_map::SnapshotCreateRequest: 0x7f82d0007510 should_complete: state=ADD_SNAPSHOT, r=0
2023-11-03 16:47:40.296 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 handle_create_object_map: r=0
2023-11-03 16:47:40.296 7f82d9ffb700  5 []librbd::SnapshotCreateRequest: 0x7f82c4016cd0 update_snap_context
2023-11-03 16:47:40.296 7f82d9ffb700 20 []librbd::ManagedLock: 0x7f82d0006c20 is_lock_owner: =1
2023-11-03 16:47:40.296 7f82d9ffb700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 unblock_writes: 0x55b9509b27d0, num=0
2023-11-03 16:47:40.296 7f82d9ffb700 10 []librbd::Request: 0x7f82c4016cd0 create_context_finisher
2023-11-03 16:47:40.296 7f82d9ffb700 10 []librbd::Request: 0x7f82c4016cd0 finish: r=0
2023-11-03 16:47:40.296 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c400 handle_update_notification: refresh_seq = 1, last_refresh = 0
2023-11-03 16:47:40.296 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::notify
2023-11-03 16:47:40.296 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200: notify_header_update
2023-11-03 16:47:40.296 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:40.297 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=0
2023-11-03 16:47:40.297 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007510 C_NotifyAck: id=558345783083, handle=140199610896496
2023-11-03 16:47:40.297 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d0005200 image header updated
2023-11-03 16:47:40.297 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c400 handle_update_notification: refresh_seq = 2, last_refresh = 0
2023-11-03 16:47:40.297 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::notify
2023-11-03 16:47:40.297 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c400 flush_update_watchers
2023-11-03 16:47:40.297 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::flush
2023-11-03 16:47:40.297 7f82d9ffb700 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::flush: completing flush
2023-11-03 16:47:40.297 7f82d9ffb700 10 []librbd::ImageWatcher: 0x7f82d000e570 C_ResponseMessage: r=0
2023-11-03 16:47:40.297 7f82d9ffb700 10 []librbd::Watcher::C_NotifyAck 0x7f82d0007510 finish: r=0
2023-11-03 16:47:40.298 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:40.298 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::Operations: handle_local_request: r=0
2023-11-03 16:47:40.298 7f83059b6c80 20 []librbd::ImageState: 0x55b950a2c400 close
2023-11-03 16:47:40.298 7f83059b6c80 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 send_close_unlock
2023-11-03 16:47:40.298 7f83059b6c80 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_block_image_watcher
2023-11-03 16:47:40.298 7f83059b6c80 10 []librbd::ImageWatcher: 0x7f82d0005200 block_notifies
2023-11-03 16:47:40.298 7f83059b6c80  5 []librbd::Watcher: 0x7f82d0005200 block_notifies: blocked_count=1
2023-11-03 16:47:40.298 7f83059b6c80 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_block_image_watcher: r=0
2023-11-03 16:47:40.298 7f83059b6c80 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_shut_down_update_watchers
2023-11-03 16:47:40.298 7f83059b6c80 20 []librbd::ImageState: 0x55b950a2c400 shut_down_update_watchers
2023-11-03 16:47:40.298 7f83059b6c80 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::shut_down
2023-11-03 16:47:40.298 7f83059b6c80 20 []librbd::ImageState: 0x55b950a2c4f0 ImageUpdateWatchers::shut_down: completing shut down
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_shut_down_update_watchers: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_shut_down_io_queue
2023-11-03 16:47:40.298 7f82d97fa700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 shut_down: shut_down: in_flight=0
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 start_op
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ImageRequest: 0x7f82d97f5450 send: aio_flush: ictx=0x55b9509b27d0, completion=0x7f82c4017930
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 set_request_count: pending=1
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4017b00
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 flush: 
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4017b00
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 complete_request: cb=1, pending=0
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 finalize: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_shut_down_io_queue: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_shut_down_exclusive_lock
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 shut_down
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 shut_down: 
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 send_shutdown: 
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 finish_op
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 send_shutdown_release: 
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 pre_release_lock_handler
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_cancel_op_requests: 
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_cancel_op_requests: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_block_writes: 
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ImageRequestWQ: 0x55b950a2d160 set_require_lock: 
2023-11-03 16:47:40.298 7f82d97fa700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 block_writes: 0x55b9509b27d0, num=1
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 start_op
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ImageRequest: 0x7f82d97f5250 send: aio_flush: ictx=0x55b9509b27d0, completion=0x7f82c4017930
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 set_request_count: pending=1
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4017b00
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 flush: 
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 send: object_dispatch_spec=0x7f82c4017b00
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 complete_request: cb=1, pending=0
2023-11-03 16:47:40.298 7f82d97fa700 20 []librbd::io::AioCompletion: 0x7f82c4017930 finalize: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_block_writes: r=0
2023-11-03 16:47:40.298 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_wait_for_ops: 
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_wait_for_ops: 
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_invalidate_cache: 
2023-11-03 16:47:40.299 7f82d97fa700  5 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 invalidate_cache: 
2023-11-03 16:47:40.299 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 invalidate_cache: 
2023-11-03 16:47:40.299 7f82d97fa700 20 []librbd::io::AsyncOperation: 0x7f82c4017a58 finish_op
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_invalidate_cache: r=0
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_flush_notifies: 
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_flush_notifies: 
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_close_object_map: 
2023-11-03 16:47:40.299 7f82d97fa700 10 []librbd::object_map::UnlockRequest: 0x7f82c4001f20 send_unlock: oid=rbd_object_map.4d296a5b7521b8
2023-11-03 16:47:40.301 7f82d9ffb700 10 []librbd::object_map::UnlockRequest: 0x7f82c4001f20 handle_unlock: r=0
2023-11-03 16:47:40.301 7f82d9ffb700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 handle_close_object_map: r=0
2023-11-03 16:47:40.301 7f82d9ffb700 10 []librbd::exclusive_lock::PreReleaseRequest: 0x7f82c4017c60 send_unlock: 
2023-11-03 16:47:40.301 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_shutdown_pre_release: r=0
2023-11-03 16:47:40.301 7f82d97fa700 10 []librbd::managed_lock::ReleaseRequest: 0x7f82c4016f20 send_unlock: entity=client.4816263, cookie=auto 140199610896496
2023-11-03 16:47:40.304 7f82d9ffb700 10 []librbd::managed_lock::ReleaseRequest: 0x7f82c4016f20 handle_unlock: r=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ExclusiveLock: 0x7f82d0006c20 post_release_lock_handler: r=0 shutting_down=1
2023-11-03 16:47:40.304 7f82d97fa700 20 []librbd::io::ImageRequestWQ: 0x55b950a2d160 set_require_lock: 
2023-11-03 16:47:40.304 7f82d97fa700  5 []librbd::io::ImageRequestWQ: 0x55b950a2d160 unblock_writes: 0x55b9509b27d0, num=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 notify released lock
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 current lock owner: [0,0]
2023-11-03 16:47:40.304 7f82d97fa700 20 []librbd::watcher::Notifier: 0x7f82d0005280 notify: pending=1
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 handle_shutdown_post_release: r=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 wait_for_tracked_ops: r=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ManagedLock: 0x7f82d0006c20 complete_shutdown: r=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_shut_down_exclusive_lock: r=0
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_unregister_image_watcher
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::ImageWatcher: 0x7f82d0005200 unregistering image watcher
2023-11-03 16:47:40.304 7f82d97fa700 10 []librbd::Watcher: 0x7f82d0005200 unregister_watch: 
2023-11-03 16:47:40.305 7f82d9ffb700  5 []librbd::Watcher: 0x7f82d0005200 notifications_blocked: blocked=1
2023-11-03 16:47:40.307 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: r=0
2023-11-03 16:47:40.307 7f82d9ffb700 20 []librbd::watcher::Notifier: 0x7f82d0005280 handle_notify: pending=0
2023-11-03 16:47:40.307 7f82cb7fe700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_unregister_image_watcher: r=0
2023-11-03 16:47:40.307 7f82cb7fe700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_flush_readahead
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_flush_readahead: r=0
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_shut_down_object_dispatcher
2023-11-03 16:47:40.307 7f82d97fa700  5 []librbd::io::ObjectDispatcher: 0x55b950a2f4f0 shut_down: 
2023-11-03 16:47:40.307 7f82d97fa700  5 []librbd::io::ObjectDispatch: 0x55b9509c04e0 shut_down: 
2023-11-03 16:47:40.307 7f82d97fa700  5 []librbd::cache::ObjectCacherObjectDispatch: 0x7f82c4002170 shut_down: 
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_shut_down_object_dispatcher: r=0
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 send_flush_op_work_queue
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_flush_op_work_queue: r=0
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::image::CloseRequest: 0x55b950a30a40 handle_flush_image_watcher: r=0
2023-11-03 16:47:40.307 7f82d97fa700 10 []librbd::ImageState: 0x55b950a2c400 0x55b950a2c400 handle_close: r=0