openstack Y版本，当 --force也无法删除volume之后

背景

因为种种原因，有一些volume，attach的vm已经被删除，但volume没有解除和vm attach状态，volume reset只能重置卷的状态，但不能清理与vm的attach信息，而openstack server remove volume 也无法解除attach状态，因为虚拟机已经不存在，最终导致这些卷无法通过命令行被删除

问题复现

尝试删除volume报错

# openstack volume delete 0d053cfe-1c20-4248-9ff2-917a39593e80
Failed to delete volume with name or ID '0d053cfe-1c20-4248-9ff2-917a39593e80': Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-d76fc35b-5b08-4d96-b7f1-5308f23aa2ba)

确认问题的volume，volume连着这虚拟机，但实际虚拟机已不存在

[root@SHA-STNC02-CCC-PSVR01 ~]# openstack volume list --all |grep 2b6f1956-b334-4828-bc27-358694cbc8e7
| 0d053cfe-1c20-4248-9ff2-917a39593e80 |                  | in-use   |   20 | Attached to 2b6f1956-b334-4828-bc27-358694cbc8e7 on /dev/vda                                   |
[root@SHA-STNC02-CCC-PSVR01 ~]#
[root@SHA-STNC02-CCC-PSVR01 ~]#
[root@SHA-STNC02-CCC-PSVR01 ~]# nova list --all |grep 2b6f1956-b334-4828-bc27-358694cbc8e7
[root@SHA-STNC02-CCC-PSVR01 ~]#

查看ceph中的rbd、

[root@SHA-STNC02-CCC-PSVR01 ~]# rbd ls -p ssd-volumes |grep 0d053cfe-1c20-4248-9ff2-917a39593e80
volume-0d053cfe-1c20-4248-9ff2-917a39593e80
[root@SHA-STNC02-CCC-PSVR01 ~]#

在数据库中查看volume的状态, attach_status状态为 attached

use cinder;
select * from volume_attachment where volume_id =  '0d053cfe-1c20-4248-9ff2-917a39593e80'\G
mysql> select * from volume_attachment where volume_id =  '0d053cfe-1c20-4248-9ff2-917a39593e80'\G
*************************** 1. row ***************************
created_at: 2024-08-21 02:34:58
updated_at: 2024-08-21 02:34:59
deleted_at: NULL
deleted: 0
id: cbc3465b-08f4-4c92-a2ab-48ce7ab00658
volume_id: 0d053cfe-1c20-4248-9ff2-917a39593e80
attached_host: SHA-STNC02-CCC-PSVR01
instance_uuid: 2b6f1956-b334-4828-bc27-358694cbc8e7
mountpoint: /dev/vda
attach_time: 2024-08-21 02:34:59
detach_time: NULL
attach_mode: rw
attach_status: attached
connection_info: {"name": "ssd-volumes/volume-0d053cfe-1c20-4248-9ff2-917a39593e80", "hosts": ["172.24.0.43", "172.24.0.44", "172.24.0.45"], "ports": ["6789", "6789", "6789"], "cluster_name": "ceph", "auth_enabled": true, "auth_username": "cinder", "secret_type": "ceph", "secret_uuid": "879766a7-7f81-452f-b857-6aa74c4f07c7", "volume_id": "0d053cfe-1c20-4248-9ff2-917a39593e80", "discard": true, "qos_specs": null, "access_mode": "rw", "encrypted": false, "cacheable": false, "driver_volume_type": "rbd", "attachment_id": "cbc3465b-08f4-4c92-a2ab-48ce7ab00658"}
connector: {"platform": "x86_64", "os_type": "linux", "ip": "172.28.2.101", "host": "SHA-STNC02-CCC-PSVR01", "multipath": false, "initiator": "iqn.1994-05.com.redhat:f8297a1a41f", "do_local_attach": false, "system uuid": "4c4c4544-0032-5210-8031-b9c04f354633", "nqn": "nqn.2014-08.org.nvmexpress:uuid:29096749-bd82-4293-878b-adaa5b413a20", "found_dsc": "", "mountpoint": "/dev/vda"}
1 row in set (0.01 sec)
mysql>
select * from volumes where id =  '0d053cfe-1c20-4248-9ff2-917a39593e80'\G;
mysql> select * from volumes where id = '0d053cfe-1c20-4248-9ff2-917a39593e80'\G;
*************************** 1. row ***************************
created_at: 2024-08-21 02:34:22
updated_at: 2024-08-21 02:34:59
deleted_at: NULL
deleted: 0
id: 0d053cfe-1c20-4248-9ff2-917a39593e80
ec2_id: NULL
user_id: 9c9309faed304feea23a097670a7e626
project_id: ba232682389b4ccead385b1ffab416fa
host: rbd:ssd_volumes@ssd#ssd
size: 20
availability_zone: nova
status: in-use
attach_status: attached
scheduled_at: 2024-08-21 02:34:22
launched_at: 2024-08-21 02:34:57
terminated_at: NULL
display_name:
display_description:
provider_location: NULL
provider_auth: NULL
snapshot_id: NULL
volume_type_id: 67630d87-1129-425e-9ba3-7bd3b7387ddd
source_volid: NULL
bootable: 1
provider_geometry: NULL
_name_id: NULL
encryption_key_id: NULL
migration_status: NULL
replication_status: NULL
replication_extended_status: NULL
replication_driver_data: NULL
consistencygroup_id: NULL
provider_id: NULL
multiattach: 0
previous_status: NULL
cluster_name: NULL
group_id: NULL
service_uuid: ca7b40c3-b30b-4be0-a709-8fcf32c99608
shared_targets: 0
use_quota: 1
1 row in set (0.00 sec)

解决办法

在数据库中修改volume的attach状态

use cinder;

1.更新volume_attachment 表中volume attach_status 的状态为 'detached'

UPDATE volume_attachment SET attach_status = 'detached' WHERE volume_id = '0d053cfe-1c20-4248-9ff2-917a39593e80';

2.更新volume 表中volume attach_status 的状态为 'detached'

UPDATE volumes SET attach_status = 'detached' WHERE id = '0d053cfe-1c20-4248-9ff2-917a39593e80';

3.重置volume状态

# cinder reset-state --state available 0d053cfe-1c20-4248-9ff2-917a39593e80

4.5.删除volume

# openstack volume delete 0d053cfe-1c20-4248-9ff2-917a39593e80