MQ服务消息超时或者任何其他中断实例创建或者删除的情况,都会造成“僵死”实例的存在。也即是这个实例并不存在,或者即便存在也是有问题的,并且在Dashboard仪表盘上删除不了。提示“you are not allowed to terminate this instance"之类的错误。即便提示删除成功,实际还是在那里死趟着。命令行同样的问题,如下
# nova list /*列表显示出来*/
+--------------------------------------+--------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------+--------+------------+-------------+---------------------+
| 09dca062-c577-469f-bff5-cccbe65a5b7c | instance_name | ERROR | deleting | NOSTATE | |
# nova reset-state 09dca062-c577-469f-bff5-cccbe65a5b7c /*重置下状态,正在删除任务消失*/
# nova delete 09dca062-c577-469f-bff5-cccbe65a5b7c /*尝试命令行删除*/
# nova list /*还是上面的显示正在删除任务的状态*/
# mysql -uroot -p /*在控制节点上数据库去看看这个实例在哪个结算节点上*/
MariaDB [(none)]> select node from nova.instances where uuid='09dca062-c577-469f-bff5-cccbe65a5b7c'; /*得到这个实例所在的计算节点*/
# ll /var/lib/nova/instances/ /*到该计算节点查看实例文件夹在不在*/
# rm -Rf 09dca062-c577-469f-bff5-cccbe65a5b7c /*在的话,给删除*/
现在再次进入数据库,进行多个表格删除对应实例ID的记录。
1. nova.instances表中,实例ID的字段名是uuid;
2. 其他外链表使用的是instance_uuid来引用。
MariaDB [nova]> delete from table_name where instance_uuid='09dca062-c577-469f-bff5-cccbe65a5b7c'; /*从下面的表格中一一删除关联的记录*/
table_name如下:
1. security_group_instance_association,实例安全组,创建默认安全组是default,但不会记录,如果你已经关联了安全组,那就删除下*/
2. instance_info_caches,实例缓存,删除
3. block_device_mapping,实例块存储,默认保存了一条volume_id为NULL的记录,删除之*/
4. instance_actions_events,实例操作结果,字段action_id引用下表instance_actions中的id,该表没有instance_uuid*/
5. instance_actions,实例操作,每个操作有一个id,作为上表instance_actions_events的action_id*/
这里我在想,真是蛋疼,两个表为啥不结合为一个表。所以只能首先在instance_actions中获取操作id然后在events中再去删
MariaDB [nova]> select id,action,instance_uuid from instance_actions where instance_uuid='09dca062-c577-469f-bff5-cccbe65a5b7c';
+-----+--------+--------------------------------------+
| id | action | instance_uuid |
+-----+--------+--------------------------------------+
| 380 | create | 09dca062-c577-469f-bff5-cccbe65a5b7c |
| 382 | delete | 09dca062-c577-469f-bff5-cccbe65a5b7c |
| 383 | delete | 09dca062-c577-469f-bff5-cccbe65a5b7c |
| 384 | delete | 09dca062-c577-469f-bff5-cccbe65a5b7c |
| 385 | delete | 09dca062-c577-469f-bff5-cccbe65a5b7c |
| 386 | delete | 09dca062-c577-469f-bff5-cccbe65a5b7c |
+-----+--------+--------------------------------------+
MariaDB [nova]> delete from instance_actions_events where action_id='380'; /*我在仪表盘操作了基础*/
MariaDB [nova]> delete from instance_actions_events where action_id='382';
MariaDB [nova]> delete from instance_actions_events where action_id='383';
MariaDB [nova]> delete from instance_actions_events where action_id='384';
MariaDB [nova]> delete from instance_actions_events where action_id='385';
MariaDB [nova]> delete from instance_actions_events where action_id='386'; /*删除instance_action_events之后再去删除instance_action*/
6. instance_faults,实例错误
7. instance_extra,没有看表的内容,不知道放的啥东西,只有一条记录
8. instance_system_metadata,实例元数据,记录蛮多的
9. instances,实例表,记录所有实例,仪表盘的实例列表就是读取这里
上面的9个表格删除完毕止之后,不要急着去刷新仪表盘,不然会提示无法获取实例列表信息 - 蛮吓人的!实例占用的资源依然被占用。
# systemctl restart openstack-nova-api openstack-nova-conductor /*重启下服务*/
# nova list /*实例删掉了,资源收回*/
最后如果每次都这么手动的去搞,麻烦,那就写个脚本:
# vim deletevm.sh
mysql -uroot -p << EOF
use nova;
delete from security_group_instance_association where instance_uuid='$1';
delete from instance_info_caches where instance_uuid='$1';
delete from block_device_mapping where instance_uuid='$1';
delete from instance_actions where instance_uuid='$1'; /*这个地方可能需要手工,SQL能力不强哎*/
delete from instance_faults where instance_uuid='$1';
delete from instance_extra where instance_uuid='$1';
delete from instance_system_metadata where instance_uuid='$1';
delete from instances where instance_uuid='$1';
EOF
# chmod +x deletevm.sh
# ./deletevm.sh instance_id /*以后用这个脚本来搞就快多了*/
# nova list /*最后可以再看看*/