解决openstack创建虚拟机失败Received a sync request from an unknown host ‘nova‘. Re-created its InstanceList.

解决openstack创建虚拟机失败Received a sync request from an unknown host ‘nova’. Re-created its InstanceList.。

---

控制节点查看 nova 日志：

tail -300f /var/log/nova/nova-scheduler.log

出现如下类似错误：

2023-04-10 19:17:36.184 25297 INFO nova.scheduler.host_manager [req-806ca4b7-9d07-4eab-9202-218ec2fbd517 - - - - -] Received a sync request from an unknown host ‘nova’. Re-created its InstanceList.
2023-04-10 19:38:59.732 25288 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.734 25287 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.734 25295 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.734 25297 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.734 25291 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.734 25283 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.736 25281 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.735 25284 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.735 25285 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.735 25282 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.736 25300 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:38:59.736 25293 INFO nova.scheduler.host_manager [req-9fce5e8c-b6cc-4490-9787-036e01fda003 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.654 25281 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.655 25288 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.655 25295 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.655 25297 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.656 25282 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.656 25300 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.656 25293 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.656 25291 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.657 25284 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.658 25287 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.658 25283 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.
2023-04-10 19:40:59.666 25285 INFO nova.scheduler.host_manager [req-5fef2ef5-ed14-480e-b5d1-23eeaeea54b3 - - - - -] The instance sync for host ‘nova’ did not match. Re-created its InstanceList.

---

计算节点查看nova日志：

tail -300f /var/log/nova/nova-compute.log

出现如下类似错误：

2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [req-46c6ce76-a57a-4840-bd4a-cd6255ac86f8 8badcefd8f904b6c8a87a15174e816b5 90259745e51b413fa1d0fd3743dbe268 - default default] [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] Failed to build and run instance: PortBindingFailed: Binding failed for port 0c66bffc-b5b0-4149-9e0c-d27f47f6f5f1, please check neutron logs for more information.
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] Traceback (most recent call last):
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 2444, in _build_and_run_instance
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] block_device_info=block_device_info)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py”, line 3550, in spawn
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] mdevs=mdevs)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py”, line 6143, in _get_guest_xml
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] network_info_str = str(network_info)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/model.py”, line 601, in str
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] return self._sync_wrapper(fn, *args, **kwargs)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/model.py”, line 584, in _sync_wrapper
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] self.wait()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/model.py”, line 616, in wait
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] self[:] = self._gt.wait()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/eventlet/greenthread.py”, line 181, in wait
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] return self._exit_event.wait()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/eventlet/event.py”, line 125, in wait
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] result = hub.switch()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py”, line 298, in switch
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] return self.greenlet.switch()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/eventlet/greenthread.py”, line 221, in main
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] result = function(*args, **kwargs)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/utils.py”, line 675, in context_wrapper
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] return func(*args, **kwargs)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1716, in _allocate_network_async
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] six.reraise(*exc_info)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1699, in _allocate_network_async
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] resource_provider_mapping=resource_provider_mapping)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 1040, in allocate_for_instance
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] bind_host_id, requested_ports_dict)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 1169, in update_ports_for_instance
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] vif.destroy()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/oslo_utils/excutils.py”, line 220, in exit
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] self.force_reraise()
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/oslo_utils/excutils.py”, line 196, in force_reraise
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] six.reraise(self.type, self.value, self.tb)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 1139, in _update_ports_for_instance
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] port_client, instance, port_id, port_req_body)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 513, in _update_port
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] _ensure_no_port_binding_failure(port)
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 236, in _ensure_no_port_binding_failure
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] raise exception.PortBindingFailed(port_id=port[‘id’])
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334] PortBindingFailed: Binding failed for port 0c66bffc-b5b0-4149-9e0c-d27f47f6f5f1, please check neutron logs for more information.
2023-04-10 19:54:04.534 51580 ERROR nova.compute.manager [instance: bfe7dbaf-bdbb-4535-9aa1-87e6d8fcb334]

---

控制节点查看 neutron 日志：

tail -300f /var/log/neutron/l3-agent.log

出现错误类似如下：

2023-04-10 19:16:16.423 24571 INFO neutron.agent.l3.agent [-] Successfully reported state after a previous failure.
2023-04-10 19:16:44.487 24571 ERROR neutron.agent.l3.agent [req-c50a372d-e2a7-4278-a808-030abfabe913 - - - - -] Server failed to return info for routers in required time, decreasing chunk size to: 128: MessagingTimeout: Timed out waiting for a reply to message ID 186cf8048ff2432ba8f95e7199744353
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task [req-c50a372d-e2a7-4278-a808-030abfabe913 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task: MessagingTimeout: Timed out waiting for a reply to message ID 186cf8048ff2432ba8f95e7199744353
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py”, line 222, in run_periodic_tasks
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task task(self, context)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py”, line 815, in periodic_sync_routers_task
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task self.fetch_and_sync_all_routers(context, ns_manager)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/osprofiler/profiler.py”, line 160, in wrapper
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task result = f(*args, **kwargs)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/osprofiler/profiler.py”, line 160, in wrapper
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task result = f(*args, **kwargs)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py”, line 828, in fetch_and_sync_all_routers
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task router_ids = self.plugin_rpc.get_router_ids(context)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_utils/timeutils.py”, line 386, in wrapper
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task result = func(*args, **kwargs)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron/common/utils.py”, line 982, in wrapper
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task ret = f(*args, **kwargs)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py”, line 144, in get_router_ids
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task return cctxt.call(context, ‘get_router_ids’, host=self.host)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron_lib/rpc.py”, line 180, in call
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task time.sleep(wait)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_utils/excutils.py”, line 220, in exit
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task self.force_reraise()
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_utils/excutils.py”, line 196, in force_reraise
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task six.reraise(self.type_, self.value, self.tb)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/neutron_lib/rpc.py”, line 157, in call
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task return self._original_context.call(ctxt, method, **kwargs)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py”, line 181, in call
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task transport_options=self.transport_options)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/transport.py”, line 129, in _send
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task transport_options=transport_options)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py”, line 646, in send
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task transport_options=transport_options)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py”, line 634, in _send
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task call_monitor_timeout)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py”, line 523, in wait
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task File “/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py”, line 401, in get
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task ‘to message ID %s’ % msg_id)
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 186cf8048ff2432ba8f95e7199744353
2023-04-10 19:16:44.488 24571 ERROR oslo_service.periodic_task
2023-04-10 19:16:44.491 24571 WARNING oslo.service.loopingcall [req-c50a372d-e2a7-4278-a808-030abfabe913 - - - - -] Function ‘neutron.service.Service.periodic_tasks’ run outlasted interval by 78.12 sec
2023-04-10 19:16:47.040 24571 INFO neutron.agent.l3.agent [-] Starting router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, action 3, priority 2, update_id 8c10fca2-1af0-4e58-95e9-f0df792583d1. Wait time elapsed: 0.001
2023-04-10 19:16:47.043 24571 INFO neutron.agent.l3.agent [-] Starting router update for 1052c04f-39d4-46bb-9a7d-81b2473a1f8c, action 3, priority 2, update_id e00be860-77b9-47ac-9804-4f06a3ebc58c. Wait time elapsed: 0.003
2023-04-10 19:16:47.045 24571 INFO neutron.agent.l3.agent [-] Starting router update for 8431f20d-ed83-42c9-a044-5d614eaeebbd, action 3, priority 2, update_id 9896951f-85b4-4912-abaa-448783504997. Wait time elapsed: 0.004
2023-04-10 19:16:47.046 24571 INFO neutron.agent.l3.agent [-] Starting router update for 9eec05ba-efcc-4515-9279-8ab275ca25c4, action 3, priority 2, update_id e30a9f8c-76e4-41da-9c04-8debd7d5721b. Wait time elapsed: 0.006
2023-04-10 19:16:47.337 24571 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=26492
2023-04-10 19:16:52.701 24571 INFO neutron.agent.l3.agent [req-112c4e29-8a01-439d-894c-aa842e3638fe - ae108251e738443a9d86c7e6421a7942 - - -] Finished a router update for 8431f20d-ed83-42c9-a044-5d614eaeebbd, update_id 9896951f-85b4-4912-abaa-448783504997. Time elapsed: 5.656
2023-04-10 19:16:52.854 24571 INFO neutron.agent.l3.agent [req-06362116-8885-44c2-924b-855bbc952d6d - 90259745e51b413fa1d0fd3743dbe268 - - -] Finished a router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, update_id 8c10fca2-1af0-4e58-95e9-f0df792583d1. Time elapsed: 5.813
2023-04-10 19:16:52.905 24571 INFO neutron.agent.l3.agent [req-e98919dd-6ab6-4617-919b-8fff765f623e - ae108251e738443a9d86c7e6421a7942 - - -] Finished a router update for 1052c04f-39d4-46bb-9a7d-81b2473a1f8c, update_id e00be860-77b9-47ac-9804-4f06a3ebc58c. Time elapsed: 5.862
2023-04-10 19:16:52.921 24571 INFO neutron.agent.l3.agent [req-469f13be-980d-4b17-b915-095fc1dad0f5 - 90259745e51b413fa1d0fd3743dbe268 - - -] Finished a router update for 9eec05ba-efcc-4515-9279-8ab275ca25c4, update_id e30a9f8c-76e4-41da-9c04-8debd7d5721b. Time elapsed: 5.875
2023-04-10 19:38:54.398 24571 INFO neutron.agent.l3.agent [-] Starting router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, action 3, priority 1, update_id 8e4e644c-b446-4d98-8d70-af4074974514. Wait time elapsed: 0.001
2023-04-10 19:38:56.697 24571 INFO neutron.agent.l3.agent [req-37b2c50e-4b6e-4926-a7bf-cb30d11f24b1 - 90259745e51b413fa1d0fd3743dbe268 - - -] Finished a router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, update_id 8e4e644c-b446-4d98-8d70-af4074974514. Time elapsed: 2.299
2023-04-10 19:40:40.699 24571 INFO neutron.agent.l3.agent [-] Starting router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, action 3, priority 1, update_id a1c8a381-a84e-464d-8bc4-320fb3f5c6f2. Wait time elapsed: 0.000
2023-04-10 19:40:42.955 24571 INFO neutron.agent.l3.agent [req-08c5f14a-02c5-44ea-83b6-8a82fa896e37 - 90259745e51b413fa1d0fd3743dbe268 - - -] Finished a router update for 0f2de474-0776-4fe6-ba2f-e497c457e9df, update_id a1c8a381-a84e-464d-8bc4-320fb3f5c6f2. Time elapsed: 2.256
2023-04-10 19:52:29.425 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [924f64e2-2990-47df-a24b-35c5b6bfb58a] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
2023-04-10 19:52:29.430 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [54f11509-eada-4646-8b7a-17cd1cb4bf20] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
2023-04-10 19:52:29.535 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [d46f84e2-b68b-4027-a1b0-18e0f5e17892] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
2023-04-10 19:52:29.547 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [870ddcc6-2ed4-4f64-a585-cc276ca23fc8] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
2023-04-10 19:52:30.433 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [924f64e2-2990-47df-a24b-35c5b6bfb58a] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:30.439 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [54f11509-eada-4646-8b7a-17cd1cb4bf20] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:30.541 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [d46f84e2-b68b-4027-a1b0-18e0f5e17892] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:30.557 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [870ddcc6-2ed4-4f64-a585-cc276ca23fc8] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:31.104 24571 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2023-04-10 19:52:31.113 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] ECONNREFUSED (retrying in 2.0 seconds): error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:32.441 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [924f64e2-2990-47df-a24b-35c5b6bfb58a] AMQP server on 10.0.5.3:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <RecoverableConnectionError: unknown error>
2023-04-10 19:52:32.448 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [54f11509-eada-4646-8b7a-17cd1cb4bf20] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:32.549 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [d46f84e2-b68b-4027-a1b0-18e0f5e17892] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:32.569 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [870ddcc6-2ed4-4f64-a585-cc276ca23fc8] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:33.128 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 111] ECONNREFUSED (retrying in 4.0 seconds): error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:33.453 24571 ERROR oslo.messaging._drivers.impl_rabbit [-] [924f64e2-2990-47df-a24b-35c5b6bfb58a] AMQP server on 10.0.5.3:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.: error: [Errno 111] ECONNREFUSED
2023-04-10 19:52:35.468 24571 INFO oslo.messaging._drivers.impl_rabbit [-] [924f64e2-2990-47df-a24b-35c5b6bfb58a] Reconnected to AMQP server on 10.0.5.3:5672 via [amqp] client with port 45090.
2023-04-10 19:52:36.622 24571 INFO oslo.messaging._drivers.impl_rabbit [-] [54f11509-eada-4646-8b7a-17cd1cb4bf20] Reconnected to AMQP server on 10.0.5.3:5672 via [amqp] client with port 45486.
2023-04-10 19:52:37.640 24571 INFO oslo.messaging._drivers.impl_rabbit [-] [d46f84e2-b68b-4027-a1b0-18e0f5e17892] Reconnected to AMQP server on 10.0.5.3:5672 via [amqp] client with port 46002.
2023-04-10 19:52:37.650 24571 INFO oslo.messaging._drivers.impl_rabbit [-] [870ddcc6-2ed4-4f64-a585-cc276ca23fc8] Reconnected to AMQP server on 10.0.5.3:5672 via [amqp] client with port 46096.

解决方案：

计算节点停止服务：

 systemctl stop $(systemctl list-unit-files |grep -i nova | grep enabled | awk '{print $1}')

控制节点停止服务：

 systemctl stop $(systemctl list-unit-files |grep -i nova | grep enabled | awk '{print $1}')

控制节点删除对应的host：

 nova service-list
 
 nova service-delete id-xxxxxxxx

控制节点重启服务：

systemctl start $(systemctl list-unit-files |grep -i nova | grep enabled | awk '{print $1}')

计算节点重启服务：

systemctl start $(systemctl list-unit-files |grep -i nova | grep enabled | awk '{print $1}')

控制节点发现服务：

nova-manage cell_v2 discover_hosts --verbose

还有额外需要注意的事placement为admin角色：

openstack role add --project service --user placement admin