nova/cinder等组件报连不上 rabbitmq

only火车头

已于 2023-07-12 11:38:57 修改

阅读量1.5k

点赞数

文章标签： rabbitmq qt 分布式

于 2023-07-12 10:06:00 首次发布

本文链接：https://blog.csdn.net/Hello_NB1/article/details/131674957

版权

该文描述了一个OpenStackNova计算节点在启动时遇到的问题，具体表现为尝试连接RabbitMQ服务时收到AccessRefused错误，原因是认证机制AMQPLAIN失败。解决方案包括检查和更新RabbitMQ的用户、权限和虚拟主机设置，确保Nova计算节点能够正确访问RabbitMQ集群。

摘要由CSDN通过智能技术生成

问题现象:

# kubectl logs nova-compute-g4pnl -n openstack -c nova-compute
...
+ exec nova-compute --config-file /etc/nova/nova.conf --config-file /tmp/pod-shared/nova-vnc.ini --config-dir /opt/nova-compute/conf
Deprecated: Option "notification_format" from group "DEFAULT" is deprecated. Use option "notification_format" from group "notifications".
2023-07-11 20:06:08.393 1617 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
2023-07-11 20:06:08.500 1617 WARNING oslo_config.cfg [req-fc476544-5edf-461a-bb5a-f3f8aee4bffe - - - - -] Deprecated: Option "heartbeat_in_pthread" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2023-07-11 20:06:08.530 1617 CRITICAL nova [req-fc476544-5edf-461a-bb5a-f3f8aee4bffe - - - - -] Unhandled error: amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2023-07-11 20:06:08.530 1617 ERROR nova Traceback (most recent call last):
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/bin/nova-compute", line 10, in <module>
2023-07-11 20:06:08.530 1617 ERROR nova     sys.exit(main())
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/nova/cmd/compute.py", line 59, in main
2023-07-11 20:06:08.530 1617 ERROR nova     topic=compute_rpcapi.RPC_TOPIC)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/nova/service.py", line 256, in create
2023-07-11 20:06:08.530 1617 ERROR nova     periodic_interval_max=periodic_interval_max)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/nova/service.py", line 115, in __init__
2023-07-11 20:06:08.530 1617 ERROR nova     conductor_api.wait_until_ready(context.get_admin_context())
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/nova/conductor/api.py", line 68, in wait_until_ready
2023-07-11 20:06:08.530 1617 ERROR nova     timeout=timeout)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/nova/baserpc.py", line 58, in ping
2023-07-11 20:06:08.530 1617 ERROR nova     return cctxt.call(context, 'ping', arg=arg_p)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2023-07-11 20:06:08.530 1617 ERROR nova     transport_options=self.transport_options)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send
2023-07-11 20:06:08.530 1617 ERROR nova     transport_options=transport_options)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send
2023-07-11 20:06:08.530 1617 ERROR nova     transport_options=transport_options)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 626, in _send
2023-07-11 20:06:08.530 1617 ERROR nova     msg.update({'_reply_q': self._get_reply_q()})
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 607, in _get_reply_q
2023-07-11 20:06:08.530 1617 ERROR nova     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 598, in _get_connection
2023-07-11 20:06:08.530 1617 ERROR nova     purpose=purpose)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/common.py", line 425, in __init__
2023-07-11 20:06:08.530 1617 ERROR nova     self.connection = connection_pool.create(purpose)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/pool.py", line 146, in create
2023-07-11 20:06:08.530 1617 ERROR nova     return self.connection_cls(self.conf, self.url, purpose)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 627, in __init__
2023-07-11 20:06:08.530 1617 ERROR nova     self.ensure_connection()
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 744, in ensure_connection
2023-07-11 20:06:08.530 1617 ERROR nova     self.connection.ensure_connection(errback=on_error)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 383, in ensure_connection
2023-07-11 20:06:08.530 1617 ERROR nova     self._ensure_connection(*args, **kwargs)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 439, in _ensure_connection
2023-07-11 20:06:08.530 1617 ERROR nova     callback, timeout=timeout
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
2023-07-11 20:06:08.530 1617 ERROR nova     return fun(*args, **kwargs)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 866, in _connection_factory
2023-07-11 20:06:08.530 1617 ERROR nova     self._connection = self._establish_connection()
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 801, in _establish_connection
2023-07-11 20:06:08.530 1617 ERROR nova     conn = self.transport.establish_connection()
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection
2023-07-11 20:06:08.530 1617 ERROR nova     conn.connect()
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 329, in connect
2023-07-11 20:06:08.530 1617 ERROR nova     self.drain_events(timeout=self.connect_timeout)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 523, in drain_events
2023-07-11 20:06:08.530 1617 ERROR nova     while not self.blocking_read(timeout):
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 529, in blocking_read
2023-07-11 20:06:08.530 1617 ERROR nova     return self.on_inbound_frame(frame)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/method_framing.py", line 53, in on_frame
2023-07-11 20:06:08.530 1617 ERROR nova     callback(channel, method_sig, buf, None)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 536, in on_inbound_method
2023-07-11 20:06:08.530 1617 ERROR nova     method_sig, payload, content,
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/abstract_channel.py", line 143, in dispatch_method
2023-07-11 20:06:08.530 1617 ERROR nova     listener(*args)
2023-07-11 20:06:08.530 1617 ERROR nova   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 666, in _on_close
2023-07-11 20:06:08.530 1617 ERROR nova     (class_id, method_id), ConnectionError)
2023-07-11 20:06:08.530 1617 ERROR nova amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2023-07-11 20:06:08.530 1617 ERROR nova

检查 rabbitmq 集群状态:

root@rabbitmq-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local ...
Basics

Cluster name: rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local

Disk Nodes

rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local

Running Nodes

rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local

Versions

rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local: RabbitMQ 3.8.0 on Erlang 22.1.5
rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local: RabbitMQ 3.8.0 on Erlang 22.1.5
rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local: RabbitMQ 3.8.0 on Erlang 22.1.5

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-1.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-2.rabbitmq-discovery.openstack.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled

集群状态是好的

检查用户

root@rabbitmq-0:/# rabbitmqctl list_users
Listing users ...

root@rabbitmq-0:/#

缺少用户

新增用户，通过其它环境查看用户名，并增加用户，密码可以在当前环境 nova 的配置文件中获取:
示例:

[root@node-1 ~]# kubectl get cm nova-etc -n openstack -o yaml | grep transport_url
    transport_url = rabbit://rabbitmq:yHkxhtkD@rabbitmq.openstack.svc.cluster.local:5672/

则 rabbitmq是用户名；yHkxhtkD是密码。

root@rabbitmq-0:/# rabbitmqctl add_user rabbitmq
Adding user "rabbitmq" ...
Password: yHkxhtkD
root@rabbitmq-0:/#

之后再次检查问题 pod，发现 rabbitmq 用户权限问题，解决方法如下:
新增 vhost

root@rabbitmq-0:/# rabbitmqctl add_vhost /
Adding vhost "/" ...

设置权限

root@rabbitmq-0:/# rabbitmqctl set_permissions rabbitmq ".*" ".*" ".*"
Setting permissions for user "rabbitmq" in vhost "/" ...
root@rabbitmq-0:/#

设置 rabbitmq 用户 administrator tag

root@rabbitmq-0:/# rabbitmqctl set_user_tags rabbitmq administrator
Setting tags for user "rabbitmq" to [administrator] ...
root@rabbitmq-0:/#

再次检查问题 pod，可以正常连接 rabbitmq，并启动成功。

only火车头

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫