解决ERROR oslo_messaging.rpc.server UnixHTTPConnectionPool(host=‘localhost‘, port=None) Read timed out

zun.common.exception.ZunException: Unexpected error: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)

2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server [req-6ff62c08-fd25-4df6-8a81-d144956cbbd9 7db25ffff6314194822f5ada52569de5 ce76b778ff0b453aa7b486f9934f694e default - -] Exception during message handling: zun.common.exception.ZunException: Unexpected error: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 445, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server six.raise_from(e, None)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “”, line 3, in raise_from
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 440, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server httplib_response = conn.getresponse()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 1346, in getresponse
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server response.begin()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 307, in begin
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server version, status, reason = self._read_status()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/http/client.py”, line 268, in _read_status
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/lib64/python3.6/socket.py”, line 586, in readinto
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._sock.recv_into(b)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 371, in recv_into
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 359, in _recv_loop
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._read_trampoline()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 331, in _read_trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server timeout_exc=socket_timeout(‘timed out’))
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/greenio/base.py”, line 210, in _trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server mark_as_closed=self._mark_as_closed)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/hubs/init.py”, line 159, in trampoline
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return hub.switch()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/eventlet/hubs/hub.py”, line 313, in switch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.greenlet.switch()
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server socket.timeout: timed out
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/adapters.py”, line 449, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server timeout=timeout
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 756, in urlopen
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py”, line 532, in increment
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise six.reraise(type(error), error, _stacktrace)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py”, line 735, in reraise
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise value
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 706, in urlopen
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server chunked=chunked,
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 447, in _make_request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 337, in _raise_timeout
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self, url, “Read timed out. (read timeout=%s)” % timeout_value
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/common/utils.py”, line 226, in decorated_function
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/compute/manager.py”, line 577, in container_show
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server container = self.driver.show(context, container)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/container/docker/driver.py”, line 616, in show
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server response = docker.inspect_container(container.container_id)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py”, line 19, in wrapped
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return f(self, resource_id, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/api/container.py”, line 774, in inspect_container
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server self._get(self._url(“/containers/{0}/json”, container)), True
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py”, line 46, in inner
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/docker/api/client.py”, line 237, in _get
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.get(url, **self._set_request_timeout(kwargs))
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 555, in get
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self.request(‘GET’, url, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 542, in request
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server resp = self.send(prep, **send_kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/sessions.py”, line 655, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server r = adapter.send(request, **kwargs)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/requests/adapters.py”, line 529, in send
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise ReadTimeout(e, request=request)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/server.py”, line 165, in _process_incoming
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py”, line 309, in dispatch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py”, line 229, in _do_dispatch
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server File “/usr/local/lib/python3.6/site-packages/zun/common/utils.py”, line 232, in decorated_function
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server raise e
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server zun.common.exception.ZunException: Unexpected error: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
2023-09-13 17:26:08.649 8468 ERROR oslo_messaging.rpc.server

---

解决方法：

查看mq连接数量：

rabbitmqctl list_connections | wc -l


rabbitmqctl list_connections

如果具柄不够，修改相关具柄数量：

1. 在 /etc/security/limits.conf 添加：

mysql hard nofile 65535
mysql soft nofile 65535
 
* soft nofile 655350
* hard nofile 655350
 
* soft nproc 655350
* hard nproc 655350
 
* soft memlock unlimited
* hard memlock unlimited

2. 创建文件夹：mkdir -p /etc/systemd/system/mariadb.service.d/
   添加文件 vim /etc/systemd/system/mariadb.service.d/limits.conf ， 内容如下：

[Service]
LimitNOFILE=65535

3. 修改 vim /etc/my.cnf 添加 max_connections 项，如下：

[mysqld]
max_connections=30000

4. 刷新配置：systemctl daemon-reload ;

5. 重启服务：systemctl restart mariadb

2.Nova 计算节点发现

nova-manage cell_v2 discover_hosts --verbose

3.rabbitmq 句柄满了

1. 编辑 vim /usr/lib/systemd/system/rabbitmq-server.service 文件，
   在 [Service] 下添加 LimitNOFILE=300000 ;如下：

[Service]
LimitNOFILE=300000

2. 刷新配置：systemctl daemon-reload ;

3. 重启服务：systemctl restart rabbitmq-server ;

在控制节点重新创建 keystone 认证用户：

source ~/keystonerc_admin

openstack user del zun
openstack user create --domain default --password zun zun
openstack role add --project services --user zun admin

openstack user del kuryr
openstack user create --domain default --password kuryr kuryr
openstack role add --project services --user kuryr admin

重启控制节点的zun服务：
systemctl restart zun-api zun-wsproxy

重启计算节点的zun服务：
systemctl restart kuryr-libnetwork
systemctl restart zun-compute

---

在 RabbitMQ 中使用 rabbitmqctl list_connections 命令查看TCP/IP连接的统计信息，命令格式：

rabbitmqctl list_connections [connectioninfoitem...]

注意，connectioninfoitem 参数用于指示哪些信息项会包含在结果集中，结果集的列顺序将匹配参数的顺序。connectioninfoitem 可选值如下：

pid：与连接相关的 Erlang 进程 ID

name：连接的名称。

port：服务器端口

host：返回反向 DNS 获取的服务器主机名称，或者 IP 地址，或者未启用。

peer_port：服务器对端端口。当一个客户端与服务器连接时，这个客户端的端口就是 peer_port。

peer_host：返回反向 DNS 获取的对端主机名称，或者 IP 地址，或者未启用。

ssl：是否启用 SSL

ssl_protocol：SSL 协议，如 tlsvl

ssl_key_exchange: SSL 密钥交换算法，如：rsa

ssl_cipher: SSL 加密算法，如：aes_256_cbc

ssl_hash: SSL 哈希算法，如：sha

peer_cert_subject：对端的 SSL 安全证书的主题，基于 RFC4514 的形式

peer_cert_issuer：对端 SSL 安全证书的发行者，基于 RFC4514 的形式

peer_cert_validity：对端 SSL 安全证书的有效期。

state：连接状态，包括 starting、tuning、opening、running、flow、blocking、blocked、closing 和 closed 这几种。

channels：该连接中的信道个数。

protocol：使用的 AMQP 协议的版本，当前是｛0,9,1｝或者｛0,8,0｝。注意，如果客户端请求的是 AMQP 0-9 的连接，RabbitMQ 也会将其视为 0-9-1。

auth_mechanism：使用的 SASL 认证机制，如 PLAIN、AMQPLAIN、EXTERNAL、RABBIT-CR-DEMO 等

user：与连接相关的用户名。

vhost：与连接相关的 vhost 的名称。

timeout：连接超时/协商的心跳间隔，单位为秒。

frame_max：最大传输帧的大小，单位为 B

channel_max：此连接上信道的最大数量。如果值 0，则表示无上限，但客户端一般会将 0 转变为 65535

client_properties：在建立连接期间由客户端发送的信息属性。

recv_oct：收到的字节数。

recv_cnt：收到的数据包个数。

send_oct：发送的字节数。

send_cnt：发送的数据包个数。

send_pend：发送队列大小。

connected_at：连接建立的时间戳。

示例

（1）如果没有指定 connectioninfoitem 信息，那么默认会显示 user、peer_host、peer_port 和 state 这几项信息。

D:\server\rabbitmq_server-3.9.11\sbin> rabbitmqctl list_connections
Listing connections ...
user    peer_host       peer_port       state
root    127.0.0.1       9365    running

（2）手动指定 connectioninfoitem 信息，例如：

D:\server\rabbitmq_server-3.9.11\sbin> rabbitmqctl list_connections user peer_host peer_port state
Listing connections ...
user    peer_host       peer_port       state
root    127.0.0.1       9365    running