OpenStack环境出现MySQL ‘Too many connections’

最新推荐文章于 2021-02-05 03:15:53 发布

weixin_33709219

最新推荐文章于 2021-02-05 03:15:53 发布

阅读量415

点赞数

文章标签：数据库运维 python

原文链接：https://my.oschina.net/sannychan/blog/843707

版权

2019独角兽企业重金招聘Python工程师标准>>>

之前在虚拟机环境搭建过几次openstack环境，除了因为网络策略导致除网络节点能与实例连通之外，其他一切正常，所以我们申请了交换机和三台物理机来搭建Openstack，避免现有网络策略的影响。

经过分组件搭建完成后，在创建实例这一步，dashboard出现大量异常提示，经过查看nova日志，发现mysql报出 ‘Too many connections’

2017-02-22 11:30:10.764 24509 INFO nova.osapi_compute.wsgi.server [req-7d5b04d4-8b4a-4953-94f2-4b3d284e5301 c2dda1f21faa49ef9ff589b98f6b5fd3 166a0459c60f4953b5597974b5b2e3d7 - 5fafae76caaa43aaa2290f24ad147411 5fafae76caaa43aaa2290f24ad147411] 10.60.30.53 "GET /v2.1/166a0459c60f4953b5597974b5b2e3d7/os-services HTTP/1.1" status: 200 len: 1174 time: 0.0233388
ine 385, in connect
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in 
Connect
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694,
 in __init__
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     self.connect()
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916,
 in connect
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     self._request_authentication()
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124
, in _request_authentication
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     auth_packet = self._read_packet()
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991,
 in _read_packet
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     packet.check_error()
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393,
 in check_error
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in rais
e_mysql_exception
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connecti
ons')
2017-02-21 19:25:17.245 24542 ERROR nova.api.openstack.extensions

在mysql中查看当前连接，发现有大量来自controller节点的连接，由于当时急于将整套环境部署OK，就将mysql的连接数改大并重启。

重启后dashboard正常，创建实例正常。但查看mysql的连接，发现当前有1000多个连接，太不正常了。

后面经过一系列的分析，发现controller节点的nova-api进程有98个，neutron进程53，nova相关进程有151个。

通过查看，发现几乎所有组件的进程都有很多，这也能解释为啥mysql连接数那么高了。

通过查看nova.conf，发现nova进程和cpu核数有关，以前是4核的虚拟机上跑，现在是在48核的物理机上跑，所以出现了这个问题。

[DEFAULT]
# Number of workers for metadata service. The default will be the number of CPUs
# available. (integer value)
#metadata_workers=<None>


# Number of workers for OpenStack API service. The default will be the number of
# CPUs available. (integer value)
#osapi_compute_workers=<None>


[conductor]
#
# Number of workers for OpenStack Conductor service. The default will be the
# number of CPUs available.
#  (integer value)
#workers=<None>

如果 show variables like "max_connections";还是没有变化，则需要修改centos7的限制：

解决办法：修改mariadb服务的配置，增加下面两行后重启服务配置就生效了。

[root@controller ~]# less /usr/lib/systemd/system/mariadb.service
[Service]
Type=simple
User=mysql
Group=mysql
LimitNOFILE=65535 （增加在service标签下）
LimitNPROC=65535 （增加在service标签下）

转载于:https://my.oschina.net/sannychan/blog/843707