mysql 访问失败_MySQL 性能瓶颈导致访问失败

作者：Maxwell Li

日期：2017/01/20

未经作者允许，禁止转载本文任何内容。如需转载请留言。

问题发现

近段时间 Compass4NFV 物理部署 OpenStack Newton Ubuntu 极其不稳定，各种常见的不常见的概率性问题频发，提高代码稳定性与部署成功率刻不容缓。

1月20日，发现 aodh 在进行数据库同步时失败，而且仅仅出现在 huawei-pod2 和 intel-pod8 上，huawei-pod1 上的物理部署一切顺利。另外，在慕尼黑的 huawei-pod5 部署 OpenStack Newton CentOS 也没有出现过这个问题。

JIRA单请见：COMPASS-521

问题定位

Step1 查看 ERROR log

进入控制节点，手动初始化 aodh 数据库，ERROR 如下：

root@host1:~# su -s /bin/sh -c "aodh-dbsync" aodh

2017-01-19 22:07:57.990 4179 DEBUG aodh.storage [-] looking for 'mysql' driver in 'aodh.storage' get_connection_from_config /usr/lib/python2.7/dist-packages/aodh/storage/__init__.py:68

2017-01-19 22:07:58.078 4179 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:261

2017-01-19 22:07:58.081 4179 INFO alembic.runtime.migration [-] Context impl MySQLImpl.

2017-01-19 22:07:58.082 4179 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.

CRITI [aodh] DBError: (pymysql.err.InternalError) (1135, u'Can\'t create a new thread (errno 11 "Resource temporarily unavailable"); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug')

Traceback (most recent call last):

File "/usr/bin/aodh-dbsync", line 10, in

sys.exit(dbsync())

File "/usr/lib/python2.7/dist-packages/aodh/cmd/storage.py", line 29, in dbsync

storage.get_connection_from_config(conf).upgrade()

File "/usr/lib/python2.7/dist-packages/aodh/storage/impl_sqlalchemy.py", line 108, in upgrade

command.upgrade(cfg, "head")

File "/usr/lib/python2.7/dist-packages/alembic/command.py", line 174, in upgrade

script.run_env()

File "/usr/lib/python2.7/dist-packages/alembic/script/base.py", line 407, in run_env

util.load_python_file(self.dir, 'env.py')

File "/usr/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file

module = load_module_py(module_id, path)

File "/usr/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py

mod = imp.load_source(module_id, path, fp)

File "/usr/lib/python2.7/dist-packages/aodh/storage/sqlalchemy/alembic/env.py", line 92, in

run_migrations_online()

File "/usr/lib/python2.7/dist-packages/aodh/storage/sqlalchemy/alembic/env.py", line 75, in run_migrations_online

with connectable.connect() as connection:

File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2018, in connect

return self._connection_cls(self, **kwargs)

File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 72, in __init__

if connection is not None else engine.raw_connection()

File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection

self.pool.unique_connection, _connection)

File