openstack基于MySQL_openstack基于mysql的问题修复

最新推荐文章于 2022-11-08 22:34:54 发布

渊亭无迹静涛君

最新推荐文章于 2022-11-08 22:34:54 发布

阅读量321

点赞数

文章标签： openstack基于MySQL

本文链接：https://blog.csdn.net/weixin_28808097/article/details/113205480

版权

在排查一台openstack单节点服务器故障时，发现由于mysqld服务未启动导致服务异常。错误日志显示数据库非正常关闭，进而影响了crash recovery。通过删除并备份ibdata*和ib_logfile*文件，重启mysqld服务，虽然数据库启动成功，但因数据不一致导致nova和keystone服务报错。为恢复服务，采取了mysqldump导出所有数据库，关闭强制恢复设置，删除数据库文件，再重启mysqld导入数据的方法，成功解决了问题。

摘要由CSDN通过智能技术生成

openstack基于mysql的问题修复

同事一台openstack单节点服务器由于未知原因，无法正常工作。

远程引导同事检查故障。

[root@localhost ~]# nova-manage service list

出现错误:SQL connection failed 。貌似没有连接数据库,看来可能是服务或数据库有问题。

引导同事检测openstack服务及数据库：[root@sxun init.d]# /etc/init.d/openstack-keystone status

keystone (pid 12593) is running...

[root@sxun init.d]# /etc/init.d/openstack-nova-api status

openstack-nova-api (pid 7446) is running...

[root@sxun init.d]# /etc/init.d/openstack-nova-compute status

openstack-nova-compute (pid 11728) is running...

[root@sxun init.d]# /etc/init.d/openstack-nova-conductor status

openstack-nova-conductor (pid 7704) is running...

[root@sxun init.d]# /etc/init.d/mysqld statusmysql is stoppd

[root@sxun init.d]# /etc/init.d/mysqld start

starting mysqld: [FAILED]

restart也是如此，看来是mysqld服务没有启动的问题。为什么没有启动了，找到了mysqld的日志查看：

tail -100 /var/lib/mysql/sxun.err

错误日志报大致错误如下:

170420 15:00:20 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

......

##############以下引用#############130508 16:30:53 mysqld_safe Starting mysqlddaemon with databases from /data/mysql

130508 16:30:54 [Warning] The syntax'--log-slow-queries' is deprecated and will be removed in a future release.Please use '--slow-query-log'/'--slow-query-log-file' instead.

130508 16:30:54 InnoDB: The InnoDB memoryheap is disabled

130508 16:30:54 InnoDB: Mutexes andrw_locks use GCC atomic builtins

130508 16:30:54 InnoDB: Compressed tablesuse zlib 1.2.3

130508 16:30:54 InnoDB: Initializing bufferpool, size = 500.0M

130508 16:30:54 InnoDB: Completedinitialization of buffer pool

130508 16:30:54 InnoDB: highest supportedfile format is Barracuda.

InnoDB: Log scan progressed past thecheckpoint lsn 27219795678

130508 16:30:54 InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

InnoDB: Reading tablespace information fromthe .ibd files...

InnoDB: Restoring possible half-writtendata pages from the doublewrite

InnoDB: buffer...

InnoDB: Doing recovery: scanned up to logsequence number 27219928522

##############引用，中间省略#############

解决办法：删除掉ibdata* ,

ib_logfile* ,所有的日志。重启后可启动数据库，因数据库是非正常关闭引起的。所有会造成数据丢失。要做好备份工作

##############以上引用#############

[root@sxun mysql]# ll

total 225356

drwx------ 2 mysql mysql 4096 Feb 15 16:06 cinder

drwx------ 2 mysql mysql 4096 Oct 1 2016 glance

-rw-rw---- 1 mysql mysql 220200960 Apr 20 18:08 ibdata1

-rw-rw---- 1 mysql mysql 5242880 Apr 20 18:08 ib_logfile0

-rw-rw---- 1 mysql mysql 5242880 Apr 20 18:08 ib_logfile1

drwx------ 2 mysql mysql 4096 Oct 1 2016 keystone

drwx------ 2 mysql mysql 4096 Oct 1 2016 mysql

srwxrwxrwx 1 mysql mysql 0 Feb 16 14:52 mysql.sock

drwx------ 2 mysql mysql 4096 Oct 1 2016 nova

drwx------ 2 mysql mysql 4096 Oct 1 2016 ovs_neutron

-rw-r----- 1 mysql root 38548 Feb 16 14:52 sxun.err

drwx------ 2 mysql mysql 4096 Oct 1 2016 test

生产环境不敢删除，先移动其它地方备份记[root@sxun init.d]#mv ibdata* /root/

[root@sxun init.d]#mv ib_logfile* /root/

[root@sxun init.d]# /etc/init.d/mysqld start

starting mysqld: [OK]

服务器的数据库已经是启动起来了，也生成了新的ib*文件。测试一下keystone和nova服务，但是：[root@sxun instances]# nova-manage service list

CRITICAL nova[req-97fd8f67-8a06-4ab7-b421-01d36bc6ad02 None None] (ProgrammingError) (1146,"Table 'nova.services' doesn't exist") 'SELECT services.created_at ASservices_created_at, services.updated_at AS services_updated_at,services.deleted_at AS services_deleted_at, services.deleted ASservices_deleted, services.id AS services_id, services.host AS services_host,services.`binary` AS services_binary, services.topic AS services_topic,services.report_count AS services_report_count, services.disabled ASservices_disabled, services.disabled_reason AS services_disabled_reason \nFROMservices \nWHERE services.deleted = %s' (0,)

[root@sxun instances]# keystone user-list

Authorization Failed: An unexpected errorprevented the server from fulfilling your request. (ProgrammingError) (1146,"Table 'keystone.domain' doesn't exist") 'SELECT domain.id ASdomain_id, domain.name AS domain_name, domain.enabled AS domain_enabled,domain.extra AS domain_extra \nFROM domain \nWHERE domain.id = %s' ('default',)(HTTP 500)

mysqld起来了,但由于ib*文件已经更换，数据不一样了。nova等各项服务已经出错了，尝试reboot重启了一下服务器，仍然不行，ib*得还原，但还原又启动不了mysqld了。

直接抄袭主义不行，关键还得想办法：InnoDB: Database was not shut down normally!

InnoDB: Starting crash recovery.

问题在于这个错误，数据库非正常关机，然后数据库重启后有一个恢复操作，恢复不了就挂了。

后续的解决方案：

1、还原ib*文件，

2、在配置文件加入不强制恢复数据的关掉 #innodb_force_recover = 6。

3、启动数据库，mysqldump将数据库导出。

4、删除数据库文件，mysql启动完成后重新导入mysql数据库

同事操的刀，具体操作如下：[root@sxun01 mysql(keystone_admin)]#mysqldump -uroot --all-database >> my_mysql.dump

-- Warning: Skipping the data of tablemysql.event. Specify the --events option explicitly.

[root@sxun01 mysql(keystone_admin)]#/etc/init.d/mysqld stop

Stopping mysqld: [ OK ]

[root@sxun01 mysql(keystone_admin)]# rm -rfibdata1 ib_logfile0 ib_logfile1

[root@sxun01 mysql(keystone_admin)]# vi/etc/my.cnf

[client]

port = 3306