最近某套MySQL数据库服务器异常关机,导致MySQL不能正常拉起来,启动过程中,error日志中记录了如下的信息,可以看到,数据库因为异常关闭,此时会进行实例恢复的操作,
[Note] InnoDB: Database was not shutdown normally!
[Note] InnoDB: Starting crash recovery.
...
[Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
[Note] InnoDB: 32 non-redo rollback segment(s) are active.
[Note] InnoDB: Waiting for purge to start
...
[ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
...
[ERROR] Can't open and lock privilege tables: Unknown error 1146
...
Status information:
Current dir: /opt/app/mysql/data/
Running threads: 0 Stack size: 262144
Current locks:
lock: 0x35f53f0:
lock: 0x35ad680:
lock: 0x35a5560:
lock: 0x359e5a0:
lock: 0x3589bd0:
...
Events status:
LLA = Last Locked At LUA = Last Unlocked At
WOC = Waiting On Condition DL = Data Locked
Event scheduler status:
State : INITIALIZED
Thread id : 0
...
Event queue status:
Element count : 0
Data locked : NO
Attempting lock : NO
LLA : init_queue:96
LUA : init_queue:104
WOC : NO
Next activation : never
2021-10-14T10:26:05.258906Z 0 [Warning] InnoDB: Cannot open table mysql/servers from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2021-10-14T10:26:05.258952Z 0 [ERROR] Can't open and lock privilege tables: Unknown error 1146
根据一些资料,从文件夹的权限、启动命令等角度尝试解决均未果,但是数据文件目录还在本地磁盘中,既然旧库很难拉起来,能不能通过重建一套新的数据库,带动这些旧的数据文件?
尝试一下,为了避免版本不同导致的问题,首先从官网下载一个同版本的数据库软件,mysql-5.7.22-el7-x86_64.tar.gz,不得不说,MySQL这种二进制的安装确实很简单,具体参考《小白学习MySQL - 数据库软件和初始化安装
直接将旧的数据库文件夹拷贝至新库目录下,登录数据库,能进入这个“旧”的数据库,但是访问所有的表都提示1146不存在"'xxx' doesn't exist",此时要将旧的ibdata1拷到新的库路径,重新登陆,虽然日志中报了一堆错,
[Warning] InnoDB: Cannot open table test/AO_0456E7_BAM_MARKER from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
...
[ERROR] InnoDB: Page [page id: space=0, page number=7] log sequence number 43749257776 is in the future! Current system log sequence number 2589302.
2021-10-15T10:32:46.280173+08:00 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html for information about forcing recovery.
...
InnoDB: Operating system error number 2 in a file operation.
2021-10-15T10:32:46.393644+08:00 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2021-10-15T10:32:46.393648+08:00 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2021-10-15T10:32:46.393652+08:00 0 [ERROR] InnoDB: Cannot open datafile for read-only: './test/AO_187CCC_SIDEBAR_LINK.ibd' OS error: 71
2021-10-15T10:32:46.393658+08:00 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2021-10-15T10:32:46.393662+08:00 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2021-10-15T10:32:46.393665+08:00 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2021-10-15T10:32:46.393670+08:00 0 [ERROR] InnoDB: Could not find a valid tablespace file for `test/AO_187CCC_SIDEBAR_LINK`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
但是此时能登录到旧的数据库访问表了,惊不惊喜?激不激动?
不得不感叹一句,MySQL可真耐倒腾。
尽管如此,由于旧的ibdata1还是存在一些不兼容,mysql数据库中的一些表此时就可能无法访问了,还可能出现其他的问题,
[ERROR] InnoDB: Tablespace for table `mysql`.`innodb_table_stats` is missing.
2021-10-15T10:34:13.283461+08:00 2 [ERROR] InnoDB: Fetch of persistent statistics requested for table `test`.`AO_0456E7_RESOURCE_MAPPING` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
因此稳妥的方式,是将这个旧库的数据导出来,重新导入一个新的数据库,保证环境的一致,
mysqldump -uroot -p test > /opt/app/mysql/dump/test.sql
重新初始化库
create database test;
mysqldump -uroot -p test </opt/app/mysql/dump/test.sql
创建一个新用户,允许远程访问,
create user 'test'@'%' identified by 'test123';
grant all privileges on test.* to 'test'@'%';
flush privileges;
如果数据库内容太多了,登录较慢,
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
可以用-A参数,实现快速登录,
mysql -utest -p -A
由于异常关机时存在一些未决事务,此时新的库中可能存在一些不一致的事务,这是需要注意的,如果无伤大雅,其实这种workaround还算行,此时就可以在这个新的数据库访问旧的数据了。
库虽然起来了,但是这个工作,还未完成,出现异常关机,我们可能躲不开,但是数据备份机制我们是需要的,这是保障业务运行的基础,环境再出问题,我们有数据备份,就什么都不怕了。
数据备份有很多的形式,最简单直接的,就是写个脚本,加到定时任务中,
0 0 * * * /opt/app/mysql/dump/test_backup.sh >> /opt/app/mysql/dump/run.log 2>&1
建议,
1. 备份的数据文件最好通过gzip压缩存储,压缩比能到1/5,节省存储空间。
2. 备份逻辑中需要增加历史数据删除策略,存储有限时间段的数据备份文件。
3. 日志写的细致些,执行异常时,才可以快速定位。
因此,无论你使用的是什么数据库,如果数据很重要,或者应用运行依赖这些数据,就创建好数据的保障机制,这才是王道。
参考资料,
http://www.geekapp.cn/archives/390.html
https://www.jianshu.com/p/d424f46a12ae
https://blog.csdn.net/lctmei/article/details/86520214
https://blog.csdn.net/qq_37960324/article/details/82228161
https://jingyan.baidu.com/article/3a2f7c2e0a0d2626afd611a7.html
小白学习MySQL,
《小白学习MySQL - InnoDB支持optimize table?
《小白学习MySQL - table_open_cache的作用
《小白学习MySQL - only_full_group_by的校验规则
《小白学习MySQL - max_allowed_packet》
《小白学习MySQL - mysqldump保证数据一致性的参数差异
《小白学习MySQL - MySQL会不会受到“高水位”的影响?
近期更新的文章:
《知乎的彩蛋》
文章分类和索引: