Linux系统中MySQL Server因only_full_group_by导致自动升级失败，引发上游系统接口失效的问题

MySQL Server自动升级导致上游系统崩溃的问题

问题情景

事情是这样的，早上大概九点钟的时候，被告知系统登录验证码接口失效，我当时就想到可能是数据库出了问题。

1. SSH 连接上业务服务器终端，通过tail -n 20 xxxlog查看最新的日志发现，后端应用连接不上 MySQL 数据库了(Communications link failure)。

2023-05-09 06:54:40.373 ERROR 305711 --- [Druid-ConnectionPool-Create-712974096] com.alibaba.druid.pool.DruidDataSource   : create connection SQLException, url: jdbc:mysql://xxxxxxxx/xxxx?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai, errorCode 0, state 08S01

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

2. 之后通过查看 cat /var/log/mysql/error.log 错误日志发现早上6点54分的时候，MySQL Server 8.0.32被迫开始关闭，接着就那几个线程就被关闭，Server 于56分完成关闭。
3. 紧接着mysqld 8.0.33开始启动(注意这个地方的版本号和上边的版本号不同，其实就暗示着升级)，然后InnoDB开始进行初始化工作。
4. 重头戏来了：[Server] Server upgrade from '80032' to '80033' started. ，MySQL Server开始升级。

2023-05-08T22:54:34.495393Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.32-0ubuntu0.20.04.2).
2023-05-08T22:54:36.567898Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14752  user: 'root'.
2023-05-08T22:54:36.568071Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14745  user: 'root'.
2023-05-08T22:54:36.568139Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14747  user: 'root'.
2023-05-08T22:54:36.568228Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14718  user: 'root'.
2023-05-08T22:54:36.568288Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14679  user: 'root'.
2023-05-08T22:54:36.568381Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14759  user: 'root'.
2023-05-08T22:54:36.568538Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 14735  user: 'root'.
2023-05-08T22:56:18.635353Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32-0ubuntu0.20.04.2)  (Ubuntu).
2023-05-08T22:56:30.847093Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.1) starting as process 729947
2023-05-08T22:56:30.976467Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-08T22:57:05.368518Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-08T23:57:26.306400Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.

5. 不知道什么原因，升级一直卡着，我尝试杀死进程之后，但是它会重新启动，继续升级。
6. 没办法，为了尽快恢复，我查询了MySQL官方文档，发现可以用--upgrade=none指定不升级的模式启动MySQL，但是遭到了拒绝。

Server shutting down because upgrade is required, yet prohibited by the command line option '--upgrade=NONE'.

7. 后来尝试以 mysqld --upgrade=minimal模式启动了一个实例。

Server upgrade is required, but skipped by command line option '--upgrade=MINIMAL'.

---

解决问题

1. 接着我就耐下心来继续分析日志，发现他在升级过程中由于一个SQL语句执行失败，导致升级失败。

2023-05-09T02:15:00.808085Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement 'ALTER TABLE tables_priv   MODIFY Db char(64) NOT NULL default '',   MODIFY User char(32) NOT NULL default '',   MODIFY Table_name char(64) NOT NULL default '',   CONVERT TO CHARACTER SET utf8mb3 COLLATE utf8mb3_bin; ' failed with error code = 3664, error message = 'Failed to delete SDI 'mysql.tables_priv' in tablespace 'mysql'.'.
2023-05-09T02:15:00.808085Z 4 [ERROR] [MY-013178] [Server] Failed to upgrade server.

2. 在前边提到为了不影响客户的使用，我临时以--upgrade=minimal模式启动了一个实例，所以我直接在Navicat中执行了这条语句，发现报了以下错误：

   [Err] 1055 - Expression #1 of ORDER BY clause is not in GROUP BY clause and contains nonaggregated column 'information_schema. PROFILING .SEQ' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
   

3. 看到了only_full_group_by，立马知道怎么搞了，之前一直拖着没修改这个配置，果不其然还是出了问题。

4. 通过select @@global.sql_mode在Navicat中查询出sql_mode，然后把only_full_group_by删除掉，把剩下组合在一起的写到mysqld.cnf中。

5. 最后以普通模式启动(不加--upgrade=...) MySQL Server，数据库就会更新成功。

2023-05-09T03:46:42.518474Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.1) starting as process 3228
2023-05-09T03:46:42.531758Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-09T03:46:44.657204Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-09T03:46:46.185053Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.
2023-05-09T03:49:18.353587Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' completed.
2023-05-09T03:49:20.925513Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2023-05-09T03:49:20.944206Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2023-05-09T03:49:21.852286Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-05-09T03:49:21.852370Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-05-09T03:49:21.881574Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-05-09T03:49:21.881651Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.20.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).

补充材料：MySQL官网对于升级的一些描述

升级过程

MySQL升级主要涉及到两部分的升级：

Data dictionary upgrade(数据字典升级)

• mysql schema 中的数据字典表。 如果实际数据字典版本低于当前预期版本，服务器会创建具有更新定义的数据字典表，将持久化的元数据复制到新表中，以原子操作的方式用新表替换旧表，并重新初始化数据字典。
• performance schema、INFORMATION_SCHEMA 和 ndbinfo。

Server upgrade 服务器升级

此步骤包括所有其他升级任务。 如果现有 MySQL 安装的服务器版本低于新安装的 MySQL 版本，则必须升级其他所有内容：

• mysql schema中的系统表（非数据字典表）。
• sys schema
• user schema

--upgrade参数

下边提到的：
steps 1 表示数据字典升级
step 2 表示服务器升级

As of MySQL 8.0.16, the --upgrade server option controls whether and how the server performs an automatic upgrade at startup:

• With no option or with --upgrade=AUTO, the server upgrades anything it determines to be out of date (steps 1 and 2).

• With --upgrade=NONE, the server upgrades nothing ( skips steps 1 and 2), but also exits with an error if the data dictionary must be upgraded . It is not possible to run the server with an out-of-date data dictionary; the server insists on either upgrading it or exiting.

• With --upgrade=MINIMAL, the server upgrades the data dictionary, the Performance Schema, and the INFORMATION_SCHEMA, if necessary (step 1). Note that following an upgrade with this option, Group Replication cannot be started, because system tables on which the replication internals depend are not updated, and reduced functionality might also be apparent in other areas.

• With --upgrade=FORCE, the server upgrades the data dictionary, the Performance Schema, and the INFORMATION_SCHEMA, if necessary (step 1), and forces an upgrade of everything else (step 2). Expect server startup to take longer with this option because the server checks all objects in all schemas.