日常工作记录--2019-01-18

2019-01-18 14:31:30 0x7f6c221b6700  InnoDB: Assertion failure in thread 140102405416704 in file trx0trx.cc line 2348
InnoDB: Failing assertion: trx->lock.n_active_thrs == 1
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:31:30 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

 

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=2
max_threads=1000
thread_count=4
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 406113 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6a64005ed0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6c184a7bff thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf2bb6b]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x7bb9b1]
/lib64/libpthread.so.0(+0xf5e0)[0x7f6c38f695e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f6c3795c1f7]
/lib64/libc.so.6(abort+0x148)[0x7f6c3795d8e8]
/usr/sbin/mysqld[0x788a04]
/usr/sbin/mysqld(_Z30trx_commit_or_rollback_prepareP5trx_t+0xad)[0x10e0d1d]
/usr/sbin/mysqld(_Z17trx_rollback_stepP9que_thr_t+0x22a)[0x10caafa]
/usr/sbin/mysqld(_Z15que_run_threadsP9que_thr_t+0xed1)[0x1007741]
/usr/sbin/mysqld[0x10cc016]
/usr/sbin/mysqld[0x10cc9e0]
/usr/sbin/mysqld(_Z22trx_rollback_for_mysqlP5trx_t+0x137)[0x10ccd87]
/usr/sbin/mysqld[0xf59adb]
/usr/sbin/mysqld(_Z15ha_rollback_lowP3THDb+0x137)[0x81bb67]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG8rollbackEP3THDb+0x65)[0xec8375]
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0x66)[0x81bdb6]
/usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x3e)[0xd9928e]
/usr/sbin/mysqld(_Z21wsrep_client_rollbackP3THD+0xd2)[0x7d3972]
/usr/sbin/mysqld[0x7d4011]
/usr/sbin/mysqld(start_wsrep_THD+0x1c6)[0x7ae206]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x1253fa4]
/lib64/libpthread.so.0(+0x7e25)[0x7f6c38f61e25]
/lib64/libc.so.6(clone+0x6d)[0x7f6c37a1f34d]

 

mysql  5.7.21 galera-3-25.3.23-2.el7  频繁重启

重启发生在存储过程、事务发生的节点

 

查询galera官网得到以下信息

MySQL 5.7
New release of Galera Cluster for MySQL 5.7, consisting of MySQL-wsrep 5.7.24 and wsrep API version 25.Notable features and bug fixes in MySQL 5.7.24:

• Support for Ubuntu/Bionic was added in this release.
• Auth_pam and auth_dialog plugins were added in this release.
• Rsync SST was not copying tablespace from the donor node (mysql-wsrep#334).
• Fixes for transaction replaying from stored procedures (mysql-wsrep#336).
• Fixed a regression where transaction replaying caused a crash when AUTOCOMMIT=OFF (mysql-wsrep#344).

 

标红部分:应该是和我们现在的现象比较吻合。

现在的解决方式是将高频的统计存储过程停止。明天查看效果。

如果不行的话估计要升级到最近的galera集群了。