2023-11-01T08:21:22.574390Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 13014483
2023-11-01T08:21:22.574401Z 0 [Note] InnoDB: Database was not shutdown normally!
2023-11-01T08:21:22.574470Z 0 [Note] InnoDB: Starting crash recovery.
2023-11-01 16:21:22 0x7f8f2d4ed700 InnoDB: Assertion failure in thread 140252917192448 in file os0file.cc line 3124
[root@sealos-k8s-node-08 ~]# docker logs dae5f49046b8
2023-11-01 16:21:21+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01 16:21:22+08:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-01 16:21:22+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01T08:21:22.487449Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-11-01T08:21:22.497761Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.497865Z 0 [Note] mysqld (mysqld 5.7.37) starting as process 1 ...
2023-11-01T08:21:22.509062Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.515635Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-x7b75.lower-test
2023-11-01T08:21:22.517729Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-11-01T08:21:22.517753Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-11-01T08:21:22.517757Z 0 [Note] InnoDB: Uses event mutexes
2023-11-01T08:21:22.517761Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-11-01T08:21:22.517763Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-11-01T08:21:22.517766Z 0 [Note] InnoDB: Using Linux native AIO
2023-11-01T08:21:22.518076Z 0 [Note] InnoDB: Number of pools: 1
2023-11-01T08:21:22.518206Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-11-01T08:21:22.520016Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-11-01T08:21:22.528689Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-11-01T08:21:22.531479Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-11-01T08:21:22.565931Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-11-01T08:21:22.574332Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 13014474
2023-11-01T08:21:22.574390Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 13014483
2023-11-01T08:21:22.574401Z 0 [Note] InnoDB: Database was not shutdown normally!
2023-11-01T08:21:22.574470Z 0 [Note] InnoDB: Starting crash recovery.
2023-11-01 16:21:22 0x7f8f2d4ed700 InnoDB: Assertion failure in thread 140252917192448 in file os0file.cc line 3124
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:21:22 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68197 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0x7f8f47dafb6c]
mysqld(handle_fatal_signal+0x501)[0x7f8f476aff31]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f8f46f75730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f8f46a507bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f8f46a3b535]
mysqld(+0x6c3df1)[0x7f8f47676df1]
mysqld(_Z18os_file_flush_funci+0x20c)[0x7f8f47fcb43c]
mysqld(_Z9fil_flushm+0x320)[0x7f8f48186b10]
mysqld(_Z15log_io_completeP11log_group_t+0x459)[0x7f8f47fa4f69]
mysqld(_Z12fil_aio_waitm+0xe0)[0x7f8f481803e0]
mysqld(io_handler_thread+0xa8)[0x7f8f4807b158]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f8f46f6afa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8f46b124cf]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
-bash-4.2# kubectl get po | grep mysql
dbapi-mysql-m6xlq 0/1 CrashLoopBackOff 1 (69s ago) 14s
hive-mysql-metastore-qs6r2 1/1 Running 2 (24d ago) 226d
mysql-hive-metastore-glusterfs-4lxjf 1/1 Running 0 7d6h
innodb_force_recovery=1
http://blog.chinaunix.net/uid-26950862-id-5593732.html
-bash-4.2# cat mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
data:
mysqld.cnf: |-
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
#log-error = /var/log/mysql/error.log
## By default we only accept connections from localhost
##bind-address = 127.0.0.1
## Disabling symbolic-links is recommended to prevent assorted security risks
#symbolic-links=0
symbolic-links=0
log-bin=/var/lib/mysql/mysql-bin
server-id=1
group_concat_max_len = 10240
# 最大睡眠时间
wait_timeout=300
# 超时时间设置
interactive_timeout = 500
####
innodb_force_recovery=1
-bash-4.2# cat mysql-dbapi.yaml
apiVersion: v1
kind: ReplicationController
metadata:
labels:
app: dbapi-mysql
name: dbapi-mysql
spec:
replicas: 1
selector:
app: dbapi-mysql
template:
metadata:
labels:
app: dbapi-mysql
spec:
nodeSelector:
doris: doris-fe
containers:
- name: dbapi-mysql
image: docker-registry-node:5000/mysql:5.7.37
ports:
- containerPort: 3306
env:
- name: TZ
value: Asia/Shanghai
- name: MYSQL_ROOT_PASSWORD
value: "123456"
volumeMounts:
- name: mysql-configmap-volume
mountPath: /etc/mysql/mysql.conf.d
- name: mysql-volume
mountPath: /var/lib/mysql
volumes:
- name: mysql-volume
persistentVolumeClaim:
claimName: mysql-dbapi-pvc
- name: mysql-configmap-volume
configMap:
name: mysql-config
---
apiVersion: v1
kind: Service
metadata:
labels:
app: dbapi-mysql-svc
name: dbapi-mysql-svc
spec:
type: NodePort
ports:
- port: 3306
targetPort: 3306
nodePort: 32310
name: mysql-port
selector:
app: dbapi-mysql
Errcode: 30 - Read-only file system
-bash-4.2# kubectl logs dbapi-mysql-m6xlq
2023-11-01 17:15:27+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01 17:15:28+08:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-11-01 17:15:28+08:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
2023-11-01T09:15:28.563686Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-11-01T09:15:28.589541Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
2023-11-01T09:15:28.589666Z 0 [Note] mysqld (mysqld 5.7.37-log) starting as process 1 ...
2023-11-01T09:15:28.604726Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
2023-11-01T09:15:28.615831Z 0 [Warning] Can't create test file /var/lib/mysql/dbapi-mysql-m6xlq.lower-test
mysqld: File '/var/lib/mysql/mysql-bin.index' not found (Errcode: 30 - Read-only file system)
2023-11-01T09:15:28.628082Z 0 [ERROR] Aborting
2023-11-01T09:15:28.628112Z 0 [Note] Binlog end
2023-11-01T09:15:28.631272Z 0 [Note] mysqld: Shutdown complete
挂载节点经常性的断电重启、Errcode: 30 - Read-only file system
重启挂载节点重启处理了
-bash-4.2# kubectl get po | grep mysql
dbapi-mysql-bcjmp 1/1 Running 0 90s
...
延展
尝试执行 update
命令,报错如下 Operation not allowed when innodb_forced_recovery > 0
, 去除这个innodb_force_recovery=1
配置