系统自己弹出诸如 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s [mysqld:2875]

系统在没有人使用的情况下自己弹出诸如以下关于内核的报错

[root@bkce tmp]#
Message from syslogd@bkce at Oct 13 14:25:00 …
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [mysqld:2875]

Message from syslogd@bkce at Oct 13 14:25:00 …
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [xfsaild/dm-0:1059]

Message from syslogd@bkce at Oct 13 14:25:00 …
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [consul:3503]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [basereport:18282]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 34s! [kworker/u16:0:10815]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 35s! [cmdb_cloudserve:26778]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 34s! [consul:31586]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [cmdb_toposerver:2713]

Message from syslogd@bkce at Oct 13 15:01:32 …
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 25s! [basereport:15883]

Message from syslogd@bkce at Oct 13 15:01:33 …
kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 25s! [cmdb_procserver:26557]

Message from syslogd@bkce at Oct 13 15:04:28 …
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [1_scheduler:9550]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [celery:32038]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [supervisord:31880]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [basereport:22621]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [python:32031]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [supervisord:30968]

Message from syslogd@bkce at Oct 13 15:05:27 …
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 34s! [xfsaild/dm-0:1059]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s! [kworker/2:2:2083]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 29s! [python:31737]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 29s! [kworker/5:2:16355]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 29s! [cmdb_authserver:26540]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 29s! [exceptionbeat:21758]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 34s! [ksoftirqd/0:6]

Message from syslogd@bkce at Oct 13 15:06:21 …
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 29s! [kworker/u16:0:10815]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [6_scheduler:9579]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 21s! [uwsgi:29155]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [consul:8168]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [python:32031]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [gunicorn:23944]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [dataWorker:8717]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [xfsaild/dm-0:1059]

Message from syslogd@bkce at Oct 13 15:27:56 …
kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [SyncThread:0:5078]

内核软死锁（soft lockup）

Soft lockup：这个bug没有让系统彻底死机，但是若干个进程（或者kernel thread）被锁死在了某个状态（一般在内核区域），很多情况下这个是由于内核锁的使用的问题。

出现死锁原因

1、CPU高负载时间过长
2、服务器电源供电不足，导致CPU电压不稳定
3、vcpus超过物理cpu cores
4、虚机所在的宿主机的CPU太忙或磁盘IO太高
5、虚机机的CPU太忙或磁盘IO太高
6、VM网卡驱动存在bug，处理高水位流量时存在bug导致CPU死锁
7、BIOS开启了超频，导致超频时电压不稳，容易出现CPU死锁
8、Linux kernel或KVM存在bug
9、BIOS Intel C-State开启导致，关闭可解决
10、BIOS spread spectrum开启导致

解决办法

echo 30 > /proc/sys/kernel/watchdog_thresh
echo “kernel.watchdog_thresh=30” >> /etc/sysctl.conf
sysctl -w kernel.watchdog_thresh=30
sysctl -q vm.swappiness
sysctl -p

然后重启系统