NetApp FAS系列一个CIFS bug引起的控制器重启案例分享

最新推荐文章于 2024-03-23 09:21:33 发布

存储服务专家StorageExpert

最新推荐文章于 2024-03-23 09:21:33 发布

阅读量1.1k

点赞数

分类专栏： NetApp存储设备文章标签：运维服务器网络系统架构硬件架构

本文链接：https://blog.csdn.net/m0_72255440/article/details/125943136

版权

NetApp存储设备专栏收录该内容

24 篇文章 9 订阅

订阅专栏

客户有一台IBM的N3240，对应到NetApp就是FAS2240，经常发生控制器重启，如果顺利重启，客户基本上就没有感受到，但是有些情况，重启没有成功，一个控制器掉了，客户业务就有明显的感知。客户的做法就是把控制器再重启一下，就是在loader下，Boot_ontap 一下，机器又顺利启动了。这样反反复复，终于有一天，最终用户怒了，要维保商分析原因。下面就是这个case的分析过程分享。

查看系统的event日志会发现，这个控制器几乎每天都有重启，甚至每天有好几次，2022半年来，重启了40多次，当然这个里面有些是处理问题重启的，有些是系统自动重启的。如下是部分event log。

Record 3180: Sun Jan 16 05:33:45 2022 [SP.critical]: Filer Reboots

Record 3195: Mon Jan 17 15:21:30 2022 [SP.critical]: Filer Reboots

Record 3226: Sat Feb 5 05:54:55 2022 [SP.critical]: Filer Reboots

Record 3241: Mon Feb 7 15:07:24 2022 [SP.critical]: Filer Reboots

Record 3280: Sun Mar 6 03:43:36 2022 [SP.critical]: Filer Reboots

Record 3294: Tue Mar 8 01:52:48 2022 [SP.critical]: Filer Reboots

Record 3307: Tue Mar 8 05:42:47 2022 [SP.critical]: Filer Reboots

Record 3328: Mon Mar 14 15:43:53 2022 [SP.critical]: Filer Reboots

Record 3349: Wed Mar 23 07:33:02 2022 [SP.critical]: Filer Reboots

Record 3362: Wed Mar 23 10:08:18 2022 [SP.critical]: Filer Reboots

Record 3377: Fri Mar 25 05:57:46 2022 [SP.critical]: Filer Reboots

Record 3393: Mon Mar 28 05:42:58 2022 [SP.critical]: Filer Reboots

Record 3411: Sat Apr 2 02:12:31 2022 [SP.critical]: Filer Reboots

Record 3429: Tue Apr 5 15:05:40 2022 [SP.critical]: Filer Reboots

Record 3449: Wed Apr 13 00:53:43 2022 [SP.critical]: Filer Reboots

Record 3476: Wed Apr 27 13:09:58 2022 [SP.critical]: Filer Reboots

Record 3493: Sun May 1 13:18:01 2022 [SP.critical]: Filer Reboots

Record 3524: Thu May 19 01:49:50 2022 [SP.critical]: Filer Reboots

Record 3539: Sat May 21 06:40:10 2022 [SP.critical]: Filer Reboots

Record 3553: Sun May 22 16:17:47 2022 [SP.critical]: Filer Reboots

Record 3568: Tue May 24 13:24:54 2022 [SP.critical]: Filer Reboots

Record 3598: Fri Jun 10 13:26:04 2022 [SP.critical]: Filer Reboots

Record 3615: Wed Jun 15 00:14:00 2022 [SP.critical]: Filer Reboots

Record 3629: Thu Jun 16 11:33:34 2022 [SP.critical]: Filer Reboots

Record 3644: Fri Jun 17 05:47:57 2022 [SP.critical]: Filer Reboots

Record 3657: Fri Jun 17 12:12:42 2022 [SP.critical]: Filer Reboots

Record 3676: Fri Jun 24 00:05:47 2022 [SP.critical]: Filer Reboots

Record 3690: Sat Jun 25 16:26:57 2022 [SP.critical]: Filer Reboots

Record 3705: Mon Jun 27 05:35:27 2022 [SP.critical]: Filer Reboots

Record 3720: Wed Jun 29 10:53:57 2022 [SP.critical]: Filer Reboots

Record 3736: Sat Jul 2 12:43:12 2022 [SP.critical]: Filer Reboots

Record 3750: Mon Jul 4 03:23:30 2022 [SP.critical]: Filer Reboots

Record 3766: Thu Jul 7 10:30:59 2022 [SP.critical]: Filer Reboots

Record 3779: Thu Jul 7 12:00:53 2022 [SP.critical]: Filer Reboots

Record 3794: Fri Jul 8 07:10:19 2022 [SP.critical]: Filer Reboots

Record 3807: Sat Jul 9 01:27:50 2022 [SP.critical]: Filer Reboots

Record 3822: Sun Jul 10 11:46:48 2022 [SP.critical]: Filer Reboots

Record 3836: Tue Jul 12 04:32:41 2022 [SP.critical]: Filer Reboots

Record 3850: Wed Jul 13 22:39:10 2022 [SP.critical]: Filer Reboots

Record 3864: Thu Jul 14 01:28:26 2022 [SP.critical]: Filer Reboots

Record 3877: Thu Jul 14 07:43:41 2022 [SP.critical]: Filer Reboots

Record 3892: Sat Jul 16 12:42:43 2022 [SP.critical]: Filer Reboots

Record 3906: Mon Jul 18 03:35:09 2022 [SP.critical]: Filer Reboots

Record 3919: Mon Jul 18 04:23:21 2022 [SP.critical]: Filer Reboots

Record 3933: Tue Jul 19 04:24:35 2022 [SP.critical]: Filer Reboots

Record 3946: Tue Jul 19 10:15:06 2022 [SP.critical]: Filer Reboots

然后我们再来看重启时候的日志，基本上都是下面的panic信息，都是类似的

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 06
fault virtual address   = 0x28
fault code       = supervisor read data, page not present
instruction pointer   = 0x8:0xffffffff842d9ef3
stack pointer   = 0x10:0xfffffe0008764bd0
frame pointer   = 0x10:0xfffffe0008764bf8
code segment       = base 0x0, limit 0xfffff, type 0x1b
           = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process       = 1553 (ontap: cpu2)
trap number       = 12
PANIC : page fault (supervisor read, page not present) on VA 0x28 cs:rip = 0x8:0xffffffff842d9ef3 rflags = 0x10246
version: 8.1.2: Tue Oct 30 19:56:51 PDT 2012
conf : x86_64
cpuid = 2
Uptime: 44m28s

PANIC: page fault (supervisor read, page not present) on VA 0x28 cs:rip = 0x8:0xffffffff842d9ef3 rflags = 0x10246 in SK process Auth09 on release 8.1.2 on Mon Jul 18 12:20:08 CST 2022

version: 8.1.2: Tue Oct 30 19:56:51 PDT 2012
compile flags: x86_64
HA: current time (in sk_msecs) 2648596 (in sk_cycles) 4702800163010
DUMPCORE: START
Dumping to disks: 0a.00.11
Writing panic info to sparecore disk.

导致上述panic的根本原因是 NetApp ONTAP 8.1版本的CIFS（SMB 2.0） bug 552397导致。

下面是NetApp对这个bug的官方解释：

If durable handles are enabled and all the following conditions are met, controller disruption might occur: 1. Multiple SMB 2 sessions from a workstation share the same TCP connection; and 2. There are open files on at least one of those sessions; and 3. Network error detected by the workstation triggers session reconnects.

好了，找到panic的问题了，下一步就是解决方案。其实根本的解决方案就是升级Ontap操作系统，至少到 8.1.4P9以后。如果不想升级，就这样对付，但也不想让panic天天发生，就是disable durable handles的属性或者干脆把smb 2.0 或者2.1都disable掉，不使用。

详细解决方案可以咨询博主@ wechat : StorageExpert.