场景
WCF应用程序部署在IIS7中,使用net.tcp协议对外给几百台客户端提供服务,应用程序池不断崩溃重启。
分析过程
- 在事件查看器中看到的错误信息类似于
为应用程序池“XX”提供服务的进程在与 Windows Process Activation Service 通信时出现严重错误
- 使用windbg在崩溃时进行分析,找到引发崩溃的堆栈的最顶部信息进行搜索,找到两个补丁包安装上
用于搜索的顶部异常信息为:IOCompletionThunk.UnhandledExceptionFrame BeginReadCore
0:084> !analyze -v ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* ***** OS symbols are WRONG. Please fix symbols to do analysis. ************************************************************************* *** *** *** *** *** Either you specified an unqualified symbol, or your debugger *** *** doesn't have full symbol information. Unqualified symbol *** *** resolution is turned off by default. Please either specify a *** *** fully qualified symbol module!symbolname, or enable resolution *** *** of unqualified symbols by typing ".symopt- 100". Note that *** *** enabling unqualified symbol resolution with network symbol *** *** server shares in the symbol path may cause the debugger to *** *** appear to hang for long periods of time when an incorrect *** *** symbol name is typed or the network symbol server is down. *** *** *** *** For some commands to work properly, your symbol path *** *** must point to .pdb files that have full type information. *** *** *** *** Certain .pdb files (such as the public OS symbols) do not *** *** contain the required information. Contact the group that *** *** provided you with these symbols if you need this command to *** *** work. *** *** *** *** Type referenced: ntdll!_PEB *** *** *** ************************************************************************* ************************************************************************* *** *** *** *** *** Either you specified an unqualified symbol, or your debugger *** *** doesn't have full symbol information. Unqualified symbol *** *** resolution is turned off by default. Please either specify a *** *** fully qualified symbol module!symbolname, or enable resolution *** *** of unqualified symbols by typing ".symopt- 100". Note that *** *** enabling unqualified symbol resolution with network symbol *** *** server shares in the symbol path may cause the debugger to *** *** appear to hang for long periods of time when an incorrect *** *** symbol name is typed or the network symbol server is down. *** *** *** *** For some commands to work properly, your symbol path *** *** must point to .pdb files that have full type information. *** *** *** *** Certain .pdb files (such as the public OS symbols) do not *** *** contain the required information. Contact the group that *** *** provided you with these symbols if you need this command to *** *** work. *** *** *** *** Type referenced: nt!IMAGE_NT_HEADERS32 *** *** *** ************************************************************************* FAULTING_IP: KERNELBASE!RaiseException+3d 000007fe`fdbfac3d 4881c4c8000000 add rsp,0C8h EXCEPTION_RECORD: 0000000000000001 -- (.exr 0x1) Cannot read Exception record @ 0000000000000001 DEFAULT_BUCKET_ID: WRONG_SYMBOLS PROCESS_NAME: w3wp.exe ADDITIONAL_DEBUG_TEXT: You can run '.symfix; .reload' to try to fix the symbol path and load symbols. FAULTING_MODULE: 0000000077930000 ntdll DEBUG_FLR_IMAGE_TIMESTAMP: 0 ERROR_CODE: (NTSTATUS) 0xe0434352 - <Unable to get error code text> EXCEPTION_CODE: (NTSTATUS) 0xe0434352 - <Unable to get error code text> EXCEPTION_PARAMETER1: ffffffff80131501 EXCEPTION_PARAMETER2: 0000000000000000 EXCEPTION_PARAMETER3: 0000000000000000 EXCEPTION_PARAMETER4: 0 APP: w3wp.exe MANAGED_STACK: SP IP Function 0000000009DC9CE0 000007FEE1BBF617 System_ServiceModel_ni!System.ServiceModel.Channels.StreamConnection.BeginRead(Int32, Int32, System.TimeSpan, System.Threading.WaitCallback, System.Object)+0xfa7b67 0000000009DCE2B0 000007FEE0C3AE5E System_ServiceModel_ni!System.ServiceModel.Channels.SessionConnectionReader.BeginReceive(System.TimeSpan, System.Threading.WaitCallback, System.Object)+0x1ee 0000000009DCE340 000007FEE0C3AC56 System_ServiceModel_ni!System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.PerformOperation(System.TimeSpan)+0x46 0000000009DCE380 000007FEE0C3AB1B System_ServiceModel_ni!System.ServiceModel.Channels.SynchronizedMessageSource+SynchronizedAsyncResult`1[[System.__Canon, mscorlib]]..ctor(System.ServiceModel.Channels.SynchronizedMessageSource, System.TimeSpan, System.AsyncCallback, System.Object)+0x12b 0000000009DCE410 000007FEE0C3A997 System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0xe7 0000000009DCE4E0 000007FEE0C3A846 System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel+TryReceiveAsyncResult..ctor(System.ServiceModel.Channels.FramingDuplexSessionChannel, System.TimeSpan, System.AsyncCallback, System.Object)+0x76 0000000009DCE550 000007FEE0C3A7BB System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginTryReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0x3b 0000000009DCE5B0 000007FEE132B224 System_ServiceModel_ni!System.ServiceModel.Channels.InputChannelWrapper.BeginTryReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0x114 0000000009DCE650 000007FEE13386E8 System_ServiceModel_ni!System.ServiceModel.Channels.ReplyOverDuplexChannelBase`1[[System.__Canon, mscorlib]].BeginTryReceiveRequest(System.TimeSpan, System.AsyncCallback, System.Object)+0x18 0000000009DCE680 000007FEE162EE18 System_ServiceModel_ni!System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].StartInnerReceive()+0x88 0000000009DCE6D0 000007FEE162ED30 System_ServiceModel_ni!System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Start()+0x10 0000 EXCEPTION_OBJECT: !pe 1bffc9ab8 Exception object: 00000001bffc9ab8 Exception type: System.ServiceModel.CommunicationException Message: 套接字连接已中止。这可能是由于处理消息时出错或远程主机超过接收超时或者潜在的网络资源问题导致的。本地套接字超时是“10675199.02:48:05.4775807”。 InnerException: System.IO.IOException, Use !PrintException 00000001bffc9878 to see more. StackTrace (generated): SP IP Function 0000000009DC9CE0 000007FEE1BBF617 System_ServiceModel_ni!System.ServiceModel.Channels.StreamConnection.BeginRead(Int32, Int32, System.TimeSpan, System.Threading.WaitCallback, System.Object)+0xfa7b67 0000000009DCE2B0 000007FEE0C3AE5E System_ServiceModel_ni!System.ServiceModel.Channels.SessionConnectionReader.BeginReceive(System.TimeSpan, System.Threading.WaitCallback, System.Object)+0x1ee 0000000009DCE340 000007FEE0C3AC56 System_ServiceModel_ni!System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.PerformOperation(System.TimeSpan)+0x46 0000000009DCE380 000007FEE0C3AB1B System_ServiceModel_ni!System.ServiceModel.Channels.SynchronizedMessageSource+SynchronizedAsyncResult`1[[System.__Canon, mscorlib]]..ctor(System.ServiceModel.Channels.SynchronizedMessageSource, System.TimeSpan, System.AsyncCallback, System.Object)+0x12b 0000000009DCE410 000007FEE0C3A997 System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0xe7 0000000009DCE4E0 000007FEE0C3A846 System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel+TryReceiveAsyncResult..ctor(System.ServiceModel.Channels.FramingDuplexSessionChannel, System.TimeSpan, System.AsyncCallback, System.Object)+0x76 0000000009DCE550 000007FEE0C3A7BB System_ServiceModel_ni!System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginTryReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0x3b 0000000009DCE5B0 000007FEE132B224 System_ServiceModel_ni!System.ServiceModel.Channels.InputChannelWrapper.BeginTryReceive(System.TimeSpan, System.AsyncCallback, System.Object)+0x114 0000000009DCE650 000007FEE13386E8 System_ServiceModel_ni!System.ServiceModel.Channels.ReplyOverDuplexChannelBase`1[[System.__Canon, mscorlib]].BeginTryReceiveRequest(System.TimeSpan, System.AsyncCallback, System.Object)+0x18 0000000009DCE680 000007FEE162EE18 System_ServiceModel_ni!System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].StartInnerReceive()+0x88 0000000009DCE6D0 000007FEE162ED30 System_ServiceModel_ni!System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Start()+0x10 0000000009DCE700 000007FEE615EDB1 System_Runtime_DurableInstancing_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x71 0000000009DCE760 000007FEE6236A10 System_Runtime_DurableInstancing_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0xdbc70 0000000009DCE7C0 000007FEF7F95DA6 mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x96 StackTraceString: <none> HResult: 80131501 There are nested exceptions on this thread. Run with -nested for details EXCEPTION_OBJECT: !pe 1bffc9878 Exception object: 00000001bffc9878 Exception type: System.IO.IOException Message: 读取操作失败,请参见内部异常。 InnerException: System.ServiceModel.CommunicationException, Use !PrintException 0000000120009910 to see more. StackTrace (generated): SP IP Function 0000000009DCBD10 000007FEF697945C System_ni!System.Net.Security._SslStream.ProcessRead(Byte[], Int32, Int32, System.Net.AsyncProtocolRequest)+0x28c