ASP.NET web app performance issue - ThreadPool

 

One of my bank customers planned to roll out a new ASP.NET web application serving as their critical internet banking portal. They faced some weird performance issue in their UAT environment. They noticed the CPU utilization was really high (above 90%) without any fluctuation. After checking the IIS log, I found that time-taken value was really small which showed IIS can respond promptly. The stress testing was conducted in the same machine as IIS server so any suspicion on network can be diverted. As usual, I took three rounds of hang dump on IIS server as well as performance monitor log.

 

The CPU utilization on IIS server was really high and most of them was User Mode time. I also noticed the Worker Thread usage had a running number of 24. As the number shown here might not be accurate, we need to dig further on whether it was a thread pool issue.

Work Request in Queue: 0

--------------------------------------

Number of Timers: 9

--------------------------------------

CPU utilization 91%

--------------------------------------

Worker Thread: Total: 27 Running: 24 Idle: 3 MaxLimit: 200 MinLimit: 2

Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2

 

With that in mind, I dumped the System.Web.RequestQueue object and found the minExternFreeThreads value of 176 which can in turn calculate the worker threads of 24.

 

0:010> !do 06641d20

Name: System.Web.RequestQueue

MethodTable: 68a20dc8

EEClass: 68a20d58

Size: 64(0x40) bytes

GC Generation: 2

(C:/WINDOWS/assembly/GAC_32/System.Web/2.0.0.0__b03f5f7f11d50a3a/System.Web.dll)

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

790fed1c  4001134       14         System.Int32  1 instance      176 _minExternFreeThreads

790fed1c  4001135       18         System.Int32  1 instance      152 _minLocalFreeThreads

790fed1c  4001136       1c         System.Int32  1 instance     5000 _queueLimit

7910ca9c  4001137       2c      System.TimeSpan  1 instance 06641d4c _clientConnectedTime

79104f64  4001138       28       System.Boolean  1 instance        1 _iis6

79108964  4001139        4 ...Collections.Queue  0 instance 06641d60 _localQueue

79108964  400113a        8 ...Collections.Queue  0 instance 06641e14 _externQueue

790fed1c  400113b       20         System.Int32  1 instance      275 _count

79113c8c  400113c        c ...ding.WaitCallback  0 instance 06641e38 _workItemCallback

790fed1c  400113d       24         System.Int32  1 instance        0 _workItemCount

79104f64  400113e       29       System.Boolean  1 instance        0 _draining

7910ca9c  400113f       34      System.TimeSpan  1 instance 06641d54 _timerPeriod

7910d0dc  4001140       10 ...m.Threading.Timer  0 instance 06641e78 _timer

 

The number of worker threads = Total - minExternFreeThreads = 100*CPUs - 176 = 100*2 – 176 = 24. So in our case, the 24 limit has reached.

 

So the resolution is quite straightforward as well. We need to either add more nodes to offload the client requests or we need to reduce minFreeThreads (minExternFreeThreads) to 140 or less.

 

Here I posted some good articles for your reference:

 

ASP.NET 2.0 uses AutoConfig=true for some settings by default:
http://msdn.microsoft.com/en-us/library/7w2sway1.aspx

821268 Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
http://support.microsoft.com/default.aspx?scid=kb;EN-US;821268

 

Don’t forget to set the AutoConfig=false (the blog is using Maxconnection, for other parameters, we also need to set AutoConfig=false):

http://blogs.msdn.com/asiatech/archive/2009/09/24/maxconnection-failed-to-take-effect.aspx