如何分析thread dump(另一篇)

原文链接:https://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html


This article will teach you how to analyze a JVM Thread Dump and pinpoint the root cause of your problem(s). From my perspective, Thread Dump analysis is the most important skillset to master for any individual involved in Java EE production support. The amount of information that you can derive from Thread Dump snapshots is often much beyond than what you can think of.

My goal is to share with you my knowledge on Thread Dump analysis that I accumulated over the last 10 years e.g. hundreds of Thread Dump analysis cycles with dozens of common problem patterns across many JVM versions and JVM vendors.

Please bookmark this page and stay tuned for weekly articles.
Please also feel free to share this Thread Dump training plan with your work colleagues and friends.

Sounds good, I really need to improve my Thread Dump skills… so where do we start?

What I’m proposing to you is a complete Thread Dump training plan. The following items will be covered. I will also provide you with real life Thread Dump examples that you can study and understand.

1)  Thread Dump overview & fundamentals
2)  Thread Dump generation techniques and available tools
3)  Thread Dump format differences between Sun HotSpot, IBM JRE and Oracle JRockit
4)  Thread Stack Trace explanation and interpretation
5)  Thread Dump analysis and correlation techniques
6)  Thread Dump common problem patterns (Thread race, deadlock, hanging IO calls, garbage collection / OutOfMemoryError problems, infinite looping etc.)
7)  Thread Dump examples via real life case studies

I really hope this Thread Dump analysis training plan will be beneficial for you so please stay tuned for weekly updates and articles!

But what if I still have questions or still struggling to understand these training articles?

Don’t worry and please consider me as your trainer. I strongly encourage you to ask me any question on Thread Dump (remember, there are no stupid questions) so I propose the following options to you for free; simply chose the communication model that you are more comfortable with:

1)  Submit your Thread Dump related question(s) by posting your comment(s) below the article (please feel free to remain Anonymous)
2)  Submit your Thread Dump data to the Root Cause Analysis forum
3)  Email me your Thread Dump related question(s) @phcharbonneau@hotmail.com

Can I send you my Thread Dump data from my production environment / servers?

Yes, please feel free to send me your generated Thread Dump data via email or Root Cause Analysis forum if you wish to discuss the root cause of your problem(s). Real life Thread Dump analysis is always the best way to learn.

I really hope that you will enjoy and share this Thread Dump analysis training plan. I will do my very best to provide you with quality material and answers to any question.

Before going deeper into Thread Dump analysis and problem patterns, it is very important that you understand the fundamentals. The post will cover the basics and allow you to better your JVM and middleware interaction with your Java EE container.

Java VM overview

The Java virtual machine is really the foundation of any Java EE platform. This is where your middleware and applications are deployed and active.

The JVM provides the middleware software and your Java / Java EE program with:

– A runtime environment for your Java / Java EE program (bytecode format)
– Several program features and utilities (IO facilities, data structure, Threads management, security, monitoring etc.)
– Dynamic memory allocation and management via the garbage collector

Your JVM can reside on many OS (Solaris, AIX, Windows etc.) and depending of your physical server specifications, you can install 1…n JVM processes per physical / virtual server.

JVM and Middleware software interactions

Find below a diagram showing you a high level interaction view between the JVM, middleware and application(s).

This is showing you a typical and simple interaction diagram between the JVM, middleware and application. As you can see, the Threads allocation for a standard Java EE application are done mainly between the middleware kernel itself and JVM (there are some exceptions when application itself or some APIs create Threads directly but this is not common and must be done very carefully).

Also, please note that certain Threads are managed internally within the JVM itself such as GC (garbage collection) Threads in order to handle concurrent garbage collections.

Since most of the Thread allocations are done by the Java EE container, it is important that you understand and recognize the Thread Stack Trace and identify it properly from the Thread Dump data. This will allow you to understand quickly the type of request that the Java EE container is attempting to execute.

From a Thread Dump analysis perspective, you will learn how to differentiate between the different Thread Pools found from the JVM and identify the request type.

This last section will provide you with an overview of what is a JVM Thread Dump for the HotSpot VM and the different Threads that you will find. Detail for the IBM VM Thread Dump format will be provided in the part 4.

Please note that you will find the Thread Dump sample used for this article from the root cause analysis forum.

JVM Thread Dump – what is it?

A JVM Thread Dump is a snapshot taken at a given time which provides you with a complete listing of all created Java Threads.

Each individual Java Thread found gives you information such as:

– Thread name; often used by middleware vendors to identify the Thread Id along with its associated Thread Pool name and state (running, stuck etc.)

      – Thread type & priority ex: daemon prio=3 ** middleware softwares typically create their Threads as daemon meaning their Threads are running in background; providing services to its user e.g. your Java EE application **

       – Java Thread ID ex: tid=0x000000011e52a800 ** This is the Java Thread Id obtained via java.lang.Thread.getId() and usually implemented as an auto-incrementing long 1..n**

       – Native Thread ID ex: nid=0x251c** Crucial information as this native Thread Id allows you to correlate for example which Threads from an OS perspective are using the most CPU within your JVM etc. **

       – Java Thread State and detail ex: waiting for monitor entry [0xfffffffea5afb000] java.lang.Thread.State: BLOCKED (on object monitor)
** Allows to quickly learn about Thread state and its potential current blocking condition **

        – Java Thread Stack Trace; this is by far the most important data that you will find from the Thread Dump. This is also where you will spent most of your analysis time since the Java Stack Trace provides you with 90% of the information that you need in order to pinpoint root cause of many problem pattern types as you will learn later in the training sessions

        – Java Heap breakdown; starting with HotSpot VM 1.6, you will also find at the bottom of the Thread Dump snapshot a breakdown of the HotSpot memory spaces utilization such as your Java Heap (YoungGen, OldGen) & PermGen space. This is quite useful when excessive GC is suspected as a possible root cause so you can do out-of-the-box correlation with Thread data / patterns found

1 Heap
2 PSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
3 eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
4 from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
5 to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
6 PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
7 object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
8 PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
9 object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

Thread Dump breakdown overview

In order for you to better understand, find below a diagram showing you a visual breakdown of a HotSpot VM Thread Dump and its common Thread Pools found:

As you can there are several pieces of information that you can find from a HotSpot VM Thread Dump. Some of these pieces will be more important than others depending of your problem pattern(problem patterns will be simulated and explained in future articles).

For now, find below a detailed explanation for each Thread Dump section as per our sample HotSpot Thread Dump:

# Full thread dump identifier
This is basically the unique keyword that you will find in your middleware / standalong Java standard output log once you generate a Thread Dump (ex: via kill -3 <PID> for UNIX). This is the beginning of the Thread Dump snapshot data.

1 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

# Java EE middleware, third party & custom application Threads
This portion is the core of the Thread Dump and where you will typically spend most of your analysis time. The number of Threads found will depend on your middleware software that you use, third party libraries (that might have its own Threads) and your application (if creating any custom Thread, which is generally not a best practice).

In our sample Thread Dump, Weblogic is the middleware used. Starting with Weblogic 9.2, a self-tuning Thread Pool is used with unique identifier “’weblogic.kernel.Default (self-tuning)”

1 "[STANDBY] ExecuteThread: '414' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010916a800 nid=0x2613 in Object.wait() [0xfffffffe9edff000]
2    java.lang.Thread.State: WAITING (on object monitor)
3         at java.lang.Object.wait(Native Method)
4         - waiting on <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
5         at java.lang.Object.wait(Object.java:485)
6         at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:160)
7         - locked <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
8         at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

# HotSpot VM Thread
This is an internal Thread managed by the HotSpot VM in order to perform internal native operations. Typically you should not worry about this one unless you see high CPU(via Thread Dump & prstat / native Thread id correlation).

1 "VM Periodic Task Thread" prio=3 tid=0x0000000101238800 nid=0x19 waiting on condition

# HotSpot GC Thread
When using HotSpot parallel GC (quite common these days when using multi physical cores hardware), the HotSpot VM create by default or as per your JVM tuning a certain # of GC Threads. These GC Threads allow the VM to perform its periodic GC cleanups in a parallel manner, leading to an overall reduction of the GC time; at the expense of increased CPU utilization.

1 "GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100120000 nid=0x3 runnable
2 "GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100131000 nid=0x4 runnable
3 ………………………………………………………………………………………………………………………………………………………………

This is crucial data as well since when facing GC related problems such as excessive GC, memory leaks etc, you will be able to correlate any high CPU observed from the OS / Java process(es) with these Threads using their native id value (nid=0x3). You will learn how to identify and confirm this problem is future articles.

# JNI global references count
JNI (Java Native Interface) global references are basically Object references from the native code to a Java object managed by the Java garbage collector. Its role is to prevent collection of an object that is still in use by native code but technically with no “live” references in the Java code.

It is also important to keep an eye on JNI references in order to detect JNI related leaks. This can happen if you program use JNI directly or using third party tools like monitoring tools which are prone to native memory leaks.

1 JNI global references: 1925

# Java Heap utilization view
This data was added back to JDK 1 .6 and provides you with a short and fast view of your HotSpot Heap. I find it quite useful when troubleshooting GC related problems along with HIGH CPU since you get both Thread Dump & Java Heap in a single snapshot allowing you to determine (or to rule out) any pressure point in a particular Java Heap memory space along with current Thread computing currently being done at that time. As you can see in our sample Thread Dump, the Java Heap OldGen is maxed out!

1 Heap
2  PSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
3   eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
4   from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
5   to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
6  PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
7   object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
8  PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
9   object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

I hope this article has helped to understand the basic view of a HotSpot VM Thread Dump.The next article will provide you this same Thread Dump overview and breakdown for the IBM VM.

Please feel free to post any comment or question.

Reference: How to analyze Thread Dump – part 1 How to analyze Thread Dump – Part2: JVM Overview & How to analyze Thread Dump – Part 3: HotSpot VM from our JCG partner Pierre-Hugues Charbonneau at the Java EE Support Patterns & Java Tutorial blog.

展开阅读全文

weblogic 9.22 如何通过 full thread dump 分析问题?

11-04

今天客户反馈应用服务响应太别慢,就登录到了 weblogic 控制台查看服务器的运行状态,可用堆大小已经是0了,难怪反应慢,cpu也是占用太别高,只好重启应用了,唉。。。rn 这已经是第3次了,所以得找个解决的方法才行,还好通过控制还能查看thread dump,上次直接提示服务不可用。于是得到了以下信息,没有研究过dump, 把内容贴出来,请大侠们帮忙分析下问题出在哪些程序中:rnrn服务器 Server-0 的当前线程堆栈 rn此页中显示了每个线程的当前堆栈。rn===== FULL THREAD DUMP ===============rnTue Nov 04 16:56:52 2008rn BEA JRockit(R) R26.4.0-63_CR302700-72606-1.5.0_06-20061127-1108-win-ia32rn "Main Thread" id=1 idx=0x2 tid=772 prio=5 alive, in native, waitingrn -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x0099C300[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)rn at java/lang/Object.wait()V(Native Method)rn at weblogic/t3/srvr/T3Srvr.waitForDeath()V(T3Srvr.java:730)rn ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x0099C300[fat lock]rn at weblogic/t3/srvr/T3Srvr.run([Ljava/lang/String;)I(T3Srvr.java:380)rn at weblogic/Server.main([Ljava/lang/String;)V(Server.java:67)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of trace rn "(Signal Handler)" id=2 idx=0x4 tid=1368 prio=5 alive, in native, daemonrn "(Code Generation Thread 1)" id=3 idx=0x6 tid=3156 prio=5 alive, in native, native_waiting, daemonrn "(Code Optimization Thread 1)" id=4 idx=0x8 tid=2840 prio=5 alive, in native, native_waiting, daemonrn "(GC Main Thread)" id=5 idx=0xa tid=4124 prio=5 alive, in native, daemonrn "(GC Worker Thread 1)" id=? idx=0xc tid=4128 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 2)" id=? idx=0xe tid=4116 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 3)" id=? idx=0x10 tid=4176 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 4)" id=? idx=0x12 tid=4100 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 5)" id=? idx=0x14 tid=4200 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 6)" id=? idx=0x16 tid=4148 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 7)" id=? idx=0x18 tid=4240 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 8)" id=? idx=0x1a tid=4236 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 9)" id=? idx=0x1c tid=6088 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 10)" id=? idx=0x1e tid=4252 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 11)" id=? idx=0x20 tid=4248 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 12)" id=? idx=0x22 tid=4264 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 13)" id=? idx=0x24 tid=4280 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 14)" id=? idx=0x26 tid=4276 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 15)" id=? idx=0x28 tid=4244 prio=5 alive, native_waiting, daemonrn "(GC Worker Thread 16)" id=? idx=0x2a tid=672 prio=5 alive, native_waiting, daemonrn "(VM Periodic Task)" id=6 idx=0x2c tid=856 prio=10 alive, in native, native_blocked, daemonrn "Finalizer" id=7 idx=0x2e tid=4272 prio=8 alive, in native, native_waiting, daemonrn at jrockit/memory/Finalizer.getPendingFinalizee(Z)Ljava/lang/Object;(Native Method)rn at jrockit/memory/Finalizer.access$100(Z)Ljava/lang/Object;(Unknown Source)rn at jrockit/memory/Finalizer$4.run()V(Unknown Source)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "Reference Handler" id=8 idx=0x30 tid=2460 prio=10 alive, in native, native_waiting, daemonrn at java/lang/ref/Reference.getPending()Ljava/lang/ref/Reference;(Native Method)rn at java/lang/ref/Reference.access$000()Ljava/lang/ref/Reference;(Unknown Source)rn at java/lang/ref/Reference$ReferenceHandler.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "Timer-0" id=11 idx=0x32 tid=3116 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: java/util/TaskQueue@0x00D47E68[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)rn at java/lang/Object.wait()V(Native Method)rn at java/util/TimerThread.mainLoop()V(Timer.java:483)rn ^-- Lock released while waiting: java/util/TaskQueue@0x00D47E68[fat lock]rn at java/util/TimerThread.run()V(Timer.java:462)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "Timer-1" id=12 idx=0x34 tid=4284 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: java/util/TaskQueue@0x015C6540[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)[optimized]rn at java/util/TimerThread.mainLoop()V(Timer.java:509)rn ^-- Lock released while waiting: java/util/TaskQueue@0x015C6540[fat lock]rn at java/util/TimerThread.run()V(Timer.java:462)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=13 idx=0x36 tid=4632 prio=5 alive, in native, daemonrn at java/net/PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V(Native Method)rn ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x062068C0[recursive]rn at java/net/PlainDatagramSocketImpl.receive(Ljava/net/DatagramPacket;)V(PlainDatagramSocketImpl.java:136)rn ^-- Holding lock: java/net/PlainDatagramSocketImpl@0x062068C0[thin lock]rn at java/net/DatagramSocket.receive(Ljava/net/DatagramPacket;)V(DatagramSocket.java:712)rn ^-- Holding lock: java/net/MulticastSocket@0x06206908[thin lock]rn ^-- Holding lock: java/net/DatagramPacket@0x0C8EA790[thin lock]rn at weblogic/cluster/FragmentSocket.receive([B)I(FragmentSocket.java:202)rn at weblogic/cluster/MulticastManager.run()V(MulticastManager.java:400)rn at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "weblogic.time.TimeEventGenerator" id=14 idx=0x38 tid=5140 prio=9 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/time/common/internal/TimeTable@0x015C6610[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at weblogic/time/common/internal/TimeTable.snooze()V(TimeTable.java:289)[optimized]rn ^-- Lock released while waiting: weblogic/time/common/internal/TimeTable@0x015C6610[fat lock]rn at weblogic/time/common/internal/TimeEventGenerator.run()V(TimeEventGenerator.java:117)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "weblogic.timers.TimerThread" id=15 idx=0x3a tid=4584 prio=9 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/timers/internal/TimerThread@0x015C6658[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)[optimized]rn at weblogic/timers/internal/TimerThread$Thread.run()V(TimerThread.java:260)rn ^-- Lock released while waiting: weblogic/timers/internal/TimerThread@0x015C6658[fat lock]rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=16 idx=0x3c tid=4588 prio=5 alive, in native, daemonrn at jrockit/net/SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)rn at jrockit/net/SocketNativeIO.socketWrite(Ljava/io/FileDescriptor;[BII)V(Unknown Source)[inlined]rn at java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(Unknown Source)[inlined]rn at java/net/SocketOutputStream.socketWrite([BII)V(SocketOutputStream.java:92)[inlined]rn at java/net/SocketOutputStream.write([BII)V(SocketOutputStream.java:137)[optimized]rn at oracle/net/ns/DataPacket.send(I)V(Unknown Source)[inlined]rn at oracle/net/ns/NetOutputStream.flush()V(Unknown Source)[inlined]rn at oracle/net/ns/NetInputStream.getNextPacket()V(Unknown Source)[optimized]rn at oracle/net/ns/NetInputStream.read([BII)I(Unknown Source)[inlined]rn at oracle/net/ns/NetInputStream.read([B)I(Unknown Source)[inlined]rn at oracle/net/ns/NetInputStream.read()I(Unknown Source)[optimized]rn at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1()S(T4CMAREngine.java:1099)[optimized]rn at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1()B(T4CMAREngine.java:1070)[inlined]rn at oracle/jdbc/driver/T4C8Oall.receive()V(T4C8Oall.java:449)[optimized]rn at oracle/jdbc/driver/T4CPreparedStatement.doOall8(ZZZZ)V(T4CPreparedStatement.java:216)[inlined]rn at oracle/jdbc/driver/T4CPreparedStatement.executeForDescribe()V(T4CPreparedStatement.java:833)[optimized]rn at oracle/jdbc/driver/OracleStatement.executeMaybeDescribe()V(OracleStatement.java:1037)[optimized]rn at oracle/jdbc/driver/T4CPreparedStatement.executeMaybeDescribe()V(T4CPreparedStatement.java:839)[optimized]rn at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout()V(OracleStatement.java:1132)[optimized]rn at oracle/jdbc/driver/OraclePreparedStatement.executeInternal()I(OraclePreparedStatement.java:3316)[optimized]rn at oracle/jdbc/driver/OraclePreparedStatement.executeQuery()Ljava/sql/ResultSet;(OraclePreparedStatement.java:3361)[optimized]rn ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x0C619320[thin lock]rn ^-- Holding lock: oracle/jdbc/driver/T4CPreparedStatement@0x3C5E3CD8[thin lock]rn at weblogic/jdbc/wrapper/PreparedStatement.executeQuery()Ljava/sql/ResultSet;(PreparedStatement.java:100)[optimized]rn at org/loushang/commons/jdbc/core/JdbcTemplate$DefaultQueryExecutor.executeQuery(Ljava/sql/PreparedStatement;)Ljava/sql/ResultSet;(JdbcTemplate.java:1008)[optimized]rn at org/loushang/bsp/monitor/jdbc/JdbcMonitorTemplate.query(Lorg/loushang/commons/jdbc/core/PreparedStatementCreator;Lorg/loushang/commons/jdbc/core/ResultSetExtractor;)V(JdbcMonitorTemplate.java:109)[optimized]rn at org/loushang/commons/jdbc/core/JdbcTemplate.query(Lorg/loushang/commons/jdbc/core/PreparedStatementCreator;Lorg/loushang/commons/jdbc/core/RowCallbackHandler;)V(JdbcTemplate.java:390)[inlined]rn at org/loushang/commons/jdbc/object/SqlQueryNoPage.execute([Ljava/lang/Object;Ljava/util/Map;)Ljava/util/List;(SqlQueryNoPage.java:70)[inlined]rn at org/loushang/commons/jdbc/object/SqlQueryNoPage.execute([Ljava/lang/Object;)Ljava/util/List;(SqlQueryNoPage.java:84)[inlined]rn at org/loushang/commons/jdbc/object/SqlQueryNoPage.execute()Ljava/util/List;(SqlQueryNoPage.java:101)[optimized]rn at com/lc/v3/crm/subinterface/CustClusterTypeDaoImpl.getCustGroupTypeAll_new(Ljava/lang/String;)Ljava/util/List;(CustClusterTypeDaoImpl.java:262)rn at com/lc/v3/crm/subinterface/CrmInterfaceDomainImpl.getCustGroupTypeAll_new(Ljava/lang/String;)Ljava/util/List;(CrmInterfaceDomainImpl.java:215)rn at com/lc/v3/subinterface/crm/CRMFactory.getCustGroupTypeAll_new(Ljava/lang/String;)Ljava/util/List;(CRMFactory.java:237)rn at com/lc/v3/rm/license/cust/view/querycust/QueryCustQueryPageInitCmd.query(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/loushang/waf/mvc/QueryHelper;Ljava/util/Map;)Ljava/lang/String;(Unknown Source)[optimized]rn at org/loushang/waf/mvc/QueryCommandSupportWithInit.doExecute()Ljava/lang/String;(QueryCommandSupportWithInit.java:287)[optimized]rn at org/loushang/waf/mvc/CommandSupport.execute()V(CommandSupport.java:141)[optimized]rn at org/loushang/waf/mvc/ServletDispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(ServletDispatcher.java:164)[optimized]rn at javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:856)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run()Ljava/lang/Object;(StubSecurityHelper.java:223)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/http/HttpServletRequest;Lweblogic/servlet/internal/ServletRequestImpl;Ljavax/servlet/ServletResponse;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/Servlet;)Ljava/lang/Throwable;(StubSecurityHelper.java:125)[inlined]rn at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:230)[optimized]rn at weblogic/servlet/internal/TailFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(TailFilter.java:26)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/intercept/web/SecurityEnforcementFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SecurityEnforcementFilter.java:105)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/ui/AbstractIntegrationFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(AbstractIntegrationFilter.java:215)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/util/SetCharacterEncodingFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SetCharacterEncodingFilter.java:75)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[inlined]rn at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(WebAppServletContext.java:3231)[optimized]rn at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)[optimized]rn at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(WebAppServletContext.java:2003)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(WebAppServletContext.java:1862)[optimized]rn at weblogic/servlet/internal/ServletRequestImpl.run()V(ServletRequestImpl.java:1359)[optimized]rn at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)[optimized]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "weblogic.store.WLS_DIAGNOSTICS" id=17 idx=0x3e tid=4572 prio=10 alive, in native, parked, daemonrn -- Waiting for notification on: java/lang/Object@0x0470CD60[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at java/lang/Object.wait()V(Native Method)[optimized]rn at java/util/concurrent/locks/LockSupport.whPark(ZJ)V(Native Method)rn ^-- Lock released while waiting: java/lang/Object@0x0470CD60[fat lock]rn at java/util/concurrent/locks/LockSupport.park()V(Unknown Source)rn at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await()V(AbstractQueuedSynchronizer.java:1767)rn at java/util/concurrent/LinkedBlockingQueue.take()Ljava/lang/Object;(LinkedBlockingQueue.java:359)rn at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take()Ljava/lang/Object;(JDK15ConcurrentBlockingQueue.java:89)rn at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork()[Lweblogic/store/internal/StoreRequest;(PersistentStoreImpl.java:570)rn at weblogic/store/internal/PersistentStoreImpl.run()V(PersistentStoreImpl.java:618)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "GC event thread" id=18 idx=0x40 tid=4524 prio=5 alive, in native, native_waiting, daemonrn at jrockit/mapi/GarbageCollectorImpl.waitForEvent()I(Native Method)rn at jrockit/mapi/GarbageCollectorImpl.access$100()I(Unknown Source)rn at jrockit/mapi/GarbageCollectorImpl$GCEventThread.getNextEvent()Ljava/lang/Object;(Unknown Source)rn at jrockit/mapi/AsyncEventProvider$EventThread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=19 idx=0x42 tid=4520 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=20 idx=0x44 tid=4464 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=21 idx=0x46 tid=4460 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=22 idx=0x48 tid=4444 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" id=23 idx=0x4a tid=4440 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '5' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x4c tid=2828 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '6' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x4e tid=4360 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x50 tid=4372 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '8' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x52 tid=2348 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '9' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x54 tid=736 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '10' for queue: 'weblogic.socket.Muxer'" id=29 idx=0x56 tid=1724 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '11' for queue: 'weblogic.socket.Muxer'" id=30 idx=0x58 tid=1944 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '12' for queue: 'weblogic.socket.Muxer'" id=31 idx=0x5a tid=3056 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '13' for queue: 'weblogic.socket.Muxer'" id=32 idx=0x5c tid=4320 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '14' for queue: 'weblogic.socket.Muxer'" id=33 idx=0x5e tid=4316 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '15' for queue: 'weblogic.socket.Muxer'" id=34 idx=0x60 tid=2884 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "ExecuteThread: '16' for queue: 'weblogic.socket.Muxer'" id=35 idx=0x62 tid=2436 prio=5 alive, in native, daemonrn at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)rn at weblogic/socket/NTSocketMuxer.processSockets()V(NTSocketMuxer.java:81)rn at weblogic/socket/SocketReaderRequest.run()V(SocketReaderRequest.java:29)rn at weblogic/socket/SocketReaderRequest.execute(Lweblogic/kernel/ExecuteThread;)V(SocketReaderRequest.java:42)rn at weblogic/kernel/ExecuteThread.execute(Lweblogic/kernel/ExecuteRequest;)V(ExecuteThread.java:145)rn at weblogic/kernel/ExecuteThread.run()V(ExecuteThread.java:117)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "VDE Transaction Processor Thread" id=38 idx=0x64 tid=4636 prio=2 alive, in native, waiting, daemonrn -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x0470CD88[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)rn at java/lang/Object.wait()V(Native Method)rn at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange()V(TransactionProcessor.java:367)rn ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x0470CD88[fat lock]rn at com/octetstring/vde/backend/standard/TransactionProcessor.run()V(TransactionProcessor.java:212)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "LDAPConnThread-0 ldap://10.98.129.55:1001" id=42 idx=0x66 tid=5044 prio=5 alive, in native, daemonrn at jrockit/net/SocketNativeIO.readBytesPinned(I[BIII)I(Native Method)rn at jrockit/net/SocketNativeIO.socketRead(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)rn at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)rn at java/net/SocketInputStream.read([BII)I(SocketInputStream.java:129)rn at java/io/BufferedInputStream.fill()V(BufferedInputStream.java:218)rn at java/io/BufferedInputStream.read()I(BufferedInputStream.java:235)rn ^-- Holding lock: java/io/BufferedInputStream@0x0471E4A8[thin lock]rn at netscape/ldap/ber/stream/BERElement.getElement(Lnetscape/ldap/ber/stream/BERTagDecoder;Ljava/io/InputStream;[I)Lnetscape/ldap/ber/stream/BERElement;(BERElement.java:101)rn at netscape/ldap/LDAPConnThread.run()V(LDAPConnThread.java:538)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "DoSManager" id=41 idx=0x68 tid=4160 prio=6 alive, in native, sleeping, daemonrn -- Waiting for notification on: java/lang/Object@0x0470CD80[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)[optimized]rn at java/lang/Thread.sleep(J)V(Unknown Source)rn ^-- Lock released while waiting: java/lang/Object@0x0470CD80[fat lock]rn at com/octetstring/vde/DoSManager.run()V(DoSManager.java:433)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "LDAPConnThread-1 ldap://10.98.129.55:1001" id=43 idx=0x6a tid=5108 prio=5 alive, in native, daemonrn at jrockit/net/SocketNativeIO.readBytesPinned(I[BIII)I(Native Method)rn at jrockit/net/SocketNativeIO.socketRead(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)rn at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)rn at java/net/SocketInputStream.read([BII)I(SocketInputStream.java:129)rn at java/io/BufferedInputStream.fill()V(BufferedInputStream.java:218)rn at java/io/BufferedInputStream.read()I(BufferedInputStream.java:235)rn ^-- Holding lock: java/io/BufferedInputStream@0x00EC9DF8[thin lock]rn at netscape/ldap/ber/stream/BERElement.getElement(Lnetscape/ldap/ber/stream/BERTagDecoder;Ljava/io/InputStream;[I)Lnetscape/ldap/ber/stream/BERElement;(BERElement.java:101)rn at netscape/ldap/LDAPConnThread.run()V(LDAPConnThread.java:538)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "weblogic.store._WLS_Server-0" id=44 idx=0x6c tid=5068 prio=10 alive, in native, parked, daemonrn -- Waiting for notification on: java/lang/Object@0x00BF25D8[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)rn at java/lang/Object.wait()V(Native Method)rn at java/util/concurrent/locks/LockSupport.whPark(ZJ)V(Native Method)rn ^-- Lock released while waiting: java/lang/Object@0x00BF25D8[fat lock]rn at java/util/concurrent/locks/LockSupport.park()V(Unknown Source)rn at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await()V(AbstractQueuedSynchronizer.java:1767)rn at java/util/concurrent/LinkedBlockingQueue.take()Ljava/lang/Object;(LinkedBlockingQueue.java:359)rn at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take()Ljava/lang/Object;(JDK15ConcurrentBlockingQueue.java:89)rn at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork()[Lweblogic/store/internal/StoreRequest;(PersistentStoreImpl.java:570)rn at weblogic/store/internal/PersistentStoreImpl.run()V(PersistentStoreImpl.java:618)rn at java/lang/Thread.run()V(Unknown Source)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "FSCacheRefQueueThread" id=45 idx=0x6e tid=5376 prio=1 alive, in native, waiting, daemonrn -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x0150DD78[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)rn at java/lang/ref/ReferenceQueue.remove(J)Ljava/lang/ref/Reference;(ReferenceQueue.java:116)rn ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x0150DD78[fat lock]rn at java/lang/ref/ReferenceQueue.remove()Ljava/lang/ref/Reference;(ReferenceQueue.java:132)rn at workshop/util/filesystem/FSCache$FSCacheRefQueueThread.run()V(FSCache.java:65)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=46 idx=0x70 tid=2428 prio=1 alive, in native, native_waiting, daemonrn at jrockit/vm/Allocator.allocLargeArray(IIZ)Ljava/lang/Object;(Native Method)rn at jrockit/vm/Allocator.getNewTLAAndAlloc(IIIZ)Ljava/lang/Object;(Unknown Source)[inlined]rn at jrockit/vm/Allocator.getMoreMemoryAndAlloc(IIIIZ)Ljava/lang/Object;(Unknown Source)[optimized]rn at java/lang/AbstractStringBuilder.expandCapacity(I)V(AbstractStringBuilder.java:99)[inlined]rn at java/lang/AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder;(AbstractStringBuilder.java:393)[inlined]rn at java/lang/StringBuffer.append(Ljava/lang/Object;)Ljava/lang/StringBuffer;(StringBuffer.java:220)[inlined]rn at com/lc/v3/sd/dist/unitedist/view/DistView.toString()Ljava/lang/String;(Unknown Source)[optimized]rn ^-- Holding lock: java/lang/StringBuffer@0x088F4660[thin lock]rn at java/lang/String.valueOf(Ljava/lang/Object;)Ljava/lang/String;(Unknown Source)[optimized]rn at java/lang/StringBuffer.append(Ljava/lang/Object;)Ljava/lang/StringBuffer;(StringBuffer.java:220)[inlined]rn at org/apache/commons/beanutils/BeanUtils.setProperty(Ljava/lang/Object;Ljava/lang/String;Ljava/lang/Object;)V(BeanUtils.java:849)[optimized]rn ^-- Holding lock: java/lang/StringBuffer@0x088F46C0[thin lock]rn ^-- Holding lock: java/lang/StringBuffer@0x088F46C0[thin lock]rn at org/loushang/web/taglib/util/BeanUtil.populate(Ljava/lang/Object;Ljava/util/Map;)V(BeanUtil.java:47)[optimized]rn at org/loushang/web/taglib/util/MappingTool.populate(Ljava/lang/Object;Ljava/lang/String;Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;)V(MappingTool.java:137)[optimized]rn at org/loushang/web/taglib/util/MappingTool.populate(Ljava/lang/Object;Ljavax/servlet/http/HttpServletRequest;)V(MappingTool.java:41)[inlined]rn at org/loushang/waf/mvc/ViewCommandSupport.doExecute()Ljava/lang/String;(ViewCommandSupport.java:65)[optimized]rn at org/loushang/waf/mvc/CommandSupport.execute()V(CommandSupport.java:141)[optimized]rn at org/loushang/waf/mvc/ServletDispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(ServletDispatcher.java:164)[optimized]rn at javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:856)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run()Ljava/lang/Object;(StubSecurityHelper.java:223)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/http/HttpServletRequest;Lweblogic/servlet/internal/ServletRequestImpl;Ljavax/servlet/ServletResponse;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/Servlet;)Ljava/lang/Throwable;(StubSecurityHelper.java:125)[inlined]rn at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:230)[optimized]rn at weblogic/servlet/internal/TailFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(TailFilter.java:26)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/intercept/web/SecurityEnforcementFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SecurityEnforcementFilter.java:105)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/ui/AbstractIntegrationFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(AbstractIntegrationFilter.java:215)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/util/SetCharacterEncodingFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SetCharacterEncodingFilter.java:75)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[inlined]rn at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(WebAppServletContext.java:3231)[optimized]rn at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)[optimized]rn at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(WebAppServletContext.java:2003)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(WebAppServletContext.java:1862)[optimized]rn at weblogic/servlet/internal/ServletRequestImpl.run()V(ServletRequestImpl.java:1359)[optimized]rn at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)[optimized]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=47 idx=0x72 tid=5752 prio=5 alive, in native, native_waiting, daemonrn at jrockit/vm/Allocator.allocLargeArray(IIZ)Ljava/lang/Object;(Native Method)rn at jrockit/vm/Allocator.getNewTLAAndAlloc(IIIZ)Ljava/lang/Object;(Unknown Source)[inlined]rn at jrockit/vm/Allocator.getMoreMemoryAndAlloc(IIIIZ)Ljava/lang/Object;(Unknown Source)[optimized]rn at java/lang/AbstractStringBuilder.expandCapacity(I)V(AbstractStringBuilder.java:99)[inlined]rn at java/lang/AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder;(AbstractStringBuilder.java:393)[inlined]rn at java/lang/StringBuffer.append(Ljava/lang/String;)Ljava/lang/StringBuffer;(StringBuffer.java:226)[optimized]rn ^-- Holding lock: java/lang/StringBuffer@0x3C1A1250[recursive]rn at weblogic/logging/ConsoleFormatter.formatForConsole(Lweblogic/management/configuration/LogMBean;Lweblogic/logging/LogEntry;Lweblogic/logging/DateFormatter;)Ljava/lang/String;(ConsoleFormatter.java:79)rn ^-- Holding lock: java/lang/StringBuffer@0x3C1A1250[thin lock]rn at weblogic/logging/ConsoleFormatter.toString(Lweblogic/logging/LogEntry;)Ljava/lang/String;(ConsoleFormatter.java:159)rn at weblogic/logging/ConsoleFormatter.format(Ljava/util/logging/LogRecord;)Ljava/lang/String;(ConsoleFormatter.java:150)rn at java/util/logging/StreamHandler.publish(Ljava/util/logging/LogRecord;)V(StreamHandler.java:179)rn ^-- Holding lock: weblogic/logging/ConsoleHandler@0x0132C278[recursive]rn at weblogic/logging/ConsoleHandler.publish(Ljava/util/logging/LogRecord;)V(ConsoleHandler.java:94)rn ^-- Holding lock: weblogic/logging/ConsoleHandler@0x0132C278[thin lock]rn at java/util/logging/Logger.log(Ljava/util/logging/LogRecord;)V(Unknown Source)rn at weblogic/logging/WLLogger.log(Ljava/util/logging/LogRecord;)V(WLLogger.java:40)rn at weblogic/logging/MessageLogger.log(Ljava/lang/String;[Ljava/lang/Object;Ljava/lang/String;)V(MessageLogger.java:110)rn at weblogic/t3/srvr/T3SrvrLogger.logWarnPossibleStuckThread(Ljava/lang/String;JLjava/lang/String;JLjava/lang/String;)Ljava/lang/String;(T3SrvrLogger.java:1841)rn at weblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer.logStuckThreads([Lweblogic/management/runtime/ExecuteThread;ILjava/lang/String;)Z(CoreHealthService.java:278)rn at weblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer.checkStuckThreads()V(CoreHealthService.java:241)rn at weblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer.access$300(Lweblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer;)V(CoreHealthService.java:159)rn at weblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer$1.run()Ljava/lang/Object;(CoreHealthService.java:184)rn at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)[inlined]rn at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)[optimized]rn at weblogic/t3/srvr/CoreHealthService$ThreadMonitoringTimer.timerExpired(Lweblogic/timers/Timer;)V(CoreHealthService.java:180)rn at weblogic/timers/internal/TimerImpl.run()V(TimerImpl.java:265)[optimized]rn at weblogic/work/ServerWorkManagerImpl$WorkAdapterImpl.run()V(ServerWorkManagerImpl.java:518)[optimized]rn at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)[optimized]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "Timer-2" id=48 idx=0x74 tid=6956 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: java/util/TaskQueue@0x0625DA78[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)[optimized]rn at java/util/TimerThread.mainLoop()V(Timer.java:509)rn ^-- Lock released while waiting: java/util/TaskQueue@0x0625DA78[fat lock]rn at java/util/TimerThread.run()V(Timer.java:462)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "Cron" id=50 idx=0x76 tid=6960 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: org/loushang/commons/schedule/Cron@0x06343840[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at java/lang/Object.wait(J)V(Native Method)[optimized]rn at org/loushang/commons/schedule/Cron.run()V(Cron.java:170)rn ^-- Lock released while waiting: org/loushang/commons/schedule/Cron@0x06343840[fat lock]rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=51 idx=0x78 tid=7008 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/work/ExecuteThread@0x06186B60[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at jrockit/vm/Locks.wait(Ljava/lang/Object;)V(Unknown Source)[inlined]rn at weblogic/work/ExecuteThread.waitForRequest()V(ExecuteThread.java:168)[optimized]rn ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06186B60[fat lock]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:186)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=52 idx=0x7a tid=7012 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/work/ExecuteThread@0x06185DB0[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at jrockit/vm/Locks.wait(Ljava/lang/Object;)V(Unknown Source)[inlined]rn at weblogic/work/ExecuteThread.waitForRequest()V(ExecuteThread.java:168)[optimized]rn ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x06185DB0[fat lock]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:186)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STANDBY] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=53 idx=0x7c tid=7016 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/work/ExecuteThread@0x0617FD08[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at jrockit/vm/Locks.wait(Ljava/lang/Object;)V(Unknown Source)[inlined]rn at weblogic/work/ExecuteThread.waitForRequest()V(ExecuteThread.java:168)[optimized]rn ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0617FD08[fat lock]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:186)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" id=54 idx=0x7e tid=7020 prio=1 alive, in native, native_waiting, daemonrn at jrockit/vm/Allocator.allocLargeArray(IIZ)Ljava/lang/Object;(Native Method)rn at jrockit/vm/Allocator.getNewTLAAndAlloc(IIIZ)Ljava/lang/Object;(Unknown Source)[inlined]rn at jrockit/vm/Allocator.getMoreMemoryAndAlloc(IIIIZ)Ljava/lang/Object;(Unknown Source)[optimized]rn at java/lang/AbstractStringBuilder.expandCapacity(I)V(AbstractStringBuilder.java:99)[inlined]rn at java/lang/AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder;(AbstractStringBuilder.java:393)[inlined]rn at java/lang/StringBuffer.append(Ljava/lang/Object;)Ljava/lang/StringBuffer;(StringBuffer.java:220)[inlined]rn at com/lc/v3/sd/dist/unitedist/view/DistView.toString()Ljava/lang/String;(Unknown Source)[optimized]rn ^-- Holding lock: java/lang/StringBuffer@0x088E0950[thin lock]rn at java/lang/String.valueOf(Ljava/lang/Object;)Ljava/lang/String;(Unknown Source)[optimized]rn at java/lang/StringBuffer.append(Ljava/lang/Object;)Ljava/lang/StringBuffer;(StringBuffer.java:220)[inlined]rn at org/apache/commons/beanutils/BeanUtils.setProperty(Ljava/lang/Object;Ljava/lang/String;Ljava/lang/Object;)V(BeanUtils.java:849)[optimized]rn ^-- Holding lock: java/lang/StringBuffer@0x088E09B0[thin lock]rn ^-- Holding lock: java/lang/StringBuffer@0x088E09B0[thin lock]rn at org/loushang/web/taglib/util/BeanUtil.populate(Ljava/lang/Object;Ljava/util/Map;)V(BeanUtil.java:47)[optimized]rn at org/loushang/web/taglib/util/MappingTool.populate(Ljava/lang/Object;Ljava/lang/String;Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;)V(MappingTool.java:137)[optimized]rn at org/loushang/web/taglib/util/MappingTool.populate(Ljava/lang/Object;Ljavax/servlet/http/HttpServletRequest;)V(MappingTool.java:41)[inlined]rn at org/loushang/waf/mvc/ViewCommandSupport.doExecute()Ljava/lang/String;(ViewCommandSupport.java:65)[optimized]rn at org/loushang/waf/mvc/CommandSupport.execute()V(CommandSupport.java:141)[optimized]rn at org/loushang/waf/mvc/ServletDispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V(ServletDispatcher.java:164)[optimized]rn at javax/servlet/http/HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(HttpServlet.java:856)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run()Ljava/lang/Object;(StubSecurityHelper.java:223)[optimized]rn at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(Ljavax/servlet/ServletRequest;Ljavax/servlet/http/HttpServletRequest;Lweblogic/servlet/internal/ServletRequestImpl;Ljavax/servlet/ServletResponse;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/Servlet;)Ljava/lang/Throwable;(StubSecurityHelper.java:125)[inlined]rn at weblogic/servlet/internal/ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V(ServletStubImpl.java:230)[optimized]rn at weblogic/servlet/internal/TailFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(TailFilter.java:26)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/intercept/web/SecurityEnforcementFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SecurityEnforcementFilter.java:105)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/bsp/permit/pep/ui/AbstractIntegrationFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(AbstractIntegrationFilter.java:215)[optimized]rn at org/loushang/bsp/permit/pep/util/FilterToBeanProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(FilterToBeanProxy.java:47)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[optimized]rn at org/loushang/util/SetCharacterEncodingFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V(SetCharacterEncodingFilter.java:75)[optimized]rn at weblogic/servlet/internal/FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(FilterChainImpl.java:42)[inlined]rn at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(WebAppServletContext.java:3231)[optimized]rn at weblogic/security/acl/internal/AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(AuthenticatedSubject.java:321)[optimized]rn at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(SecurityManager.java:121)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(WebAppServletContext.java:2003)[inlined]rn at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(WebAppServletContext.java:1862)[optimized]rn at weblogic/servlet/internal/ServletRequestImpl.run()V(ServletRequestImpl.java:1359)[optimized]rn at weblogic/work/ExecuteThread.execute(Ljava/lang/Runnable;)V(ExecuteThread.java:209)[optimized]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:181)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=55 idx=0x80 tid=7024 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/work/ExecuteThread@0x0616BE98[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at jrockit/vm/Locks.wait(Ljava/lang/Object;)V(Unknown Source)[inlined]rn at weblogic/work/ExecuteThread.waitForRequest()V(ExecuteThread.java:168)[optimized]rn ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0616BE98[fat lock]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:186)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STANDBY] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=56 idx=0x82 tid=7028 prio=5 alive, in native, waiting, daemonrn -- Waiting for notification on: weblogic/work/ExecuteThread@0x0616B460[fat lock]rn at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)rn at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined]rn at jrockit/vm/Locks.wait(Ljava/lang/Object;)V(Unknown Source)[inlined]rn at weblogic/work/ExecuteThread.waitForRequest()V(ExecuteThread.java:168)[optimized]rn ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0616B460[fat lock]rn at weblogic/work/ExecuteThread.run()V(ExecuteThread.java:186)rn at jrockit/vm/RNI.c2java(IIII)V(Native Method)rn -- end of tracern "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" id=57 idx=0x84 tid=7032 prio=1 alive, in native, native_waiting, daemonrn at jrockit/vm/Allocator.allocLargeArray(IIZ)Ljava/lang/Object;(Native Method)rn at jrockit/vm/Allocator.getNewTLAAndAlloc(IIIZ)Ljava/lang/Object;(Unknown Source)[inlined]rn at jrockit/vm/Allocator.getMoreMemoryAndAlloc(IIIIZ)Ljava/lang/Object;(Unknown Source)[optimized]rn at java/lang/AbstractStringBuilder.expandCapacity(I)V(AbstractStringBuilder.java:99)[inlined]rn at java/lang/AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder;(AbstractStringBuilder.java:393)[inlined]rn at java/lang/StringBuffer.append(Ljava/lang/Object;)Ljava/lang/StringBuffer;(StringBuffer.java:220)[inlined]rn at com/lc/v3/sd/dist/unitedist/view/DistView.toString()Ljava/lang/String;(Unknown Source)[optimized]rn ^-- Holding lock 问答

如何分析weblogic DUMP日志

01-03

rn 在网上搜了有关DUMP日志如何分析,但是在实际项目中看到DUMP日志就蒙了rn  rn 哪位帮忙解答下:rn 1       A6vkX6Q5pALcgXm2PTpA3Arn "ExecuteThread: '60' for queue: 'weblogic.kernel.Default'" id=69 idx=0x138 tid=411728 prio=5 alive, in native, native_blocked, daemonrn     at jrockit/vm/Allocator.nativeGetNewTLA()V(Native Method)rn     at jrockit/vm/Allocator.getNewTLA(Allocator.java:805)rn     at jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:833)rn     at jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:947)rn     at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:980)rn     at jrockit/vm/Allocator.allocArray(Allocator.java:762)rn     at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:665)rn     at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)rn     at oracle/jdbc/driver/T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)rn     at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)rn     at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285)rn     at oracle/jdbc/driver/OraclePreparedStatement.execute(OraclePreparedStatement.java:3390)rn     ^-- Holding lock: oracle/jdbc/driver/T4CPreparedStatement@0x4d91df08[thin lock]rn     ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x4cf9e310[thin lock]rn     at weblogic/jdbc/wrapper/PreparedStatement.execute(PreparedStatement.java:102)rn     at com/ibatis/sqlmap/engine/execution/SqlExecutor.executeUpdate(SqlExecutor.java:84)rn     at com/ibatis/sqlmap/engine/mapping/statement/GeneralStatement.sqlExecuteUpdate(GeneralStatement.java:200)rn     at com/ibatis/sqlmap/engine/mapping/statement/GeneralStatement.executeUpdate(GeneralStatement.java:78)rn     at com/ibatis/sqlmap/engine/impl/SqlMapExecutorDelegate.update(SqlMapExecutorDelegate.java:500)rn     at com/ibatis/sqlmap/engine/impl/SqlMapSessionImpl.update(SqlMapSessionImpl.java:85)rn     at com/***app/lwc/service/persistence/dao/ibatis/SqlMapClientTemplate$10.doInSqlMapClient(SqlMapClientTemplate.java:521)rn     at com/***app/lwc/service/persistence/dao/ibatis/SqlMapClientTemplate.execute(SqlMapClientTemplate.java:193)rn     at com/***app/lwc/service/persistence/dao/ibatis/SqlMapClientTemplate.update(SqlMapClientTemplate.java:516)rn     at com/***core/integration/dao/impl/***IbatisDAO.update(***IbatisDAO.java:206)rn     at com/***core/biz/bo/CustomerBO.update(CustomerBO.java:394)rn     at com/***core/biz/service/impl/RegisterPojoService.updateCustomerInfo(RegisterPojoService.java:248)rn     at com/***core/biz/action/login/LoginLogDealAction.handleRequest(LoginLogDealAction.java:113)rn     at com/***core/biz/action/***Action.perform(***Action.java:105)rn     at com/***app/biz/ac/impl/ApplicationControllerBean.handleRequest(ApplicationControllerBean.java:101)rn     at com/***app/biz/ac/impl/pafaAC_q25x5e_EOImpl.handleRequest(pafaAC_q25x5e_EOImpl.java:46)rn     at com/***app/biz/ac/impl/pafaAC_q25x5e_EOImpl_WLSkel.invoke(ILweblogic/rmi/spi/InboundRequest;Lweblogic/rmi/spi/OutboundResponse;Ljava/lang/Object;)Lweblogic/rmi/spi/OutboundResponse;(Unknown Source)rn     at weblogic/rmi/internal/BasicServerRef.invoke(BasicServerRef.java:492)rn     at weblogic/rmi/cluster/ReplicaAwareServerRef.invoke(ReplicaAwareServerRef.java:108)rn     at weblogic/rmi/internal/BasicServerRef$1.run(BasicServerRef.java:435)rn     at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)rn     at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:147)rn     at weblogic/rmi/internal/BasicServerRef.handleRequest(BasicServerRef.java:430)rn     at weblogic/rmi/internal/BasicExecuteRequest.execute(BasicExecuteRequest.java:35)rn     at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:224)rn     at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:183)rn     at jrockit/vm/RNI.c2java(IIIII)V(Native Method)rn     -- end of tracern 1       W4p/Iplh9oGwXyRzId2Y4Qrn "ExecuteThread: '63' for queue: 'weblogic.kernel.Default'" id=72 idx=0x144 tid=411731 prio=5 alive, in native, blocked, daemonrn     -- Blocked trying to get lock: weblogic/utils/classloaders/GenericClassLoader@0x166c74a0[thin lock]rn     at jrockit/vm/Threads.sleep(I)V(Native Method)rn     at jrockit/vm/Locks.waitForThinRelease(Locks.java:1233)rn     at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1307)rn     at jrockit/vm/Locks.monitorEnter(Locks.java:2389)rn     at jrockit/vm/Classes.loadClassInternal(Classes.java:41)rn     at jrockit/vm/RNI.c2java(IIIII)V(Native Method)rn     at jrockit/vm/ExceptionHandler.throwMissingCDPReference(III)V(Native Method)rn     at com/***biz/ac/ACRequestTool.requestAfter(ACRequestTool.java:50)rn     at com/***biz/ac/impl/ApplicationControllerBean.handleRequest(ApplicationControllerBean.java:141)rn     at com/***biz/ac/impl/****AC_q25x5e_EOImpl.handleRequest(pafaAC_q25x5e_EOImpl.java:46)rn     at sun/reflect/GeneratedMethodAccessor276.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)rn     at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)rn     at java/lang/reflect/Method.invoke(Method.java:324)rn     at com/***lwc/service/remoting/access/rmi/RmiClientInterceptorUtils.doInvoke(RmiClientInterceptorUtils.java:103)rn     at com/***lwc/service/remoting/access/ejb/SimpleRemoteSlsbInvokerInterceptor$MethodInvokedAction.run(SimpleRemoteSlsbInvokerInterceptor.java:149)rn     at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)rn     at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)rn     at weblogic/security/Security.runAs(Security.java:41)rn     at com/***lwc/service/remoting/access/ejb/SimpleRemoteSlsbInvokerInterceptor.invokeFromWeblogic(SimpleRemoteSlsbInvokerInterceptor.java:133)rn     at com/***lwc/service/remoting/access/ejb/SimpleRemoteSlsbInvokerInterceptor.doInvoke(SimpleRemoteSlsbInvokerInterceptor.java:97)rn     at com/***lwc/service/remoting/access/ejb/SmartRemoteSlsbInvokerInterceptor.doInvoke(SmartRemoteSlsbInvokerInterceptor.java:134)rn     at com/***lwc/service/remoting/access/ejb/AbstractRemoteSlsbInvokerInterceptor.invoke(AbstractRemoteSlsbInvokerInterceptor.java:116)rn     at com/***lwc/core/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:138)rn     at com/***lwc/core/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:148)rn     at $Proxy10.handleRequest(Lcom/***dto/ServiceRequest;)Lcom/***dto/ServiceResponse;(Unknown Source)rn     at com/***core/biz/service/ejb/MessageControllerBean.onMessage(***MessageControllerBean.java:133)rn     at weblogic/ejb20/internal/MDListener.execute(MDListener.java:400)rn     at weblogic/ejb20/internal/MDListener.transactionalOnMessage(MDListener.java:333)rn     at weblogic/ejb20/internal/MDListener.onMessage(MDListener.java:298)rn     at weblogic/jms/client/JMSSession.onMessage(JMSSession.java:2686)rn     at weblogic/jms/client/JMSSession.execute(JMSSession.java:2598)rn     at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:224)rn     at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:183)rn     at jrockit/vm/RNI.c2java(IIIII)V(Native Method)rn     -- end of tracern  rn 问答

没有更多推荐了,返回首页