JVM java.lang.OutOfMemoryError: PermGen space处理方法

   近期weblogic 11g永久代内存溢出，分析JVM dump文件是没有用处的，因为那只是堆内存，永久代不在里面。目前永久代设置是1G,遥想当年，只有400M，这么多年来一直在涨，现在一次full gc需要10多秒。可以增大到1.5G，但从GC日志中可以看到永久代在不断的增长，如果二周不重启weblogic，内存又不够用了，不是长久之计。
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.6.0_30]
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427) ~[na:1.6.0_30]
at java.lang.Class.getDeclaredMethod(Class.java:1935) ~[na:1.6.0_30]
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382) ~[na:1.6.0_30]
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52) ~[na:1.6.0_30]
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438) ~[na:1.6.0_30]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_30]
可以看到是方法区有问题，创建了大量的类，从GC日志上看，启动之后永久代有700M，随着时间的推移，会缓慢增长，这就是内存泄露，堆内存泄露也是这样。如何定位哪些类一直在涨呢，此时就用到两个重要的JVM参数，一个显示每个加载的类，一个显示每个卸载的类：
-XX:+TraceClassLoading  -XX:+TraceClassUnloading

weblogic里面会出现这种日志，需要自己写程序解析最后还有哪些类存活着。
[Loaded MM.statistics.mmbillstatistics.exception.MmBillStatisticalTypeException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/mmbillstatistics/exception/MmBillStatisticalTypeException.class]
[Loaded MM.statistics.mmbillstatistics.appservice.impl.MmBillStatisticalTypeSes_1k61gv_IMmBillStatisticalTypeServiceImpl_1035_WLStub from file:/data/wls1035/modules/com.bea.core.utils.wrapper_1.4.0.0.jar]
[Loaded MM.statistics.amfmHuiXuFinance.appservice.AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf from file:/data/Domain/mm_Domain/servers/mmServer1/cache/EJBCompilerCache/1nhs7towub2hs/mm/statistics/amfmHuiXuFinance/appservice/AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf.class]
[Loaded MM.statistics.amfmHuiXuFinance.exception.AmfmHuiXuFinanceException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/amfmHuiXuFinance/exception/AmfmHuiXuFinanceException.class]
[Unloading class mm.purchase.purchaseorder.model.PurchaseItemVO]
[Unloading class mm.inventory.reservereplenishplan.appservice.impl.ReserveReplenishPlanBizService]
[Unloading class mm.inventory.reservequota.dao.ReserveQuotaBatchDAO]

解析程序开始：
drop table load_class purge;
truncate table load_class;
create table load_class
(
  nu number,
  action varchar2(20),
  class_name varchar2(1000),
  class_file varchar2(1000)
);

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
/**
 * [Loaded java.lang.ClassNotFoundException from /usr/local/jdk1.6/jre/lib/rt.jar]
 * [Unloading class com.MM.externalinterface.fmis.impl.FmisBizService]
 *
 */
public class PermAla {
    static final String driver_class  = "oracle.jdbc.driver.OracleDriver";
    static final String connectionURL = "jdbc:oracle:thin:@10.10.15.164:1521:orcl";
    static final String userID        = "SPROC";
    static final String userPassword  = "SPROC";
    public static void readTxtFile(String filePath){
        Connection  con = null;
        String  s_sql = "insert into load_class values(?,?,?,?)";
        PreparedStatement pstmt = null;
        int i=0;
        try {
            Class.forName (driver_class).newInstance();
            con = DriverManager.getConnection(connectionURL, userID, userPassword);
            pstmt = con.prepareStatement(s_sql);
            con.setAutoCommit(false);
            String encoding="GBK";
            File file=new File(filePath);
            InputStreamReader read = new InputStreamReader(
                    new FileInputStream(file),encoding);//考虑到编码格式
            BufferedReader bufferedReader = new BufferedReader(read);
            String lineTxt = null;
            String[] lineTxtArray= null;
            while((lineTxt = bufferedReader.readLine()) != null){
                if(lineTxt.indexOf("[Loaded")==0  ) {
                    lineTxtArray = lineTxt.split(" ");
                    pstmt.setInt(1, i);
                    pstmt.setString(2, lineTxtArray[0]);
                    pstmt.setString(3, lineTxtArray[1]);
                    if(lineTxtArray.length ==4) {
                        pstmt.setString(4, lineTxtArray[3]);
                    }else{
                        System.out.println(lineTxt);
                    }
                    pstmt.addBatch();
                    i++;
                } else if(lineTxt.indexOf("[Unloading")==0){
                    lineTxtArray = lineTxt.split(" ");
                    pstmt.setInt(1, i);
                    pstmt.setString(2, lineTxtArray[0]);
                    pstmt.setString(3, lineTxtArray[2]);
                    pstmt.addBatch();
                    i++;
                }
                if(i % 10000 == 0){
                    pstmt.executeBatch();
                    con.commit();
                }
            }
            con.commit();
            read.close();
        } catch (Exception e) {
            System.out.println("读取文件内容出错,行数："+i);
            e.printStackTrace();
        }finally{
            if(pstmt != null){
                try {
                    pstmt.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }finally{
                    pstmt = null;
                }
            }
            if(con != null){
                try {
                    con.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }finally{
                    con = null;
                }
            }
        }


    }


    public static void main(String argv[]){
        String filePath = "E:\\永久代内存溢出\\20161213\\Server1.log";
        readTxtFile(filePath);
    }
}

--执行java代码导入之后，处理一些特殊的格式
update load_class
   set action     = replace(action, '[', ''),
       class_file = replace(class_file, ']', ''),
       class_name = replace(class_name, ']', '');
commit;
update load_class
   set class_name = replace(class_name, 'file:', ''),
       class_file = replace(class_file, 'file:', '');
commit;


SQL> select * from load_class where rownum <10;
 NU ACTION  CLASS_NAME                               CLASS_FILE
--- ------- --------------------------------------- --------------------------------
 96 Loaded  sun.misc.SharedSecrets                  /data/jdk1.6.0_45/jre/lib/rt.jar
 97 Loaded  sun.misc.Unsafe                         /data/jdk1.6.0_45/jre/lib/rt.jar
 98 Loaded  java.lang.IncompatibleClassChangeError  /data/jdk1.6.0_45/jre/lib/rt.jar
 99 Loaded  java.lang.NoSuchMethodError             /data/jdk1.6.0_45/jre/lib/rt.jar
100 Loaded  sun.reflect.Reflection                  /data/jdk1.6.0_45/jre/lib/rt.jar
101 Loaded  java.util.Collections                   /data/jdk1.6.0_45/jre/lib/rt.jar
102 Loaded  java.lang.Iterable                      /data/jdk1.6.0_45/jre/lib/rt.jar
103 Loaded  java.util.Collection                    /data/jdk1.6.0_45/jre/lib/rt.jar
104 Loaded  java.util.Set                           /data/jdk1.6.0_45/jre/lib/rt.jar
       
--显示加载且没有卸载的类 
drop table  purge;
create table  as
with res as
(select count(*) over(partition by action,class_name order by nu asc) rn,
               t.*
          from load_class t)
,res1 as (select rn,class_name from res a where a.action='Loaded' 
minus 
select rn,class_name from res b where b.action='Unloading')
select b.* from res1 a, res b where a.rn= b.rn and a.class_name= b.class_name;


查看数据，对数据摸底                            
select count(1) from  where class_file='__JVM_DefineClass__';
select count(1) from  where class_file like
       '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/lib%';
select count(1) from  where class_file like
       '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/classes%'
      or class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/ejb%';
select count(1) from  where class_file like
       '/data/wls1035/%';   
select count(1) from  where class_file like
       '/data/Domain/MMDomain/servers/MMServer1/cache/EJBCompilerCache/%'; 
select count(1) from  where class_file like
       '/data/jdk1.6.0_45/jre%';         
select count(1)  from 
 where class_file in ('weblogic.utils.classloaders.GenericClassLoader',
        'sun.misc.Launcher$AppClassLoader',
        'weblogic.utils.classloaders.ChangeAwareClassLoader','instance');
select count(1) from  where class_file like
       '/data/Domain/MMDomain/jsp_temp%';             

可以看到反射的类很多，重点研究__JVM_DefineClass__和sun.reflect.GeneratedMethodAccessor，发现是反射产生的一个问题。参考http://stackoverflow.com/questions/16130292/java-lang-outofmemoryerror-permgen-space-java-reflection

      When using Java reflection, the JVM has two methods of accessing the information on the class being reflected. It can use a JNI accessor, or a Java bytecode accessor. 

      If it uses a Java bytecode accessor, then it needs to have its own Java class and classloader (sun/reflect/GeneratedMethodAccessor class and sun/reflect/DelegatingClassLoader). Theses classes and classloaders use native memory. 

      The accessor bytecode can also get JIT compiled, which will increase the native memory use even more. 

      If Java reflection is used frequently, this can add up to a significant amount of native memory use. The JVM will use the JNI accessor first, then after some number of accesses on the same class, will change to use the Java bytecode accessor. This is called inflation, when the JVM changes from the JNI accessor to the bytecode accessor. Fortunately, we can control this with a Java property. 

      The sun.reflect.inflationThreshold property tells the JVM what number of times to use the JNI accessor. If it is set to 0, then the JNI accessors are always used. Since the bytecode accessors use more native memory than the JNI ones, if we are seeing a lot of Java reflection, we will want to use the JNI accessors. To do this, we just need to set the inflationThreshold property to zero.

如果节点使用sun公司jdk，配置-Dsun.reflect.inflationThreshold=2147483647 

如果是IBM的jdk，配置-Dsun.reflect.inflationThreshold=0

以下是配置的前后类数量对比，反射类有明显的下降，从gc日志上看也有回收的迹象：

加载类的出处	Server1		server2
	加参数前	加参数后	加参数前	加参数后
__JVM_DefineClass__(反射产生的类)	42372	3632	19961	2822
EAR/APP-INF/lib	25552	23162	23613	23579
EAR/APP-INF/classes	21531	15684	15167	15176
wls1035	19017	19420	19661	19168
MMServer1/cache/EJBCompilerCache	5996	5996	5996	5996
/data/jdk1.6.0_45/jre	3494	3493	3175	3458
weblogic.utils.classloaders.GenericClassLoader sun.misc.Launcher$AppClassLoader weblogic.utils.classloaders.ChangeAwareClassLoader	2550	2574	2508	2516
jsp_temp	843	1395	843	849

当然我遇到的情况是反射类引起的问题，如果你的永久代设置太小，不用纠结，直接把永久代内存加大；如果是类太多导致（不是反射类），那就想办法瘦身，想办法 改造去掉一些依赖的jar包，可能是一个大工程。