首先,问题出现在春结后,测试环境的大部分模块都不能使用,报如下错误。
3353394284 [163714][k4oi4rk2kxn1nfyu1] ERROR com.alibaba.dubbo.rpc.filter.ExceptionFilter - [DUBBO] Got unchecked and undeclared exception which called by 10.1.20.3. service: com.xxxx.yyyy.service.user.TblUserService, method: getVerificationCodeByMobile, exception: java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil, dubbo version: 2.5.3, current host: 10.1.20.3
java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil
at com.xxxx.yyyy.service.user.TblUserServiceImpl.getVerificationCodeByMobile(TblUserServiceImpl.java:420)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
省略。。。
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
查找原因:把日志取下来,查找发现多次报错:
第一次报错:
3353348894 [163714][uryn8cuudltaamddh] ERROR com.alibaba.dubbo.rpc.filter.ExceptionFilter - [DUBBO] Got unchecked and undeclared exception which called by 10.1.20.3. service: com.xxxx.yyyy.service.user.TblUserService, method: getVerificationCodeByMobile, exception: java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil, dubbo version: 2.5.3, current host: 10.1.20.3
java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil
at com.xxxx.yyyy.service.user.TblUserServiceImpl.getVerificationCodeByMobile(TblUserServiceImpl.java:420)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
省略。。。
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.xxxx.yyyy.util.RandomUtil
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:450)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:403)
... 51 more
之后报错:
3353394284 [163714][k4oi4rk2kxn1nfyu1] ERROR com.alibaba.dubbo.rpc.filter.ExceptionFilter - [DUBBO] Got unchecked and undeclared exception which called by 10.1.20.3. service: com.xxxx.yyyy.service.user.TblUserService, method: getVerificationCodeByMobile, exception: java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil, dubbo version: 2.5.3, current host: 10.1.20.3
java.lang.NoClassDefFoundError: com/xxxx/yyyy/util/RandomUtil
at com.xxxx.yyyy.service.user.TblUserServiceImpl.getVerificationCodeByMobile(TblUserServiceImpl.java:420)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
省略。。。。
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
发现第一次有WebAppClassLoader报出ClassNotFoundException的异常,之后没有,
即使JVM做过FullGC,也应该再次找到类,并加载,为什么没有找到呢?
所以从类加载查起。使用Arthas调试工具,进入调试交互界面。
直接检查未加载到的类名:
$ sc -d com/xxxx/yyyy/util/RandomUtil
class-info com.gdpi.sgtc.util.RandomUtil
code-source /tmp/jetty-0.0.0.0-8084-messager.war-_messager-any-992374700122651111.dir/webapp/WEB-INF/classes/
name com.xxxx.yyyy.util.RandomUtil
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RandomUtil
modifier public
annotation
interfaces
super-class +-java.lang.Object
class-loader +-WebAppClassLoader=sgtcMessager@4fe3c938
+-startJarLoader@4eec7777
+-sun.misc.Launcher$AppClassLoader@42a57993
+-sun.misc.Launcher$ExtClassLoader@54bedef2
classLoaderHash 4fe3c938
Affect(row-cnt:1) cost in 11 ms.
检索到 code-source的来源在/tmp,jetty加载war包时,将临时文件解压在/tmp文件夹下,
如果找不到加载类,就到临时文件中获取.class文件,重新加载,因为/tmp文件夹内会做
定时清理,所以临时文件已被清理掉,所以无法加载。查找tmp文件夹清理策略:
vi /usr/lib/tmpfiles.d/tmp.conf
# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d
# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp
策略为清理10天前的数据,春节期间10天没有被使用过,所以被清理,
当WebAppClassLoader再去/tmp文件夹加载时,已经无法找到.class文件,所以报错。
解决方法:
查找到jetty位置时有一个机制:当jetty的根目录下存在work文件夹时,会自动将临时文件创建在work文件夹下,
所以简单有效的方法就是在jetty的根目录下,创建work文件夹,再次部署war包,就会将临时文件加载到此目录下。
后记:
该方法是一个Utils方法,如:
public class XXUtils{
public static void print(){
System.out.println("hello world!");
}
}
这样的方法长期不使用时,会被GC回收。这也是引起此问题的原因。
参考Spring里面定义的Utils类,都在类名上加了abstract,也是为了使工具类不被回收。