服务器磁盘爆满引发的dubbo请求超时问题

最新推荐文章于 2023-04-17 10:22:06 发布

bruce128

最新推荐文章于 2023-04-17 10:22:06 发布

阅读量1.5w

点赞数 2

分类专栏： Java

本文链接：https://blog.csdn.net/bruce128/article/details/50669053

版权

Java 专栏收录该内容

35 篇文章

订阅专栏

今天遇到了运用dubbo开发过程中经常遇到的问题，请求超时

Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method subscribe in the service com.alibaba.dubbo.registry.RegistryService. Tried 3 times of the providers [192.168.146.125:6060] (1/1) from the registry 192.168.146.125:6060 on the consumer 10.14.137.59 using the dubbo version 2.4.5. Last error is: Invoke remote method timeout. method: subscribe, provider: dubbo://192.168.146.125:6060/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&check=false&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, cause: Waiting server-side response timeout. start time: 2016-02-15 15:36:42.643, end time: 2016-02-15 15:36:52.644, client elapsed: 0 ms, server elapsed: 10001 ms, timeout: 10000 ms, request: Request [id=2, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=subscribe, parameterTypes=[class com.alibaba.dubbo.common.URL, interface com.alibaba.dubbo.registry.NotifyListener], arguments=[consumer://10.14.137.59/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, com.alibaba.dubbo.registry.integration.RegistryDirectory@c4d412], attachments={sys_callback_arg-1=12899346, path=com.alibaba.dubbo.registry.RegistryService, interface=com.alibaba.dubbo.registry.RegistryService, timeout=10000, version=0.0.0}]], channel: /10.14.137.59:56595 -> /192.168.146.125:6060
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
	at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
	at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
	at com.alibaba.dubbo.common.bytecode.proxy0.subscribe(proxy0.java)
	at com.alibaba.dubbo.registry.dubbo.DubboRegistry.doSubscribe(DubboRegistry.java:138)
	at com.alibaba.dubbo.registry.support.FailbackRegistry.subscribe(FailbackRegistry.java:189)
	... 45 more
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2016-02-15 15:36:42.643, end time: 2016-02-15 15:36:52.644, client elapsed: 0 ms, server elapsed: 10001 ms, timeout: 10000 ms, request: Request [id=2, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=subscribe, parameterTypes=[class com.alibaba.dubbo.common.URL, interface com.alibaba.dubbo.registry.NotifyListener], arguments=[consumer://10.14.137.59/com.alibaba.dubbo.registry.RegistryService?application=im-tracker&callbacks=10000&connect.timeout=10000&dubbo=2.4.5&interface=com.alibaba.dubbo.registry.RegistryService&lazy=true&methods=register,subscribe,unregister,unsubscribe,lookup&organization=chat.jd.com&owner=lvsheng&pid=6820&reconnect=false&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1455521782295&unsubscribe.1.callback=false, com.alibaba.dubbo.registry.integration.RegistryDirectory@c4d412], attachments={sys_callback_arg-1=12899346, path=com.alibaba.dubbo.registry.RegistryService, interface=com.alibaba.dubbo.registry.RegistryService, timeout=10000, version=0.0.0}]], channel: /10.14.137.59:56595 -> /192.168.146.125:6060
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:107)
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84)
	at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)
	at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
	at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
	at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
	... 51 more

正常思路是，服务方压力过大，导致请求阻塞，长时间没有返回。但是照这条线下去排查根本不行。

这个时候用df命令查机器磁盘

[root@localhost /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        20G   20G     0 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
/dev/sda1       9.9G  172M  9.2G   2% /boot
/dev/sda6        71G  503M   67G   1% /data
/dev/sda2        40G  282M   38G   1% /usr/local

根目录所在磁盘满了，把磁盘清理了下，问题得以修复。

一个合理的解释是，当我发出请求后，服务提供方进入服务流程，但是需要向OS申请磁盘资源。可惜OS没有，于是服务提供方进入阻塞状态。