{timestamp: 1528113688276, status: 500, error: "Internal Server Error",…}
error
:
"Internal Server Error"
exception
:
"com.alibaba.dubbo.rpc.***”
message
:
"Failed to invoke the method getById in the service com.shuwen.production.media.manage.client.media.MediaServiceFacade. Tried 3 times of the providers [172.20.3.5:20880, 172.20.3.15:20880, 172.20.3.11:20880] (3/3) from the registry 0.0.0.0:9090 on the consumer 172.20.30.48 using the dubbo version 2.8.6. Last error is: Invoke remote method timeout. method: getById, provider: dubbo://172.20.3.15:20880/
cause: Waiting server-side response timeout by scan timer. start time: 2018-06-04 20:01:27.251, end time: 2018-06-04 20:01:28.273, client elapsed: 0 ms, server elapsed: 1022 ms, timeout: 1000 ms, request: Request [id=178349, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=getById, parameterTypes=[class com.shuwen.production.media.manage.dto.media.MediaDTO], arguments=[MediaDTO{mediaId='c82e54c8-f6cc-407a-8b35-bc09f6b947f6', userId=null, enterpriseId=null, mediaType='image', openType='null', title='null', url='null', categoryId='null', folderId='null', parentMediaId='null', thumbImage='null', fileSize=null, occurTimestamp=null, publishStatus='null', source='null', content='null', outId='null', outKeyword='null', dupMediaId='null', quality=null, subMediaType='null', wordCount=null, shot='null', scene='null', sizeScope='null', duration=null, bitrate=null, durationScope='null', definitionScope='null', subMediaCount=0, entityList=null, tagList=null, extra='null'} BaseDTO{gmtCreate=null, gmtModified=null, criteria=null}], attachments={path=com.shuwen.production.media.manage.client.media.MediaServiceFacade, interface=com.shuwen.production.media.manage.client.media.MediaServiceFacade, version=1.0}]], channel: /172.20.30.48:33914 -> /172.20.3.15:20880"
path
:
"/api/***/***”
status
:
500
timestamp
:
1528113688276
解析与解决方法:
Dubbo是阿里开源的分布式远程调用方案(RPC),由于网络或服务端不可靠,会导致调用出现一种不确定的中间状态(超时)。为了避免超时导致客户端资源(线程)挂起耗尽,必须设置超时时间。Provider可以配置的Consumer端主要属性有timeout、retries、loadbalance、actives和cluster。Provider上应尽量多配置些Consumer端的属性,让Provider实现者一开始就思考Provider的服务特点与服务质量。配置之间存在着覆盖,具体规则如下:
1. 方法级配置别优于接口级别,即小Scope优先
2. Consumer端配置优于Provider配置,优于全局配置
3. Dubbo Hard Code的配置值(默认)根据规则2,纵使消费端配置优于服务端配置,但消费端配置超时时间不能随心所欲,需要根据业务实际情况来设定。如果超时时间设置得太短,复杂业务本来就需要很长时间完成,服务端无法在设定的超时时间内完成业务处理;如果超时时间设置太长,会由于服务端或者网络问题导致客户端资源大量线程挂起。
超时配置:
Dubbo消费端
1---全局超时配置:<dubbo:consumer timeout="5000" /> 调高一点
2--指定接口以及特定方法超时配置
<dubbo:reference interface="com.asia.service" timeout="2000"> <dubbo:method name="getOne" timeout="3000" /></dubbo:reference>
Dubbo服务端
1--全局超时配置 :<dubbo:provider timeout="5000" />
<dubbo:provider interface="com.asis.service" timeout="2000">
<dubbo:method name="getOne" timeout="3000" />
</dubbo:provider>
3
Dubbo协议超时实现
Dubbo协议超时实现使用了Future模式,主要涉及类DubboInvoker,ResponseFuture, DefaultFuture。
ResponseFuture.get()在请求还未处理完或未到超时前一直是wait状态;响应达到后,设置请求状态,并进行notify唤醒。
public Object get() throws RemotingException {
return get(timeout);
}
public Object get(int timeout) throws RemotingException {
if (timeout <= 0) {
timeout = Constants.DEFAULT_TIMEOUT;
}
if (! isDone()) {
long start = System.currentTimeMillis();
lock.lock();
try {
while (! isDone()) {
done.await(timeout, TimeUnit.MILLISECONDS);
if (isDone() || System.currentTimeMillis() - start > timeout) {
break;
}
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
} finally {
lock.unlock();
}
if (! isDone()) {
throw new TimeoutException(sent > 0, channel, getTimeoutMessage(false));
}
}
return returnFromResponse();
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
public static void received(Channel channel, Response response) {
try {
DefaultFuture future = FUTURES.remove(response.getId());
if (future != null) {
future.doReceived(response);
} else {
logger.warn("The timeout response finally returned at "
+ (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS").format(new Date()))
+ ", response " + response
+ (channel == null ? "" : ", channel: " + channel.getLocalAddress()
+ " -> " + channel.getRemoteAddress()));
}
} finally {
CHANNELS.remove(response.getId());
}
}
private void doReceived(Response res) {
lock.lock();
try {
response = res;
if (done != null) {
done.signal();
}
} finally {
lock.unlock();
}
if (callback != null) {
invokeCallback(callback);
}
}