OkHttp无限重试问题

背景:

公司值班同学收到反馈,xxx-tc-tke-prodd请求频繁超时导致页面查询超时问题。

现象

1.调用方耗时长。

2.调用方发送一次请求,服务提供方收到大量重复请求。如下图

调查步骤:
1.当请求参数过长时,问题可以重现。注:问题发生时,请求经过的链路为:xxx-tc-tke-prod->腾讯LB
->MK网关->gis
2.查看okhttp源码,重试逻辑在okhttp3.internal.http.RetryAndFollowUpInterceptor类中。
关键逻辑介绍:

1. RetryAndFollowUplnterceptor拦截器会对服务端返回的异常信息做分析。判断异常是否可通过重试解决(代码
   逻辑,okhttp3.internal.http.RetryAndFollowUplnterceptor#reecover)。
2. RetryAndFollowUpInterceptor拦截器收到的异常类为:IOException,异常详细信息:“stream was reset:
   CANCEL”,符合重试逻辑。
   
   3.请求重试后,拦截器再次收到同样的异常,再次重试,并无限循环下去。

问题:
1.为什么okhttp会不断收到"stream was reset:CANCEL"异常常?
与
结论如下:
a.腾讯云LBHTTP2协议默认最大的HEADER大小为4kb,数据超过最最大值后,LB直接拒绝并发送FIN包给调用方。然
而,调用方因请求数据并未发送完,所以会继续发送数据,LB4收到额外数据后,发送RESET包给调用方,故,okhttp
收到"stream was reset: CANCEL"异常。
b.以上分析与抓包数据吻合

2.为什么okHttp会无间断的重试?
经测试,此为okhttp低版本的bug,虽然okhttp也有最大重试次数的判断逻辑,但无法触达。高版本okhttp已修复该问题。详情见解决方案。

问题复现条件:

1.使用HTTP2协议。
2.流量经过腾讯LB网关,且请求头信息超过LB配置的最大值(当前值为4kb)。
3.使用okHttp的问题版本。

import okhttp3.*;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class TestController {

    private static final String s = "https://api-les-gis-test.nioint.com/v1/fence/batch?fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&fenceNos=119898&";
    private static final String s1 = "https://api-les-gis-test.nioint.com/v1/fence/batch?fenceNos=119898";

    public static void main(String[] args) throws IOException {
        OkHttpClient.Builder builder = new OkHttpClient().newBuilder();
        builder.connectTimeout(20, TimeUnit.SECONDS);
        builder.readTimeout(10000, TimeUnit.SECONDS);
        builder.retryOnConnectionFailure(true);
        List<Protocol> protocolList = new ArrayList<Protocol>();
        protocolList.add(Protocol.HTTP_2);
        //protocolList.add(Protocol.HTTP_1_1);
        builder.connectionPool(new ConnectionPool(10, 60,TimeUnit.MINUTES));
        final OkHttpClient client = builder.build();
        //构建请求
        Request.Builder requestBuilder = new Request.Builder();
        requestBuilder.get();
        requestBuilder.url(s);
        Request request = requestBuilder.build();
        Call call = client.newCall(request);
        Response response = call.execute();
        if(response.code() > 200){
            System.out.println(response);
        }
        response.close();
    }
}

解决方案:

问题版本:OkHttp4.2.2及以下。
修复版本:OkHttp4.3.0及以上。
经过测试,升级OkHttp修复版本后,问题消失。

Okhttp3 相关文章：
OKhttp3 默认拦截器
深入Okhttp之网络连接
HTTP2 详解
HTTP/2协议中二进制桢的示例分析