OkHttp工具类在微服高并发场景中问题实践总结

OkHttp工具类在微服高并发场景中问题实践总结

问题场景

1. 我的应用是一个中间业务应用XXApp，一个交易请求进来需要依赖下游应用，采用http协议通讯方式，需要调用3-4次下游请求。
2. 老XXApp在生产上有大量Close_wait问题，（可以请求交易没有及时关闭，会造成系统有限资源port被占用完）导致“too many open files” SocketExcepiton.
3. 我改成以前常用的OKhttp3，有个高并发场景的应用Utils类。但是让我没想到的在压力测试环境下20qps，XXApp系统不能连接后端服务，异常表现是大量的“time out”。
4. 开始判断是后端服务能力不够或者是业务服务线程数不够，

OKHttp的部分代码

  private static final OkHttpClient OK_HTTP_CLIENT =new OkHttpClient().newBuilder().connectTimeout(5,TimeUnit.SECONDS).build();


 /**
     * POST请求，参数为json格式。
     *
     * @param url
     * @param json
     * @return Optional<String>
     */
    public static String post(String url, String json, int timeout) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(JSON, json);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(timeout).newCall(request).execute().body().string();
        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }


    }

    public static OkHttpClient getHttpClient(int timeout) {

            if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
                return OK_HTTP_CLIENT;
            } else {
                OkHttpClient okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();

                return okHttpClient;
            }
        }

1. 为了支持不同业务调用需要不同timeout，提供自定义okHttpClient与默认的OK_HTTP_CLIENT。
2. XXApp在业务调用的时候采用不同接口自定义timeout时间。

压测现象分析

1. top -H -P 进程ID， 查看进程的线程数，发现totalThread数上15000多，还在不停的增加，这就有些奇怪了，以前高并发应用也没有这么高的线程数。

2. netstat -n | awk ‘/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’ ； 发现也有CLOSE_WAIT的连接状态在增长。

---

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-CGSrfyGC-1582808024865)(./image/close_wait_issue.jpg)]

---

---

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-I65oGUiM-1582808024867)(./image/many_conn.jpg)]

---

3. 开始怀疑是不是每个请求都创建okHttpClient实例造成线程数量增加。然后在关闭连接过程造成CLOSE_WAIT数量波动在3000-4000多。
4. 查询一下http协议1.1 keep-alive长连接应用可以减少tcp 连接，关闭的时间。OKHttp也有一个ConnectionPool类来支持并发请求，查看builder类，果然给okHttpClient创建一个默认
5. 如果每次都要创建okHttpClient的确不很好提升性能，所有先改成默认OK_HTTP_CLIENT实例试下，压测效果瞬间提升到100qps。 进程的线程数也稳定在500左右，CLOSE_WAIT问题也没有了.

再次压测现象分析

---

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Mp6JHBlo-1582808024867)(./image/ok-conn-count.jpg)]

---

1. 各种压测指标，监控指标都正常了。

2. 继续分析源码来总结，查看OkHttpClients源码

     * Factory for {@linkplain Call calls}, which can be used to send HTTP requests and read their
     * responses.
     *
     * <h3>OkHttpClients should be shared</h3>
     *
     * <p>OkHttp performs best when you create a single {@code OkHttpClient} instance and reuse it for
     * all of your HTTP calls. This is because each client holds its own connection pool and thread
     * pools. Reusing connections and threads reduces latency and saves memory. Conversely, creating a
     * client for each request wastes resources on idle pools.
     *
     * <p>Use {@code new OkHttpClient()} to create a shared instance with the default settings:
     * <pre>   {@code
     *
     *   // The singleton HTTP client.
     *   public final OkHttpClient client = new OkHttpClient();
     * }</pre>
     *
   

3. 已经明确描述希望OkHttpClient是一个单例，因为它有自己连接池与线程池。也解释为什么不停调用会thread增加。

总结

1. 我在构建一个utils的时候，除参考网络上的代码例子，还应该参考官方example，及api的comments。
   https://github.com/square/okhttp/blob/master/okhttp/src/test/java/okhttp3/OkHttpClientTest.java
2. 关注点

• 注意测试工具的timeout问题，一定要写单元测试。
• 注意工具的连接关闭问题，参考官方demo是最好的。netstat 观察
• 了解工具调用连接结构，方便调优，已解决在高并发场景下的调优。

修改后的OKHttp3Utils3

import lombok.extern.slf4j.Slf4j;
import okhttp3.*;
import okhttp3.internal.Util;
import okio.BufferedSink;
import okio.Okio;
import okio.Source;

import java.io.IOException;
import java.io.InputStream;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;

@Slf4j
public final class OKHttp3Utils3 {

    public static int DEFAULT_TIME_OUT = 10;

    /**
     * 全局实例可以保持http1.1 连接复用，线程池复用， 减少tcp的网络连接，关闭，
     * 如果每次一个请求，在高并发下，thread增多到1W，close_wait持续增加到6k。
     */
    private static final OkHttpClient OK_HTTP_CLIENT = new OkHttpClient.Builder()
            .connectionPool(new ConnectionPool(50, 5, TimeUnit.MINUTES))
            .connectTimeout(10, TimeUnit.SECONDS).readTimeout(10, TimeUnit.SECONDS).writeTimeout(10, TimeUnit.SECONDS).build();

    private static final MediaType JSON = MediaType.parse("application/json; charset=utf-8");

    private static final MediaType FORM_TYPE = MediaType.parse("application/x-www-form-urlencoded; charset=UTF-8");


    /**
     * 不同timeout的连接池
     */
    public static ConcurrentHashMap<Integer, OkHttpClient> cacheClients = new ConcurrentHashMap();


    public static OkHttpClient getHttpClient(int timeout) {

        if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
            return OK_HTTP_CLIENT;
        } else {
            OkHttpClient okHttpClient = cacheClients.get(timeout);
            if (okHttpClient == null) {
                return syncCreateClient(timeout);
            }
            return okHttpClient;
        }
    }

    private static synchronized OkHttpClient syncCreateClient(int timeout) {
        OkHttpClient okHttpClient;

        okHttpClient = cacheClients.get(timeout);
        if (okHttpClient != null) {
            return okHttpClient;
        }

        okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();
        cacheClients.put(timeout, okHttpClient);
        return okHttpClient;

    }


    /**
     * GET请求
     *
     * @param url
     * @return Optional<String>
     */
    public static String get(String url, int timeout) throws Exception {
        Request request = new Request.Builder().url(url)
                .build();
        return getHttpClient(timeout).newCall(request).execute().body().string();

    }

    public static String get(String url) throws Exception {
        return get(url, 0);
    }

    /**
     * POST请求，参数为json格式。
     *
     * @param url
     * @param json
     * @return Optional<String>
     */
    public static String post(String url, String json, int timeout) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(JSON, json);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(timeout).newCall(request).execute().body().string();
        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }


    }


    public static String post(String url, String json) throws Exception {
        return post(url, json, 0);
    }


    public static String postByFormType(String url, String form) throws Exception {
        long start = System.currentTimeMillis();
        try {
            RequestBody body = RequestBody.create(FORM_TYPE, form);
            Request request = new Request.Builder().url(url).post(body).build();
            return getHttpClient(0).newCall(request).execute().body().string();

        } catch (Exception e) {
            throw e;
        } finally {
            log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));

        }
    }


    /**
     * 根据不同的类型和requestbody类型来接续参数
     *
     * @param url
     * @param mediaType
     * @param inputStream
     * @return
     * @throws Exception
     */
    public static String post(String url, MediaType mediaType, InputStream inputStream) throws Exception {
        RequestBody body = createRequestBody(mediaType, inputStream);
        Request request = new Request.Builder().url(url).post(body).build();
        return OK_HTTP_CLIENT.newCall(request).execute().body().string();
    }

    private static RequestBody createRequestBody(final MediaType mediaType, final InputStream inputStream) {
        return new RequestBody() {
            // @Nullable
            @Override
            public MediaType contentType() {
                return mediaType;
            }

            @Override
            public long contentLength() throws IOException {
                try {
                    return inputStream.available();
                } catch (IOException e) {
                    return 0;
                }
            }

            @Override
            public void writeTo(BufferedSink sink) throws IOException {
                Source source = null;
                try {
                    source = Okio.source(inputStream);
                    sink.writeAll(source);
                } finally {
                    Util.closeQuietly(source);
                }
            }
        };
    }

}