DataX二次开发——Doris写入时报Content-Length header already present异常处理

DataX Doris的写入模板：

             "writer": {
                    "name": "doriswriter",
                    "parameter": {
                        "loadUrl": ["10.252.226.7:8130"],
                        "loadProps": {
                        },
                        "column": [ "dept_name", "date", "balance"],
                        "username": "root",
                        "password": "",
                        "postSql": [],
                        "preSql": [],
                        "flushInterval":30000,
                        "connection": [
                          {
                            "jdbcUrl": "jdbc:mysql://10.252.226.7:9030/datax_test",
                            "selectedDatabase": "datax_test",
                            "table": ["balance_info"]
                          }
                        ]
                    }
                }

写入时爆了异常，异常信息如下：

org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
……
Caused by: org.apache.http.ProtocolException: Content-Length header already present
at org.apache.http.protocol.RequestContent.process(RequestContent.java:67)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:108)
at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:174)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:462)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 27 more

• 出现上面的问题是发送请求的时候设置了请求头Content-Length参数。
• 解决办法：把设置的请求头Content-Length参数去掉。
• 分析原因：跟踪代码，查看HttpClient中RequestContent类的process方法，看到当body非空时，会自动加上Content-Length请求头及其对应值，不需要自己手动加上它。加上就会抛出上面的异常。

源码：
 

public class RequestContent implements HttpRequestInterceptor {
    private final boolean overwrite;

    public RequestContent() {
        this(false);
    }

    public RequestContent(boolean overwrite) {
        this.overwrite = overwrite;
    }

    public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
        Args.notNull(request, "HTTP request");
        if (request instanceof HttpEntityEnclosingRequest) {
            if (this.overwrite) {
                request.removeHeaders("Transfer-Encoding");
                request.removeHeaders("Content-Length");
            } else {
                if (request.containsHeader("Transfer-Encoding")) {
                    throw new ProtocolException("Transfer-encoding header already present");
                }

                if (request.containsHeader("Content-Length")) {
                    throw new ProtocolException("Content-Length header already present");
                }
            }

            ProtocolVersion ver = request.getRequestLine().getProtocolVersion();
            HttpEntity entity = ((HttpEntityEnclosingRequest)request).getEntity();
            if (entity == null) {
                request.addHeader("Content-Length", "0");
                return;
            }

            if (!entity.isChunked() && entity.getContentLength() >= 0L) {
            	//计算请求头的长度
                request.addHeader("Content-Length", Long.toString(entity.getContentLength()));
            } else {
                if (ver.lessEquals(HttpVersion.HTTP_1_0)) {
                    throw new ProtocolException("Chunked transfer encoding not allowed for " + ver);
                }

                request.addHeader("Transfer-Encoding", "chunked");
            }

            if (entity.getContentType() != null && !request.containsHeader("Content-Type")) {
                request.addHeader(entity.getContentType());
            }

            if (entity.getContentEncoding() != null && !request.containsHeader("Content-Encoding")) {
                request.addHeader(entity.getContentEncoding());
            }
        }

    }
}

• 解决办法：根据源码可以知道，overwrite这个参数为false，所以最走到下面判断请求头信息，所以我们在创建httpClientBuilder对象的时候 加上这句 .addInterceptorLast(new RequestContent(true)); 最后代码如下

  httpClientBuilder = HttpClients.custom().setRedirectStrategy(new DefaultRedirectStrategy() {
      @Override
      protected boolean isRedirectable(String method) {
          return true;
      }
  }).addInterceptorLast(new RequestContent(true));
  

  这样问题就解决了！