解决Jsoup 访问 https协议网站时产生connect reset错误

最新推荐文章于 2024-06-29 14:17:24 发布
最新推荐文章于 2024-06-29 14:17:24 发布
阅读量6.3k 收藏 11
点赞数 4
分类专栏： java
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ningzhao/article/details/107629058
版权

目录

一、问题现象

二、问题分析

三、解决问题过程

 四、参考内容

一、问题现象

1.通过jsoup访问https://www.xncx.gov.cn/Item/13754.aspx 时报connect reset错误（jsoup 已经设置了忽略证书，信任所有主机）

2.通过chrome浏览器,fiddler访问目标URL正常。

 

二、问题分析

 

Connection Reset——其中一端主动断开连接

Connection Reset是在建立TCP连接之后，其中一方的TCP标志位使用了Reset标志主动重置了连接

客户端Or服务器端

而我这里既然是客户端报的错误信息，那势必是服务器主动断开了连接，为什么它要断开连接？

服务器主动断开连接的原因：

1.服务器异常

2.服务器和客户端长短连接不匹配

3.Https连接，服务器和客户端的TLS版本不一致

 

三、解决问题过程

 

首先怀疑是客户端与服务器端使用的TLS协议版本不同导致。

 

1.确认客户端tls版本(jdk1.8)

设置-Djavax.net.debug=all ,再次用jsoup 访问时返回以下日志：

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_DH_anon_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DH_anon_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_anon_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_DH_anon_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

trustStore is: C:\Program Files\Java\jdk1.8.0_131\jre\lib\security\cacerts

trustStore type is : jks

trustStore provider is :

init truststore

adding as trusted cert:

Subject: CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US

Issuer: CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US

Algorithm: RSA; Serial number: 0xc3517

Valid from Mon Jun 21 12:00:00 CST 1999 until Mon Jun 22 12:00:00 CST 2020



adding as trusted cert:

Subject: CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US

Issuer: CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US

Algorithm: EC; Serial number: 0xa68b79290000000050d091f9

Valid from Tue Dec 18 23:25:36 CST 2012 until Fri Dec 18 23:55:36 CST 2037



adding as trusted cert:

Subject: CN=SecureTrust CA, O=SecureTrust Corporation, C=US

Issuer: CN=SecureTrust CA, O=SecureTrust Corporation, C=US

Algorithm: RSA; Serial number: 0xcf08e5c0816a5ad427ff0eb271859d0

Valid from Wed Nov 08 03:31:18 CST 2006 until Tue Jan 01 03:40:55 CST 2030



keyStore is :

keyStore type is : jks

keyStore provider is :

init keystore



init keymanager of type SunX509

trigger seeding of SecureRandom

done seeding SecureRandom

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256

Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

trigger seeding of SecureRandom

done seeding SecureRandom

Allow unsafe renegotiation: false

Allow legacy hello messages: true

Is initial handshake: true

Is secure renegotiation: false

main, setSoTimeout(30000) called

Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 for TLSv1

Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1

Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 for TLSv1.1

%% No cached client session

*** ClientHello, TLSv1.2

RandomCookie: GMT: 1595891766 bytes = { 13, 206, 82, 66, 76, 253, 107, 89, 152, 84, 212, 156, 85, 89, 140, 88, 170, 187, 24, 20, 10, 89, 235, 192, 196, 128, 57, 63 }

Session ID: {}

Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

Compression Methods: { 0 }

Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}

Extension ec_point_formats, formats: [uncompressed]

Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA

***

[write] MD5 and SHA1 hashes: len = 161

0000: 01 00 00 9D 03 03 5F 1F 60 36 0D CE 52 42 4C FD ......_.`6..RBL.

0010: 6B 59 98 54 D4 9C 55 59 8C 58 AA BB 18 14 0A 59 kY.T..UY.X.....Y

0020: EB C0 C4 80 39 3F 00 00 3A C0 23 C0 27 00 3C C0 ....9?..:.#.'.<.

0030: 25 C0 29 00 67 00 40 C0 09 C0 13 00 2F C0 04 C0 %.).g.@...../...

0040: 0E 00 33 00 32 C0 2B C0 2F 00 9C C0 2D C0 31 00 ..3.2.+./...-.1.

0050: 9E 00 A2 C0 08 C0 12 00 0A C0 03 C0 0D 00 16 00 ................

0060: 13 00 FF 01 00 00 3A 00 0A 00 16 00 14 00 17 00 ......:.........

0070: 18 00 19 00 09 00 0A 00 0B 00 0C 00 0D 00 0E 00 ................

0080: 16 00 0B 00 02 01 00 00 0D 00 16 00 14 06 03 06 ................

0090: 01 05 03 05 01 04 03 04 01 04 02 02 03 02 01 02 ................

00A0: 02 .

main, WRITE: TLSv1.2 Handshake, length = 161

[Raw write]: length = 166

0000: 16 03 03 00 A1 01 00 00 9D 03 03 5F 1F 60 36 0D ..........._.`6.

0010: CE 52 42 4C FD 6B 59 98 54 D4 9C 55 59 8C 58 AA .RBL.kY.T..UY.X.

0020: BB 18 14 0A 59 EB C0 C4 80 39 3F 00 00 3A C0 23 ....Y....9?..:.#

0030: C0 27 00 3C C0 25 C0 29 00 67 00 40 C0 09 C0 13 .'.<.%.).g.@....

0040: 00 2F C0 04 C0 0E 00 33 00 32 C0 2B C0 2F 00 9C ./.....3.2.+./..

0050: C0 2D C0 31 00 9E 00 A2 C0 08 C0 12 00 0A C0 03 .-.1............

0060: C0 0D 00 16 00 13 00 FF 01 00 00 3A 00 0A 00 16 ...........:....

0070: 00 14 00 17 00 18 00 19 00 09 00 0A 00 0B 00 0C ................

0080: 00 0D 00 0E 00 16 00 0B 00 02 01 00 00 0D 00 16 ................

0090: 00 14 06 03 06 01 05 03 05 01 04 03 04 01 04 02 ................

00A0: 02 03 02 01 02 02 ......

main, handling exception: java.net.SocketException: Connection reset

main, SEND TLSv1.2 ALERT: fatal, description = unexpected_message

main, WRITE: TLSv1.2 Alert, length = 2

main, Exception sending alert: java.net.SocketException: Connection reset by peer: socket write error

main, called closeSocket()

main, called close()

main, called closeInternal(true)



java.net.SocketException: Connection reset



at java.net.SocketInputStream.read(SocketInputStream.java:210)

at java.net.SocketInputStream.read(SocketInputStream.java:141)

at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)

at sun.security.ssl.InputRecord.read(InputRecord.java:503)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:653)

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:630)

at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:262)

通过日志发现

ClientHello, TLSv1.2，确认客户端使用的是TLSV1.2

2.通过https://www.ssllabs.com/ssltest/index.html 查服务器端支持的TLS版本，最终发现也是支持TLS1.2

 

 

3.使用chrome 访问https://www.xncx.gov.cn/Item/13754.aspx 查看协议版本也是TLS1.2

通过以上验证发现使用的协议版本全是TLS1.2,基本上确认不是协议版本的问题。

再次决定通过fiddler抓包分析问题

1.配置jsoup 使用fiddler作为代理,访问目标地址，结果竟然没有异常。

通过对比正常与异常结果的日志发现：

正常访问时有extension server_name 值指定，指定-Djsse.enableSNIExtension=true,再次测试，还是connect reset 异常，同时日志中也没有看到Server Name Identification (SNI) 内容。最后决定修改HttpConnection.java 增加Server Name Identification(SNI)，最终问题解决。

附修改后的HttpConnection.java

package org.jsoup.helper;


import org.jsoup.*;

import org.jsoup.nodes.Document;

import org.jsoup.parser.Parser;

import org.jsoup.parser.TokenQueue;

import javax.net.ssl.*;

import java.io.*;

import java.net.*;

import java.nio.ByteBuffer;

import java.nio.charset.Charset;

import java.nio.charset.IllegalCharsetNameException;

import java.security.KeyManagementException;

import java.security.NoSuchAlgorithmException;

import java.security.cert.X509Certificate;

import java.util.*;

import java.util.regex.Pattern;

import java.util.zip.GZIPInputStream;



import static org.jsoup.Connection.Method.HEAD;

import static org.jsoup.internal.Normalizer.lowerCase;



/**

* Implementation of {@link Connection}.

* @see org.jsoup.Jsoup#connect(String)

*/

public class HttpConnection implements Connection {

public static final String CONTENT_ENCODING = "Content-Encoding";

/**

* Many users would get caught by not setting a user-agent and therefore getting different responses on their desktop

* vs in jsoup, which would otherwise default to {@code Java}. So by default, use a desktop UA.

*/

public static final String DEFAULT_UA =

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36";

private static final String USER_AGENT = "User-Agent";

private static final String CONTENT_TYPE = "Content-Type";

private static final String MULTIPART_FORM_DATA = "multipart/form-data";

private static final String FORM_URL_ENCODED = "application/x-www-form-urlencoded";

private static final int HTTP_TEMP_REDIR = 307; // http/1.1 temporary redirect, not in Java's set.



public static Connection connect(String url) {

Connection con = new HttpConnection();

con.url(url);

return con;

}



public static Connection connect(URL url) {

Connection con = new HttpConnection();

con.url(url);

return con;

}



/**

* Encodes the input URL into a safe ASCII URL string

* @param url unescaped URL

* @return escaped URL

*/

private static String encodeUrl(String url) {

try {

URL u = new URL(url);

return encodeUrl(u).toExternalForm();

} catch (Exception e) {

return url;

}

}



private static URL encodeUrl(URL u) {

try {

// odd way to encode urls, but it works!

final URI uri = new URI(u.toExternalForm());

return new URL(uri.toASCIIString());

} catch (Exception e) {

return u;

}

}



private static String encodeMimeName(String val) {

if (val == null)

return null;

return val.replaceAll("\"", "%22");

}



private Connection.Request req;

private Connection.Response res;



private HttpConnection() {

req = new Request();

res = new Response();

}



public Connection url(URL url) {

req.url(url);

return this;

}



public Connection url(String url) {

Validate.notEmpty(url, "Must supply a valid URL");

try {

req.url(new URL(encodeUrl(url)));

} catch (MalformedURLException e) {

throw new IllegalArgumentException("Malformed URL: " + url, e);

}

return this;

}



public Connection proxy(Proxy proxy) {

req.proxy(proxy);

return this;

}



public Connection proxy(String host, int port) {

req.proxy(host, port);

return this;

}



public Connection userAgent(String userAgent) {

Validate.notNull(userAgent, "User agent must not be null");

req.header(USER_AGENT, userAgent);

return this;

}



public Connection timeout(int millis) {

req.timeout(millis);

return this;

}



public Connection maxBodySize(int bytes) {

req.maxBodySize(bytes);

return this;

}



public Connection followRedirects(boolean followRedirects) {

req.followRedirects(followRedirects);

return this;

}



public Connection referrer(String referrer) {

Validate.notNull(referrer, "Referrer must not be null");

req.header("Referer", referrer);

return this;

}



public Connection method(Method method) {

req.method(method);

return this;

}



public Connection ignoreHttpErrors(boolean ignoreHttpErrors) {

req.ignoreHttpErrors(ignoreHttpErrors);

return this;

}



public Connection ignoreContentType(boolean ignoreContentType) {

req.ignoreContentType(ignoreContentType);

return this;

}



public Connection validateTLSCertificates(boolean value) {

req.validateTLSCertificates(value);

return this;

}



public Connection data(String key, String value) {

req.data(KeyVal.create(key, value));

return this;

}



public Connection data(String key, String filename, InputStream inputStream) {

req.data(KeyVal.create(key, filename, inputStream));

return this;

}



public Connection data(Map<String, String> data) {

Validate.notNull(data, "Data map must not be null");

for (Map.Entry<String, String> entry : data.entrySet()) {

req.data(KeyVal.create(entry.getKey(), entry.getValue()));

}

return this;

}



public Connection data(String... keyvals) {

Validate.notNull(keyvals, "Data key value pairs must not be null");

Validate.isTrue(keyvals.length %2 == 0, "Must supply an even number of key value pairs");

for (int i = 0; i < keyvals.length; i += 2) {

String key = keyvals[i];

String value = keyvals[i+1];

Validate.notEmpty(key, "Data key must not be empty");

Validate.notNull(value, "Data value must not be null");

req.data(KeyVal.create(key, value));

}

return this;

}



public Connection data(Collection<Connection.KeyVal> data) {

Validate.notNull(data, "Data collection must not be null");

for (Connection.KeyVal entry: data) {

req.data(entry);

}

return this;

}



public Connection.KeyVal data(String key) {

Validate.notEmpty(key, "Data key must not be empty");

for (Connection.KeyVal keyVal : request().data()) {

if (keyVal.key().equals(key))

return keyVal;

}

return null;

}



public Connection requestBody(String body) {

req.requestBody(body);

return this;

}



public Connection header(String name, String value) {

req.header(name, value);

return this;

}



public Connection headers(Map<String,String> headers) {

Validate.notNull(headers, "Header map must not be null");

for (Map.Entry<String,String> entry : headers.entrySet()) {

req.header(entry.getKey(),entry.getValue());

}

return this;

}



public Connection cookie(String name, String value) {

req.cookie(name, value);

return this;

}



public Connection cookies(Map<String, String> cookies) {

Validate.notNull(cookies, "Cookie map must not be null");

for (Map.Entry<String, String> entry : cookies.entrySet()) {

req.cookie(entry.getKey(), entry.getValue());

}

return this;

}



public Connection parser(Parser parser) {

req.parser(parser);

return this;

}



public Document get() throws IOException {

req.method(Method.GET);

execute();

return res.parse();

}



public Document post() throws IOException {

req.method(Method.POST);

execute();

return res.parse();

}



public Connection.Response execute() throws IOException {

res = Response.execute(req);

return res;

}



public Connection.Request request() {

return req;

}



public Connection request(Connection.Request request) {

req = request;

return this;

}



public Connection.Response response() {

return res;

}



public Connection response(Connection.Response response) {

res = response;

return this;

}



public Connection postDataCharset(String charset) {

req.postDataCharset(charset);

return this;

}



@SuppressWarnings({"unchecked"})

private static abstract class Base<T extends Connection.Base> implements Connection.Base<T> {

URL url;

Method method;

Map<String, String> headers;

Map<String, String> cookies;



private Base() {

headers = new LinkedHashMap<String, String>();

cookies = new LinkedHashMap<String, String>();

}



public URL url() {

return url;

}



public T url(URL url) {

Validate.notNull(url, "URL must not be null");

this.url = url;

return (T) this;

}



public Method method() {

return method;

}



public T method(Method method) {

Validate.notNull(method, "Method must not be null");

this.method = method;

return (T) this;

}



public String header(String name) {

Validate.notNull(name, "Header name must not be null");

String val = getHeaderCaseInsensitive(name);

if (val != null) {

// headers should be ISO8859 - but values are often actually UTF-8. Test if it looks like UTF8 and convert if so

val = fixHeaderEncoding(val);

}

return val;

}



private static String fixHeaderEncoding(String val) {

try {

byte[] bytes = val.getBytes("ISO-8859-1");

if (!looksLikeUtf8(bytes))

return val;

return new String(bytes, "UTF-8");

} catch (UnsupportedEncodingException e) {

// shouldn't happen as these both always exist

return val;

}

}



private static boolean looksLikeUtf8(byte[] input) {

int i = 0;

// BOM:

if (input.length >= 3 && (input[0] & 0xFF) == 0xEF

&& (input[1] & 0xFF) == 0xBB & (input[2] & 0xFF) == 0xBF) {

i = 3;

}



int end;

for (int j = input.length; i < j; ++i) {

int o = input[i];

if ((o & 0x80) == 0) {

continue; // ASCII

}



// UTF-8 leading:

if ((o & 0xE0) == 0xC0) {

end = i + 1;

} else if ((o & 0xF0) == 0xE0) {

end = i + 2;

} else if ((o & 0xF8) == 0xF0) {

end = i + 3;

} else {

return false;

}



while (i < end) {

i++;

o = input[i];

if ((o & 0xC0) != 0x80) {

return false;

}

}

}

return true;

}



public T header(String name, String value) {

Validate.notEmpty(name, "Header name must not be empty");

Validate.notNull(value, "Header value must not be null");

removeHeader(name); // ensures we don't get an "accept-encoding" and a "Accept-Encoding"

headers.put(name, value);

return (T) this;

}



public boolean hasHeader(String name) {

Validate.notEmpty(name, "Header name must not be empty");

return getHeaderCaseInsensitive(name) != null;

}



/**

* Test if the request has a header with this value (case insensitive).

*/

public boolean hasHeaderWithValue(String name, String value) {

return hasHeader(name) && header(name).equalsIgnoreCase(value);

}



public T removeHeader(String name) {

Validate.notEmpty(name, "Header name must not be empty");

Map.Entry<String, String> entry = scanHeaders(name); // remove is case insensitive too

if (entry != null)

headers.remove(entry.getKey()); // ensures correct case

return (T) this;

}



public Map<String, String> headers() {

return headers;

}



private String getHeaderCaseInsensitive(String name) {

Validate.notNull(name, "Header name must not be null");

// quick evals for common case of title case, lower case, then scan for mixed

String value = headers.get(name);

if (value == null)

value = headers.get(lowerCase(name));

if (value == null) {

Map.Entry<String, String> entry = scanHeaders(name);

if (entry != null)

value = entry.getValue();

}

return value;

}



private Map.Entry<String, String> scanHeaders(String name) {

String lc = lowerCase(name);

for (Map.Entry<String, String> entry : headers.entrySet()) {

if (lowerCase(entry.getKey()).equals(lc))

return entry;

}

return null;

}



public String cookie(String name) {

Validate.notEmpty(name, "Cookie name must not be empty");

return cookies.get(name);

}



public T cookie(String name, String value) {

Validate.notEmpty(name, "Cookie name must not be empty");

Validate.notNull(value, "Cookie value must not be null");

cookies.put(name, value);

return (T) this;

}



public boolean hasCookie(String name) {

Validate.notEmpty(name, "Cookie name must not be empty");

return cookies.containsKey(name);

}



public T removeCookie(String name) {

Validate.notEmpty(name, "Cookie name must not be empty");

cookies.remove(name);

return (T) this;

}



public Map<String, String> cookies() {

return cookies;

}

}



public static class Request extends HttpConnection.Base<Connection.Request> implements Connection.Request {

private Proxy proxy; // nullable

private int timeoutMilliseconds;

private int maxBodySizeBytes;

private boolean followRedirects;

private Collection<Connection.KeyVal> data;

private String body = null;

private boolean ignoreHttpErrors = false;

private boolean ignoreContentType = false;

private Parser parser;

private boolean parserDefined = false; // called parser(...) vs initialized in ctor

private boolean validateTSLCertificates = true;

private String postDataCharset = DataUtil.defaultCharset;



private Request() {

timeoutMilliseconds = 30000; // 30 seconds

maxBodySizeBytes = 1024 * 1024; // 1MB

followRedirects = true;

data = new ArrayList<Connection.KeyVal>();

method = Method.GET;

headers.put("Accept-Encoding", "gzip");

headers.put(USER_AGENT, DEFAULT_UA);

parser = Parser.htmlParser();

}



public Proxy proxy() {

return proxy;

}



public Request proxy(Proxy proxy) {

this.proxy = proxy;

return this;

}



public Request proxy(String host, int port) {

this.proxy = new Proxy(Proxy.Type.HTTP, InetSocketAddress.createUnresolved(host, port));

return this;

}



public int timeout() {

return timeoutMilliseconds;

}



public Request timeout(int millis) {

Validate.isTrue(millis >= 0, "Timeout milliseconds must be 0 (infinite) or greater");

timeoutMilliseconds = millis;

return this;

}



public int maxBodySize() {

return maxBodySizeBytes;

}



public Connection.Request maxBodySize(int bytes) {

Validate.isTrue(bytes >= 0, "maxSize must be 0 (unlimited) or larger");

maxBodySizeBytes = bytes;

return this;

}



public boolean followRedirects() {

return followRedirects;

}



public Connection.Request followRedirects(boolean followRedirects) {

this.followRedirects = followRedirects;

return this;

}



public boolean ignoreHttpErrors() {

return ignoreHttpErrors;

}



public boolean validateTLSCertificates() {

return validateTSLCertificates;

}



public void validateTLSCertificates(boolean value) {

validateTSLCertificates = value;

}



public Connection.Request ignoreHttpErrors(boolean ignoreHttpErrors) {

this.ignoreHttpErrors = ignoreHttpErrors;

return this;

}



public boolean ignoreContentType() {

return ignoreContentType;

}



public Connection.Request ignoreContentType(boolean ignoreContentType) {

this.ignoreContentType = ignoreContentType;

return this;

}



public Request data(Connection.KeyVal keyval) {

Validate.notNull(keyval, "Key val must not be null");

data.add(keyval);

return this;

}



public Collection<Connection.KeyVal> data() {

return data;

}



public Connection.Request requestBody(String body) {

this.body = body;

return this;

}



public String requestBody() {

return body;

}



public Request parser(Parser parser) {

this.parser = parser;

parserDefined = true;

return this;

}



public Parser parser() {

return parser;

}



public Connection.Request postDataCharset(String charset) {

Validate.notNull(charset, "Charset must not be null");

if (!Charset.isSupported(charset)) throw new IllegalCharsetNameException(charset);

this.postDataCharset = charset;

return this;

}



public String postDataCharset() {

return postDataCharset;

}

}



public static class Response extends HttpConnection.Base<Connection.Response> implements Connection.Response {

private static final int MAX_REDIRECTS = 20;

private static SSLSocketFactory sslSocketFactory;

private static final String LOCATION = "Location";

private int statusCode;

private String statusMessage;

private ByteBuffer byteData;

private String charset;

private String contentType;

private boolean executed = false;

private int numRedirects = 0;

private Connection.Request req;



/*

* Matches XML content types (like text/xml, application/xhtml+xml;charset=UTF8, etc)

*/

private static final Pattern xmlContentTypeRxp = Pattern.compile("(application|text)/\\w*\\+?xml.*");



Response() {

super();

}



private Response(Response previousResponse) throws IOException {

super();

if (previousResponse != null) {

numRedirects = previousResponse.numRedirects + 1;

if (numRedirects >= MAX_REDIRECTS)

throw new IOException(String.format("Too many redirects occurred trying to load URL %s", previousResponse.url()));

}

}



static Response execute(Connection.Request req) throws IOException {

return execute(req, null);

}



static Response execute(Connection.Request req, Response previousResponse) throws IOException {

Validate.notNull(req, "Request must not be null");

String protocol = req.url().getProtocol();

if (!protocol.equals("http") && !protocol.equals("https"))

throw new MalformedURLException("Only http & https protocols supported");

final boolean methodHasBody = req.method().hasBody();

final boolean hasRequestBody = req.requestBody() != null;

if (!methodHasBody)

Validate.isFalse(hasRequestBody, "Cannot set a request body for HTTP method " + req.method());



// set up the request for execution

String mimeBoundary = null;

if (req.data().size() > 0 && (!methodHasBody || hasRequestBody))

serialiseRequestUrl(req);

else if (methodHasBody)

mimeBoundary = setOutputContentType(req);



HttpURLConnection conn = createConnection(req);

Response res;

try {

conn.connect();

if (conn.getDoOutput())

writePost(req, conn.getOutputStream(), mimeBoundary);



int status = conn.getResponseCode();

res = new Response(previousResponse);

res.setupFromConnection(conn, previousResponse);

res.req = req;



// redirect if there's a location header (from 3xx, or 201 etc)

if (res.hasHeader(LOCATION) && req.followRedirects()) {

if (status != HTTP_TEMP_REDIR) {

req.method(Method.GET); // always redirect with a get. any data param from original req are dropped.

req.data().clear();

req.requestBody(null);

req.removeHeader(CONTENT_TYPE);

}



String location = res.header(LOCATION);

if (location != null && location.startsWith("http:/") && location.charAt(6) != '/') // fix broken Location: http:/temp/AAG_New/en/index.php

location = location.substring(6);

URL redir = StringUtil.resolve(req.url(), location);

req.url(encodeUrl(redir));



for (Map.Entry<String, String> cookie : res.cookies.entrySet()) { // add response cookies to request (for e.g. login posts)

req.cookie(cookie.getKey(), cookie.getValue());

}

return execute(req, res);

}

if ((status < 200 || status >= 400) && !req.ignoreHttpErrors())

throw new HttpStatusException("HTTP error fetching URL", status, req.url().toString());



// check that we can handle the returned content type; if not, abort before fetching it

String contentType = res.contentType();

if (contentType != null

&& !req.ignoreContentType()

&& !contentType.startsWith("text/")

&& !xmlContentTypeRxp.matcher(contentType).matches()

)

throw new UnsupportedMimeTypeException("Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml",

contentType, req.url().toString());



// switch to the XML parser if content type is xml and not parser not explicitly set

if (contentType != null && xmlContentTypeRxp.matcher(contentType).matches()) {

// only flip it if a HttpConnection.Request (i.e. don't presume other impls want it):

if (req instanceof HttpConnection.Request && !((Request) req).parserDefined) {

req.parser(Parser.xmlParser());

}

}



res.charset = DataUtil.getCharsetFromContentType(res.contentType); // may be null, readInputStream deals with it

if (conn.getContentLength() != 0 && req.method() != HEAD) { // -1 means unknown, chunked. sun throws an IO exception on 500 response with no content when trying to read body

InputStream bodyStream = null;

try {

bodyStream = conn.getErrorStream() != null ? conn.getErrorStream() : conn.getInputStream();

if (res.hasHeaderWithValue(CONTENT_ENCODING, "gzip"))

bodyStream = new GZIPInputStream(bodyStream);



res.byteData = DataUtil.readToByteBuffer(bodyStream, req.maxBodySize());

} finally {

if (bodyStream != null) bodyStream.close();

}

} else {

res.byteData = DataUtil.emptyByteBuffer();

}

} finally {

// per Java's documentation, this is not necessary, and precludes keepalives. However in practise,

// connection errors will not be released quickly enough and can cause a too many open files error.

conn.disconnect();

}



res.executed = true;

return res;

}



public int statusCode() {

return statusCode;

}



public String statusMessage() {

return statusMessage;

}



public String charset() {

return charset;

}



public Response charset(String charset) {

this.charset = charset;

return this;

}



public String contentType() {

return contentType;

}



public Document parse() throws IOException {

Validate.isTrue(executed, "Request must be executed (with .execute(), .get(), or .post() before parsing response");

Document doc = DataUtil.parseByteData(byteData, charset, url.toExternalForm(), req.parser());

byteData.rewind();

charset = doc.outputSettings().charset().name(); // update charset from meta-equiv, possibly

return doc;

}



public String body() {

Validate.isTrue(executed, "Request must be executed (with .execute(), .get(), or .post() before getting response body");

// charset gets set from header on execute, and from meta-equiv on parse. parse may not have happened yet

String body;

if (charset == null)

body = Charset.forName(DataUtil.defaultCharset).decode(byteData).toString();

else

body = Charset.forName(charset).decode(byteData).toString();

byteData.rewind();

return body;

}



public byte[] bodyAsBytes() {

Validate.isTrue(executed, "Request must be executed (with .execute(), .get(), or .post() before getting response body");

return byteData.array();

}



// set up connection defaults, and details from request

private static HttpURLConnection createConnection(Connection.Request req) throws IOException {

final HttpURLConnection conn = (HttpURLConnection) (

req.proxy() == null ?

req.url().openConnection() :

req.url().openConnection(req.proxy())

);



conn.setRequestMethod(req.method().name());

conn.setInstanceFollowRedirects(false); // don't rely on native redirection support

conn.setConnectTimeout(req.timeout());

conn.setReadTimeout(req.timeout());



if (conn instanceof HttpsURLConnection) {

if (!req.validateTLSCertificates()) {

//modify by zhxj 200728

initUnSecureTSL(req.url());

((HttpsURLConnection)conn).setSSLSocketFactory(sslSocketFactory);

((HttpsURLConnection)conn).setHostnameVerifier(getInsecureVerifier());

}

}



if (req.method().hasBody())

conn.setDoOutput(true);

if (req.cookies().size() > 0)

conn.addRequestProperty("Cookie", getRequestCookieString(req));

for (Map.Entry<String, String> header : req.headers().entrySet()) {

conn.addRequestProperty(header.getKey(), header.getValue());

}

return conn;

}



/**

* Instantiate Hostname Verifier that does nothing.

* This is used for connections with disabled SSL certificates validation.

*

*

* @return Hostname Verifier that does nothing and accepts all hostnames

*/

private static HostnameVerifier getInsecureVerifier() {

return new HostnameVerifier() {

public boolean verify(String urlHostName, SSLSession session) {

return true;

}

};

}



/**

* Initialise Trust manager that does not validate certificate chains and

* add it to current SSLContext.

* <p/>

* please not that this method will only perform action if sslSocketFactory is not yet

* instantiated.

*

* @throws IOException

*/

private static synchronized void initUnSecureTSL(final URL url) throws IOException {

if (sslSocketFactory == null) {

// Create a trust manager that does not validate certificate chains

final TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() {



public void checkClientTrusted(final X509Certificate[] chain, final String authType) {

}



public void checkServerTrusted(final X509Certificate[] chain, final String authType) {

}



public X509Certificate[] getAcceptedIssuers() {

return null;

}

}};



// Install the all-trusting trust manager

final SSLContext sslContext;

try {

//modify by zhxj 200728

sslContext = SSLContext.getInstance("TLS");

sslContext.init(null, trustAllCerts, new java.security.SecureRandom());



// Create an ssl socket factory with our all-trusting manager



//modify by zhxj 200728

//sslSocketFactory = sslContext.getSocketFactory();

SSLParameters sslParameters = new SSLParameters();

List sniHostNames = new ArrayList(1);

sniHostNames.add(new SNIHostName(url.getHost()));

sslParameters.setServerNames(sniHostNames);

sslSocketFactory = new SSLSocketFactoryWrapper(sslContext.getSocketFactory(), sslParameters);

} catch (NoSuchAlgorithmException e) {

throw new IOException("Can't create unsecure trust manager");

} catch (KeyManagementException e) {

throw new IOException("Can't create unsecure trust manager");

}

}



}



// set up url, method, header, cookies

private void setupFromConnection(HttpURLConnection conn, Connection.Response previousResponse) throws IOException {

method = Method.valueOf(conn.getRequestMethod());

url = conn.getURL();

statusCode = conn.getResponseCode();

statusMessage = conn.getResponseMessage();

contentType = conn.getContentType();



Map<String, List<String>> resHeaders = createHeaderMap(conn);

processResponseHeaders(resHeaders);



// if from a redirect, map previous response cookies into this response

if (previousResponse != null) {

for (Map.Entry<String, String> prevCookie : previousResponse.cookies().entrySet()) {

if (!hasCookie(prevCookie.getKey()))

cookie(prevCookie.getKey(), prevCookie.getValue());

}

}

}



private static LinkedHashMap<String, List<String>> createHeaderMap(HttpURLConnection conn) {

// the default sun impl of conn.getHeaderFields() returns header values out of order

final LinkedHashMap<String, List<String>> headers = new LinkedHashMap<String, List<String>>();

int i = 0;

while (true) {

final String key = conn.getHeaderFieldKey(i);

final String val = conn.getHeaderField(i);

if (key == null && val == null)

break;

i++;

if (key == null || val == null)

continue; // skip http1.1 line



if (headers.containsKey(key))

headers.get(key).add(val);

else {

final ArrayList<String> vals = new ArrayList<String>();

vals.add(val);

headers.put(key, vals);

}

}

return headers;

}



void processResponseHeaders(Map<String, List<String>> resHeaders) {

for (Map.Entry<String, List<String>> entry : resHeaders.entrySet()) {

String name = entry.getKey();

if (name == null)

continue; // http/1.1 line



List<String> values = entry.getValue();

if (name.equalsIgnoreCase("Set-Cookie")) {

for (String value : values) {

if (value == null)

continue;

TokenQueue cd = new TokenQueue(value);

String cookieName = cd.chompTo("=").trim();

String cookieVal = cd.consumeTo(";").trim();

// ignores path, date, domain, validateTLSCertificates et al. req'd?

// name not blank, value not null

if (cookieName.length() > 0)

cookie(cookieName, cookieVal);

}

} else { // combine same header names with comma: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

if (values.size() == 1)

header(name, values.get(0));

else if (values.size() > 1) {

StringBuilder accum = new StringBuilder();

for (int i = 0; i < values.size(); i++) {

final String val = values.get(i);

if (i != 0)

accum.append(", ");

accum.append(val);

}

header(name, accum.toString());

}

}

}

}



private static String setOutputContentType(final Connection.Request req) {

String bound = null;

if (req.hasHeader(CONTENT_TYPE)) {

// no-op; don't add content type as already set (e.g. for requestBody())

// todo - if content type already set, we could add charset or boundary if those aren't included

}

else if (needsMultipart(req)) {

bound = DataUtil.mimeBoundary();

req.header(CONTENT_TYPE, MULTIPART_FORM_DATA + "; boundary=" + bound);

} else {

req.header(CONTENT_TYPE, FORM_URL_ENCODED + "; charset=" + req.postDataCharset());

}

return bound;

}



private static void writePost(final Connection.Request req, final OutputStream outputStream, final String bound) throws IOException {

final Collection<Connection.KeyVal> data = req.data();

final BufferedWriter w = new BufferedWriter(new OutputStreamWriter(outputStream, req.postDataCharset()));



if (bound != null) {

// boundary will be set if we're in multipart mode

for (Connection.KeyVal keyVal : data) {

w.write("--");

w.write(bound);

w.write("\r\n");

w.write("Content-Disposition: form-data; name=\"");

w.write(encodeMimeName(keyVal.key())); // encodes " to %22

w.write("\"");

if (keyVal.hasInputStream()) {

w.write("; filename=\"");

w.write(encodeMimeName(keyVal.value()));

w.write("\"\r\nContent-Type: application/octet-stream\r\n\r\n");

w.flush(); // flush

DataUtil.crossStreams(keyVal.inputStream(), outputStream);

outputStream.flush();

} else {

w.write("\r\n\r\n");

w.write(keyVal.value());

}

w.write("\r\n");

}

w.write("--");

w.write(bound);

w.write("--");

} else if (req.requestBody() != null) {

// data will be in query string, we're sending a plaintext body

w.write(req.requestBody());

}

else {

// regular form data (application/x-www-form-urlencoded)

boolean first = true;

for (Connection.KeyVal keyVal : data) {

if (!first)

w.append('&');

else

first = false;



w.write(URLEncoder.encode(keyVal.key(), req.postDataCharset()));

w.write('=');

w.write(URLEncoder.encode(keyVal.value(), req.postDataCharset()));

}

}

w.close();

}



private static String getRequestCookieString(Connection.Request req) {

StringBuilder sb = new StringBuilder();

boolean first = true;

for (Map.Entry<String, String> cookie : req.cookies().entrySet()) {

if (!first)

sb.append("; ");

else

first = false;

sb.append(cookie.getKey()).append('=').append(cookie.getValue());

// todo: spec says only ascii, no escaping / encoding defined. validate on set? or escape somehow here?

}

return sb.toString();

}



// for get url reqs, serialise the data map into the url

private static void serialiseRequestUrl(Connection.Request req) throws IOException {

URL in = req.url();

StringBuilder url = new StringBuilder();

boolean first = true;

// reconstitute the query, ready for appends

url

.append(in.getProtocol())

.append("://")

.append(in.getAuthority()) // includes host, port

.append(in.getPath())

.append("?");

if (in.getQuery() != null) {

url.append(in.getQuery());

first = false;

}

for (Connection.KeyVal keyVal : req.data()) {

Validate.isFalse(keyVal.hasInputStream(), "InputStream data not supported in URL query string.");

if (!first)

url.append('&');

else

first = false;

url

.append(URLEncoder.encode(keyVal.key(), DataUtil.defaultCharset))

.append('=')

.append(URLEncoder.encode(keyVal.value(), DataUtil.defaultCharset));

}

req.url(new URL(url.toString()));

req.data().clear(); // moved into url as get params

}

}



private static boolean needsMultipart(Connection.Request req) {

// multipart mode, for files. add the header if we see something with an inputstream, and return a non-null boundary

boolean needsMulti = false;

for (Connection.KeyVal keyVal : req.data()) {

if (keyVal.hasInputStream()) {

needsMulti = true;

break;

}

}

return needsMulti;

}



public static class KeyVal implements Connection.KeyVal {

private String key;

private String value;

private InputStream stream;



public static KeyVal create(String key, String value) {

return new KeyVal().key(key).value(value);

}



public static KeyVal create(String key, String filename, InputStream stream) {

return new KeyVal().key(key).value(filename).inputStream(stream);

}



private KeyVal() {}



public KeyVal key(String key) {

Validate.notEmpty(key, "Data key must not be empty");

this.key = key;

return this;

}



public String key() {

return key;

}



public KeyVal value(String value) {

Validate.notNull(value, "Data value must not be null");

this.value = value;

return this;

}



public String value() {

return value;

}



public KeyVal inputStream(InputStream inputStream) {

Validate.notNull(value, "Data input stream must not be null");

this.stream = inputStream;

return this;

}



public InputStream inputStream() {

return stream;

}



public boolean hasInputStream() {

return stream != null;

}



@Override

public String toString() {

return key + "=" + value;

}

}

}

 附SSLSocketFactoryWrapper.java

package org.jsoup.helper;

import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import java.io.IOException;
import java.net.InetAddress;
import java.net.Socket;
import java.net.UnknownHostException;

public class SSLSocketFactoryWrapper extends SSLSocketFactory {

    private final SSLSocketFactory wrappedFactory;
    private final SSLParameters sslParameters;

    public SSLSocketFactoryWrapper(SSLSocketFactory factory, SSLParameters sslParameters) {
        this.wrappedFactory = factory;
        this.sslParameters = sslParameters;
    }

    @Override
    public Socket createSocket(String host, int port) throws IOException, UnknownHostException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket(host, port);
        setParameters(socket);
        return socket;
    }

    @Override
    public Socket createSocket(String host, int port, InetAddress localHost, int localPort)
            throws IOException, UnknownHostException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket(host, port, localHost, localPort);
        setParameters(socket);
        return socket;
    }


    @Override
    public Socket createSocket(InetAddress host, int port) throws IOException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket(host, port);
        setParameters(socket);
        return socket;
    }

    @Override
    public Socket createSocket(InetAddress address, int port, InetAddress localAddress, int localPort) throws IOException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket(address, port, localAddress, localPort);
        setParameters(socket);
        return socket;

    }

    @Override
    public Socket createSocket() throws IOException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket();
        setParameters(socket);
        return socket;
    }

    @Override
    public String[] getDefaultCipherSuites() {
        return wrappedFactory.getDefaultCipherSuites();
    }

    @Override
    public String[] getSupportedCipherSuites() {
        return wrappedFactory.getSupportedCipherSuites();
    }

    @Override
    public Socket createSocket(Socket s, String host, int port, boolean autoClose) throws IOException {
        SSLSocket socket = (SSLSocket) wrappedFactory.createSocket(s, host, port, autoClose);
        setParameters(socket);
        return socket;
    }

    private void setParameters(SSLSocket socket) {
        socket.setSSLParameters(sslParameters);
    }

}

 

四、参考内容

https://www.cnblogs.com/chengweijun/p/12156790.html

http://javabreaks.blogspot.com/2015/12/java-ssl-handshake-with-server-name.html

https://blog.csdn.net/myle69/article/details/82914770

 

ningzhao
关注
专栏目录
异常记录——Connection reset
chengw315的博客
01-03 2192
连接重置Connection reset 异常java.net.SocketException: Connection reset 详细信息 java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.So...
java https 报错,Java 服务请求 https 接口报错 connection reset 问题
weixin_34650199的博客
03-13 3018
问题现象：和客户对接时，客户请求我们的 https 接口报错 java.net.SocketException：Connection reset。原因分析：根据客户提供的服务异常信息，可以大略猜到是 SSL 的问题。我们这边的 Nginx SSL 配置为 ：ssl_protocols TLSv1.1 TLSv1.2;或者使用命令 nmap --script ssl-enum-ciphers -p ...
Jsoup 无法联网时如何处理
关注移动端、Web、微信小程序等大前端领域
12-02 1737
Jsoup 在无法联网时会在调用get()时抛出 UnkownHostException： try { Connection con = Jsoup.connect(someurl).timeout(10000); Document doc = con.get(); } catch (IOException e) { // 无法连接网络，此处写处理代码
解析connectionReset异常的原因与解决方案
最新发布
微赚开发者技术分享博客
06-29 1760
（ConnectionResetException）通常指的是在尝试通过网络连接发送数据时，远程连接方显式地关闭了连接，或者网络中断导致连接重置。有时候connectionReset异常可能与操作系统或网络设备的配置有关，可以检查操作系统的网络设置、防火墙规则等，确保没有意外的连接关闭行为。这种情况下，操作系统或网络设备可能会强制关闭连接，导致connectionReset异常。确保网络传输过程中的稳定性和可靠性，可以通过优化代码逻辑、增加超时设置或者使用保持活动连接的方式来减少连接被重置的可能性。
https请求报Connection reset问题
m0_37825799的博客
07-10 1万+
背景： 使用HttpsURLconnection或者HttpURLConnection进行https请求时，有时会报Connection reset异常 原因： 这是因为客户端的TLS版本服务端不支持的原因。 对于JDK1.6，支持SSLv2、SSLv3、TLSv1，默认使用TLSv1 对于JDK1.7，支持SSLv2、SSLv3、TLSv1、TLSv1.1、TLSv1.2，默认使...
HTTPS请求报CONNECTION RESET问题,TLS版本服务端不支持
Menahem
06-20 1467
对于JDK1.7，支持SSLv2、SSLv3、TLSv1、TLSv1.1、TLSv1.2，默认使用TLSv1.1。如果服务端配置的支持的SSL协议为TLSv1.2，那么默认情况下只有JDK1.8才能正常的访问.，支持SSLv2、SSLv3、TLSv1、TLSv1.1、TLSv1.2，默认使用。解决：在nginx里修改网站 站点配置文件对应的TLS版本，修改为TLSv1.2。对于JDK1.6，支持SSLv2、SSLv3、TLSv1，默认使用TLSv1。这是因为客户端的TLS版本服务端不支持的原因。
javax.net.ssl.SSLException: java.net.SocketException: Connection reset，Linux环境下https请求失败
weixin_48184989的博客
08-04 5971
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
使用JSoup解析HTML文件
Yoga-jiang
03-11 778
HTML是WEB的核心，互联网中你看到的所有页面都是HTML，不管它们是由JavaScript，JSP，PHP,ASP或者是别的什么WEB技术动态生成的。你的浏览器会去解析HTML并替你去渲染它们。不过如果你需要自己在Java程序中解析HTML文档并查找某些元素，标签，属性或者检查某个特定的元素是否存在的话，那又该如何呢？如果你已经使用Java编程多年了，我相信你肯定试过去解析XML，也使用过类似
DC音乐播放器开发日记
huanglong0438的专栏
04-09 2445
DC音乐播放器开发日记   2016-3-31 总结： 用Application存储系统级的全局变量，这样每个activity都可以得到service的实例，然后对音乐进行操作了 //onBackPressed()会在其它activity在上面的时候也执行（这条是错的，见4-4）   2016-4-1 总结： create table favorite (id integer p
Java面试题
HackAge4619的博客
03-02 6701
N年前收集的面试题都放在这了，Java基础，前端，框架，数据库等…… 针对面试初级 一起复制的，有点乱，自己整理一下吧！ jdbc的原理，怎么连接数据库的？ 需要深入的了解 1）在开发环境中加载指定的数据库驱动程序  导入数据库驱动包 2）加载驱动程序  java程序中，通过Class.forName来指定数据库的驱动程序  3）创建数据库连接对象  通过DriverManager类创建
Android框架——dagger简单实践使用
qq_24442769的博客
02-24 645
一、概述 上一篇主要介绍了dagger的基本使用方法，这篇则分享一下我之前基于mvp架构的项目中的简单应用，也作为一个记录；最近才了解到mvpclean，把dagger的依赖注入部分放在presentation层比较合适。 二、dagger架构思路 由于还不算很熟悉dagger，所以只是将网络请求的httpclient以及数据库采用依赖注入的方式，并没有涉及到一些对象的作用域甚至包含、继...
读书笔记-Java网络编程安全教程-微课版
u011225581的专栏
01-10 393
Java网络编程安全教程 清华大学出版社 ISBN-9787302482833 仅供参考, 自建索引, 以备后查 ================================================================= 一: IO流 java.io.* java.nio.* java.nio.channels.* InputStream OutputStream 字节流 Reader Writer 字符流 InputStreamReade...
建立https链接的SLL验证证书失效问题
Inner Peace
10-14 7942
爬取网页遇到的目标站点证书不合法问题。 使用jsoup爬取解析网页时，出现了如下的异常情况。 javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBu
请求https提示connect rest
DamonLe的博客
03-04 3297
最近公司项目升级，进行系统改造，调用https接口第一天接口程序调用能够正常返回信息，隔天再次调用接口提示 connect rest，经过多方查找论证，这是因为客户端的TLS版本服务端不支持的原因(客户端ssl版本落后于服务端ssl版本) 解决方法：将项目编译版本提升至于服务端相匹配的jdk环境 转载：https://blog.csdn.net/m0_37825799/article/details/95367343 ps在此过程中可能遇到升级编译版本jdk报错问题 转...
soapui测试https报connection reset的解决方案
nongyan90的专栏
11-22 7267
这个问题搞了我几天，想死的心都有了！搞好了，心情也顿时舒畅了。 报这个异常，具体原因肯定有好多，记录一下我这个原因，以便给后来人提供一种参考 or 借鉴 or 思路 java.net.socketException:connection reset 报这个异常时因为客户端与服务器，有一方已经断开链接，另一方还在读。 我们服务器使用了 linux操作系统 nginx代理
Http请求并发高时报Connection reset错误
没有伞的孩子必须努力奔跑！—小林
07-20 3666
1、报错信息： 2021-07-14 10:03:16,318 ERROR [WebContainer : 39] [com.elitecrm.webchat.util.PoolingHttpService] - java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:221) ~[?:1.8.0] at java.net.SocketInputStre
javax.net.ssl.SSLHandshakeException:(HTTPS绕过安全认证进行请求链接)
qq_41615095的博客
02-20 1357
1、问题： 在使用jsoup爬取数据时，碰到了使用https的网站，遇到报错:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed 2、解决 后自己在网上查了一些资料,找到相关解决代码: package com
Connection reset原因分析及解决思路
热门推荐
大叶子不小的博客
02-09 2万+
Connection reset原因分析及解决思路 - lwli - 博客园 我们在开发过程中经常会出现Connection reset问题，包括http调用，数据库连接等场景。出现Connection reset的原因很多，本文从tcp层面简单介绍下Connection reset出现的原因，以及在实际开发过程中如何排查这类问题。 1、什么是Connection reset 　　在TCP首部中有6个标志位，其中一个标志位为RST，用于“复位”的。无论何时一个报文 段发往基准的连接（ referenc
jsoup Connection reset
08-13
您好！遇到“jsoup Connection reset”错误通常是由于网络连接问题引起的。这个错误意味着与服务器之间的连接被意外地中断了。 有几种可能的原因导致这个错误，您可以尝试以下解决方法： 1. 检查网络连接：确保您的网络连接正常，并且没有任何阻碍访问目标网站的防火墙或代理设置。 2. 增加连接超时时间：使用Jsoup时，可以通过设置连接超时时间来增加与服务器建立连接的时间。例如：`Connection.timeout(5000)`，将超时时间设置为5秒。 3. 降低请求频率：如果您的程序发送了大量的请求，服务器可能会将您的IP地址列为异常访问者并且拒绝连接。您可以尝试减少请求的频率，或者在每次请求之间增加延迟时间。 4. 使用代理服务器：如果访问目标网站受到限制，您可以尝试使用代理服务器来进行连接。Jsoup支持设置代理服务器，例如：`Connection.proxy(proxy)`，其中`proxy`是代理服务器的地址和端口。 5. 更新Jsoup版本：确保您正在使用最新版本的Jsoup库，以便获得最新的bug修复和改进。 如果以上方法都无法解决问题，可能是因为目标网站有特殊的防护机制或限制访问。在这种情况下，您可以尝试与网站管理员联系，了解是否有任何限制或要求。 希望这些解决方法能帮助您解决问题！如果您有更多的问题，请随时提问。
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值