When I try to get document from a website using jsoup, I got the error after seconds of stucking.
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:453)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:434)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:181)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:170)
at com.yystar.crown.adapter.service.CrownScraper.main(CrownScraper.java:12)
Code is as simple as following:
public static void main(String[] args) {
String url = "http://xxx.com/";
try {
Document doc = Jsoup.connect(url).get();
System.out.println(doc.data());
} catch (IOException e) {
e.printStackTrace();
}
}
What could cause this error? Given that we can access the page normally using browser.
After comparison with browser's HTTP request header(METHODOLOGY), I set the USER AGENT and issue the request again, surprised, the server responded with the right document. The following snippet works:
public static void main(String[] args) {
String url = "http://xxx.com/xxx";
try {
Document doc = Jsoup
.connect(url)
.followRedirects(true)
.referrer("http://xxx.com/xxx")
.userAgent(
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36")
.get();
System.out.println(doc.data());
} catch (IOException e) {
e.printStackTrace();
}
}
Note that the user agent value copied from browser developer tool.