jsoup请求头添加cookie_Jsoup的源码修改-支持Cookie等头信息的设置 - Java综合 - Java - ......

Jsoup是一个开源的html解析工具，使用css子选择器的语法解析元素，比古老的HtmlParser好用很多。在最新版本中，不支持设置Cookie等头信息来访问url，这样一些需要设置Cookie的网页就无法抓取了。为方便使用，我稍微修改了源码。

版本1.2.3中，访问url的api为：

Java代码

String url ="";

Document doc = Jsoup.parse(newURL(url),3000);

String url = "";

Document doc = Jsoup.parse(new URL(url), 3000);

不能设置Cookie等头信息。

下面我们进行源码的修改。先浏览源码：

Jsoup.parse的源码为：

Java代码

publicstaticDocument parse(URL url,inttimeoutMillis)throwsIOException {

returnDataUtil.load(url, timeoutMillis);

}

public static Document parse(URL url, int timeoutMillis) throws IOException {

return DataUtil.load(url, timeoutMillis);

}

DataUtil.load的源码为：

Java代码

staticDocument load(URL url,inttimeoutMillis)throwsIOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"),"Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

conn.connect();

static Document load(URL url, int timeoutMillis) throws IOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"), "Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

conn.connect();

很明显，调用HttpURLConnection.setRequestProperty来设置头信息即可。

我在DataUtil.java中重载了load函数，并抽取了一个公用方法，如下：

Java代码

staticDocument load(URL url, Map requestMap,inttimeoutMillis)throwsIOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"),"Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

// set request prop

for(String key : requestMap.keySet()) {

conn.setRequestProperty(key, requestMap.get(key));

}

conn.connect();

returnload(url, conn);

}

staticDocument load(URL url,inttimeoutMillis)throwsIOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"),"Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

conn.connect();

returnload(url, conn);

}

privatestaticDocument load(URL url, HttpURLConnection conn)throwsIOException {

intres = conn.getResponseCode();

if(res != HttpURLConnection.HTTP_OK)

thrownewIOException(res +" error loading URL "+ url.toString());

String contentType = conn.getContentType();

if(contentType ==null|| !contentType.startsWith("text/"))

thrownewIOException(String.format("Unhandled content type \"%s\" on URL %s. Must be text/*",

contentType, url.toString()));

InputStream inStream =newBufferedInputStream(conn.getInputStream());

String charSet = getCharsetFromContentType(contentType);// may be null, readInputStream deals with it

Document doc = readInputStream(inStream, charSet, url.toExternalForm());

inStream.close();

returndoc;

}

static Document load(URL url, Map requestMap, int timeoutMillis) throws IOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"), "Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

// set request prop

for (String key : requestMap.keySet()) {

conn.setRequestProperty(key, requestMap.get(key));

}

conn.connect();

return load(url, conn);

}

static Document load(URL url, int timeoutMillis) throws IOException {

String protocol = url.getProtocol();

Validate.isTrue(protocol.equals("http") || protocol.equals("https"), "Only http & https protocols supported");

HttpURLConnection conn = (HttpURLConnection) url.openConnection();

conn.setInstanceFollowRedirects(true);

conn.setConnectTimeout(timeoutMillis);

conn.setReadTimeout(timeoutMillis);

conn.connect();

return load(url, conn);

}

private static Document load(URL url, HttpURLConnection conn) throws IOException {

int res = conn.getResponseCode();

if (res != HttpURLConnection.HTTP_OK)

throw new IOException(res + " error loading URL " + url.toString());

String contentType = conn.getContentType();

if (contentType == null || !contentType.startsWith("text/"))

throw new IOException(String.format("Unhandled content type \"%s\" on URL %s. Must be text/*",

contentType, url.toString()));

InputStream inStream = new BufferedInputStream(conn.getInputStream());

String charSet = getCharsetFromContentType(contentType); // may be null, readInputStream deals with it

Document doc = readInputStream(inStream, charSet, url.toExternalForm());

inStream.close();

return doc;

}

我在Jsoup.java中重载了parse函数：

Java代码

publicstaticDocument parse(URL url, Map requestMap,inttimeoutMillis)throwsIOException {

returnDataUtil.load(url, requestMap, timeoutMillis);

}

public static Document parse(URL url, Map requestMap, int timeoutMillis) throws IOException {

return DataUtil.load(url, requestMap, timeoutMillis);

}

完成以上修改了，编写ant脚本打jar包即可。

调用例子：

Java代码

String url ="";

Map requestMap =newHashMap();

requestMap.put("Cookie","");

Document doc = Jsoup.parse(newURL(url), requestMap,3000);

String url = "";

Map requestMap = new HashMap();

requestMap.put("Cookie", "");

Document doc = Jsoup.parse(new URL(url), requestMap, 3000);

这样就可以设置Cookie信息了，可解决需要登录验证的页面爬取问题。