1.准备工作
需要下载两个jar包:commons-httpclient和commons-codes
commons-httpclient下载地址:http://hc.apache.org/downloads.cgi 4.0版的现在还没有正式版,我下的是3.1版的
commons-codes下载地址:http://commons.apache.org/downloads/download_codec.cgi
2.获取一个网页
HttpClient httpClient = new HttpClient(); GetMethod getMethod = new GetMethod("http://www.baidu.com/"); try { int statusCode = httpClient.executeMethod(getMethod); if (statusCode != HttpStatus.SC_OK) { System.err.println("Method failed: " + getMethod.getStatusLine()); } // 读取内容 byte[] responseBody = getMethod.getResponseBody(); // 处理内容
String html = new String(responseBody);
System.out.println(html); } catch (Exception e) { System.err.println("页面无法访问"); } getMethod.releaseConnection();
⑴说明一
GetMethod getMethod = new GetMethod(http://www.baidu.com/);
这是一个完整的网页路径请求,也可以先设置Host,然后使用路径直接访问:
httpClient.getHostConfiguration().setHost("www.baidu.com", 80);
GetMethod getMethod = new GetMethod("/index.htm");
⑵说明二
// 读取内容
byte[] responseBody = getMethod.getResponseBody();
// 处理内容
html = new String(responseBody);
如果只是简单的获取页面html,可以直接这么使用:
html = getMethod.getResponseBodyAsString();
三.以Post的方式获取网页
上面的例子是以Get的方式获取网页,这个例子是以Post的方式
HttpClient httpClient = new HttpClient();
PostMethod postMethod = new PostMethod("http://127.0.0.1:8080/site/page.jsp");
postMethod.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,new DefaultHttpMethodRetryHandler());
NameValuePair[] data = { new NameValuePair("username", "aaa"),new NameValuePair("password", "bbb") };
postMethod.setRequestBody(data);
try {
int statusCode = httpClient.executeMethod(postMethod);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: "
+ postMethod.getStatusLine());
}
//读取内容
byte[] responseBody = postMethod.getResponseBody();
//处理内容
String html = new String(responseBody);
System.out.println(html);
} catch (Exception e) {
System.err.println("页面无法访问");
}
postMethod.releaseConnection();
这个例子传递了两个Post参数:username为aaa,password为bbb,传递给页面http://127.0.0.1:8080/site/page.jsp
4.获取经过gzip的网页
HttpClient httpClient = new HttpClient();
httpClient.getHostConfiguration().setHost("127.0.0.1", 8080);
GetMethod getMethod = new GetMethod("/page.jsp");
String acceptEncoding = "";
if (getMethod.getResponseHeader("Content-Encoding") != null)
acceptEncoding = getMethod.getResponseHeader("Content-Encoding")
.getValue();
if (acceptEncoding.toLowerCase().indexOf("gzip") > -1) {
getMethod.setRequestHeader("Accept-Encoding", "gzip, deflate");
try {
int statusCode = httpClient.executeMethod(getMethod);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: "
+ getMethod.getStatusLine());
}
InputStream is = getMethod.getResponseBodyAsStream();
GZIPInputStream gzin = new GZIPInputStream(is);
InputStreamReader isr = new InputStreamReader(gzin, "utf-8"); // 设置读取流的编码格式,自定义编码
java.io.BufferedReader br = new java.io.BufferedReader(isr);
StringBuffer sb = new StringBuffer();
String tempbf;
while ((tempbf = br.readLine()) != null) {
sb.append(tempbf);
sb.append("\r\n");
}
isr.close();
gzin.close();
String html = sb.toString();
System.out.println(html);
} catch (Exception e) {
System.err.println("页面无法访问");
}
getMethod.releaseConnection();
5.获取图片或者其他二进制文件
HttpClient httpClient = new HttpClient();
GetMethod getMethod = new GetMethod("http://127.0.0.1:8080/site/file.gif");
try {
InputStream inputStream = getMethod.getResponseBodyAsStream();
// 这里处理 inputStream
} catch (Exception e) {
System.err.println("页面无法访问");
}
getMethod.releaseConnection();
6.解析获取的html
比较优秀的解析方式有htmlparser或者nekohtml,需要了解的请参考相关资料
7.模拟登录网站
很多站点都是需要身份验证的,通过简单的设置cookie可以达到这样的效果:
getMethod.addRequestHeader("Cookie","cookiename1=aa;cookiename2=bb");
当然也可以以Post登录系统的时候,获取cookie,动态的写上去,具体的就先不写了,这篇文章写的好累,先到这里吧。
参考资料:http://www.ibm.com/developerworks/cn/opensource/os-httpclient/
http://www.cnblogs.com/yesun/archive/2008/10/31/1323432.html