有的网站限制网络爬虫的抓取,例如javaeye。会出现错误提示:
您可能使用了网络爬虫抓取JavaEye网站页面!
<p>JavaEye网站不允许您使用网络爬虫对JavaEye进行恶意的网页抓取,请您立刻停止该抓取行为!</p>
<p>如果您的网络爬虫不属于恶意抓取行为,希望JavaEye网站允许你进行网页抓取,请和JavaEye管理员联系,取得授权: webmaster<img src='/images/email.gif' alt="Email" />support.iteye.com</p>
<p>如果您确实使用浏览器访问,但是被错误的识别为网络爬虫,请将您浏览器发送的“User Agent”信息告知我们,帮助我们解决错误: webmaster<img src='/images/email.gif' alt="Email" />support.iteye.com</p>
呵呵,不过可以设置下链接的参数。
URL url=new URL(pathString);
URLConnection con=url.openConnection();
con.setDoOutput(true);
con.setRequestProperty("User-Agent", "");
加上上面红色的一句,就可以了。
或者使用httpclient是,添加
HttpClient httpClient=new HttpClient();
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000);
httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5");
这样就可以了。
--------------------
我的部分代码参考:
private HttpClient httpClient = null;
public SuNingHtmlparseUtil(String host){
if(httpClient == null){
httpClient = new HttpClient();
}
httpClient.getHostConfiguration().setHost(host, 80, "http");
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000);
httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5");
}
public SuNingHtmlparseUtil(){
if(httpClient == null){
httpClient = new HttpClient();
}
httpClient.getHostConfiguration().setHost("", 80, "http");
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000);
httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5");
}
/**
* 返回String流
* @param url
* @return
* @throws Exception
*/
public String getConnectAsString(String url,String charset) throws Exception {
GetMethod get = new GetMethod(url);
httpClient.executeMethod(get);
if("utf-8".equals(charset.toLowerCase())){
return new String(get.getResponseBodyAsString().getBytes("iso-8859-1"),charset);
}else{
return new String(get.getResponseBodyAsString().getBytes(),charset);
}
}