java 抓取_java抓取网站

作业要求：

1.给定一个网站的URL，抓取下整个网站，包括网页、图片等等。

2.对该网站的所有对象进行一个统计，比如链接、图片、按钮等等分别有多少。

我上次提交的

import java.io.BufferedReader;

import java.io.IOException;

import java.io.InputStreamReader;

import java.net.MalformedURLException;

import java.net.URL;

import java.net.URLConnection;

public class UrlContent {

public static String getHtmlSource(String url)

{

StringBuffer codeBuffer = null;

BufferedReader in=null;

try

{

URLConnection uc = new URL(url).openConnection();

// 读取url流内容

in = new BufferedReader(new InputStreamReader(uc

.getInputStream(), "gb2312"));

codeBuffer = new StringBuffer();

String tempCode = "";

// 把buffer内的值读取出来

while ((tempCode = in.readLine()) != null)

{

codeBuffer.append(tempCode).append("\n");

}

in.close();

}

catch (MalformedURLException e)

{

e.printStackTrace();

}

catch (IOException e)

{

e.printStackTrace();

}

return codeBuffer.toString();

}

public static void main(String[] args)

{

System.out.println(UrlContent.getHtmlSource("http://sports.163.com/zc/"));

}

}

一个站还是一个页啊,一个站有千八百页,那还有办法啊.?要从网页的URL取得另一页,另一页又有,................. URL url; HttpURLConnection conn=null; try{ url=new URL("http://baike.baidu.com/view/2125344.htm"); conn=(HttpURLConnection)url.openConnection(); conn.setRequestProperty("Accept-Charset","utf-8"); conn.setRequestProperty("contentType","utf-8"); BufferedReader reader=new BufferedReader(new InputStreamReader(conn .getInputStream(),"gbk")); String downloadURL; StringBuffer sb=new StringBuffer(); while((downloadURL=reader.readLine())!=null){ sb.append(downloadURL); } System.out.println(sb.toString()); 这样就可以打印出这个网页了,至于里用正则如何替换,找出按钮,图片的,那还得再弄下. }catch(Exception e){ }finally{ conn.disconnect(); }