java爬虫（一）--下载html

需要的包Jsoup1.6.jar

效果图：

第一步、先建一个类，将操作部分封装，传入网址u和目录mkd

public class DownLoadPage {
    public static  void downloadPage(String u,String mkd){
    }
}

1.1 传入的网址不一定是域名格式，需要我们解析成域名

URL url=new URL(u);
String uri=url.getHost();

1.2 获取网页标题，使用DOM解析，有的网页标题含有特殊符号需要正则表达式匹配出来替换，这里就不过多讲解

Document doc=Jsoup.connect(u).get();
Elements ele=doc.getElementsByTag("title");
String title=ele.text();

1.3 创建目录

        File mk=new File(mkd);
        if(mk.exists()){
            System.out.println("目录"+mk+"已经存在");
        }else{
            mk.mkdir();
            System.out.println("创建目录"+mk+"成功");
        }

1.4 url.openStream()获取网页内容，存放br里，编码和网页的编码要相同，从br里读出来再存入到bw里，最后关闭流释放资源

    try {
            BufferedReader br=new BufferedReader(new InputStreamReader(url.openStream(),"utf8"));
            BufferedWriter bw=new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File(mk+"/"+title+".html")),"utf8"));
            String line;
            while((line=br.readLine())!=null){
                bw.write(line);
            }
            br.close();
            bw.close();
            System.out.println("下载完毕！");
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }

二、完整代码

package test;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;

public class DownLoadPage {
    public static  void downloadPage(String u,String mkd) throws UnsupportedEncodingException, IOException{
        URL url=new URL(u);
        String uri=url.getHost();
        System.out.println("URI:"+uri);
        Document doc=Jsoup.connect(u).get();
        Elements ele=doc.getElementsByTag("title");
        String title=ele.text();
        System.out.println("网页标题："+title);
        File mk=new File(mkd);
        if(mk.exists()){
            System.out.println("目录"+mk+"已经存在");
        }else{
            mk.mkdir();
            System.out.println("创建目录"+mk+"成功");
        }
        System.out.println("MK:"+mk);
        try {
            BufferedReader br=new BufferedReader(new InputStreamReader(url.openStream(),"utf8"));
            BufferedWriter bw=new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File(mk+"/"+title+".html")),"utf8"));
            String line;
            while((line=br.readLine())!=null){
                bw.write(line);
            }
            br.close();
            bw.close();
            System.out.println("下载完毕！");
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }
    }
}

main方法

package test;

public class TestPc {
    public static void main(String[] args) throws Exception {
        DownLoadPage.downloadPage("http://www.xuecheyi.com/Info/show-27818.html","D:/新建网页");
    }
}

三、需要的知识点
1.目录和文件的创建
2.文件流的操作（输入流、输出流、字节流、字符流）
3.面向对象的知识，封装、继承、多态
4.xml解析方法