java爬虫 之 搜狐新闻爬虫（一）

最近开始学习java爬虫，网上很多教程，自己找的时候花了好久的时间去理解别人的思路。

打算将自己最近的学习进度稍作整理，理清思路。

主要工具使用jsoup：具体用法看http://blog.csdn.net/u012315428/article/details/51135640

下面是获取一个网页中所有的超链接：

package com.sohu;

import org.jsoup.Jsoup;

import java.io.IOException;
import java.util.*;

import org.jsoup.*;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
/*
 * author :cj
 * 找到所有的超链接
 */
public class FindAllUrl {

	public static void main(String[] args) {
		// TODO Auto-generated method stub
		try {
			Document doc = Jsoup.connect("http://news.sohu.com/").get();
			Elements links = doc.select("a[href]");
			for(Element link : links){
				String StrUrl = link.attr("abs:href");
				if(StrUrl.startsWith("http://news.sohu.com/"))//以。。。开头的url
				System.out.println(StrUrl);
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
	}

}

运行结果
http://news.sohu.com/
http://news.sohu.com/mindiao/
http://news.sohu.com/scroll/
http://news.sohu.com/mindiao/
http://news.sohu.com/special.shtml
http://news.sohu.com/guoneixinwen.shtml
http://news.sohu.com/shehuixinwen.shtml
http://news.sohu.com/guojixinwen.shtml
http://news.sohu.com/matrix/
http://news.sohu.com/newsmaker_list/
http://news.sohu.com/photo/
http://news.sohu.com/wurenji/
http://news.sohu.com/#
http://news.sohu.com/#
http://news.sohu.com/#
http://news.sohu.com/20160414/n444127123.shtml
http://news.sohu.com/20160414/n444127800.shtml
http://news.sohu.com/20160414/n444193395.shtml
http://news.sohu.com/20160414/n444148450.shtml
http://news.sohu.com/20160414/n444133304.shtml
http://news.sohu.com/20160414/n444199124.shtml
http://news.sohu.com/20160413/n444107224.shtml
http://news.sohu.com/20160414/n444127800.shtml
http://news.sohu.com/20160413/n444105842.shtml
http://news.sohu.com/20160414/n444140620.shtml
http://news.sohu.com/20160414/n444126073.shtml
http://news.sohu.com/20160413/n444086783.shtml
http://news.sohu.com/20160414/n444187234.shtml
http://news.sohu.com/20160414/n444193015.shtml
http://news.sohu.com/20160414/n444207393.shtml
http://news.sohu.com/20160414/n444148450.shtml
http://news.sohu.com/20160414/n444193395.shtml