java爬虫新闻网站_java爬虫 之 搜狐新闻爬虫（一）

最近开始学习java爬虫，网上很多教程，自己找的时候花了好久的时间去理解别人的思路。

打算将自己最近的学习进度稍作整理，理清思路。

主要工具使用jsoup：具体用法看http://blog.csdn.net/u012315428/article/details/51135640

下面是获取一个网页中所有的超链接：

package com.sohu;

import org.jsoup.Jsoup;

import java.io.IOException;

import java.util.*;

import org.jsoup.*;

import org.jsoup.nodes.Document;

import org.jsoup.nodes.Element;

import org.jsoup.select.Elements;

/*

* author :cj

* 找到所有的超链接

*/

public class FindAllUrl {

public static void main(String[] args) {

// TODO Auto-generated method stub

try {

Document doc = Jsoup.connect("http://news.sohu.com/").get();

Elements links = doc.select("a[href]");

for(Element link : links){

String StrUrl = link.attr("abs:href");

if(StrUrl.startsWith("http://news.sohu.com/"))//以。。。开头的url

System.out.println(StrUrl);

}

} catch (IOException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

}

}

运行结果 http://news.sohu.com/ http://news.sohu.com/mindiao/ http://news.sohu.com/scroll/ http://news.sohu.com/mindiao/ http://news.sohu.com/special.shtml http://news.sohu.com/guoneixinwen.shtml http://news.sohu.com/shehuixinwen.shtml http://news.sohu.com/guojixinwen.shtml http://news.sohu.com/matrix/ http://news.sohu.com/newsmaker_list/ http://news.sohu.com/photo/ http://news.sohu.com/wurenji/ http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/20160414/n444127123.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160414/n444193395.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444133304.shtml http://news.sohu.com/20160414/n444199124.shtml http://news.sohu.com/20160413/n444107224.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160413/n444105842.shtml http://news.sohu.com/20160414/n444140620.shtml http://news.sohu.com/20160414/n444126073.shtml http://news.sohu.com/20160413/n444086783.shtml http://news.sohu.com/20160414/n444187234.shtml http://news.sohu.com/20160414/n444193015.shtml http://news.sohu.com/20160414/n444207393.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444193395.shtml