最近开始学习java爬虫,网上很多教程,自己找的时候花了好久的时间去理解别人的思路。
打算将自己最近的学习进度稍作整理,理清思路。
主要工具使用jsoup:具体用法看http://blog.csdn.net/u012315428/article/details/51135640
下面是获取一个网页中所有的超链接:
package com.sohu;
import org.jsoup.Jsoup;
import java.io.IOException;
import java.util.*;
import org.jsoup.*;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
/*
* author :cj
* 找到所有的超链接
*/
public class FindAllUrl {
public static void main(String[] args) {
// TODO Auto-generated method stub
try {
Document doc = Jsoup.connect("http://news.sohu.com/").get();
Elements links = doc.select("a[href]");
for(Element link : links){
String StrUrl = link.attr("abs:href");
if(StrUrl.startsWith("http://news.sohu.com/"))//以。。。开头的url
System.out.println(StrUrl);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
运行结果 http://news.sohu.com/ http://news.sohu.com/mindiao/ http://news.sohu.com/scroll/ http://news.sohu.com/mindiao/ http://news.sohu.com/special.shtml http://news.sohu.com/guoneixinwen.shtml http://news.sohu.com/shehuixinwen.shtml http://news.sohu.com/guojixinwen.shtml http://news.sohu.com/matrix/ http://news.sohu.com/newsmaker_list/ http://news.sohu.com/photo/ http://news.sohu.com/wurenji/ http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/# http://news.sohu.com/20160414/n444127123.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160414/n444193395.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444133304.shtml http://news.sohu.com/20160414/n444199124.shtml http://news.sohu.com/20160413/n444107224.shtml http://news.sohu.com/20160414/n444127800.shtml http://news.sohu.com/20160413/n444105842.shtml http://news.sohu.com/20160414/n444140620.shtml http://news.sohu.com/20160414/n444126073.shtml http://news.sohu.com/20160413/n444086783.shtml http://news.sohu.com/20160414/n444187234.shtml http://news.sohu.com/20160414/n444193015.shtml http://news.sohu.com/20160414/n444207393.shtml http://news.sohu.com/20160414/n444148450.shtml http://news.sohu.com/20160414/n444193395.shtml