今天看到一篇关于简单爬虫的代码,本来只是想简单看看写法,却看到了不懂的语法:
public static void crawler(String startingURL) {
LinkedList<String> listOfPendingURLs = new LinkedList<>();
HashSet<String> listOfTraversedURLs = new HashSet<>();
listOfPendingURLs.add(startingURL);
while (!listOfPendingURLs.isEmpty() && listOfTraversedURLs.size() <= 100) {
String url = listOfPendingURLs.pop();
if (!listOfTraversedURLs.contains(url)) {
listOfTraversedURLs.add(url);
System.out.println("Craw " + url);
getSubURLs(url).stream().filter(s -> !listOfTraversedURLs.contains(s)).forEach(listOfPendingURLs::push);
}
}
}
getSubURLs(url).stream().filter(s -> !listOfTraversedURLs.contains(s)).forEach(listOfPendingURLs::push);
其中这句代码,看到双冒号的时候第一时间想起了C语言里面的用类名加双冒号引用方法的用法,但是又不确定,看到forEach又觉得是个for循环的,后来查了资料,确实跟自己想的差不多。不过因为很少接触这种写法,还是不太熟悉。资料如下:
首先是前面的stream()、filter():
解释的还是比较好懂的。
接下来就是后边的双冒号,看到知乎上有这么一个问题:
这样一看就很清晰了,原来还是把for循环换了一种写法。评论下有一个比较简单的解释:
大神的解释有点复杂就不放出来了,链接如下:
以下是完整的代码:
package Chapter_21;
import java.util.*;
/**
* (Web crawler)
* Rewrite Listing 12.18, WebCrawler.java, to improve the performance
* by using appropriate new data structures for listOfPendingURLs
* and listofTraversedURLs.
*
* Created by Luiz Arantes Sa on 1/30/15.
*/
public class Exercise_14 {
public static void main(String[] args) {
java.util.Scanner input = new java.util.Scanner(System.in);
System.out.print("Enter a URL: ");
String url = input.nextLine();
crawler(url); // Traverse the Web from the a starting url
}
public static void crawler(String startingURL) {
LinkedList<String> listOfPendingURLs = new LinkedList<>();
HashSet<String> listOfTraversedURLs = new HashSet<>();
listOfPendingURLs.add(startingURL);
while (!listOfPendingURLs.isEmpty() && listOfTraversedURLs.size() <= 100) {
String url = listOfPendingURLs.pop();
if (!listOfTraversedURLs.contains(url)) {
listOfTraversedURLs.add(url);
System.out.println("Craw " + url);
getSubURLs(url).stream().filter(s -> !listOfTraversedURLs.contains(s)).forEach(listOfPendingURLs::push);
}
}
}
public static ArrayList<String> getSubURLs(String urlString) {
ArrayList<String> list = new ArrayList<>(1000);
try {
java.net.URL url = new java.net.URL(urlString);
Scanner input = new Scanner(url.openStream());
int current = 0;
while (input.hasNext()) {
String line = input.nextLine();
current = line.indexOf("http:", current);
while (current > 0) {
int endIndex = line.indexOf("\"", current);
if (endIndex > 0) { // Ensure that a correct URL is found
list.add(line.substring(current, endIndex));
current = line.indexOf("http:", endIndex);
}
else
current = -1;
}
}
}
catch (Exception ex) {
System.out.println("Error: " + ex.getMessage());
}
return list;
}
}
更新补充一篇关于Java 8 Lambda表达式的文章: