java调用spark_详解Java编写并运行spark应用程序的方法

我们首先提出这样一个简单的需求：

现在要分析某网站的访问日志信息，统计来自不同IP的用户访问的次数，从而通过Geo信息来获得来访用户所在国家地区分布状况。这里我拿我网站的日志记录行示例，如下所示：

121.205.198.92 - - [21/Feb/2014:00:00:07 +0800] "GET /archives/417.html HTTP/1.1" 200 11465 "http://shiyanjun.cn/archives/417.html/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"

121.205.198.92 - - [21/Feb/2014:00:00:11 +0800] "POST /wp-comments-post.php HTTP/1.1" 302 26 "http://shiyanjun.cn/archives/417.html/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0"

121.205.198.92 - - [21/Feb/2014:00:00:12 +0800] "GET /archives/417.html/ HTTP/1.1" 301 26 "http://shiyanjun.cn/archives/417.html/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"

121.205.198.92 - - [21/Feb/2014:00:00:12 +0800] "GET /archives/417.html HTTP/1.1" 200 11465 "http://shiyanjun.cn/archives/417.html" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"

121.205.241.229 - - [21/Feb/2014:00:00:13 +0800] "GET /archives/526.html HTTP/1.1" 200 12080 "http://shiyanjun.cn/archives/526.html/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"

121.205.241.229 - - [21/Feb/2014:00:00:15 +0800] "POST /wp-comments-post.php HTTP/1.1" 302 26 "http://shiyanjun.cn/archives/526.html/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0"

Java实现Spark应用程序(Application)

我们实现的统计分析程序，有如下几个功能点：

从HDFS读取日志数据文件

将每行的第一个字段(IP地址)抽取出来

统计每个IP地址出现的次数

根据每个IP地址出现的次数进行一个降序排序

根据IP地址，调用GeoIP库获取IP所属国家

打印输出结果，每行的格式：[国家代码] IP地址 频率

下面，看我们使用Java实现的统计分析应用程序代码，如下所示：

package org.shirdrn.spark.job;

import java.io.File;

import java.io.IOException;

import java.util.Arrays;

import java.util.Collections;

import java.util.Comparator;

import java.util.List;

import java.util.regex.Pattern;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.apache.spark.api.java.JavaPairRDD;

import org.apache.spark.api.java.JavaRDD;

import org.apache.spark.api.java.JavaSparkContext;

import org.apache.spark.api.java.function.FlatMapFunction;

import org.apache.spark.api.java.function.Function2;

import org.apache.spark.api.java.function.PairFunction;

import org.shirdrn.spark.job.maxmind.Country;

import org.shirdrn.spark.job.maxmind.LookupService;

import scala.Serializable;

import scala.Tuple2;

public class IPAddressStats implements Serializable {

private static final long serialVersionUID = 8533489548835413763L;

private static final Log LOG = LogFactory.getLog(IPAddressStats.class);

private static final Pattern SPACE = Pattern.compile(" ");

private transient LookupService lookupService;

private transient final String geoIPFile;

public IPAddressStats(String geoIPFile) {

this.geoIPFile = geoIPFile;

try {

// lookupService: get country code from a IP address

File file = new File(this.geoIPFile);

LOG.info("GeoIP file: " + file.getAbsolutePath());

lookupService = new AdvancedLookupService(file, LookupService.GEOIP_MEMORY_CACHE);

} catch (IOException e) {

throw new RuntimeException(e);

}

}

@SuppressWarnings("serial")

public void stat(String[] args) {

JavaSparkContext ctx = new JavaSparkContext(args[0], "IPAddressStats",

System.getenv("SPARK_HOME"), JavaSparkContext.jarOfClass(IPAddressStats.class));

JavaRDD lines = ctx.textFile(args[1], 1);

// splits and extracts ip address filed

JavaRDD words = lines.flatMap(new FlatMapFunction() {

@Override

public Iterable call(String s) {

// 121.205.198.92 - - [21/Feb/2014:00:00:07 +0800] "GET /archives/417.html HTTP/1.1" 200 11465 "http://shiyanjun.cn/archives/417.html/" "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"

// ip address

return Arrays.asList(SPACE.split(s)[0]);

}

});

// map

JavaPairRDD ones = words.map(new PairFunction() {

@Override

public Tuple2 call(String s) {

return new Tuple2(s, 1);

}

});

// reduce

JavaPairRDD counts = ones.reduceByKey(new Function2() {

@Overri