Spark二次排序Java示例讲解

示例文件内容（ps：为什么给这么一大串的内容呢，没错，就是为了大家真正看懂其中的奥秘，哈哈哈）：

1 23
3 22
3 31
1 12
2 11
4 45
1 2
1 89
2 56
2 6
2 9
3 56
3 2
3 8
3 16
4 66
4 3
4 8

什么时候使用spark二次排序？

当你的项目中需要对多个字段进行排序的时候，你可以使用二次排序来解决。

先按第一列排序，若第一列相同按照第二列排序，依次类推。

讲清楚为什么使用二次排序以后，接下来用一个java的小demo来进行测试验证，大家也可以copy到本地进行测试

Firstly：创建SecondarySort类

package com.leo.OrderedTest;

import java.io.Serializable;

import scala.math.Ordered;

public class SecondarySort implements Ordered<SecondarySort>,Serializable{
	
	private static final long serialVersionUID = 1L;
	//自定义二次排序的key
	private int first;
	private int second;
	
	public int getFirst() {
		return first;
	}
	public void setFirst(int first) {
		this.first = first;
	}
	public int getSecond() {
		return second;
	}
	public void setSecond(int second) {
		this.second = second;
	}
	public SecondarySort(int first, int second) {
		super();
		this.first = first;
		this.second = second;
	}
	
	@Override
	public int hashCode() {
		final int prime = 31;
		int result = 1;
		result = prime * result + first;
		result = prime * result + second;
		return result;
	}
	@Override
	public boolean equals(Object obj) {
		if (this == obj)
			return true;
		if (obj == null)
			return false;
		if (getClass() != obj.getClass())
			return false;
		SecondarySort other = (SecondarySort) obj;
		if (first != other.first)
			return false;
		if (second != other.second)
			return false;
		return true;
	}
	
	
	
	@Override
	public boolean $greater(SecondarySort other) {
		if(this.getFirst() > other.getFirst()) {
			return true;
		}else if (this.getFirst() == other.getFirst() && this.getSecond() > other.getSecond()) {
			return true;
		}
		return false;
	}
	@Override
	public boolean $greater$eq(SecondarySort other) {
		if($greater(other) || this.getFirst() == other.getFirst() && this.getSecond() == other.getSecond()) {
			return true;
		}
		return false;
	}
	@Override
	public boolean $less(SecondarySort other) {
		if(this.getFirst() < other.getFirst()) {
			return true;
		}else if(this.getFirst() == other.getFirst() && this.getSecond() < other.getSecond()) {
			return true;
		}
		return false;
	}
	@Override
	public boolean $less$eq(SecondarySort other) {
		if($less(other) || this.getFirst() == other.getFirst() && this.getSecond() == other.getSecond()) {
			return true;
		}
		return false;
	}
	@Override
	public int compare(SecondarySort other) {
		if(this.getFirst() != other.getFirst()) {
			return this.getFirst() - other.getFirst();
		} else {
			return this.getSecond() - other.getSecond();
		}
	}
	@Override
	public int compareTo(SecondarySort other) {
		if(this.getFirst() != other.getFirst()) {
			return this.getFirst() - other.getFirst();
		} else {
			return this.getSecond() - other.getSecond();
		}
	}

	
}

Secondly：编写本地测试类

package com.leo.OrderedTest;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

/**
 * Spark二次排序的具体实现步骤：
 * 第一步: 自定义key 实现scala.math.Ordered接口，和Java的Serializeable接口
 * 第二步：将要进行二次排序的数据加载，按照<key，value>格式的RDD
 * 第三步：使用sortByKey 基于自定义的key进行二次排序
 * 第四步：去掉排序的key,只保留排序的结果
 * @author FU
 *
 */
public class SecordSortTest {
	
	private static JavaSparkContext jsc;

	public static void main(String[] args) {
		
		SparkConf conf = new SparkConf()
				.setMaster("local")
				.setAppName("SecordSortTest");
		
		jsc = new JavaSparkContext(conf);
		JavaRDD<String> lines = jsc.textFile("C:\\Users\\Desktop\\test.txt");
		
		JavaPairRDD<SecondarySort, String> pairLines = lines.mapToPair(new PairFunction<String, SecondarySort, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<SecondarySort, String> call(String lines) throws Exception {
				SecondarySort secondarySort = new SecondarySort(
					Integer.valueOf(lines.split(" ")[0]),
					Integer.valueOf(lines.split(" ")[1]));
				return new Tuple2<SecondarySort, String>(secondarySort, lines);
			}
		});
		
		JavaPairRDD<SecondarySort, String> sortByKey = pairLines.sortByKey();
		JavaRDD<String> map = sortByKey.map(new Function<Tuple2<SecondarySort,String>, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public String call(Tuple2<SecondarySort, String> value) throws Exception {
				return value._2;
			}
		});
		
		map.foreach(new VoidFunction<String>() {
	
			private static final long serialVersionUID = 1L;

			@Override
			public void call(String line) throws Exception {
				System.out.println(line);
			}
		});
		jsc.stop();
	}
}

In The End:结果展示

1 2
1 12
1 23
1 89
2 6
2 9
2 11
2 56
3 2
3 8
3 16
3 22
3 31
3 56
4 3
4 8
4 45
4 66

很明显，先按第一列进行排序，然后第二列，还有第三列、第四列...

至此，简单的二次排序demo已经完成，大家可以完全贴代码进行本地测试。当然，在实际应用中，可以根据需要增加更多字段进行排序，这些都是ok的。