Spark Scala 二次排序

1.输入

1 5
2 4
3 5
1 8
2 1
3 2
5 1
6 2

2. 代码实现

2.1 自定义比较器

/**
 * User:leen
 * Date:2017/10/18 0018
 * Time:14:16
 *
 * 比较器：
 * 1. 继承 Ordered[T] 和Serializeable 方法
 * 2. 重写 compare 方法
 * 3. 需要传入参数
 */

class SecondarySortKey1(val first: Int, val second: Int) extends Ordered[SecondarySortKey1] with Serializable {
  override def compare(that: SecondarySortKey1): Int = {
    if (this.first - that.first != 0) {
      this.first - that.first    //第一列 排序：升序
    } else {
      that.second - this.second   //第二列 排序：降序
    }
  }
}

2.2调用比较器实现二次排序

import org.apache.spark.{SparkContext, SparkConf}

/**
 * User:leen
 * Date:2017/10/18 0018
 * Time:14:23
 */
object SecondarySort1 {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("SecondarySort1").setMaster("local");
    val sc = new SparkContext(conf)

    val lines = sc.textFile("C:\\Users\\leen\\Desktop\\sort.txt")
    //与自定义Key组合返回Tuple2
    val words = lines.map(line => {
      val ws = line.split(" ")
      (new SecondarySortKey1(ws(0).toInt, ws(1).toInt), line)
    })
    val sortRes = words.sortByKey()
    sortRes.foreach(pair => println(pair._2))

    sc.stop()
  }
}

3.结果输出

1 8
1 5
2 4
2 1
3 5
3 2
5 1
6 2