一般的二次排序,可以参考https://www.iteblog.com/archives/1819.html这篇文章,但是他的这种方式有问题。在这块代码:
item._2.toList.sortWith(_.toInt<_.toInt)
如果数据量非常大的话,会全部加在到内存中,容易造成内存溢出。
在spark中可以使用repartitionAndSortWithinPartitions这个算子,它会一边进行shuffle的时候就会对分区中的数据进行排序,要使用这个算子需要做两件事:
自定义Partitioner
提供一个隐式的Ordering,可以看一下OrderedRDDFunctions源码中该算子需要的参数
private val ordering = implicitly[Ordering[K]] def repartitionAndSortWithinPartitions(partitioner: Partitioner): RDD[(K, V)] = self.withScope { new ShuffledRDD[K, V, V](self, partitioner).setKeyOrdering(ordering) }
数据还是上面那篇文章中的数据
2015,1,24
2015,3,56
2015,1,3
2015,2,-43
2015,4,5
2015,3,46
2014,2,64
2015,1,4
2015,1,21
2015,2,35
2015,2,0
还是将年和月拼成一个字符串然后将(年-月, 总数)当做key,value其实随便。
在进行分区时,只要<年-月>相同的都分到相同分区中
在进行排序是,先比较<年-月>,如果<年-月>相同再比较总数
代码
import org.apache.spark.{Partitioner, SparkConf, SparkContext}
object SparkTest {
def main(args: Array[String]): Unit = {
implicit val ord = new Ordering[(String, Int)] {
override def compare(x: (String, Int), y: (String, Int)): Int = {
val c = x._1.compareTo(y._1)
if (c == 0) x._2.compareTo(y._2) else c
}
}
val conf = new SparkConf().setMaster("local[*]").setAppName("SparkTest")
val sc = new SparkContext(conf)
val rdd = sc.parallelize(Seq("2015,1,24", "2015,3,56", "2015,1,3", "2015,2,-43", "2015,4,5", "2015,3,46", "2014,2,64", "2015,1,4", "2015,1,21", "2015,2,35", "2015,2,0"))
rdd.map(s => {
val arr = s.split(",")
((s"${arr(0)}-${arr(1)}", arr(2).toInt), arr(2).toInt)
})
.repartitionAndSortWithinPartitions(new CustomPartitioner(3))
.map { case ((k, v), _) => (k, v) }
.saveAsTextFile("xxxxxx")
}
class CustomPartitioner(partitions: Int) extends Partitioner {
require(partitions > 0, s"Number of partitions ($partitions) cannot be negative.")
def numPartitions: Int = partitions
def getPartition(key: Any): Int = key match {
case (k: String, _: Int) => math.abs(k.hashCode % numPartitions)
case null => 0
case _ => math.abs(key.hashCode % numPartitions)
}
override def equals(other: Any): Boolean = other match {
case h: CustomPartitioner => h.numPartitions == numPartitions
case _ => false
}
override def hashCode: Int = numPartitions
}
}
最后三个分区的数据
(2014-2,64)
(2015-1,3)
(2015-1,4)
(2015-1,21)
(2015-1,24)
(2015-4,5)
(2015-2,-43)
(2015-2,0)
(2015-2,35)
(2015-3,46)
(2015-3,56)