DateSet的API详解七
sortGroup
Adds a secondary sort key to this GroupedDataSet. This will only have an effect if you use one
of the group-at-a-time, i.e. reduceGroup.
执行程序:
//1.创建 DataSet[(Int, String)]
val input: DataSet[(Int, String)] = benv.fromElements(
(20,"zhangsan"),
(22,"zhangsan"),
(22,"lisi"),
(22,"lisi"),
(22,"lisi"),
(18,"zhangsan"),
(18,"zhangsan"))
//2.用int分组,用int对分组进行排序
val sortdata = input.groupBy(0).sortGroup(0, Order.ASCENDING)
//3.对排序好的分组进行reduceGroup
val outputdata =sortdata.reduceGroup {
//将相同的元素用set去重
(in, out: Collector[(Int, String)]) =>
in.toSet foreach (out.collect)
}
//4.显示结果
outputdata.collect
执行结果:
res25: Seq[(Int, String)] = Buffer((18,zhangsan), (20,zhangsan), (22,zhangsan), (22,lisi))
web ui中的执行效果:
minBy
def minBy(fields: Int*): DataSet[T]
Applies a special case of a reduce transformation minBy on a grouped DataSet.
在分组后的数据中,获取每组最小的元素。
执行程序:
//1.定义case class
case class Student(age: Int, name: String,height:Double)
//2.创建DataSet[Student]
val input: DataSet[Student] = benv.fromElements(
Student(16,"zhangasn",194.5),
Student(17,"zhangasn",184.5),
Student(18,"zhangasn",174.5),
Student(16,"lisi",194.5),
Student(17,"lisi",184.5),
Student(18,"lisi",174.5))
//3.以name进行分组,获取age最小的元素
val output0: DataSet[Student] = input.groupBy(_.name).minBy(0)
output0.collect
//4.以name进行分组,获取height和age最小的元素
val output1: DataSet[Student] = input.groupBy(_.name).minBy(2,0)
output1.collect
执行结果:
Scala-Flink> output0.collect
res73: Seq[Student] = Buffer(Student(16,lisi,194.5), Student(16,zhangasn,194.5))
Scala-Flink> output1.collect
res74: Seq[Student] = Buffer(Student(18,lisi,174.5), Student(18,zhangasn,174.5))
web ui中的执行效果:
maxBy
def maxBy(fields: Int*): DataSet[T]
def max(field: Int): AggregateDataSet[T]
Applies a special case of a reduce transformation maxBy on a grouped DataSet
在分组后的数据中,获取每组最大的元素。
执行程序:
//1.定义case class
case class Student(age: Int, name: String,height:Double)
//2.创建DataSet[Student]
val input: DataSet[Student] = benv.fromElements(
Student(16,"zhangasn",194.5),
Student(17,"zhangasn",184.5),
Student(18,"zhangasn",174.5),
Student(16,"lisi",194.5),
Student(17,"lisi",184.5),
Student(18,"lisi",174.5))
//3.以name进行分组,获取age最大的元素
val output0: DataSet[Student] = input.groupBy(_.name).maxBy(0)
output0.collect
//4.以name进行分组,获取height和age最大的元素
val output1: DataSet[Student] = input.groupBy(_.name).maxBy(2,0)
output1.collect
执行结果:
Scala-Flink> output0.collect
res75: Seq[Student] = Buffer(Student(18,lisi,174.5), Student(18,zhangasn,174.5))
Scala-Flink> output1.collect
res76: Seq[Student] = Buffer(Student(16,lisi,194.5), Student(16,zhangasn,194.5))
web ui中的执行效果: