object SparkStreaming_StateFul {

窗口查询

object def mainStringUnit getLogger"org.apache.spark"WARNgetLogger"org.eclipse.jetty.server"OFFval new "local[2]"this"spark.executor.memory", "2g""spark.cores.max", "8"Array"E:\\ScalaSpace\\Spark_Streaming\\out\\artifacts\\Spark_Streaming.jar"val new //step1 create streaming context
    val new"."//step2 create a networkInputStream on get ip:port and count the words in input stream of \n delimited text
    val "218.193.154.79",12345val " "val ,1

    //使用updateStateByKey 来更新状态
    val Int

其输出结果如下所示，对全部的结果进行统计

-------------------------------------------

Time: 1459156160000 ms

-------------------------------------------

(B,1)

(F,1)

(D,4)

(G,1)

(A,1)

(C,5)

现在就可以，最热关键词进行统计，其统计代码如下所示：

那么此处为什么会有transform呢操作呢，我们看transform的介绍如下所示

/**
 * Return a new DStream in which each RDD is generated by applying a function
 * on each RDD of 'this' DStream.
 */
def transform[U: ClassTag](transformFunc: RDD[T] => RDD[U]): DStream[U] = ssc.withScope {
// because the DStream is reachable from the outer object here, and because
  // DStreams can't be serialized with closures, we can't proactively check
  // it for serializability and so we pass the optional false to SparkContext.clean
  val cleanedF = context.sparkContext.clean(transformFunc, false)
  transform((r: RDD[T], t: Time) => cleanedF(r))
}

/**
 * Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling
 * `collect` or `save` on the resulting RDD will return or output an ordered list of records
 * (in the `save` case, they will be written to multiple `part-X` files in the filesystem, in
 * order of the keys).
 */
// TODO: this currently doesn't work on P other than Tuple2!
def sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length)
    : RDD[(K, V)] = self.withScope
{
val part = new RangePartitioner(numPartitions, self, ascending)
new ShuffledRDD[K, V, V](self, part)
    .setKeyOrdering(if (ascending) ordering else ordering.reverse)
}

通过上述注释我们可以知道，sort是对RDD内所有partition数据进行排序，而并非针对所有RDD，因为SparkStreaming 是操作多个RDD，因此我们需要将使用transform 操作，对所有的RDD进行排序操作。 

stateDstream.map{
case (char,count) => (count,char)
}.transform(_.sortByKey(false))

From WizNote

转载于:https://www.cnblogs.com/zDanica/p/5471594.html