updateStateByKey函数详解及worldcount例子

updateStateByKey操作允许您在使用新的信息持续更新时保持任意状态。

1、定义状态 - 状态可以是任意数据类型。 

2、定义状态更新功能 - 使用函数指定如何使用上一个状态更新状态，并从输入流中指定新值。 

如何使用该函数，spark文档写的很模糊，网上资料也不够详尽，自己翻阅源码总结一下，并给一个完整的例子

updateStateBykey函数有6种重载函数：

1、只传入一个更新函数，最简单的一种。

更新函数两个参数 Seq [ V ], Option [ S ]，前者是每个key新增的值的集合，后者是当前保存的状态，

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Seq [ V ], Option [ S ]) => Option [ S ]

): DStream [( K , S )] = ssc . withScope {

updateStateByKey (updateFunc, defaultPartitioner ())

}

例如，对于wordcount，我们可以这样定义更新函数：

(values: Seq [Int],state: Option [Int])=>{

//创建一个变量，用于记录单词出现次数

var newValue =state. getOrElse ( 0 ) //getOrElse相当于if....else.....

for ( value <- values){

newValue += value //将单词出现次数累计相加

}

Option ( newValue )

}

2、传入更新函数和分区数

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Seq [ V ], Option [ S ]) => Option [ S ],

numPartitions: Int

): DStream [( K , S )] = ssc . withScope {

updateStateByKey (updateFunc, defaultPartitioner (numPartitions))

}

3、传入更新函数和自定义分区

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Seq [ V ], Option [ S ]) => Option [ S ],

partitioner: Partitioner

): DStream [( K , S )] = ssc . withScope {

val cleanedUpdateF = sparkContext . clean (updateFunc)

val newUpdateFunc = (iterator: Iterator [( K , Seq [ V ], Option [ S ])]) => {

iterator. flatMap (t => cleanedUpdateF (t._2, t._3). map (s => (t._1, s)))

}

updateStateByKey ( newUpdateFunc , partitioner, true )

}

4、传入完整的状态更新函数

前面的函数传入的都是不完整的更新函数，只是针对一个key的，他们在执行的时候也会生成一个完整的状态更新函数。

Iterator [( K , Seq [ V ], Option [ S ])]) => Iterator [( K , S )] 入参是一个迭代器，参数1是key，参数2是这个key在这个batch中更新的值的集合，参数3是当前状态，最终得到key-->newvalue

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Iterator [( K , Seq [ V ], Option [ S ])]) => Iterator [( K , S )],

partitioner: Partitioner ,

rememberPartitioner: Boolean

): DStream [( K , S )] = ssc . withScope {

new StateDStream (self, ssc . sc . clean (updateFunc), partitioner, rememberPartitioner, None )

}

例如，对于wordcount：

val newUpdateFunc = (iterator: Iterator [( String , Seq [Int], Option [Int])]) => {

iterator. flatMap (t => function1 (t._2, t._3). map (s => (t._1, s)))

}

5、加入初始状态

initialRDD: RDD [( K , S )] 初始状态集合

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Seq [ V ], Option [ S ]) => Option [ S ],

partitioner: Partitioner ,

initialRDD: RDD [( K , S )]

): DStream [( K , S )] = ssc . withScope {

val cleanedUpdateF = sparkContext . clean (updateFunc)

val newUpdateFunc = (iterator: Iterator [( K , Seq [ V ], Option [ S ])]) => {

iterator. flatMap (t => cleanedUpdateF (t._2, t._3). map (s => (t._1, s)))

}

updateStateByKey ( newUpdateFunc , partitioner, true , initialRDD)

}

6、是否记得当前的分区

def updateStateByKey [ S : ClassTag ](

updateFunc: ( Iterator [( K , Seq [ V ], Option [ S ])]) => Iterator [( K , S )],

partitioner: Partitioner ,

rememberPartitioner: Boolean,

initialRDD: RDD [( K , S )]

): DStream [( K , S )] = ssc . withScope {

new StateDStream (self, ssc . sc . clean (updateFunc), partitioner,

rememberPartitioner, Some (initialRDD))

}

完整的例子：

def testUpdate ={

val sc = SparkUtils . getSpark ( "test" , "db01" ).sparkContext

val ssc = new StreamingContext ( sc , Seconds ( 5 ))

ssc . checkpoint ( "hdfs://ns1/config/checkpoint" )

val initialRDD = sc . parallelize ( List (( "hello" , 1 ), ( "world" , 1 )))

val lines = ssc . fileStream [ LongWritable , Text , TextInputFormat ]( "hdfs://ns1/config/data/" )

val words = lines . flatMap (x=>x._2. toString . split ( "," ))

val wordDstream : DStream [( String , Int)]= words . map (x => (x, 1 ))

val result = wordDstream . reduceByKey (_ + _)

def function1 (newValues: Seq [Int], runningCount: Option [Int]): Option [Int] = {

val newCount = newValues. sum + runningCount. getOrElse ( 0 ) // add the new values with the previous running count to get the new count

Some ( newCount )

}

val newUpdateFunc = (iterator: Iterator [( String , Seq [Int], Option [Int])]) => {

iterator. flatMap (t => function1 (t._2, t._3). map (s => (t._1, s)))

}

val stateDS = result . updateStateByKey ( newUpdateFunc , new HashPartitioner ( sc . defaultParallelism ), true , initialRDD )

stateDS . print ()

ssc . start ()

ssc . awaitTermination ()

}