updateStateByKey操作,可以让每个key维护一份state,并持续不断的更新该state。
1、首先,要定义一个state,可以是任意的数据类型;
2、其次,要定义state更新函数——指定一个函数如何使用之前的state和新值来更新state。
对于每个batch,Spark都会为每个之前已经存在的key去应用一次state更新函数,无论这个key在batch中是否有新的数据。如果state更新函数返回none,那么key对应的state就会被删除。
当然,对于每个新出现的key,也会执行state更新函数。
注意,updateStateByKey操作,要求必须开启Checkpoint机制。
Java版本wordcount全局计数:
public static void main(String[] args) {
SparkConf conf = new SparkConf()
.setMaster("local[2]")
.setAppName("UpdateStateByKeyWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(5));
jssc.checkpoint("hdfs://hadoop01:9000/wordcount_checkpoint");
// 命令: nc -lk 9999
JavaReceiverInputDStream<String> lines = jssc.socketTextStream("localhost", 9999);
JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
private static final long serialVersionUID = 1L;
@Override
public Iterable<String> call(String line) throws Exception {
return Arrays.asList(line.split(" "));
}
});
JavaPairDStream<String, Integer> pairs = words.mapToPair(
new PairFunction<String, String, Integer>() {
private static final long serialVersionUID = 1L;
@Override
public Tuple2<String, Integer> call(String word)
throws Exception {
return new Tuple2<String, Integer>(word, 1);
}
});
JavaPairDStream<String, Integer> wordCounts = pairs.updateStateByKey(
// 这里的Optional,相当于Scala中的样例类,就是Option,可以这么理解
// 它代表了一个值的存在状态,可能存在,也可能不存在
new Function2<List<Integer>, Optional<Integer>, Optional<Integer>>() {
private static final long serialVersionUID = 1L;
@Override
public Optional<Integer> call(List<Integer> values,
Optional<Integer> state) throws Exception {
Integer newValue = 0;
if(state.isPresent()) {
newValue = state.get();
}
for(Integer value : values) {
newValue += value;
}
return Optional.of(newValue);
}
});
wordCounts.print();
jssc.start();
jssc.awaitTermination();
jssc.close();
}
Scala版本wordcount全局计数:
def main(args: Array[String]): Unit = {
val conf = new SparkConf()
.setMaster("local[2]")
.setAppName("UpdateStateByKeyWordCount")
val ssc = new StreamingContext(conf, Seconds(5))
ssc.checkpoint("hdfs://spark1:9000/wordcount_checkpoint")
val lines = ssc.socketTextStream("spark1", 9999)
val words = lines.flatMap { _.split(" ") }
val pairs = words.map { word => (word, 1) }
val wordCounts = pairs.updateStateByKey((values: Seq[Int], state: Option[Int]) => {
var newValue = state.getOrElse(0)
for(value <- values) {
newValue += value
}
Option(newValue)
})
wordCounts.print()
ssc.start()
ssc.awaitTermination()
}