待深入......代码地址:https://github.com/vicotorz/sparkStreaming
- 带状态的算子:Update StateByKey 状态的累加
如果使用了stateful的算子,就必须设置checkpoint(可供检查每个批次状态的临时文件)
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
* 使用Spark Streaming完成有状态统计
*/
object StatefulWordCount {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setAppName("StatefulWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(5))
// 如果使用了stateful的算子,必须要设置checkpoint
// 在生产环境中,把checkpoint设置到HDFS的某个文件夹中
ssc.checkpoint(".")
val lines = ssc.socketTextStream("localhost", 6789)
val result = lines.flatMap(_.split(" ")).map((_,1))
val state = result.updateStateByKey[Int](updateFunction _)
state.print()
ssc.start()
ssc.awaitTermination()
}
/**
* 把当前的数据去更新已有的或者是老的数据
* @param currentValues 当前的
* @param preValues 老的
* @return
*/
def updateFunction(currentValues: Seq[Int], preValues: Option[Int]): Option[Int] = {
val current = currentValues.sum
val pre = preValues.getOrElse(0)
Some(current + pre)
}
}
2. 将统计结果写入到mysql中(foreachRDD)
示例表:create table wordcount(word varchar(50) default null, wordcount int(10) default null);
result.foreachRDD(rdd => {
rdd.foreachPartition(partionOfRecords => {
val connection = createConnection()
partionOfRecords.foreach(record => {
val sql = "insert into wordcount(word,wordcount) values('" + record._1 + "'," + record._2 + ")"
connection.createStatement().execute(sql)
})
connection.close()
})
})
编码过程中的异常:
org.apache.spark.SparkException: Spark Streaming.note
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2287)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:924)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:924)
at SparkStreamingDemo.ForeachRDD$$anonfun$main$1.apply(ForeachRDD.scala:31)
at SparkStreamingDemo.ForeachRDD$$anonfun$main$1.apply(ForeachRDD.scala:30)
... ....
序列化异常:检查代码中是否有不符合序列化的内容
3. Window:定时执行一个时间段的操作
核心概念:
- Window length:窗口的长度
- Sliding interval:窗口的间隔
每隔多久计算某个范围内的数据:每隔sliding interval 统计前window length的值
例:每隔10秒计算前30秒的值:
val windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) =>(a+b), Seconds(30), Seconds(10))
4. 黑名单过滤(DStream 与 RDD之间操作) 待补充,不深刻
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
* 黑名单过滤
*/
object TransformApp {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
/**
* 创建StreamingContext需要两个参数:SparkConf和batch interval
*/
val ssc = new StreamingContext(sparkConf, Seconds(5))
/**
* 构建黑名单:transform, leftjoin
*/
val blacks = List("zs", "ls")
val blacksRDD = ssc.sparkContext.parallelize(blacks).map(x => (x, true))
val lines = ssc.socketTextStream("localhost", 6789)
val clicklog = lines.map(x => (x.split(",")(1), x)).transform(rdd => {
rdd.leftOuterJoin(blacksRDD)
.filter(x=> x._2._2.getOrElse(false) != true)
.map(x=>x._2._1)
})
clicklog.print()
ssc.start()
ssc.awaitTermination()
}
}
5. Spark SQL 与 Spark Streaming 的整合
(1) foreachRDD ==> 转化为RDD
(2) RDD转化为DataFrame
(3) DataFrame 注册成临时表
(4) 最后通过SQL的方式从表中获取值