Spark Streaming输出操作【Output Operations on DStreams】

Spark Streaming输出操作 可以将DStream中的数据写入到外部的存储系统，如：数据库或者文件系统

Output Operation	Meaning
print()	Prints the first ten elements of every batch of data in a DStream on the driver node running the streaming application. This is useful for development and debugging. Python API This is called pprint() in the Python API.
saveAsTextFiles(prefix, [suffix])	Save this DStream’s contents as text files. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”.
saveAsObjectFiles(prefix, [suffix])	Save this DStream’s contents as SequenceFiles of serialized Java objects. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”. Python API This is not available in the Python API.
saveAsHadoopFiles(prefix, [suffix])	Save this DStream’s contents as Hadoop files. The file name at each batch interval is generated based on prefix and suffix: “prefix-TIME_IN_MS[.suffix]”. Python API This is not available in the Python API.
foreachRDD(func)	The most generic output operator that applies a function, func, to each RDD generated from the stream. This function should push the data in each RDD to an external system, such as saving the RDD to files, or writing it over the network to a database. Note that the function func is executed in the driver process running the streaming application, and will usually have RDD actions in it that will force the computation of the streaming RDDs.

print()

打印DStream每个批次中前10个元素

package example1
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

object QuickExample {
  def main(args: Array[String]): Unit = {
    // StreamingContext是所有Spark Streaming应用的入口
    val conf = new SparkConf().setMaster("local[2]").setAppName("quick example")
    val ssc = new StreamingContext(conf, Seconds(5))
    // 通过TCP Source创建DStream对象
    val lines = ssc.socketTextStream("spark", 9999)
    // 将此DStream中生成的每个RDD的前十个元素打印到控制台
    lines.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _).print()
    // 启动计算
    ssc.start()
    // 等待计算终止
    ssc.awaitTermination()
  }
}

saveAsTextFiles(prefix, [suffix])

将DStream的内容保存到文本文件中

// 保存到文本文件中
lines.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _).saveAsTextFiles("result", "txt")

saveAsHadoopFiles(prefix, [suffix])

将DStream的内容保存到Hadoop文件系统中

  // --------------------------------------------------------------------------
   // 保存到hadoop文件中
   val configuration = new Configuration()
   configuration.set("fs.defaultFS", "hdfs://gaozhy:9000")
   val jobConf = new JobConf(configuration)
   lines.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
      .saveAsNewAPIHadoopFiles("result", "log", classOf[Text], classOf[IntWritable], classOf[TextOutputFormat[Text, IntWritable]], jobConf)

foreachRDD(func)

遍历处理DStream中的批次对应的RDD，可以将每个RDD的数据写入到外部的存储系统，如数据库、Redis等

如：将Stream的计算结果存储到MySQL中

// --------------------------------------------------------------------------
// 保存到数据库中
lines.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _).foreachRDD(rdd => {
  rdd.foreachPartition(partitionOfRecords => {
    classOf[Driver]
    val connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/test", "root", "root")
    val selectSql: String = "select *  from t_word where word = ?"
    val updateSql = "update t_word set count = ? where word = ?"
    val insertSql = "insert t_word(word,count) values(?,1)"
    partitionOfRecords.foreach(record => {
      val pstm = connection.prepareStatement(selectSql)
      pstm.setString(1, record._1)
      val rs = pstm.executeQuery()
      // word存在
      if (rs.next()) {
        val count = rs.getInt("count")
        val updateStatement = connection.prepareStatement(updateSql)
        updateStatement.setInt(1, count + record._2)
        updateStatement.setString(2, record._1)
        updateStatement.executeUpdate()
      } else {
        val insertStatement = connection.prepareStatement(insertSql)
        insertStatement.setString(1, record._1)
        insertStatement.executeUpdate()
      }
    })
    connection.close()
  })
})