SparkStreaming怎么保存数据到关系型数据库

和Core的时候没什么区别,就是调用foreachRDD,下面是流处理做wordcount保存到mysql的例子

object TestStream2 {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
    val ssc = new StreamingContext(conf, Seconds(5))

    ssc.sparkContext.setLogLevel("WARN")

    val dataDS: ReceiverInputDStream[String] = ssc.socketTextStream("192.168.182.147",9998)

    val wordDS: DStream[String] = dataDS.flatMap(_.split(" "))

    val tupleDS: DStream[(String, Int)] = wordDS.map((_,1))

    tupleDS.reduceByKey(_+_).foreachRDD(rdd=>{
      rdd.foreach(word=>{
        Class.forName("com.mysql.jdbc.Driver")
        //获取mysql连接
        val conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/test", "root", "")
        //把数据写入mysql
        try {
          var totalcount = word._2
          var sql = "";
          var querysql="select count from wordcount where titleName='"+word._1+"'"
          val queryresult: ResultSet = conn.prepareStatement(querysql).executeQuery()
          if(queryresult.next()){
            totalcount = queryresult.getString("count").toInt+word._2
            sql = "update wordcount set count='"+totalcount+"' where titleName='"+word._1+"'"
          }else{
            sql = "insert into wordcount (titleName,count) values ('" + word._1 + "','" + totalcount + "')"
          }

          conn.prepareStatement(sql).executeUpdate()
          println("保存结束--------------------------------------------------------------")
        } finally {
          conn.close()
        }
      })
    })

    ssc.start()
    ssc.awaitTermination()
  }
}
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值