SparkStreaming例题

最新推荐文章于 2024-06-23 23:34:15 发布

橙以

最新推荐文章于 2024-06-23 23:34:15 发布

阅读量827

点赞数

文章标签： SparkStreaming

本文链接：https://blog.csdn.net/qq_42721694/article/details/85252693

版权

本文主要介绍了SparkStreaming的两个实际应用案例，包括如何与集群协同工作，深入探讨了SparkStreaming在大数据实时处理中的应用。

摘要由CSDN通过智能技术生成

取材自官网http://Spark.apache.org

案例1：和集群搭配使用

package SparkStreaming

import java.io.{BufferedReader, InputStreamReader}
import java.net.Socket
import java.nio.charset.StandardCharsets

import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.receiver.Receiver

class CustomReceiver(host: String, port: Int)
  extends Receiver[String](StorageLevel.MEMORY_AND_DISK_2) {

  def onStart() {
    // Start the thread that receives data over a connection
    new Thread("Socket Receiver") {
      override def run() { receive() }
    }.start()
  }

  def onStop() {
    // There is nothing much to do as the thread calling receive()
    // is designed to stop by itself if isStopped() returns false
  }

  /** Create a socket connection and receive data until receiver is stopped */
  private def receive() {
    var socket: Socket = null
    var userInput: String = null
    try {
      // Connect to host:port
      socket &#