sparkStreaming:实时流代码案例(实现Wordcount)

最新推荐文章于 2023-04-09 11:13:17 发布
最新推荐文章于 2023-04-09 11:13:17 发布
阅读量875 收藏 1
点赞数 1
文章标签： kafka spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_43012693/article/details/111564455
版权

目录

依赖包

<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>2.4.5</version>
      <!--<scope>provided</scope>-->
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
      <dependency>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>jackson-databind</artifactId>
          <version>2.6.6</version>
      </dependency>

从文件中采集

val conf = new SparkConf().setMaster("local[2]").setAppName("demo1")
val ssc = new StreamingContext(conf,Seconds(5))

val fileDStream = ssc.textFileStream("data/file")
val wordCount = fileDStream.flatMap(line=>line.split("\\s+"))
    .map((_,1))
    .reduceByKey(_+_)

wordCount.print()
ssc.start()
ssc.awaitTermination()

自定义采集

import java.io.{BufferedReader, InputStreamReader}
import org.apache.spark.{SparkConf}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.receiver.Receiver

class MyReceiver(host:String,port:Int) extends Receiver[String](StorageLevel.MEMORY_ONLY){
    var socket:java.net.Socket=null
    def receive():Unit={
    val socket = new java.net.Socket(host,port)
    val reader = new BufferedReader(new InputStreamReader(socket.getInputStream,"UTF-8"))
    var line:String=null;
    while((line=reader.readLine())!=null){
        if(line.equals("end")){
            return
    }else{
        this.store(line)
    }
    }
}
    override def onStart():Unit = {
        new Thread(new Runnable{
            override def run():Unit={
                receive()
            }
        }).start()
    }
    override def onStop():Unit = {
        if(socket != null){
        socket.close()
        socket=null
    }
    }
}
object MyReceiverDemo{
    def main(args:Array[String]):Unit = {
        val conf = new SparkConf().setMaster("local[2]").setAppName("demo3")
        val ssc = new StreamingContext(conf,Seconds(5))
        val receiverStream = ssc.receiverStream(new MyReceiver("192.168.136.10",9999))
        val streamCount = receiverStream.flatMap(_split("\\s+"))
            .map((_,1))
            .reduceByKey(_+_)
        streamCount.print()
        ssc.start()
        ssc.awaitTermination()
    }
}

从kafka收集数据

无状态

	val conf = new SparkConf().setMaster("local[2]").setAppName("spark-kafka")
    val ssc = new StreamingContext(conf, Seconds(5))
	//与kafka建立连接
    val kafkaParams: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG->"spark-kafka")
    )
 	//接收producer的信息
    val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      //分区设置策略:1.每个Executor作为一个consumer
      LocationStrategies.PreferConsistent,
      
      // LocationStrategies.PreferBrokers
      // LocationStrategies.PreferFixed()
      	//订阅固定的topic集
      ConsumerStrategies.Subscribe(Set("sparkDemo"), kafkaParams)
    )
    //需求实现，wordcount
    val wordCount = kafkaStream.flatMap(v => v.value().toString.split(" "))
      .map((_, 1))
      .reduceByKey(_ + _)
    wordCount.print()
    //启动
    ssc.start()
    ssc.awaitTermination()

producer生成消息：

 kafka-console-producer.sh --topic spark-kafka --broker-list 192.168.136.10:9092

有状态

需要设立检查点checkpoint

	val conf = new SparkConf().setMaster("local[2]").setAppName("demo3")
    val ssc = new StreamingContext(conf, Seconds(5))
    //建立checkpoint
    ssc.checkpoint("checkpoint")
    //连接kafka
    val kafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "kafkaGroup1")
    )
    //获得producer数据
    val stream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("kafkaWindow1"), kafka)
    )
    //需求实现
    val msg = stream.flatMap(v => v.value().toString.split("\\s+"))
      .map((_, 1))
      //有状态统计
    val sum: DStream[(String, Int)] = msg.updateStateByKey {
      case (seq, buffer) => {
        val allSum: Int = buffer.getOrElse(0) + seq.sum
        Option(allSum)
      }
    }
    sum.print()
    //启动
    ssc.start()
    ssc.awaitTermination()

producer生产消息：

 kafka-console-producer.sh --topic kafkaWindow1 --broker-list 192.168.136.10:9092

有状态和无状态相比，有状态的情况下，会统计历史数据，在进行单词计数时，历史的单词也会计算进去。

窗口函数

window

新窗口与旧窗口没有交集

import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}


val conf = new SparkConf().setMaster("local[2]").setAppName("demo1")
    val ssc = new StreamingContext(conf,Seconds(5))

    val sparkKafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "window1")
    )
    val KafkaWindow: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("windowStream"), sparkKafka)
    )
    val wordCount = KafkaWindow.flatMap(v => v.value().toString.split("\\s+"))
      .map((_, 1))
      //必须是前面设置的时间间隔(5s)的倍数
      .window(Seconds(5))
      .reduceByKey(_ + _)

    wordCount.print()
    ssc.start()
    ssc.awaitTermination()

producer生产消息：

kafka-console-producer.sh --topic windowStream --broker-list 192.168.136.10:9092

新窗口与旧窗口有交集

val conf = new SparkConf().setMaster("local[2]").setAppName("demo1")
    val ssc = new StreamingContext(conf,Seconds(2))

    val sparkKafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "window1")
    )
    val KafkaWindow: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("windowStream"), sparkKafka)
    )
    val wordCount = KafkaWindow.flatMap(v => v.value().toString.split("\\s+"))
      .map((_, 1))
      //必须是前面设置的时间间隔(5s)的倍数
      .window(Seconds(4),Seconds(2))
      .reduceByKey(_ + _)

    wordCount.print()
    ssc.start()
    ssc.awaitTermination()

producer生产消息：

kafka-console-producer.sh --topic windowStream --broker-list 192.168.136.10:9092

countByWindow

统计当前时间窗口的元素个数
需要设立检查点checkpoint

val conf = new SparkConf().setMaster("local[2]").setAppName("demo2")
    val ssc = new StreamingContext(conf,Seconds(2))
    ssc.checkpoint("checkpoint")
    val sparkKafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "Window2")
    )
    val kafkamsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("windowStream"), sparkKafka)
    )
    val res = kafkamsg.flatMap(v => v.value().toString.split("\\s+"))
      .map((_, 1))
      .countByWindow(Seconds(4), Seconds(2))

    res.print()
    ssc.start()
    ssc.awaitTermination()

生产者消费：

kafka-console-producer.sh --topic windowStream --broker-list 192.168.136.10:9092

随着消息的产生，它只统计总的单词的个数
在这里插入图片描述

countByValueAndWindow

统计当前时间窗口中元素值相同的元素个数
需要设立检查点checkpoint

val conf = new SparkConf().setMaster("local[2]").setAppName("demo3")
    val ssc = new StreamingContext(conf,Seconds(2))
    ssc.checkpoint("checkpoint")
    val sparkKafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "Window3")
    )
    val kafkaMsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("windowStream"), sparkKafka)
    )
    val res: DStream[(String, Long)] = kafkaMsg.flatMap(v => v.value().toString.split("\\s+"))
      .countByValueAndWindow(Seconds(4), Seconds(2))
    res
    res.print()
    ssc.start()
    ssc.awaitTermination()

生产者消费：

kafka-console-producer.sh --topic windowStream --broker-list 192.168.136.10:9092

reduceByWindow

在调用DStream上首先取窗口函数的元素形成新的DStream，然后在窗口元素形成的DStream上进行reduce

val conf = new SparkConf().setMaster("local[2]").setAppName("demo4")
    val ssc = new StreamingContext(conf,Seconds(2))

    val kafkaStream: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, " org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, " org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "window4")
    )

    val kafkaMsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("windowStream"), kafkaStream)
    )
    val wordcount = kafkaMsg.flatMap(_.value().toString.split("\\s+"))
      .reduceByWindow(_ + ":" + _, Seconds(8), Seconds(4))
    wordcount.print()

    ssc.start()
    ssc.awaitTermination()

reduceByKeyAndWindow

按照key值运算

val conf = new SparkConf().setMaster("local[2]").setAppName("demo5")
    val ssc = new StreamingContext(conf,Seconds(2))
    ssc.checkpoint("checkpoint")
    val kafkaStream = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "window5")
    )

    val kafkaMsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("sparkDemo"), kafkaStream)
    )
    val res: DStream[(String, Int)] = kafkaMsg.flatMap(_.value().toString.split("\\s+"))
      .map((_, 1))
      .reduceByKeyAndWindow((x: Int, y: Int) => {
       x + y
      }, Seconds(4), Seconds(2))
      //这个用法需要设立检查点
     // .reduceByKeyAndWindow((x:Int,y:Int)=>{x+y},(x:Int,y:Int)=>{x-y},Seconds(4),Seconds(2))
    res.print()

    ssc.start()
    ssc.awaitTermination()

生产者生产和前面一样，这里即不再赘述。

transform

支持任意的RDD到RDD的映射操作。

val conf = new SparkConf().setMaster("local[2]").setAppName("transform")
    val ssc = new StreamingContext(conf,Seconds(2))
   
    val sparkKafka: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "transform1")
    )
    val kafkamsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("sparkDemo"), sparkKafka))

    val wordCount: DStream[((String, String), Int)] = kafkamsg.transform((rdd, timestamp) => {
      val format: SimpleDateFormat = new SimpleDateFormat("yyyyMMdd HH:mm:ss")
      val time: String = format.format(timestamp.milliseconds)
      val value: RDD[((String, String), Int)] = rdd.flatMap(x => x.value().toString.split("\\s+"))
        .map(x => ((x, time), 1))
        .reduceByKey(_ + _)
        .sortBy(_._2, false)
      value
    })

    wordCount.print()
    ssc.start()
    ssc.awaitTermination()

transform源码：
传入的参数是自定义函数，自定义函数的返回值为RDD,transform整个返回值类型为DStream.

def transform[U: ClassTag](transformFunc: (RDD[T], Time) => RDD[U]): DStream[U] = ssc.withScope {
    // because the DStream is reachable from the outer object here, and because
    // DStreams can't be serialized with closures, we can't proactively check
    // it for serializability and so we pass the optional false to SparkContext.clean
    val cleanedF = context.sparkContext.clean(transformFunc, false)
    val realTransformFunc = (rdds: Seq[RDD[_]], time: Time) => {
      assert(rdds.length == 1)
      cleanedF(rdds.head.asInstanceOf[RDD[T]], time)
    }
    new TransformedDStream[U](Seq(this), realTransformFunc)
  }

SQLContext

用SQL语句来进行操作

import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.{Row, SQLContext}
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}
object streamSQLDemo {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[2]").setAppName("streamsql")
    val ssc = new StreamingContext(conf,Seconds(2))

    val kafkaStream: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.136.10:9092"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG, "streamsql")
    )
    val kafkamsg: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("sparkDemo"), kafkaStream)
    )
    kafkamsg
    val wordcount: DStream[Row] = kafkamsg.transform(rdd => {
      val sqlContext = SQLContextSinleton.getInstance(rdd.sparkContext)
      import sqlContext.implicits._
      val value = rdd.flatMap(x => x.value().toString.split("\\s+")).map((_, 1))
      value.toDF("name", "cn")
        .createOrReplaceTempView("tbword")
      val frame = sqlContext.sql("select name,count(cn) from tbword group by name")
      frame.rdd
    })
    wordcount.print()
    ssc.start()
    ssc.awaitTermination()

  }
}

object SQLContextSinleton{
  @transient private var instance:SQLContext=_
  def getInstance(sc:SparkContext):SQLContext={
    synchronized(
      if(instance == null) {
        instance = new SQLContext(sc)
      }
    )
    instance
  }
}

再来一个
这种方式输出的是表格形式：

object Demo_SQLWordCount {
  def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("demo").setMaster("local[*]")
val ssc = new StreamingContext(conf,Seconds(5))
//sparkSession需写在ssc后面
val spark = SparkSession.builder().config(conf).getOrCreate()
import spark.implicits._
val lines = ssc.socketTextStream("hadoop01",7777)
val wrods = lines.flatMap(_.split("\\s+"))
wrods.foreachRDD(rdd=>{

  if(rdd.count()!=0) {
  val df = rdd.map(x=>Word(x)).toDF()
  df.createOrReplaceTempView("tb_word")
  spark.sql("select word,count(1) from tb_word group by word")
  .show()
}
})

  //value.print()
  ssc.start()
  ssc.awaitTermination()
}

  case class Word(word:String)
}

在这里插入图片描述

小五家的二哈
关注
大数据Spark（三十六）：SparkStreaming实战案例一 WordCount
Lansonli（蓝深李）的博客
04-21 1036
SparkStreaming实时处理数据性能如何（是否可以实时处理数据）？每批次Batch数据处理总时间TD = 批次调度延迟时间SD + 批次数据处理时间PT。nc是netcat的简称，原本是用来设置路由器,我们可以利用它向某个端口发送数据。每批次数据处理时间TD
Spark Streaming实时流处理原理与代码实例讲解
最新发布
AI天才研究院
10-04 1195
随着互联网的快速发展，实时数据处理变得越来越重要。在许多场景中，如社交网络、电子商务、金融交易等，都需要对实时数据进行分析和挖掘，以便快速响应市场变化、优化业务决策。传统的批处理系统在处理实时数据时存在响应速度慢、效率低等问题。因此，实时流处理技术应运而生。Spark Streaming采用微批处理的方式处理流数据。从数据源（如Kafka、Flume等）获取数据，将其划分为小批量（batch）。对每个批次进行转换操作，如过滤、映射、转换等。
spark-streaming实践代码
qq_28286027的博客
03-20 420
实践1：wordcount 1.1本地跑wordcount代码（无状态）： package com.badou.streaming import org.apache.spark.SparkConf import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark.storage.St...
SparkStreaming wordcount demo
sundujing的博客
06-06 978
流数据统计，将每隔10s内的数据做一次单词统计package com.streamingimport org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ import org.apache.spark.SparkContext import org.apache.spark.api.jav
spark 2.2.0学习笔记5之SparkStreamingWordCountDemo
undergrowth的专栏
12-24 292
spark 2.2.0学习笔记5之SparkStreamingWordCountDemoInfospark streaming—-Spark 提供的对实时数据进行流式计算的组件/微批次架构 Spark Streaming 使用离散化流（discretized stream）作为抽象表示，叫作DStream DStream 是随时间推移而收到的数据的序列 一种是转化操作（transformation
spark streaming WordCount实例演示
斯特凡今天也很帅的博客
02-18 807
一、首先准备一台虚拟机 安装NC yum -y install nc 然后 nc -lk 9999 二、打开IDEA （1）创建一个maven项目，并修改pom文件如下 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:sc
sparkStreaming-WordCountDemo处理socket数据
无
06-06 252
步骤一：配置环境，特别注意虚拟机的hosts文件和window中的hosts文件，以及关闭防火墙     linux:   sudo vim /etc/hosts  添加ip到name的映射，从而让外网访问你时用name便可     window:   C:\Windows\System32\drivers\etc 将linux /etc/hosts 中的映射写进去,这用于在window上可以访问...
kafka +spark streaming 流计算wordcount实例实现
kyj3518861703的博客
05-24 570
一、Kafka简介 Broker Kafka集群包含一个或多个服务器，这种服务器被称为broker Topic 每条发布到Kafka集群的消息都有一个类别，这个类别被称为Topic。（物理上不同Topic的消息分开存储，逻辑上一个Topic的消息虽然保存于一个或多个broker上，但用户只需指定消息的Topic即可生产或消费数据而不必关心数据存于何处） Partition Partition是物理上的概念，每个Topic包含一个或多个Partition。 Producer 负责发布消息
SparkStreaming之WordCount案例
01-07
使用 netcat 工具向 9999 端口不断的发送数据，通过 SparkStreaming 读取端口数据并统 计不同单词出现的次数 。 二、netcat操作 1、虚拟机中安装netcat [root@hadoop1 spark]# yum install -y nc 2、启动程序并发送...
Spark 案例实操（Spark Streaming之WordCount）
qq_50678909的博客
12-15 2803
Spark Streaming 是什么 Spark Streaming 用于流式数据的处理。Spark Streaming 支持的数据输入源很多，例如：Kafka、 Flume、Twitter、ZeroMQ 和简单的 TCP 套接字等等。数据输入后可以用 Spark 的高度抽象原语 如：map、reduce、join、window 等进行运算。而结果也能保存在很多地方，如 HDFS，数据库等。和 Spark 基于 RDD 的概念很相似，Spark Streaming 使用离散化流(discretize
SparkStreaming-编写程序实例
weixin_44085996的博客
09-26 345
1.IDEA编写程序 案例所需的pom.xml依赖： <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>${spark.version}</version> <scope>provided</sco...
Apache SparkStreaming WordCount示例
大数据流浪法师的学习笔记与分享
04-10 163
需求 首先在linux服务器上安装nc工具 nc是netcat的简称，原本是用来设置路由器,我们可以利用它向某个端口发送数据 yum install -y nc 启动一个服务端并开放9999端口,等一下往这个端口发数据 nc -lk 9999 接收数据的代码 package demo13 import org.apache.spark.streaming.dstream.{DStream...
sparkStreaming：实时流计算Java案例
不花的花和尚的博客
10-18 2787
现在，网上基于spark的代码基本上都是Scala，很多书上也都是基于Scala，没办法，谁叫spark是Scala写出来的了，但是我现在还没系统的学习Scala，所以只能用java写spark程序了，spark支持java，而且Scala也基于JVM,不说了，直接上代码 这是官网上给出的例子，大数据学习中经典案例单词计数  在linux下一个终端 输入 $ nc -lk 9999 然后运行下...
Spark Streaming 基础代码实现演示（二）
MannerXu的博客
08-08 512
1.需求&准备 ●图解 ●首先在linux服务器上安装nc工具(nc命令是netcat命令的简称，原本是用来设置路由器.,我们可以利用它向某个端口发送数据) yum install -y nc ●启动一个服务端并开发9999端口,等一下往这个端口发数据 nc -lk 9999 ●发送数据 2. 代码演示 import org.apache.spark.str...
spark第七章：SparkStreaming实例
weixin_50835854的博客
04-09 1483
今天我们来完成spark的最后一次实验案例.Spark的学习就告一段落了,下一步估计要啃Flink了。
sparkstreaming实时
qq_43827516的博客
11-02 261
pom依赖 <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> </dependency> <dependency> <grou
spark大数据工程实战：实时数据流处理
m0_48858192的博客
01-08 2153
大数据工程实战：实时数据流处理 在/usr/local/flume/conf下新建一个配置文件 streaming_project.conf,配置source、sink、Channel的 各项参数，连接各个组件，其配置内容如下： 第五行是一行！！！ exec-memory-kafka.sources = exec-source exec-memory-kafka.sinks = kafka-sink exec-memory-kafka.channels = memory-channel exec-me
Spark Streaming-附有代码
weixin_39540280的博客
05-03 1248
一. 官方参考链接 Spark Streaming - Spark 3.2.1 Documentationhttp://spark.apache.org/docs/latest/streaming-programming-guide.html 二. Spark Streaming 核心抽象 DStream 简介 is an extension of the core Spark API tenables scalable, high-throughput, ...
SparkStreaming的WordCount学习
Ran_记忆犹新的博客
07-30 367
SparkStreaming的入门WordCount： object WordCount { def main(args: Array[String]): Unit = { //local[2]是启动两个线程，一个是接收socket数据，一个打印到控制台 val conf = new SparkConf().setAppName(s"${this.getClass.getSimpl...
SparkStreaming：DStream创建与WordCount实现
"本文主要介绍如何使用Spark Streaming创建DStream，并通过一个具体的WordCount实例来展示其实现过程。" 在Apache Spark Streaming中，DStream（Discretized Stream）是连续数据流的离散化表示，它由一系列连续的...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值