SparkStreaming实时日志分析--实时热搜词

最新推荐文章于 2020-10-22 14:45:30 发布
最新推荐文章于 2020-10-22 14:45:30 发布
阅读量181 收藏 1
点赞数
原文链接：http://www.cnblogs.com/wttttt/p/6852704.html
版权

Overview

  • 整个项目的整体架构如下：

　　

  • 关于SparkStreaming的部分：
    1. Flume传数据到SparkStreaming：为了简单使用的是push-based的方式。这种方式可能会丢失数据，但是简单。
    2. SparkStreaming因为micro-batch的架构，跟我们这个实时热点的应用还是比较契合的。
    3. SparkStreaming这边是基于sliding window实现实时热搜的，batch interval待定(1min左右)，window也待定(3~N* batch interval)，slide就等于batch interval。

Step1：Flume Configuration

  • Flume端将之前的配置扩展成多channel + 多sink，即sink到HDFS和Spark Streaming。关于Hadoop端的配置，参见Nginx+Flume+Hadoop日志分析，Ngram+AutoComplete
  • Flume + SparkStreaming的集成这部分，暂时选用push-based的方法。简单，但容错性不行。

Flume多channel&多sink

  • 首先，Flume支持多channel+多sink；
  • 具体的实现：
    1. 在channels和sinks下面加上要add的channel和sink即可 
      • clusterLogAgent.sinks = HDFS sink2
clusterLogAgent.channels = ch1 ch2
    2. 确定所选用的selector：
      • 关于selector，我们之前在flume源码解读篇中有了解到，这里选择的是replicating的selector，也就是把source中的events复制到各个channels中。
    3. 这个多sink应该还是配在hadoop cluster端，一个avro sink加一个hdfs sink。要是配在web server端理论上还要浪费网络带宽。
  • 用FlumeEventCount.scala测试了下，一切ok，如下~
    • web server端运行 bin/flume-ng agent -n WebAccLo-c conf -f conf/flume-avro.conf
    • spark端运行 ./bin/spark-submit --class com.wttttt.spark.FlumeEventCount --master yarn --deploy-mode client --driver-memory 1g --executor-memory 1g --executor-cores 2 /home/hhh/RealTimeLog.jar 10.3.242.99 4545 30000
-------------------------------------------
Time: 1495098360000 ms
-------------------------------------------
Received 10 flume events.

17/05/18 17:06:00 INFO JobScheduler: Finished job streaming job 1495098360000 ms.0 from job set of time 1495098360000 ms
17/05/18 17:06:00 INFO JobScheduler: Total delay: 0.298 s for time 1495098360000 ms (execution: 0.216 s)
17/05/18 17:06:00 INFO ReceivedBlockTracker: Deleting batches: 
17/05/18 17:06:00 INFO InputInfoTracker: remove old batch metadata: 
17/05/18 17:06:13 INFO BlockManagerInfo: Added input-0-1495098373000 in memory on host99:42342 (size: 319.0 B, free: 366.3 MB)
17/05/18 17:06:17 INFO BlockManagerInfo: Added input-0-1495098377200 in memory on host99:42342 (size: 620.0 B, free: 366.3 MB)
17/05/18 17:06:20 INFO BlockManagerInfo: Added input-0-1495098380400 in memory on host99:42342 (size: 620.0 B, free: 366.3 MB)
17/05/18 17:06:30 INFO JobScheduler: Added jobs for time 1495098390000 ms
17/05/18 17:06:30 INFO JobScheduler: Starting job streaming job 1495098390000 ms.0 from job set of time 1495098390000 ms
17/05/18 17:06:30 INFO SparkContext: Starting job: print at FlumeEventCount.scala:30
17/05/18 17:06:30 INFO DAGScheduler: Registering RDD 14 (union at DStream.scala:605)
17/05/18 17:06:30 INFO DAGScheduler: Got job 4 (print at FlumeEventCount.scala:30) with 1 output partitions
17/05/18 17:06:30 INFO DAGScheduler: Final stage: ResultStage 8 (print at FlumeEventCount.scala:30)
17/05/18 17:06:30 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
17/05/18 17:06:30 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 7)
17/05/18 17:06:30 INFO DAGScheduler: Submitting ShuffleMapStage 7 (UnionRDD[14] at union at DStream.scala:605), which has no missing parents
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 3.3 KB, free 399.5 MB)
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.0 KB, free 399.5 MB)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 10.3.242.99:36107 (size: 2.0 KB, free: 399.6 MB)
17/05/18 17:06:30 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:996
17/05/18 17:06:30 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 7 (UnionRDD[14] at union at DStream.scala:605)
17/05/18 17:06:30 INFO YarnScheduler: Adding task set 7.0 with 4 tasks
17/05/18 17:06:30 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 88, host99, executor 1, partition 0, NODE_LOCAL, 7290 bytes)
17/05/18 17:06:30 INFO TaskSetManager: Starting task 3.0 in stage 7.0 (TID 89, host101, executor 2, partition 3, PROCESS_LOCAL, 7470 bytes)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on host99:42342 (size: 2.0 KB, free: 366.3 MB)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on host101:45692 (size: 2.0 KB, free: 366.3 MB)
17/05/18 17:06:30 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 90, host99, executor 1, partition 1, NODE_LOCAL, 7290 bytes)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 88) in 22 ms on host99 (executor 1) (1/4)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 3.0 in stage 7.0 (TID 89) in 27 ms on host101 (executor 2) (2/4)
17/05/18 17:06:30 INFO TaskSetManager: Starting task 2.0 in stage 7.0 (TID 91, host99, executor 1, partition 2, NODE_LOCAL, 7290 bytes)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 90) in 12 ms on host99 (executor 1) (3/4)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 2.0 in stage 7.0 (TID 91) in 11 ms on host99 (executor 1) (4/4)
17/05/18 17:06:30 INFO YarnScheduler: Removed TaskSet 7.0, whose tasks have all completed, from pool 
17/05/18 17:06:30 INFO DAGScheduler: ShuffleMapStage 7 (union at DStream.scala:605) finished in 0.045 s
17/05/18 17:06:30 INFO DAGScheduler: looking for newly runnable stages
17/05/18 17:06:30 INFO DAGScheduler: running: Set(ResultStage 2)
17/05/18 17:06:30 INFO DAGScheduler: waiting: Set(ResultStage 8)
17/05/18 17:06:30 INFO DAGScheduler: failed: Set()
17/05/18 17:06:30 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[17] at map at FlumeEventCount.scala:30), which has no missing parents
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.8 KB, free 399.5 MB)
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.1 KB, free 399.5 MB)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 10.3.242.99:36107 (size: 2.1 KB, free: 399.6 MB)
17/05/18 17:06:30 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:996
17/05/18 17:06:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[17] at map at FlumeEventCount.scala:30)
17/05/18 17:06:30 INFO YarnScheduler: Adding task set 8.0 with 1 tasks
17/05/18 17:06:30 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 92, host101, executor 2, partition 0, NODE_LOCAL, 7069 bytes)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on host101:45692 (size: 2.1 KB, free: 366.3 MB)
17/05/18 17:06:30 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.3.242.101:41672
17/05/18 17:06:30 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 164 bytes
17/05/18 17:06:30 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 92) in 26 ms on host101 (executor 2) (1/1)
17/05/18 17:06:30 INFO YarnScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 
17/05/18 17:06:30 INFO DAGScheduler: ResultStage 8 (print at FlumeEventCount.scala:30) finished in 0.027 s
17/05/18 17:06:30 INFO DAGScheduler: Job 4 finished: print at FlumeEventCount.scala:30, took 0.089621 s
17/05/18 17:06:30 INFO SparkContext: Starting job: print at FlumeEventCount.scala:30
17/05/18 17:06:30 INFO DAGScheduler: Got job 5 (print at FlumeEventCount.scala:30) with 3 output partitions
17/05/18 17:06:30 INFO DAGScheduler: Final stage: ResultStage 10 (print at FlumeEventCount.scala:30)
17/05/18 17:06:30 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 9)
17/05/18 17:06:30 INFO DAGScheduler: Missing parents: List()
17/05/18 17:06:30 INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[17] at map at FlumeEventCount.scala:30), which has no missing parents
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 3.8 KB, free 399.5 MB)
17/05/18 17:06:30 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.1 KB, free 399.5 MB)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 10.3.242.99:36107 (size: 2.1 KB, free: 399.6 MB)
17/05/18 17:06:30 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:996
17/05/18 17:06:30 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 10 (MapPartitionsRDD[17] at map at FlumeEventCount.scala:30)
17/05/18 17:06:30 INFO YarnScheduler: Adding task set 10.0 with 3 tasks
17/05/18 17:06:30 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 93, host99, executor 1, partition 1, PROCESS_LOCAL, 7069 bytes)
17/05/18 17:06:30 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 94, host101, executor 2, partition 2, PROCESS_LOCAL, 7069 bytes)
17/05/18 17:06:30 INFO TaskSetManager: Starting task 2.0 in stage 10.0 (TID 95, host101, executor 2, partition 3, PROCESS_LOCAL, 7069 bytes)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on host99:42342 (size: 2.1 KB, free: 366.3 MB)
17/05/18 17:06:30 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on host101:45692 (size: 2.1 KB, free: 366.3 MB)
17/05/18 17:06:30 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.3.242.99:35937
17/05/18 17:06:30 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 93) in 19 ms on host99 (executor 1) (1/3)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 2.0 in stage 10.0 (TID 95) in 21 ms on host101 (executor 2) (2/3)
17/05/18 17:06:30 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 94) in 22 ms on host101 (executor 2) (3/3)
17/05/18 17:06:30 INFO YarnScheduler: Removed TaskSet 10.0, whose tasks have all completed, from pool 
17/05/18 17:06:30 INFO DAGScheduler: ResultStage 10 (print at FlumeEventCount.scala:30) finished in 0.025 s
17/05/18 17:06:30 INFO DAGScheduler: Job 5 finished: print at FlumeEventCount.scala:30, took 0.032431 s
-------------------------------------------
Time: 1495098390000 ms
-------------------------------------------
Received 5 flume events.
  •  下一步会用Flume先做一次filter，去除掉没有搜索记录的log event。

Step2：Spark Streaming

  • Spark Streaming这边只要编写程序：
    1. new一个StreamingContext
    2. FlumeUtils.createStream接收Flume传过来的events
    3. mapPartitions(优化)：对每个partiiton建一个Pattern和hashMap
    4. reduceByKeyAndWindow(优化): 滑动窗口对hashMap的相同key进行加减
    5. sortByKey: sort之后取前N个
  •  基于上述方式，现在本地作测试：
    • sparkStreaming处理：
    • object LocalTest {
  val logger = LoggerFactory.getLogger("LocalTest")
  def main(args: Array[String]) {

    val batchInterval = Milliseconds(10000)
    val slideInterval = Milliseconds(5000)

    val conf = new SparkConf()
      .setMaster("local[2]")
      .setAppName("LocalTest")
    // WARN StreamingContext: spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data,
    // otherwise Spark jobs will not get resources to process the received data.
    val sc = new StreamingContext(conf, Milliseconds(5000))
    sc.checkpoint("flumeCheckpoint/")

    val stream = sc.socketTextStream("localhost", 9998)

    val counts = stream.mapPartitions{ events =>
      val pattern = Pattern.compile("\\?Input=[^\\s]*\\s")
      val map = new mutable.HashMap[String, Int]()
      logger.info("Handling events, events is empty: " + events.isEmpty)
      while (events.hasNext){   // par is an Iterator!!!
      val line = events.next()
        val m = pattern.matcher(line)
        if (m.find()) {
          val words = line.substring(m.start(), m.end()).split("=")(1).toLowerCase()
          logger.info(s"Processing words $words")
          map.put(words, map.getOrElse(words, 0) + 1)
        }
      }
      map.iterator
    }

    val window = counts.reduceByKeyAndWindow(_+_, _-_, batchInterval, slideInterval)
    // window.print()

    // transform和它的变体trnasformWith运行在DStream上任意的RDD-to-RDD函数;
    // 可以用来使用那些不包含在DStrema API中RDD操作
    val sorted = window.transform(rdd =>{
      val sortRdd = rdd.map(t => (t._2, t._1)).sortByKey(false).map(t => (t._2, t._1))
      val more = sortRdd.take(2)
      more.foreach(println)
      sortRdd
    })

    sorted.print()

    sc.start()
    sc.awaitTermination()
  }
}
    • 同时，另外运行一个程序，产生log，并向9998端口发送：
    • object GenerateChar {
  def main(args: Array[String]) {
    val listener = new ServerSocket(9998)
    while(true){
      val socket = listener.accept()
      new Thread(){
        override def run() = {
          println("Got client connected from :"+ socket.getInetAddress)
          val out = new PrintWriter(socket.getOutputStream,true)
          while(true){
            Thread.sleep(3000)
            val context1 = "GET /result.html?Input=test1 HTTP/1.1"
            println(context1)
            val context2 = "GET /result.html?Input=test2 HTTP/1.1"
            println(context2)
            val context3 = "GET /result.html?Input=test3 HTTP/1.1"
            println(context3)
            out.write(context1 + '\n' + context2 + "\n" + context2 + "\n" + context3 + "\n" + context3 + "\n" + context3 + "\n" + context3 + "\n")
            out.flush()
          }
          socket.close()
        }
      }.start()
    }
  }
}
  • 以上，本地完全没有问题。但是！！！一打包到集群，就各种bug，没有输出。打的logger info也没有输出，System.out.println也没有(stdout文件为空...)。而且会报错shuffleException。
  • 基于上述问题，google了很多都说是内存的问题，但是我的数据量已经不能更小了...  我又测试了下在集群上跑，但不连flume，而是在driver本地跑了个generateLog的程序向9998端口发数据。事实是仍然可能报错如下： 
    • 17/05/24 15:07:17 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from host101:37940
java.io.IOException: Failed to connect to host101/10.3.242.101:37940

      但是结果是正确的...

 

TODO

  • 整个架构还有很多可改进的地方。因为我现在只剩两台机器了，就先不折腾了。 - -
  • 其中最大的问题还是容错：
    • flume是push-based，所以一旦有events冲击波，HDFS可能负载不了高强度的写操作，从而出问题；
    • spark-streaming那边因为也是直接使用这种push-based(没有定制receiver，我嫌麻烦)，所以也会有问题。
  • 后续的话，还是要使用经典的Kafka + Flume的架构。

 

转载于:https://www.cnblogs.com/wttttt/p/6852704.html

weixin_30907523
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
BlockManager原理和源码解密
sundujing的博客
05-15 3002
一、BlockManager运行实例观察从Application启动的角度来观察BlockManager；1，在Application启动时会在SparkEnv中注册BlockManagerMaster以及MapOutputTracker，其中a) BlockManagerMaster：对整个集群的Block数据进行管理的；b) MapOutputTracker:跟踪所有mapper的输出的；2，B
大规模数据处理中拒绝连接错误分析处理
大师兄你家猴跑啦的博客
07-04 4406
1、处理的数据有几百个G,把数据处理成按照手机号计算1万多个特征 ；2、数据处理环境：     spark-2.0.2;    --executor-memory 40g --total-executor-cores 120 --driver-memory 40g  3、报的错误org.apache.spark.shuffle.MetadataFetchFailedException: Missi...
Spark：遇到问题汇总
不花的花和尚的博客
10-07 519
问题一：Map output statuses were bytes which exceeds spark.akka.frameSize   17/10/12 00:15:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 1 to spark...
Error communicating with MapOutputTracker-问题解析2
liliang199的专栏
10-12 2596
spark.MapOutputTrackerMasterActor: Map output statuses were 14371441 bytes which exceeds spark.akka.frameSize (10485760 bytes). org.apache.spark.SparkException: Map output statuses were 14371441 bytes which exceeds spark.akka.frameSize (10485760 bytes).
Spark开发中遇到的常见问题以及解决方案（一）
zhangzeyuan56的博客
05-18 8570
问题一：ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-8] shutting down ActorSystem [sparkDriver]18/05/18 15:46:59 ERROR ActorSystemImpl: Uncaug...
苏宁基于SparkStreaming的实时日志分析系统实践
01-27
目前业界基于Hadoop技术栈的底层计算平台越发稳定成熟，计算能力不再成为主要瓶颈。...目前实时日志分析系统每天处理数十亿条流量日志，不仅需要保证：低延迟、数据不丢失等要求，还要面对复杂的分析计算逻辑，这些都
spark-streaming-kafka-0-10_2.12-2.4.0.jar
03-11
spakr streaming的kafka依赖
spark-streaming_2.11-2.4.0-cdh6.1.1.jar
05-14
spark-streaming_2.11-2.4.0-cdh6.1.1.jar
spark-streaming-kafka-0-8_2.11-2.4.0.jar
01-22
spark-streaming-kafka-0-8_2.11-2.4.0.jar
Spark平台上提交作业到集群生成的日志文件
SAP 资深技术专家 Jerry Wang 的技术分享
03-04 755
Created by Wang, Jerry on Aug 16, 2015 ./spark-submit --class “org.apache.spark.examples.JavaWordCount” --master spark://NKGV50849583FV1:7077 /root/devExpert/spark-1.4.1/example-java-build/JavaWordCou...
spark实时项目
01-03
内容包括 1、用户访问sesssion的实时分析和根据不同条件进行过滤 2、根据用户的点击次数，下单次数，支付次数进行二次排序 3、将结果实现存入数据库，并用echart进行展示
Spark Shuffle源码分析系列之MapOutputTracker
进击的数据小白
10-22 683
MapOutputTracker是spark环境的主要组件之一，其功能有三方面，首先DAGScheduler使用MapOutputTrackerMaster来管理各个ShuffleMapTask的输出数据[MapStatus]，另外根据各ShuffleMapTask结果的统计信息来进行尽可能的本地化Scheduler；其次ShuffleMapStage使用MapOutputTrackerMaster来判断是否要进行ShuffleMapTask的计算；最后ShuffleReduce任务用来获取从哪些exec.
Spark(43) -- SparkStreaming -- 案例：统计一定时间内的热门词汇TopN
erainm
10-12 379
1. 需求 2. scala代码
spark打包程序提交任务
知其然，知其所以然
01-22 873
美图欣赏： 一.操作： 拷贝一个全类名： 程序： import org.apache.spark.{SparkConf, SparkContext} /** * * Scala的word count * */ object WordCountScala extends App { //获取spark的环境,setAppName定义应用程序的名字，s...
SparkCore（7）：SparkOnYarn提交（生产环境）
小蚯蚓的博客
10-24 503
一、实现功能 将spark的wordcount任务提交到Yarn上，然后计算结果输出到hdfs上。 二、实现步骤 1.Wordcount_product代码 package _0722rdd /** * Created by Administrator on 2018/7/16. */ import org.apache.spark.rdd.RDD import org.apa...
深入理解Spark 2.1 Core （十一）：Shuffle Reduce 端的原理与源码分析
02-24 4050
http://blog.csdn.net/u011239443/article/details/56843264 在《深入理解Spark 2.1 Core （九）：迭代计算和Shuffle的原理与源码分析 》我们讲解了，以传统Hadoop MapReduce类似的从HDFS中读取数据，再到rdd.HadoopRDD.compute便可以调用函数f，即map中的函数的过程。在《深入理解Spark ...
spark学习-35-Spark的Map任务输出跟踪器MapOutputTracker
九师兄
11-22 4469
1。在sparkEnv的初始化中有这样一段代码来初始化Map任务输出跟踪器MapOutputTracker　　　mapOutputTracker用于跟踪map阶段任务的输出状态，此状态便于reduce阶段任务获取地址以及中间输出结果。每个map任务或者 reduce任务都会有唯一的标识。分别为mapId和reduceId.每个reduce任务的输入可能是多个map任务的输出，reduce会到各个ma
学习笔记:从0开始学习大数据-14. java spark编程实践
LinBSoft的专栏
11-29 494
上节搭建好了eclipse spark编程环境 在测试运行scala 或java 编写spark程序 ，在eclipse平台都可以运行，但打包导出jar，提交 spark-submit运行，都不能执行，最后确定是版本问题，就是你在eclipse调试的spark版本需和spark-submit 提交spark的运行版本一致，还有就是scala版本一致，才能正常运行。 以下是java spark程...
第1课：通过案例对SparkStreaming 透彻理解三板斧
热门推荐
u012016268的专栏
05-01 2万+
1 Spark Streaming另类在线实验 2 瞬间理解Spark Streaming本质 代码如下： import org.apache.spark.SparkConf import org.apache.spark.storage.StorageLevel import org.apache.spark.streaming.{Seconds, StreamingContext} /
sparkstreaming----复习
最新发布
06-06
Spark Streaming是Apache Spark的一个组件，它提供了实时数据处理的能力。它可以从各种数据源（如Kafka、Flume、Twitter等）接收数据流，并将其分成小批次进行处理。Spark Streaming还提供了丰富的API，可以进行数据转换、聚合、过滤等操作，同时还支持窗口操作和状态管理。通过Spark Streaming，我们可以实现实时数据处理、实时监控、实时分析等应用场景。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值