SparkStreaming 消费 Kafka 数据保存 Offset 到 Zookeeper

最新推荐文章于 2022-07-12 23:00:00 发布
最新推荐文章于 2022-07-12 23:00:00 发布
阅读量1k 收藏 1
点赞数
分类专栏： Spark 文章标签： Spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43215250/article/details/93492173
版权

Sparkstreaming程序：

package com.cloudera.saveoffsettoZk

import com.cloudera.utils.KafkaZkUtils
import kafka.utils.ZkUtils
import org.apache.kafka.clients.consumer.ConsumerConfig
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.log4j.Logger
import org.apache.spark.SparkConf
import org.apache.spark.streaming.kafka010.HasOffsetRanges
import org.apache.spark.streaming.{Seconds, StreamingContext}

object SparkSaveOffsetToZkApp {

  private val logger: Logger = Logger.getLogger(this.getClass)

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setIfMissing("spark.master", "local[2]").setAppName("Spark Save Offset To Zookeeper App")

    val streamingContext = new StreamingContext(conf, Seconds(30))
    val kafkaParams = Map[String, Object](
      ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.1.100:9092,192.168.1.101:9092,192.168.1.102:9092",
      ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> classOf[StringDeserializer],
      ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> classOf[StringDeserializer],
      ConsumerConfig.GROUP_ID_CONFIG -> "group-01",
      ConsumerConfig.AUTO_OFFSET_RESET_CONFIG -> "earliest",
      ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG -> (false: java.lang.Boolean)
    )

    val topic: String = "my-topic"
    val zkUrl = "192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181"
    val sessionTimeout = 1000
    val connectionTimeout = 1000

    val zkClient = ZkUtils.createZkClient(zkUrl, sessionTimeout, connectionTimeout)

    val kafkaStream = KafkaZkUtils.createDirectStream(zkClient, streamingContext, kafkaParams, topic)

    //开始处理批次消息
    kafkaStream.foreachRDD(rdd => {
      //获取当前批次的RDD的偏移量
      val offsetsList = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

      // 处理从获取 kafka 中的数据
      val result = rdd.map(_.value()).map(_.split("\\|\\|")).map(x => (x(0), x(1), x(2)))
      result.foreach(println(_))
      println("=============== Total " + rdd.count() + " events in this    batch ..")

      // 更新offset到zookeeper中
      KafkaZkUtils.saveOffsets(zkClient, topic, KafkaZkUtils.getZkPath(kafkaParams, topic), rdd)
    })
    streamingContext.start()
    streamingContext.awaitTermination()
    streamingContext.stop()
  }
}

KafkaZkUtils 工具类:

package com.cloudera.utils

import kafka.utils.{ZKGroupTopicDirs, ZkUtils}
import org.I0Itec.zkclient.ZkClient
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.kafka.common.TopicPartition
import org.apache.log4j.Logger
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.dstream.InputDStream
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, HasOffsetRanges, KafkaUtils}

import scala.collection.immutable.Map

object KafkaZkUtils {
  private val logger: Logger = Logger.getLogger(this.getClass)


  /**
    * 获取 consumer 在zk上的路径
    * @param kafkaParams
    * @param topic
    * @return
    */
  def getZkPath(kafkaParams: Map[String, Object], topic: String): String ={
    val topicDirs = new ZKGroupTopicDirs(kafkaParams.get(ConsumerConfig.GROUP_ID_CONFIG).toString, topic)
    s"${topicDirs.consumerOffsetDir}"
  }

  /**
    * 创建 DirectStream
    * @param zkClient
    * @param streamingContext
    * @param kafkaParams
    * @param topic
    * @return
    */
  def createDirectStream(zkClient: ZkClient,streamingContext: StreamingContext, kafkaParams: Map[String, Object], topic: String): InputDStream[ConsumerRecord[String, String]] = {


    val zkPath = getZkPath(kafkaParams,topic)

    //读取 topic 的 offset
    val storedOffsets = readOffsets(zkClient, topic, zkPath)

    val kafkaStream: InputDStream[ConsumerRecord[String, String]] = storedOffsets match {
      //上次未保存offsets
      case None =>
        KafkaUtils.createDirectStream[String, String](
          streamingContext,
          PreferConsistent,
          ConsumerStrategies.Subscribe[String, String](Array(topic), kafkaParams)
        )
      case Some(fromOffsets) => {
        KafkaUtils.createDirectStream[String, String](
          streamingContext,
          PreferConsistent,         
          // 指定分区消费，无法动态感知分区变化
//          ConsumerStrategies.Assign[String, String](fromOffsets.keys.toList, kafkaParams, fromOffsets)
          ConsumerStrategies.Subscribe[String, String](topics, kafkaParams, fromOffsets)
        )
      }
    }
    kafkaStream
  }

  /**
    * 保存 offset
    * @param zkClient
    * @param topic
    * @param zkPath
    * @param rdd
    */
  def saveOffsets(zkClient: ZkClient,topic: String, zkPath: String, rdd: RDD[_]): Unit = {

    logger.info("Saving offsets to zookeeper")

    val offsetsRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    offsetsRanges.foreach(offsetRange => logger.debug(s"Using ${offsetRange}"))

    val offsetsRangesStr = offsetsRanges.map(offsetRange => s"${offsetRange.partition}:${offsetRange.untilOffset}").mkString(",")

    logger.info(s"Writing offsets to Zookeeper: ${offsetsRangesStr}")

    ZkUtils(zkClient, false).updatePersistentPath(zkPath, offsetsRangesStr)
  }

  /**
    * 读取 offset
    * @param zkClient
    * @param topic
    * @param zkPath
    * @return
    */
  def readOffsets(zkClient: ZkClient, topic: String, zkPath: String): Option[Map[TopicPartition, Long]] = {
    logger.info("Reading offsets from zookeeper")

    val (offsetsRangesStrOpt, _) = ZkUtils(zkClient, false).readDataMaybeNull(zkPath)
    offsetsRangesStrOpt match {
      case Some(offsetsRangesStr) => {
        logger.debug(s"Read offset ranges: ${
          offsetsRangesStr
        }")
        val offsets: Map[TopicPartition, Long] = offsetsRangesStr.split(",").map(s => s.split(":"))
          .map({
            case Array(partitionStr, offsetStr) =>
              (new TopicPartition(topic, partitionStr.toInt) -> offsetStr.toLong)
          }).toMap
        Some(offsets)
      }
      case None =>
        logger.info("No offsets found in Zookeeper")
        None
    }
  }
}

KafkaProducer

package com.cloudera.saveoffsettoZk

import java.text.SimpleDateFormat
import java.util.concurrent.Future
import java.util.{Date, Properties, UUID}

import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord, RecordMetadata}
import org.apache.kafka.common.serialization.StringSerializer

import scala.util.Random

object KafkaProducerDemo {

  def main(args: Array[String]): Unit = {

    // 设置 Kafka 配置属性
    val props = new Properties
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.100:9092,192.168.1.101:9092,192.168.1.102:9092")
    props.put(ProducerConfig.ACKS_CONFIG, "all")
    props.put(ProducerConfig.RETRIES_CONFIG, "0")
    props.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384") //批量发送的字节数
    props.put(ProducerConfig.LINGER_MS_CONFIG, "1") //将会减少请求数目，但是同时会增加1ms的延迟
    props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "33554432") //用来缓存数据的内存大小
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, classOf[StringSerializer].getName)
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, classOf[StringSerializer].getName)

    val producer = new KafkaProducer[String, String](props)

    val TOPIC_NAME = "my-topic"

    try {
      // 产生并发送消息
      while (true) {
        val runtime = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date().getTime)

        // 2019-09-30 08:46:10||24016FFD664A4C32AE3A4276416329B0||6
        val message: String = s"${runtime}||${UUID.randomUUID().toString.replace("-", "").toUpperCase}||${Random.nextInt()}"

        val record: ProducerRecord[String, String] = new ProducerRecord[String, String](TOPIC_NAME, message)
        // 发送消息，并获得一个Future对象
        val metadataFuture: Future[RecordMetadata] = producer.send(record)
        //同步获得Future对象的结果
        val recordMetadata: RecordMetadata = metadataFuture.get()

        Thread.sleep(500)
        println(message)
      }
    } catch {
      case e: Exception => {
        e.printStackTrace();
      }
    }
    // 关闭producer
    producer.flush()
    producer.close()
  }
}

pom.xml

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <scala.version>2.11</scala.version>
        <kafka.version>0.10.0.1</kafka.version>
        <!--<spark.version>2.4.0-cdh6.1.1</spark.version>-->
        <spark.version>2.2.0</spark.version>
        <zkclient.version>0.11</zkclient.version>
        <redis.version>3.0.0</redis.version>
        <kafka.scope>compile</kafka.scope>
        <provided.scope>compile</provided.scope>
        <!--<provided.scope>provided</provided.scope>-->
    </properties>

    <dependencies>
        <!-- scala -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}.12</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_${scala.version}</artifactId>
            <version>${kafka.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>${kafka.version}</version>
            <scope>${provided.scope}</scope>
        </dependency>
        <dependency>
            <groupId>com.101tec</groupId>
            <artifactId>zkclient</artifactId>
            <version>${zkclient.version}</version>
        </dependency>
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>${redis.version}</version>
        </dependency>
    </dependencies>

运行结果:

...
(2019-09-30 10:09:05,0F9EE952FEC548D2B2E91E4EDCF8DDC4,99555169)
(2019-09-30 10:08:59,2837DA6C8B3A4F29973B2FC210D20D54,-591678628)
(2019-09-30 10:09:12,A19DE366F9F640CFB239BF3197C5E10B,513230648)
(2019-09-30 10:09:12,E8EAF40E636E461C86C27161B794F60A,691784811)
(2019-09-30 10:09:14,CC09A901AEA0478F8C60346150FF87FA,-1644884384)
(2019-09-30 10:09:13,1317E46EE6154D2F9EF69B8382246E93,135563951)
(2019-09-30 10:09:15,121156B463AC4A22BF5CF07E22FA9CE0,-414732657)
(2019-09-30 10:09:15,DAECF018F6384933A40364A906BC99D0,-145448624)
(2019-09-30 10:09:17,46EDA56DE868495BAC44613EBFFB0BDF,1356997081)
(2019-09-30 10:09:06,5191550588544F229C33839655D8ECB8,-1222599956)
(2019-09-30 10:09:13,65833427C68A44B79FC27B14652A916A,1140020871)
(2019-09-30 10:09:14,655D6FB67C5F43AFB3EABB842818742F,1760913134)
(2019-09-30 10:09:16,752F6AFA9EA54EF48E0D2124DFBC3BC7,1518003893)
19/09/30 10:09:30 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 622 bytes result sent to driver
19/09/30 10:09:30 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 19 ms on localhost (executor driver) (3/3)
19/09/30 10:09:30 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/09/30 10:09:30 INFO DAGScheduler: ResultStage 0 (foreach at Consumer.scala:46) finished in 0.128 s
19/09/30 10:09:30 INFO DAGScheduler: Job 0 finished: foreach at Consumer.scala:46, took 0.284038 s
=============== Total 13 events in this    batch ..
19/09/30 10:09:30 INFO KafkaZkUtils$: Saving offsets to zookeeper
19/09/30 10:09:30 INFO KafkaZkUtils$: Writing offsets to Zookeeper: 0:764,1:765,2:765
19/09/30 10:09:30 INFO JobScheduler: Finished job streaming job 1569809370000 ms.0 from job set of time 1569809370000 ms
19/09/30 10:09:30 INFO JobScheduler: Total delay: 0.465 s for time 1569809370000 ms (execution: 0.335 s)
......
(2019-09-30 10:09:17,D9BD82C2A7CD45D6BA6193DF9B44F2EC,1363055585)
19/09/30 10:10:00 INFO Executor: Finished task 0.0 in stage 1.0 (TID 3). 622 bytes result sent to driver
19/09/30 10:10:00 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 4) in 13 ms on localhost (executor driver) (1/3)
19/09/30 10:10:00 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 3) in 14 ms on localhost (executor driver) (2/3)
19/09/30 10:10:00 INFO KafkaRDD: Beginning offset 765 is the same as ending offset skipping my-topic 2
19/09/30 10:10:00 INFO Executor: Finished task 2.0 in stage 1.0 (TID 5). 622 bytes result sent to driver
19/09/30 10:10:00 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 5) in 9 ms on localhost (executor driver) (3/3)
19/09/30 10:10:00 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
19/09/30 10:10:00 INFO DAGScheduler: ResultStage 1 (foreach at Consumer.scala:46) finished in 0.018 s
19/09/30 10:10:00 INFO DAGScheduler: Job 1 finished: foreach at Consumer.scala:46, took 0.026031 s
=============== Total 1 events in this    batch ..
19/09/30 10:10:00 INFO KafkaZkUtils$: Saving offsets to zookeeper
19/09/30 10:10:00 INFO KafkaZkUtils$: Writing offsets to Zookeeper: 0:765,1:765,2:765
19/09/30 10:10:00 INFO JobScheduler: Finished job streaming job 1569809400000 ms.0 from job set of time 1569809400000 ms
19/09/30 10:10:00 INFO JobScheduler: Total delay: 0.051 s for time 1569809400000 ms (execution: 0.048 s)
...
storm_fury
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
(Scala)sparkstreaming手动提交offset到zookeeper中，kafka版本0.10
斌躏天下的博客
08-05 686
话不多说先上代码，后续再写说明 package zookeeper_offset import org.apache.kafka.clients.consumer.ConsumerRecord import org.apache.kafka.common.serialization.StringDeserializer import org.apache.spark.SparkConf impo...
sparkStreaming结合kafka取存数据
01-07
目录 一、说明 1、环境介绍 ...3、主程序Kafka2SparkStreaming2Kafka 四、启动程序测试 1、创建相关kafka主题 2、kafka生产数据到程序 3、从写入到的kafka主题中读取数据 一、说明 1、环境介绍 spark:
Kafka文件存储机制那些事
美团技术团队
01-13 892
Kafka是什么 Kafka是最初由Linkedin公司开发，是一个分布式、分区的、多副本的、多订阅者，基于zookeeper协调的分布式日志系统(也可以当做MQ系统)，常见可以用于web/nginx日志、访问日志，消息服务等等，Linkedin于2010年贡献给了Apache基金会并成为顶级开源项目。 1.前言 一个商业化消息队列的性能好坏，其文件存储机制设计是衡量一个消息队...
spark-stream-kafka offsets 保存到 zookeeper
choudan1531的博客
03-10 202
package com.neo.spark.base import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark.{SparkConf, SparkContext} import...
spark stream 3.2.1（scala版本）将kafka消费偏移记录到zookeeper
liaomingwu的专栏
02-28 472
为了实现从kafka读取数据时，避免从头开始的重复读，也避免从最新开始的跳过数据读，有必要在读取数据并处理成功后，记录消费偏移量。当系统重启或者升级维护后，就可以继续从上次处理完成的位置进行继续读取了。
SparkStreaming对接Kafka0.10+管理offset到zookeeper的方式
Demon的博客
04-09 978
版本号： spark-streaming-kafka-0-10_2.11 version:2.4.0 kafka-clients version:0.11.0.0 问题：之前都是使用的0.8版本的来保存offset，但因为线上Kafka集群版本为0.11.0.0的，保存offset的方式发生了很大的变化。 官网的方式： import org.apache.kafka.client...
Spark踩坑记——SparkStreaming+Kafka
02-25
在WeTest舆情项目中，需要对每天千万级的游戏评论信息进行词频统计，在生产者一端，我们将数据按照每天的拉取时间存入了Kafka当中，而在消费者一端，我们利用了sparkstreaming从kafka中不断拉取数据进行词频统计。...
sparkStreaming-offset-to-zk:手动管理spark streaming集成kafka的数据偏移量到zookeeper中
05-03
主要功能提供了快速使用 spark streaming + kafka 开发流式程序的骨架，示例中的代码大部分都加上了详细的注释提供了手动管理kafka的offset存储到zookeeper的方法，并解决了一些bug，如kafka扩容分区，重启实
一个手动管理spark streaming集成kafka时的偏移量到zookeeper中的小项目
02-28
一个手动管理spark streaming集成kafka时的偏移量到zookeeper中的小项目
Spark Streaming + Kafka direct 从Zookeeper中恢复offset
天天 debug
08-02 5179
在上一遍《将 Spark Streaming + Kafka direct 的 offset 保存进入Zookeeper》中，我们已经成功的将 topic 的 partition 的 offset 保存到了 Zookeeper中，使监控工具发挥了其监控效果。那现在是时候来处理《“Spark Streaming + Kafka direct + checkpoints + 代码改变” 引发的问题》中提
Spark框架之SparkStreaming消费Kafka数据手动提交offset到zookeeper
qq_32095251的博客
04-13 2104
package com.bigdata.spark import kafka.common.TopicAndPartition import kafka.message.MessageAndMetadata import kafka.serializer.StringDecoder import org.apache.kafka.clients.consumer.ConsumerConfig ...
SparkStreaming + kafka 的 offset 保存在 Zookeeper、MySQL、HBase、Redis，kafka 中
Jerry的博客
03-08 444
前言 （代码亲测） Kafka做为一款流行的分布式发布订阅消息系统，以高吞吐、低延时、高可靠的特点著称，已经成为Spark Streaming常用的流数据来源。 官方提供的思路就是，把JavaInputDStream转换为OffsetRange对象，该对象具有topic对应的分区的所有信息，每次batch处理完，Spark Streaming都会自动更新该对象，所以你只需要找个合适的地方保存...
Spark+Kafka的Direct方式将偏移量发送到Zookeeper实现
u011007180的博客
07-17 603
http://www.iteblog.com/archives/1381
spark direct kafka 将offset推到zookeeper
09-30 119
1、spark streamingdirect方式读取kafka性能要好很多，缺点是它不会去zookeeper更新offset，这将导致 基于zk的KafkaOffsetMonitor监控失效。由于我们流处...
SparkStreaming 消费 Kafka 数据保存 Offset 到 Redis
storm_fury
09-30 693
Sparkstreaming程序: package com.cloudera.savekafkaoffset import com.cloudera.utils.{JedisPoolUtils, KafkaRedisUtils, RedisConfig} import org.apache.kafka.clients.consumer.ConsumerConfig import org.apac...
SparkStreaming Kafka 自动保存offset到zookeeper
jast
07-12 212
spark使用的是1.6，SparkStreaming1.6时候使用的kafka jar包为0.8的，消费时候不记录消费到的信息，导致重复消费，故手动保存到zookeeper，SparkStreaming2.1.1时使用的kafka jar包为0.10，没有出现这种状况，以下是1.6版本的消费...
Spark-SparkStreaming+Kafka+zookeeper 手动管理offsets
oJueQiang123456的博客
04-13 450
一、概述 SparkStreaming以Kafka作为数据源，手动管理offsets保存在zookeeper中。 SparkStreamingOnKafka(入口)。 KafkaZookeeperUtils(获取数据和更新offsets工具类)。 二、代码 1、SparkStreamingOnKafka package com.cfl.spark.streaming; import ...
SparkStreaming kafka zookeeper本地环境调试安装
bitcarmanlee的博客
05-12 977
1.需求 线上需要使用流式数据实时反馈CTR，因此想用spark streaming计算相关数据。之前一直没有在本地配置streaming的调试环境，因此在本地安装一下streaming的调试环境并记录。 2.需要安装的组件 spark streaming一般会接消息队列作为数据源，以kafka为例，所以需要在本地安装kafka。kafka又依赖zookeeper，所以还需要安装zookeeper。 3. 安装zookeeper 3.1 下载zk 先去zookeeper官网下载zookeeper对应的.bi
spark streaming消费kafka数据手动管理偏移量offset到zookeeper,保证精准消费一次
weixin_43806056的博客
10-10 599
Spark Streaming消费kafka数据手动管理偏移量offset到zookeeper,保证精准一致消费,最终的处理结果保存在mysql中。 spark的版本是2.1 kafka的版本是0.8 代码： /** * @author lhq * @date 2020/10/10 10:35 * @version 1.0 */ import java.sql.{Connection, DriverManager, PreparedStatement} import kafka.com
spark——消费kafka数据保存offset到redis
最新发布
03-16
可以使用Spark Streaming消费Kafka数据，并将消费的offset保存到Redis中。具体步骤如下： 1. 创建Spark Streaming上下文，并设置Kafka参数。 2. 使用KafkaUtils.createDirectStream()方法创建一个DirectStream，该...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值