1、had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord

最新推荐文章于 2024-03-17 13:27:18 发布
最新推荐文章于 2024-03-17 13:27:18 发布
阅读量4.6k 收藏 7
点赞数 3
分类专栏： 含泪踩过的bug
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zj__zfh/article/details/86658304
版权

Spark Streaming 整合Kafka报这个错
Spark版本 2.4.0 Kafka版本0.10.2.2

首先看模拟产生日志代码

public class AccessProducer extends Thread {

    private static SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
    private static Random random = new Random();
    private static String[] sections = new String[] {"country", "international", "sport", "entertainment", "movie", "carton", "tv-show", "technology", "internet", "car"};
    private static int[] arr = new int[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
    private static String date;

    private Producer<Integer, String> producer;
    private String topic;

    public AccessProducer(String topic) {
        this.topic = topic;
        // producer = new Producer<Integer, String>(createProducerConfig());
        producer = new KafkaProducer<>(createProducerProperties());
        date = sdf.format(new Date());
    }

    private Properties createProducerProperties() {
        Properties props = new Properties();
        props.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("bootstrap.servers", "192.168.114.200:9092,192.168.114.201:9092:9092,192.168.114.202:9092");
        return props;
    }

    @Override
    public void run() {
        int counter = 0;

        while(true) {
            for(int i = 0; i < 100; i++) {
                String log = null;

                if(arr[random.nextInt(10)] == 1) {
                    log = getRegisterLog();
                } else {
                    log = getAccessLog();
                }

                producer.send(new ProducerRecord<Integer, String>(topic, i, log));

                counter++;
                if(counter == 100) {
                    counter = 0;
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        }
    }

    private static String getAccessLog() {
        StringBuffer buffer = new StringBuffer("");

        // 生成时间戳
        long timestamp = System.currentTimeMillis();

        // 生成随机userid（默认1000注册用户，每天1/10的访客是未注册用户）
        Long userid = 0L;

        int newOldUser = arr[random.nextInt(10)];
        if(newOldUser == 1) {
            userid = null;
        } else {
            userid = (long) random.nextInt(1000);
        }

        // 生成随机pageid（总共1k个页面）
        Long pageid = (long) random.nextInt(1000);

        // 生成随机版块（总共10个版块）
        String section = sections[random.nextInt(10)];

        // 生成固定的行为，view
        String action = "view";

        return buffer.append(date).append(" ")
                .append(timestamp).append(" ")
                .append(userid).append(" ")
                .append(pageid).append(" ")
                .append(section).append(" ")
                .append(action).toString();
    }

    private static String getRegisterLog() {
        StringBuffer buffer = new StringBuffer("");

        // 生成时间戳
        long timestamp = System.currentTimeMillis();

        // 新用户都是userid为null
        Long userid = null;

        // 生成随机pageid，都是null
        Long pageid = null;

        // 生成随机版块，都是null
        String section = null;

        // 生成固定的行为，view
        String action = "register";

        return buffer.append(date).append(" ")
                .append(timestamp).append(" ")
                .append(userid).append(" ")
                .append(pageid).append(" ")
                .append(section).append(" ")
                .append(action).toString();
    }

    public static void main(String[] args) {
        AccessProducer producer = new AccessProducer("news");
        producer.start();
    }
}

Kafka创建主题

kafka-topics.sh --zookeeper 192.168.114.200:2181,192.168.114.201:2181,192.168.114.202:2181 --topic news --replication-factor 2 --partitions 1 --create

Spark Streaming 整合Kafka

public class NewsRealtimeStatSpark {
    public static void main(String[] args) {
        // 创建Spark上下文
        SparkConf conf = new SparkConf()
                .setMaster("local[2]")
                .setAppName("NewsRealtimeStatSpark");
        JavaStreamingContext jssc = new JavaStreamingContext(
                conf, Durations.seconds(5));

        // 创建输入DStream
        Map<String, Object> kafkaParams = new HashMap<String, Object>();
        kafkaParams.put("bootstrap.servers", "192.168.114.200:9092,192.168.114.201:9092,192.168.114.202:9092");
        kafkaParams.put("key.deserializer", IntegerDeserializer.class);
        kafkaParams.put("value.deserializer", StringDeserializer.class);
        kafkaParams.put("group.id", "111");
        kafkaParams.put("auto.offset.reset", "latest");
        kafkaParams.put("enable.auto.commit", false);
        Collection<String> topics = Arrays.asList("news");
        JavaInputDStream<ConsumerRecord<Integer, String>> lines = KafkaUtils.createDirectStream(jssc,
                LocationStrategies.PreferConsistent(),
                ConsumerStrategies.Subscribe(topics, kafkaParams));

        lines.print();

        jssc.start();
        try {
            jssc.awaitTermination();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        jssc.close();

    }
}

报错信息如下

19/01/26 15:48:45 INFO AppInfoParser: Kafka version : 0.10.2.2
19/01/26 15:48:45 INFO AppInfoParser: Kafka commitId : cd80bc412b9b9701
19/01/26 15:48:45 INFO InternalKafkaConsumer: Initial fetch for spark-executor-111 news-0 115900
19/01/26 15:48:45 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
	- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = news, partition = 0, offset = 115900, CreateTime = 1548486965892, checksum = 3320474937, serialized key size = -1, serialized value size = 51, key = null, value = 2019-01-26 1548486965891 911 550 entertainment view))
	- element of array (index: 0)
	- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11)
	at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:450)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/01/26 15:48:45 ERROR TaskSetManager: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
	- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = news, partition = 0, offset = 115900, CreateTime = 1548486965892, checksum = 3320474937, serialized key size = -1, serialized value size = 51, key = null, value = 2019-01-26 1548486965891 911 550 entertainment view))
	- element of array (index: 0)
	- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11); not retrying
19/01/26 15:48:45 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/01/26 15:48:45 INFO TaskSchedulerImpl: Cancelling stage 0
19/01/26 15:48:45 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
19/01/26 15:48:45 INFO DAGScheduler: ResultStage 0 (print at NewsRealtimeStatSpark.java:42) failed in 0.532 s due to Job aborted due to stage failure: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
	- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = news, partition = 0, offset = 115900, CreateTime = 1548486965892, checksum = 3320474937, serialized key size = -1, serialized value size = 51, key = null, value = 2019-01-26 1548486965891 911 550 entertainment view))
	- element of array (index: 0)
	- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11)
19/01/26 15:48:45 INFO DAGScheduler: Job 0 failed: print at NewsRealtimeStatSpark.java:42, took 0.599568 s
19/01/26 15:48:45 INFO JobScheduler: Finished job streaming job 1548488925000 ms.0 from job set of time 1548488925000 ms
19/01/26 15:48:45 ERROR JobScheduler: Error running job streaming job 1548488925000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
	- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = news, partition = 0, offset = 115900, CreateTime = 1548486965892, checksum = 3320474937, serialized key size = -1, serialized value size = 51, key = null, value = 2019-01-26 1548486965891 911 550 entertainment view))
	- element of array (index: 0)
	- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.streaming.kafka010.KafkaRDD.take(KafkaRDD.scala:139)
	at org.apache.spark.streaming.kafka010.KafkaRDD.take(KafkaRDD.scala:48)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 0.0 (TID 0) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
	- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = news, partition = 0, offset = 115900, CreateTime = 1548486965892, checksum = 3320474937, serialized key size = -1, serialized value size = 51, key = null, value = 2019-01-26 1548486965891 911 550 entertainment view))
	- element of array (index: 0)
	- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.streaming.kafka010.KafkaRDD.take(KafkaRDD.scala:139)
	at org.apache.spark.streaming.kafka010.KafkaRDD.take(KafkaRDD.scala:48)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/01/26 15:48:45 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook

解决方法

创建spark上下文时设置一个属性
set(“spark.serializer”, “org.apache.spark.serializer.KryoSerializer”)

// 创建Spark上下文
        SparkConf conf = new SparkConf()
                .setMaster("local[2]")
                .setAppName("NewsRealtimeStatSpark")
                .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
ZFH__ZJ
关注
  • 3
    点赞
  • 踩
  • 7
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
org.apache.spark.SparkException: Task not serializable
01-07
报错详情 org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288) at org.apache.spark.util.ClosureCleaner$
Spark 之 Had a not serializable result: org.apache.hadoop.hbase.io.ImmutableBytesWritable
qq_32253371的博客
12-26 1502
1. 场景&amp;异常： Spark 读取 Hbase 报错，代码如下： val conf = new SparkConf().setAppName("SparkHistoryTags").setMaster("local") val sc = new SparkContext(conf) // 获取HbaseRDD val hbaseRDD = sc.newAPIHadoopRDD(ge...
多进程数据库不适合作为hive的元数据库
最新发布
weixin_73350116的博客
03-17 516
今天发现一个比较奇怪的现象，因为博主不熟悉mysql，所以在安装hive的使用了postgresql作为hive的元数据库，在测试几个连接工具对hive进行链接，后面再测试的时候发现链接不上了，并且报错日志如下：”null大概意思就是我JVM的资源不足了，内存溢出了这个就很奇怪， 我就测试了几个数据库链接工具，还没上强度，也没上job，怎么就内存溢出了呢？我查看了一下进程（看看是什么事务占用了资源），吓我一跳。
not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
wangchao_cn的专栏
04-18 884
解决方法 创建SparkContext时设置一个属性 set("spark.serializer","org.apache.spark.serializer.KryoSerializer") Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Ta...
SparkStreaming +Kafka ConsumerRecord无法序列化
CyAurora的博客
08-23 731
错误：hadanotserializableresult:org.apache.kafka.clients.consumer.ConsumerRecord 错误：hadanotserializableresult:org.apache.kafka.clients.consumer.ConsumerRecord ConsumerRecord无法序列化 可注册使用Kyro序列化 org.apache.spark.serializer.KryoSerializer Spark没有...
kafka spark java,Java Spark Kafka流中的对象不可序列化(org.apache.kafka.clients.consumer.ConsumerRecord)...
weixin_30291543的博客
02-21 546
I am pretty sure that I am pushing data only string and deserialize also as String. Record I pushed it is showing in error also.But why suddenly it is showing such type of error, Is there anything I a...
Apache Kafka 简单消费者实例
m0_37813354的博客
07-30 533
我们已经创建了一个发送消息到Kafka集群的生产者。现在让我们创建一个消费者来消费Kafka集群的消息。KafkaConsumer API用于消费来自Kafka集群的消息。KafkaConsumer类的构造函数定义如下。 public KafkaConsumer(java.util.Map<java.lang.String,java.lang.Object> configs)...
大数据生态系统基础：Apache Kafka基础（三）：最新kafka编程入门:Consumer
caridle的专栏
08-05 719
生产者将信息输入到集群中， 那么消费者就要能从集群中取出所需要的信息。         主要的类就是： KafkaConsumer 一、原理        Kafka的一个分区的每一个记录保持一个数值偏移。这个偏移量作为该分区内记录的惟一标识符，并表示该分区中的使用者的位置。例如，处于位置5的消费者使用偏移量0到4的记录，然后将使用偏移量5来接收记录。     消费者api提供了覆盖各种消
kafka
weixin_32229529的博客
05-05 397
1.1定义 Kafka是一个分布式的基于发布/订阅模式的消息队列，主要应用于大数据实时处理领域。 离线数仓中也用到。 1.2 消息队列（Message Queue） 1.2.1 传统消息队列的应用场景 注册流程的案例分析:==》发短信比较慢 (1) 同步处理，只有一个线程，发送短信后响应注册；对于并发量大的不适用； （2）异步处理： 把请求写到队列里面，异步线程发送短信 具体作用： 异步，消峰，解...
mysql数据库my.cnf配置文件
04-22
# READ-UNCOMMITTED, READ-COMMITTED, REPEATABLE-READ, SERIALIZABLE. # 如没有指定，MySQL默认采用的是REPEATABLE-READ，ORACLE默认的是READ-COMMITTED log_bin = mysql-bin binlog_format = mixed expire_logs_...
vertigo:Vert.x应用程序平台的基于流的编程
05-21
眩晕 眩晕0.7.0-RC2已发布！ Vertigo 0.7.0需要Vert.x 2.1 需要支持吗？ 查看 | | Vertigo是基于应用程序平台构建的...在Vert.x事件总线上支持可Serializable Java消息 使用随机，循环，mod哈希，公平或扇出方法
http依赖jar包.zip
07-01
import java.io.Serializable; import java.util.HashMap; import java.util.Map; import com.google.gson.JsonObject; import com.google.gson.JsonArray; import net.sf.json.JSONObject; import org.apache....
Serializable序列化步骤.pdf
11-04
Serializable序列化步骤
kafka集群消费之ConsumerRecord类
热门推荐
qq_37191690的博客
03-28 1万+
java.lang.Object继承 org.apache.kafka.clients.consumer.ConsumerRecord <K，V> public final class ConsumerRecord <K，V> extends Object 从Kafka收到的键/值对。这包括主题名称和分区号，从中接收记录以及指向Kafka分区中记录...
java kafka 消费_java编程之Kafka_消费者API详解
weixin_39986973的博客
02-12 295
1消息发送1、异步发送导入依赖org.apache.kafkakafka-clients0.11.0.0编写代码需要用到的类：KafkaProducer：需要创建一个生产者对象，用来发送数据ProducerConfig：获取所需的一系列配置参数ProducerRecord：每条数据都要封装成一个ProducerRecord对象不带回调函数的APIpackage com.heima.kafka;im...
.... from type [java.lang.String] to type [org. apache.kafka.clients.consumer.ConsumerRecord＜? ?＞
那些乐趣的博客
04-25 1504
kafka消费消息的时候，报错No converter found capable of converting from type [java.lang.String] to type [org. apache.kafka.clients.consumer.ConsumerRecord遍历可以获取多个ConsumerRecord
kafka-java代码操作kafka（producer和consumer）配置信息和解释
qq_38417913的博客
08-27 1万+
1.maven: &lt;dependency&gt; &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt; &lt;artifactId&gt;kafka_2.10&lt;/artifactId&gt; &lt;version&gt;0.8.2.1&lt;/version&gt; &lt;/dependency&gt;
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
鸭梨的博客
12-15 711
java消费kafka数据时报错 ERROR [Executor task launch worker for task 90] - Exception in task 0.0 in stage 54.0 (TID 90) java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord Serialization stack: - object not serializable (class: org.a
大数据之Kafka————java来实现kafka相关操作
ycz926940的博客
07-06 3998
生产者多线程是一种常见的技术实践，可以提高消息生产的并发性和吞吐量。通过将消息生产任务分配给多个线程来并行地发送消息，可以有效地利用系统资源，加快消息的发送速度。Java中写在生产者输入内容在kafka中可以让消费者提取。（2）、Producer进行多线程操作。一、在java中配置pom。通过java来实现消费者。(2)、设置多人访问。
object not serializable (class: org.apache.hadoop.fs.path, value: file:/tmp/
01-09
1. 检查类org.apache.hadoop.fs.Path是否实现了Serializable接口。如果没有实现，需要修改该类的定义，添加implements Serializable。 2. 如果该类来自于第三方库，无法直接修改源码，可以考虑创建一个新的类，继承...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值