canal kafka 异常问题_record: 5 is not in the range-CSDN博客

本文链接：https://blog.csdn.net/A3213383291/article/details/124724431

问题

查看canal 的日志信息
2022-05-12 01:24:20.539 [pool-4-thread-1] ERROR com.alibaba.otter.canal.kafka.CanalKafkaProducer - Invalid partition given with record: 1 is not in the range [0…1).
org.apache.kafka.common.KafkaException: Invalid partition given with record: 1 is not in the range [0…1).
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:920) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:794) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784) ~[kafka-clients-1.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:671) ~[kafka-clients-1.1.1.jar:na]
at com.alibaba.otter.canal.kafka.CanalKafkaProducer.send(CanalKafkaProducer.java:118) ~[canal.server-1.1.2.jar:na]
at com.alibaba.otter.canal.server.CanalMQStarter.worker(CanalMQStarter.java:148) [canal.server-1.1.2.jar:na]
at com.alibaba.otter.canal.server.CanalMQStarter.access $500 (C a n a l M Q S t a r t e r . j a v a : 21) [c a n a l . s e r v e r - 1.1.2 . j a r : n a] a t c o m . a l i b a b a . o t t e r . c a n a l . s e r v e r . C a n a l M Q S t a r t e r$ CanalMQRunnable.run(CanalMQStarter.java:197) [canal.server-1.1.2.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

问题原因

kafka中生产的消息超出分区
kafka 的配置文件server.properties中的分区数默认配置是1
num.partitions=1
canal的 instance.properties中配置了对应kafka的分区数为4
在这里插入图片描述

导致kafka生产的消息超出了分区数，

解决办法

将canal_test 的分区数提高到4，

 bin/kafka-topics.sh --bootstrap-server niit02:9092 --alter --partitions 4 --topic canal_test

重新启动
查日志问题解决
在这里插入图片描述