使用kafka的一些常见问题

生产者固定key,导致消息集中在某个partition

现象: 服务端接收到的消息集中在固定partition上 原因: kafka生产端默认按照消息key来路由具体的broker机器
解决: 不要指定key,如果需要保证业务上的顺序,就指定key为业务变量
错误代码示例:

	String key = "aa";
    producer.send(new ProducerRecord<>("TOIPICA",key,"这是内容"))  //代码的问题在于,把消息的key硬编码了,导致消息固定往partition发送

分区策略源码 :DefaultPartitioner#partition

如果要保证消息的有序性，可以把保证有序的一类消息设置为同一个key，这样就能保证这类消息发送到同一个 partition 上，但一定要避免，所有的消息公用一个key 当时也可以通过自定义分区策略的方式，参考 生产者基本概念解释#指定分区

消费应用明明正常,但是不消费消息问题

可能的原因:topic的partion数量小于应用数量,kafka消息并行依赖partition的个数,一个topic有多个partition,而每个partition只能分配给一个消费组下的具体实例,当partition数量小于消费组实例的个数时,自然就不够分了
解决:扩大topic的分区数,或者保证每台机器消费均衡的前提下减少consumer的并行数

应用重启,消息消费延迟问题

现象:某系统2020-04-21 21:18:24的消息,21:18:42才被消费
原因:项目发布consumer消费下来的数据未处理成功, session timeout默认30s,导致30s后才能进行rebalance,rebalance后才进行消费之前的消息,导致消息消费延迟
解决:优雅起停,进程关闭的时候回调KafkaConsumer.close方法
示例代码:

      @Bean(destroyMethod = "shutdown")
      public KafkaConsumerDealer commonKafkaDealer() {
        topicCallback.put(KafkaTopic.NOTIFY_SMS_TEMPLATE.name(), commSmsWrapperWorker);
        KafkaConsumerDealer dealer = new KafkaConsumerDealer(threadName, servers, groupId,
            topicCallback, envConstant.getWaitTime());
        dealer.init();
        return dealer;
      }

业务逻辑处理慢,consumer频繁rebalance造成重复消费问题

现象:offset提交失败
原因:业务线程拉取offset超过session timeout默认配置的30s
解决:限制consumer拉取批次数或者优化业务处理性能
实例代码:

	  props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,10); //设置每批次,拉取消息量,效率抵消,慎用
      Consumer<String, String> consumer = new KafkaConsumer<>(props);

应用关闭,导致消息丢失问题

现象:通过线程池多线程方式处理KafkaConsumer拉取的消息,应用关闭,未处理完线程池里面的数据,出现消息丢失
条件:1、应用关闭;2、消费者设置offset自动提交;3、线程池处理KafkaConsumer拉取的消息

原因:消费者设置自动提交, 调用KafkaConsumer#close的时候会自动提交一次offset, 而此时线程池有未处理完的 KafkaConsumer 拉取的消息
解决: 在关闭consumer后，优雅的进行线程池关闭，保证线程池拉取的数据处理完成

较之spring kafka ,原生客户端offset仅提供了自动offset的维护,手动offset需要自己实现,但整体维护成本低,客户端实现灵活

快速入门

jar引入

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>0.10.0.1</version>
</dependency>

版本

建议 0.10.0.1 版本与服务端保持一致

快速使用

生产者示例

@Configuration
public class KafkaConfig {

  @Autowired
  private Environment environment;

  @Bean(name = "producer", destroyMethod = "close")
  public KafkaProducer producer() {
    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092");
    // 0、1(默认的设置) 和 all
    // 0 发送出去就不管了
    // 1 意思就是说只要Partition Leader接收到消息而且写入本地磁盘了，就认为成功了，
    // 不管他其他的Follower有没有同步过去这条消息了
    // all Partition Leader接收到消息之后，还必须要求ISR列表里跟Leader保持同步的那些Follower都要把消息同步过去，
    // 才能认为这条消息是写入成功了。
    props.put(ProducerConfig.ACKS_CONFIG, "all");
    props.put(ProducerConfig.RETRIES_CONFIG, 3);
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");

    //构造Producer对象，注意，该对象是线程安全的，一般来说，一个进程内一个Producer对象即可；
    //如果想提高性能，可以多构造几个对象，但不要太多，最好不要超过5个
    return new KafkaProducer<>(props);
  }
}

//发送代码示例
 ProducerRecord record = new ProducerRecord(this.topic, JSON.toJSONString(pushInfoDto));
 Future future = producer.send(record);

消费者示例

@Configuration
public class KafkaConsumerConfig {
  @Bean
  public NewKafkaConsumerDealer kafkaDealer(
      UicPushInfoReceiveWorker uicPushInfoReceiveWorker) {

    //
    String servers = "";
    String groupId = "";
    final String threadName = "KafkaDealer-";

    Map<String, Callback> topicCallback = Maps.newHashMap();
    topicCallback.put("TOPICA", uicPushInfoReceiveWorker);

    NewKafkaConsumerDealer dealer = new NewKafkaConsumerDealer(threadName, servers, groupId,
        topicCallback);
    return dealer;
  }
 }
  
 //特别注意
 NewKafkaConsumerDealer最好在容器关闭的时候关闭
 
 
 public class NewKafkaConsumerDealer {

  private static final Logger LOG = LoggerFactory.getLogger(
      NewKafkaConsumerDealer.class);

  private static final String THREAD_POOL_NAME_PREFIX = "NewKafkaConsumerDealer-Worker-";

  private Consumer<String, String> consumer;

  private String threadName;

  // 消费线程
  private Thread consumerThread;

  private volatile boolean running = false;

  private BlockingQueue<Runnable> blockingDeque = new ArrayBlockingQueue<Runnable>(256);

  private Properties props;

  private ExecutorService pool = new ThreadPoolExecutor(
      Runtime.getRuntime().availableProcessors() * 2,
      Runtime.getRuntime().availableProcessors() * 2, 120, TimeUnit.SECONDS,
      blockingDeque, NamedThreadFactory.create(THREAD_POOL_NAME_PREFIX), new CallerRunsPolicy()
  );

  private final Map<String, Callback> callbacks;

  public NewKafkaConsumerDealer(String threadName, String servers,
      String groupId
      , Map<String, Callback> callbacks) {
    this.threadName = threadName;
    this.callbacks = Collections.unmodifiableMap(callbacks);
    Assert.notEmpty(callbacks);
    props = new Properties();
    props.put("bootstrap.servers", servers);
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("group.id", groupId);
    props.put("enable.auto.commit", "true");
    props.put("auto.commit.interval.ms", "3000");
    props.put("session.timeout.ms", "30000");
    props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);

    LOG.info("consumer|init|success|topic:{}|servers:{}|groupId:{}", this.callbacks.keySet(),
        servers,
        groupId);
  }

  public void start() {
    consumer = new KafkaConsumer<>(props);
    consumer.subscribe(Lists.newArrayList(this.callbacks.keySet()));
    running = true;
    consumerThread = new Thread(() -> {
      while (running) {
        try {
          //读取数据，读取超时时间为100ms
          ConsumerRecords<String, String> records = consumer.poll(500);

          for (ConsumerRecord<String, String> record : records) {
            Callback callback = callbacks.get(record.topic());
            if (callback == null) {
              LOG.error("invalid topic|{}", record.topic());
              continue;
            }
            pool.submit(new Worker(record, callback));
          }
        } catch (WakeupException exception) {
          // 捕获 WakeupException consumer 准备退出
          LOG.info("WakeupException 准备关闭consumer");
          // 跳出循环，关闭consumer
          break;
        } catch (Exception exception) {
          LOG.error("error cause by ", exception);
        }
      }
      consumer.close();
      boolean isTerminated = ExecutorUtils.shutdownAndAwaitTermination(pool, 1, TimeUnit.SECONDS);
      LOG.info("consumer close success,poll terminated:{}", isTerminated);
    });
    consumerThread.setName(threadName);
    consumerThread.start();
  }

  public void close() {
    if (!running) {
      LOG.info("consumer 不是运行状态，不需要关闭,threadName[{}]", threadName);
      return;
    }
    Stopwatch stopwatch = Stopwatch.createStarted();
    running = false;
    consumer.wakeup();
    try {
      // 等待counsumer线程执行完成
      consumerThread.join();
    } catch (InterruptedException e) {
      LOG.error("consumerThread.join InterruptedException", e);
    }
    LOG.info("consumer线程执行完成，threadName[{}],Spend time[{}]", threadName, stopwatch);
  }

  /**
   * 添加消费者配置，会覆盖默认配置
   */
  public void addConsumerProperties(String key, Object value) {
    props.put(key, value);
  }

}