KafkaProducer 示例-拦截器- 分区 KafkaProducer-序列化器

存在,及合理

已于 2024-10-02 22:47:11 修改

阅读量502

点赞数 6

分类专栏：中间件文章标签： linq 数据库 c# kafka

于 2024-10-02 22:44:08 首次发布

本文链接：https://blog.csdn.net/qq_26594041/article/details/142686284

版权

中间件专栏收录该内容

28 篇文章 0 订阅

订阅专栏

拦截器

日志或错误的记录

spring:
  kafka:
    producer:
      properties:
        interceptor:
          classes: com.du.producerInterceptor.MyProducerlnterceptor

implements ProducerInterceptor<String,Object>

@Slf4j
public class MyProducerlnterceptor implements ProducerInterceptor<String,Object> {

    /**
     * 消息成功发送次数
     */
    private int successCount = 0 ;
    private int errorCount = 0 ;

    /**
     * Producer确保在消息被序列化以及计算分区前调用该方法。用户可以在该方法中对消息做任何操作
     * <p> 但最好保证不要修改消息所属的topic和分区，否则会影响目标分区的计算。</p>
     *
     */
    @Override
    public ProducerRecord<String, Object> onSend(ProducerRecord<String, Object> record) {
        return record;
    }

    /**
     * 该方法会在消息从RecordAccumulator成功发送到Kafka Broker之后，或者在发送过程中失败时调用。
     * 并且通常都是在producer回调逻辑触发之前。onAcknowledgement运行在producer的IO线程中，
     * 因此不要在该方法中放入很重的逻辑，否则会拖慢producer的消息发送效率
     */
    @Override
    public void onAcknowledgement(RecordMetadata recordMetadata, Exception e) {
        log.info("拦截器执行 onAcknowledgement {}",recordMetadata ,e);

        //统计成功或失败次数
        if(e == null){
            successCount++;
        }else{
            errorCount++;
        }
        log.info("successCount = {} ; errorCount = {}" ,successCount , errorCount );

    }

    /**
     * <p>关闭interceptor，主要用于执行一些资源清理工作
     * interceptor可能被运行在多个线程中，因此在具体实现时用户需要自行确保线程安全
     * </p>
     * <p>另外倘若指定了多个interceptor则producer将按照指定顺序调用它们，并仅仅是捕获每个interceptor可能抛出的异常记录到错误日志中而非在向上传递。
     *  这在使用过程中要特别留意</p>
     */
    @Override
    public void close() {
        log.info("拦截器执行 close");
    }

    /**
     * 获取配置信息和初始化数据时调用
     * @param configs
     */
    @Override
    public void configure(Map<String, ?> configs) {
        log.info("拦截器执行 configure ；map = {}",configs);

    }
}

序列化器

实现了接org.apache.kafka.common.serialization.serializer的key序列化类
实现了接口org.apache.kafka.common.serialization.serializer的value序列化类

分区

源码

KafkaProducer

private int partition(ProducerRecord<K, V> record, byte[] serializedKey, byte[] serializedValue, Cluster cluster) {
Integer partition = record.partition();
return partition != null ?partition :partitioner.partition(
record.topic(), record.key(), serializedKey, record.value(), serializedValue, cluster);
}

DefaultPartitioner implements Partitioner 
if (keyBytes == null) {
return stickyPartitionCache.partition(topic, cluster);
}
// hash the keyBytes to choose a partition
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

自定义分区

public class SamplePartition implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        /*
            key-1
            key-2
            key-3
         */
        String keyStr = key + "";
        String keyInt = keyStr.substring(4);
        System.out.println("keyStr : "+keyStr + "keyInt : "+keyInt);

        int i = Integer.parseInt(keyInt);

        return i%2;
    }

    @Override
    public void close() {

    }

    @Override
    public void configure(Map<String, ?> configs) {

    }
}

public static void producerSendWithCallbackAndPartition(){

    properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,"com.du.config.SamplePartition");

    for(int i=0;i<10;i++){
        ProducerRecord<String,String> record =
                new ProducerRecord<>(TOPIC_NAME,"key-"+i,"value-"+i);
        producer.send(record, new Callback() {
            @Override
            public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                System.out.println(
                        "partition : "+recordMetadata.partition()+" , offset : "+recordMetadata.offset());
            }
        });
    }
    producer.close();
}

public class keyPartition implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> partitionInfos = cluster.partitionsForTopic(topic);
        int numPartitions = partitionInfos.size();

        if (null == keyBytes || !(key instanceof String)) {
            throw new InvalidRecordException("kafka message must have key");
        }

        if (numPartitions == 1) {
            return 0;
        }

        if (key.equals("name")) {
            return numPartitions - 1;
        }

        return Math.abs(Utils.murmur2(keyBytes)) % (numPartitions - 1);
    }

    @Override
    public void close() {}

    @Override
    public void configure(Map<String, ?> configs) {}
}

KafkaProducer

示例

public class ProducerSample {

  private final static String TOPIC_NAME = "du-topic";
  static Properties properties = new Properties();

  static {
    properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.237.10:9092");
    properties.put(ProducerConfig.ACKS_CONFIG, "all");
    properties.put(ProducerConfig.RETRIES_CONFIG, "0");
    properties.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384");
    properties.put(ProducerConfig.LINGER_MS_CONFIG, "1");
    properties.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "33554432");
    properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
    properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
  }

  static Producer<String, String> producer = new KafkaProducer<>(properties);

  /**
     * 异步阻塞发送
      */
  @Test
  public void syncSend() throws ExecutionException, InterruptedException {
    User du = new User("du", 90);
    ProducerRecord<String, String> stringUserProducerRecord = new ProducerRecord<>("du", JSON.toJSONString(du));
    Future<RecordMetadata> send = producer.send(stringUserProducerRecord);

    //阻塞
    RecordMetadata recordMetadata = send.get();

    System.out.println(recordMetadata.partition());
    System.out.println(recordMetadata.offset());
    producer.close();

  }
  /**
     * 异步发送带回调函数
     */
  @Test
  public void asyncSendCall() throws ExecutionException, InterruptedException {
    User du = new User("du", 90);
    ProducerRecord<String, String> stringUserProducerRecord = new ProducerRecord<>("du", JSON.toJSONString(du));

    producer.send(stringUserProducerRecord,(recordMetadata,ex)->{
      if (null==ex){
        System.out.println("消息发送成功");
      }

    });

    producer.close();

  }
}

ProducerRecord

每条数据都要封装成一个ProducerRecord对象

User du = new User("du", 90);
ProducerRecord<String, String> stringUserProducerRecord = new ProducerRecord<>("du", JSON.toJSONString(du));

KafkaProducer

需要创建一个生产者对象，用来发送数据

Producer<String, String> producer = new KafkaProducer<>(properties);

Future<RecordMetadata> send = producer.send(stringUserProducerRecord);

同步确认

User du = new User("du", 90);
ProducerRecord<String, String> stringUserProducerRecord = new ProducerRecord<>("du", JSON.toJSONString(du));
Future<RecordMetadata> send = producer.send(stringUserProducerRecord);

//阻塞
RecordMetadata recordMetadata = send.get();

System.out.println(recordMetadata.partition());
System.out.println(recordMetadata.offset());
producer.close();

异步确认

回调函数会在producer收到ack时调用，为异步调用，该方法有两个参数，分别是RecordMetadata和Exception，如果Exception为null，说明消息发送成功，如果Exception不为null，说明消息发送失败
注意：消息发送失败会自动重试，不需要我们在回调函数中手动重试

User du = new User("du", 90);
ProducerRecord<String, String> stringUserProducerRecord = new ProducerRecord<>("du", JSON.toJSONString(du));

producer.send(stringUserProducerRecord,(recordMetadata,ex)->{
  if (null==ex){
    System.out.println("消息发送成功");
  }
});

producer.close();