kafka-生产者-提高生产者吞吐量-参数配置

我们可以通过调整kafka到参数，来提高吞吐量

batch.size: 默认16k， 只有数据积累到batch.size之后，sender才会发送数据。

linger.ms: 等待时间，单位ms, 默认是0ms。修改为5-100ms,会导致数据延迟。

compression.type: snappy 压缩

recordAccumulator: 缓存区大小，默认32m，修改为64m

代码示例

/**
 * kafka生产者-提高吞吐量-参数配置
 */
public class CustomKafkaProviderParameters {
    public static void main(String[] args) {
        // 配置
        Properties properties = new Properties();
        //连接kafka broker
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        //缓冲区大小
        properties.put(ProducerConfig.BUFFER_MEMORY_CONFIG,33554432);
        //设置批次大小
        properties.put(ProducerConfig.BATCH_SIZE_CONFIG,16384);
        //设置5ms
        properties.put(ProducerConfig.LINGER_MS_CONFIG,5000);
        //设置压缩
        properties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");

        //创建kafka 生产者对象
        KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);
        // 发送数据
        for (int i = 0; i < 2; i++) {
            kafkaProducer.send(new ProducerRecord<>("quickstart-events", "record:" + i), (recordMetadata, e) -> {
                if (e == null) {
                    System.out.println("主题：" + recordMetadata.topic());
                }
            });
        }
        // 关闭数据
        kafkaProducer.close();
    }
}