Kafka集群搭建 & SpringBoot整合Kafka集群

Kafka集群搭建

准备工作

准备三台虚拟器， 装好jdk， 192.168.1.117:9092,192.168.1.128:9092,192.168.1.129:9092

kafka安装

0. systemctl stop firewalld

1.tar -zxvf kafka_2.13-3.0.2.tgz

2.修改 config/server.properties
broker.id=0
listeners=PLAINTEXT://192.168.1.117:9092
zookeeper.connect=192.168.1.117:2181,192.168.1.128:2181,192.168.1.129:2181

主要修改这三条就可以：
* broker.id 需要保证每一台kafka都有一个独立的broker；
* listeners = PLAINTEXT://当前虚拟机ip地址:9092；
* zookeeper.connect zookeeper的连接地址信息。

3.scp -r kafka_2.13-3.0.0 root@192.168.1.128:/opt
scp -r kafka_2.13-3.0.0 root@192.168.1.129:/opt
scp -r jdk-8u211-linux-x64.tar.gz root@192.168.1.185:/opt

4.启动zookeeper
./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties

5.启动kafka
./bin/kafka-server-start.sh -daemon config/server.properties
./bin/kafka-server-stop.sh  config/server.properties

6.生产消息
./bin/kafka-console-producer.sh --broker-list 192.168.1.175:9092,192.168.1.229:9092,192.168.1.185:9092 --topic test

7.消费消息
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.175:9092,192.168.1.229:9092,192.168.1.185:9092 --topic test --from-beginning

scp -r kafka_2.13-3.0.0 root@192.168.1.128:/opt
scp -r kafka_2.13-3.0.0 root@192.168.1.129:/opt

8.查看topic消费情况
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.117:9092 --topic testTopic --from-beginning

SpringBoot整合Kafka集群

pom依赖

    <dependencies>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
    </dependencies>

yaml配置

server:
  port: 7891

spring:
  kafka:
    bootstrap-servers: 192.168.1.117:9092,192.168.1.128:9092,192.168.1.129:9092
    producer: # 生产者
      retries: 3 # 设置大于0的值，则客户端会将发送失败的记录重新发送
      batch-size: 16384 # 成批发送消息，消息达到16k发一波
      buffer-memory: 33554432 #消息缓存池 32M
      acks: 1
      # 指定消息key和消息体的编解码方式
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      # 消费者组，集群环境下，同一个消费组下的消费者，只会有一个消费者进行消费；同一个group内，consumer数量不能大于partition数，多出来的收不到消息
      group-id: default-group
      enable-auto-commit: false
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    listener:
      concurrency: 3
      ack-mode: MANUAL

配置类

@Configuration
@Slf4j
public class KafkaConfig {

    @Bean
    public NewTopic batchTopic() {
        return new NewTopic("testTopic", 8, (short) 1);
    }


}

生产者

    @Autowired
    private KafkaTemplate<String, Object> kafkaTemplate;

    @GetMapping("/kafka/{message}")
    public void sendMessage(@PathVariable("message") String normalMessage) {
        kafkaTemplate.send("testTopic", normalMessage);
    }

消费者

@Slf4j
@Component
public class KafkaConsumer {
    /**
     * 消费监听
     */
    @KafkaListener(groupId = "default-group", topics = "testTopic")
    public void onMessage1(String body){
        log.info("消息内容: {}", body);
    }

}

项目启动

这里copy了两份，依次启动，依次修改端口

消费结果