Kafka集群搭建
准备工作
准备三台虚拟器, 装好jdk, 192.168.1.117:9092,192.168.1.128:9092,192.168.1.129:9092
kafka安装
0. systemctl stop firewalld
1.tar -zxvf kafka_2.13-3.0.2.tgz
2.修改 config/server.properties
broker.id=0
listeners=PLAINTEXT://192.168.1.117:9092
zookeeper.connect=192.168.1.117:2181,192.168.1.128:2181,192.168.1.129:2181
主要修改这三条就可以:
* broker.id 需要保证每一台kafka都有一个独立的broker;
* listeners = PLAINTEXT://当前虚拟机ip地址:9092;
* zookeeper.connect zookeeper的连接地址信息。
3.scp -r kafka_2.13-3.0.0 root@192.168.1.128:/opt
scp -r kafka_2.13-3.0.0 root@192.168.1.129:/opt
scp -r jdk-8u211-linux-x64.tar.gz root@192.168.1.185:/opt
4.启动zookeeper
./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
5.启动kafka
./bin/kafka-server-start.sh -daemon config/server.properties
./bin/kafka-server-stop.sh config/server.properties
6.生产消息
./bin/kafka-console-producer.sh --broker-list 192.168.1.175:9092,192.168.1.229:9092,192.168.1.185:9092 --topic test
7.消费消息
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.175:9092,192.168.1.229:9092,192.168.1.185:9092 --topic test --from-beginning
scp -r kafka_2.13-3.0.0 root@192.168.1.128:/opt
scp -r kafka_2.13-3.0.0 root@192.168.1.129:/opt
8.查看topic消费情况
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.117:9092 --topic testTopic --from-beginning
SpringBoot整合Kafka集群
pom依赖
<dependencies>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
</dependencies>
yaml配置
server:
port: 7891
spring:
kafka:
bootstrap-servers: 192.168.1.117:9092,192.168.1.128:9092,192.168.1.129:9092
producer: # 生产者
retries: 3 # 设置大于0的值,则客户端会将发送失败的记录重新发送
batch-size: 16384 # 成批发送消息,消息达到16k发一波
buffer-memory: 33554432 #消息缓存池 32M
acks: 1
# 指定消息key和消息体的编解码方式
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
consumer:
# 消费者组,集群环境下,同一个消费组下的消费者,只会有一个消费者进行消费;同一个group内,consumer数量不能大于partition数,多出来的收不到消息
group-id: default-group
enable-auto-commit: false
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
listener:
concurrency: 3
ack-mode: MANUAL
配置类
@Configuration
@Slf4j
public class KafkaConfig {
@Bean
public NewTopic batchTopic() {
return new NewTopic("testTopic", 8, (short) 1);
}
}
生产者
@Autowired
private KafkaTemplate<String, Object> kafkaTemplate;
@GetMapping("/kafka/{message}")
public void sendMessage(@PathVariable("message") String normalMessage) {
kafkaTemplate.send("testTopic", normalMessage);
}
消费者
@Slf4j
@Component
public class KafkaConsumer {
/**
* 消费监听
*/
@KafkaListener(groupId = "default-group", topics = "testTopic")
public void onMessage1(String body){
log.info("消息内容: {}", body);
}
}
项目启动
这里copy了两份,依次启动,依次修改端口
消费结果