ELK集群部署(三)之部署Kafka

部署Kafka消息中间件

单kafka安装过程如下

cd /usr/local/src
wget http://mirrors.aliyun.com/apache/kafka/3.2.0/kafka_2.12-3.2.0.tgz
tar -zxf kafka_2.12-3.2.0.tgz

完整的配置
vi /usr/local/src/kafka_2.12-3.2.0/config/server.properties
broker.id=0
num.network.threads=3
num.io.threads=8

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.dirs=/var/log/kafka
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connection.timeout.ms=18000

listeners=PLAINTEXT://192.168.56.102:9092
advertised.listeners=PLAINTEXT://192.168.56.102:9092
zookeeper.connect=192.168.56.102:2181

listeners advertised.listeners：填写kafka监听的IP和端口
zookeeper.connect：填写zookeeper监听的IP和端口

启动顺序: 1.zookeeper 2.kafk
停止顺序: 1.kafka 2.zookeeper

启动zookeeper和kafka命令

启动zookeeper
/usr/local/src/kafka_2.12-3.2.0/bin/zookeeper-server-start.sh /usr/local/src/kafka_2.12-3.2.0/config/zookeeper.properties

启动kafka
/usr/local/src/kafka_2.12-3.2.0/bin/kafka-server-start.sh /usr/local/src/kafka_2.12-3.2.0/config/server.properties

后台运行
nohup /usr/local/src/kafka_2.12-3.2.0/bin/zookeeper-server-start.sh config/zookeeper.properties &

nohup /usr/local/src/kafka_2.12-3.2.0/bin/kafka-server-start.sh /usr/local/src/kafka_2.12-3.2.0/config/server.properties &

如果提示/usr/local/src/kafka_2.12-3.2.0/bin/kafka-run-class.sh: line 342: exec: java: not found

yum install -y java-1.8.0-openjdk.x86_64

停止kafka

/usr/local/src/kafka_2.12-3.2.0/bin/kafka-server-stop.sh /usr/local/src/kafka_2.12-3.2.0/config/server.properties

停止zookeeper

/usr/local/src/kafka_2.12-3.2.0/bin/zookeeper-server-stop.sh /usr/local/src/kafka_2.12-3.2.0/config/zookeeper.properties

kafka安装完成之后，启动就可以了，filebeat配置好之后，会自动在kafka上面创建topic和发送消息。
logstash可以从topic上获取消息。