ELK + kafka + filebeat日志采集平台搭建

环境信息

节点信息如下

节点ip	功能	主机名
192.168.205.6	kafka\zookeeper\filebeat	node01
192.168.205.7	kafka\zookeeper\filebeat	node02
192.168.205.8	kafka\zookeeper\filebeat	node03
192.168.205.9	elasticsearch\logstash\kibana	es-node1
192.168.205.10	elasticsearch\logstash	es-node2
192.168.205.11	elasticsearch\logstash	es-node3

本文节点的操作系统统一为centos 7.8，系统初始化禁用selinux和关闭firewalld

软件装备

需要装备的软件有
1）jdk
下载地址为
https://mirrors.huaweicloud.com/java/jdk/

2)kafka(包含zookeeper模块)
下载地址为
http://kafka.apache.org/downloads

3）Elasticsearch、Kibana、filebeat、Logstash
下载地址为
https://www.elastic.co/cn/downloads/

所有节点安装jdk

将jdk软件包上传到机器后安装jdk

rpm -ivh jdk-8u202-linux-x64.rpm

编辑环境变量vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_202-amd64
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}

查看是否安装完成
java -version

kafka\zookeeper安装

解压软件包

将软件包上传到kafka\zookeeper节点后解压

mkdir /home/kafka
tar -xzvf kafka_2.12-3.0.0.tgz -C /home/kafka
cd /home/kafka
mv kafka_2.12-3.0.0 kafka

安装zookeeper

cd /home/kafka
mkdir zkdata
mkdir zklog

vim /home/kafka/kafka/config/zookeeper.properties修改成以下项</