环境
10.0.0.51: Logstash、Kafka、Filebeat
一、Zookkeeper和Kafka安装
1、依赖
Kafka
- Kafka依赖于Zookkeeper
- 两个都依赖于Java
Kafka依赖于Zookeeper
- 官方网站:https://zookeeper.apache.org/
- 下载ZK的二进制包
- 解压到对应目录完成安装
2、下载解压ZK,复制默认配置文件
tar -zxf zookeeper-3.4.13.tar.gz
cp /usr/local/zookeeper-3.4.13/conf/zoo_sample.cfg /usr/local/zookeeper-3.4.13/conf/zoo.cfg
3、更改配置文件,监听所有地址
vim /usr/local/zookeeper-3.4.13/conf/zoo.cfg
clientPortAddress=0.0.0.0
3、启动zookeeper
/usr/local/zookeeper-3.4.13/bin/zkServer.sh start
4、zookeeper监听的端口为2181
[root@elk-node2-51 local]# netstat -tlunp | grep 2181
tcp6 0 0 :::2181 :::* LISTEN 15295/java
5、Kafka下载解压
Kafka官网:http://kafka.apache.org/
tar -zvxf kafka_2.11-2.1.1.tgz
mv kafka_2.11-2.1.1 /usr/local/kafka_2.11
6、更改kafka的配置:更改监听地址、更改连接zk的地址
vim /usr/local/kafka_2.11/config/server.properties
listeners=PLAINTEXT://10.0.0.51:9092
zookeeper.connect=10.0.0.51:2181
7、启动
前台启动
/usr/local/kafka_2.11/bin/kafka-server-start.sh /usr/local/kafka_2.11/config/server.properties
后台启动
nohup /usr/local/kafka_2.11/bin/kafka-server-start.sh /usr/local/kafka_2.11/config/server.properties >/tmp/kafka.log 2>&1 &
8、Kafka监听9092端口
[root@elk-node2-51 local]# netstat -tlunp | grep 9092
tcp6 0 0 10.0.0.51:9092 :::* LISTEN 15475/java
二、filebeat–>kafka–>logstash
1、修改filebeat配置文件,然后重新启动filebeat,将Filebeat日志发送到Kafka
vim /usr/local/filebeat-6.6.0/filebeat.yml
[root@elk-node2-51 local]# cat /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
tail_files: true
backoff: "1s"
paths:
- /usr/local/nginx/logs/access.log
fields:
type: access
fields_under_root: true
- type: log
tail_files: true
backoff: "1s"
paths:
- /var/log/secure
fields:
type: secure
fields_under_root: true
output:
kafka:
hosts: ["10.0.0.51:9092"]
topic: wangxiaoyu
2、修改Logstash的配置,从kafka读取数据,重启Logstash
如果没有配置codec => “json”,对从kafka读取的数据进行json解析,logstash虽然可以读取消费kafka里的数据,但没法将数据发送个elasticsearch,我日志也不是json格式的呀。。。。应该是kafka默认会把数据格式转为json。
[root@elk-node2-51 local]# cat /usr/local/logstash-6.6.0/config/logstash.conf
input {
kafka {
codec => "json"
bootstrap_servers => "10.0.0.51:9092"
topics => ["wangxiaoyu"]
group_id => "wangxiaoyu"
}
}
filter {
mutate {
rename => { "[host][name]" => "host" }
}
}
output{
if [type] == "access" {
elasticsearch {
hosts => ["http://10.0.0.50:9200"]
index => "access-%{+YYYY.MM.dd}"
}
}
if [type] == "secure" {
elasticsearch {
hosts => ["http://10.0.0.50:9200"]
index => "secure-%{+YYYY.MM.dd}"
}
}
}
3、Kafka查看队列信息,有个消费者可以查看读取到哪里了。
[root@elk-node2-51 bin]# cd /usr/local/kafka_2.11/bin
[root@elk-node2-51 bin]# ./kafka-consumer-groups.sh --bootstrap-server 10.0.0.51:9092 --list
wangxiaoyu
[root@elk-node2-51 bin]# ./kafka-consumer-groups.sh --bootstrap-server 10.0.0.51:9092 --group wangxiaoyu --describe
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
wangxiaoyu 0 14 14 0 logstash-0-d5c3e806-655a-4d8c-870f-3daabffb38ce /10.0.0.51 logstash-0
4、查看日志