efk+logstash+kafka（简化版）

帅大大的架构之路

已于 2022-03-16 16:30:08 修改

阅读量781

点赞数 3

分类专栏： # elfk 文章标签： efk

于 2021-06-05 15:57:47 首次发布

本文链接：https://blog.csdn.net/ss810540895/article/details/117597724

版权

elfk 专栏收录该内容

12 篇文章 0 订阅

订阅专栏

efk（仅记录，如有问题可留言）

在这里插入图片描述

cat >> /etc/hosts << EOF
192.168.2.128 node-1
192.168.2.129 node-2
192.168.2.130 node-3
EOF

安装java环境

yum install java-1.8.0-openjdk -y
java -version
openjdk version "1.8.0_262"

安装kafka和zookeeper

mkdir -p /soft/src
cd /soft/src

下载kafka（kafka中自带zookeeper）
wget https://mirrors.bfsu.edu.cn/apache/kafka/2.8.0/kafka_2.12-2.8.0.tgz

tar xf kafka_2.12-2.8.0.tgz
mv kafka_2.12-2.8.0 kafka
scp -r kafka root@node-2:/soft/src/
scp -r kafka root@node-3:/soft/src/

配置zookeeper集群。
cd kafka/config/
[root@node-2 config]# egrep -v '^#|^$' zookeeper.properties
dataDir=/soft/src/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
server.1=node-1:2888:3888
server.2=node-2:2889:3889
server.3=node-3:2890:3890

scp /soft/src/kafka/config/zookeeper.properties root@node-2:/soft/src/kafka/config
scp /soft/src/kafka/config/zookeeper.properties root@node-3:/soft/src/kafka/config

ssh root@node-2 'mkdir /soft/src/kafka/zookeeper'
ssh root@node-2 'echo 2 > /soft/src/kafka/zookeeper/myid'

ssh root@node-3 'mkdir /soft/src/kafka/zookeeper'
ssh root@node-3 'echo 3 > /soft/src/kafka/zookeeper/myid'

启动zookeeper
/soft/src/kafka/bin/zookeeper-server-start.sh -daemon /soft/src/kafka/config/zookeeper.properties
验证是否启动，有时候没启动也不报错
ps -ef|grep zookeeper.properties

配置kafka集群
[root@localhost config]$egrep -v '^#|^$' server.properties	#### 注意，broker.id后面只能写数字，写注释一定会报错
broker.id=1     #三个节点不能一样
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/soft/src/kafka/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.106.7:2181,192.168.106.8:2181,192.168.106.9:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

scp /soft/src/kafka/config/server.properties root@node-2:/soft/src/
scp /soft/src/kafka/config/server.properties root@node-3:/soft/src/

启动kafka
/soft/src/kafka/bin/kafka-server-start.sh -daemon /soft/src/kafka/config/server.properties
ps -ef|grep server.properties

创建topic
/soft/src/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.2.128:2181,192.168.2.129:2181,192.168.2.130:2181 --replication-factor 3 --partitions 1 --topic test-topic
注意：如果只有一个kafka，--replication-factor 这个为 1


查看topic
/soft/src/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.2.128:2181,192.168.2.129:2181,192.168.2.130:2181


生产消息

/soft/src/kafka/bin/kafka-console-producer.sh --broker-list 192.168.2.128:9092,192.168.2.129:9092,192.168.2.130:9092 --topic test-topic

消费消息
/soft/src/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.2.128:9092,192.168.2.129:9092,192.168.2.130:9092 --topic test-topic --from-beginning

查看topic消费进度
/soft/src/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.16.0.15:9092 --topic prod-report --time -1

删除topic
./kafka-topics.sh --delete --zookeeper 172.16.0.15:2181 --topic prod-report

部署filebeat

cd /soft/src
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.13.1-linux-x86_64.tar.gz
tar xf filebeat-7.13.1-linux-x86_64.tar.gz
cd filebeat-7.13.1-linux-x86_64/
./filebeat modules enable nginx

配置filebeat_nginx.yml （记得注释kafka version，不然报错）
vim filebeat_nginx.yml

filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]
 
#----------------------------------Kafka output--------------------------------#
output.kafka:
  enabled: true
  hosts: ['xxx:9092', 'xxx:9092', 'xxx:9092']
  topic: 'test-topic'
  required_acks: 1  #default
  compression: gzip #default
  max_message_bytes: 1000000 #default
  codec.format:
    string: '%{[message]}'


启动filebeat
nohub ./filebeat -e -c filebeat_nginx.yml &

然后访问nginx，再启动kafka消费查看有日志输出

如果有多个项目

vim filebeat-test.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /biz-code/logs/merchant/*.log
  multiline:
      pattern: '^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}'
      negate: true
      match: after
  fields:
    log_topics: merchant
    logtype: merchant-log
- type: log
  enabled: true
  paths:
    - /biz-code/logs/report/*.log
  multiline:
      pattern: '^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}'
      negate: true
      match: after
  fields:
    log_topics: report
    logtype: report-log
output.kafka:
  enabled: true
  hosts: ["172.16.0.15:9092"]
  topic: '%{[fields][log_topics]}'

filebeat输出到kafka问题

注意：在腾讯云服务器和阿里云服务器上面做这一步的时候会有问题

2020-12-02T19:16:44.589+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-12-02T19:16:44.590+0800    INFO    [publisher]     pipeline/retry.go:223     done
2020-12-02T19:16:44.590+0800    INFO    [publisher]     pipeline/retry.go:213   retryer: send wait signal to consumer
2020-12-02T19:16:44.590+0800    INFO    [publisher]     pipeline/retry.go:217     done
2020-12-02T19:16:44.590+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-12-02T19:16:44.591+0800    INFO    [publisher]     pipeline/retry.go:223     done
2020-12-02T19:16:53.325+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-12-02T19:16:53.325+0800    INFO    [publisher]     pipeline/retry.go:223     done
2020-12-02T19:17:05.584+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-12-02T19:17:05.584+0800    INFO    [publisher]     pipeline/retry.go:223     done

它会一直循环这个报错，通过在配置文件中加入 #logging.level: debug 来进行调试，发现 `Kafka publish failed with: dial tcp: lookup VM-0-15-centos on 1.6.82.98:53: no such host` 这个报错

解决：在filebeat主机hosts中加入kafka主机的ip+主机名
vim /etc/hosts
1.1.1.1 VM-0-15-centos

然后就可以看到日志出现在kafka中

部署安装logstash

cd /soft/src
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.9.2.tar.gz
tar xf logstash-7.9.2.tar.gz
cd logstash/config
vim nginx.conf

input{
  kafka {
    bootstrap_servers => ["192.168.2.128:9092,192.168.2.129:9092,192.168.2.130:9092"]
    auto_offset_reset => "latest"
    consumer_threads => 3
    decorate_events => true
    topics => ["test-topic"]
    codec => "json"
  }
}
output {
    elasticsearch {
      hosts => ["42.193.12.10:9200"]
      index => "kafkalog-%{+YYYY.MM.dd}"	# 这里定义的index就是kibana里面显示的索引名称
    }
}

启动，等es安装完了再启动
cd ../bin
nohup ./logstash -f ../config/nginx.conf >> logstash.log &
检查
ps -ef|grep logstash

如果有多个项目，多个topic，那么可以用pipeline定义多个文件，当然也可以在一个conf文件中定义多个topic，以变量的方式，不过我这边没搞定

cd config/

vim pipelines.yml
- pipeline.id: report
  path.config: "/soft/src/logstash/config/report.conf"
- pipeline.id: merchant
  path.config: "/soft/src/logstash/config/merchant.conf"

vim report.conf
input{
  kafka {
    bootstrap_servers => ["172.16.0.15:9092"]
    auto_offset_reset => "latest"
    consumer_threads => 3
    decorate_events => true
    topics => ["report"]
    codec => "json"
  }
}
filter {
    json {
        source => "message"
    }
}

output {
    elasticsearch {
      hosts => ["172.16.0.15:9222"]
      index =>  "report-%{+YYYY-MM-dd}"
    }
}

vim merchant.conf
input{
  kafka {
    bootstrap_servers => ["172.16.0.15:9092"]
    auto_offset_reset => "latest"
    consumer_threads => 3
    decorate_events => true
    topics => ["merchant"]
    codec => "json"
  }
}
filter {
    json {
        source => "message"
    }
}
output {
    elasticsearch {
      hosts => ["172.16.0.15:9222"]
      index =>  "merchant-%{+YYYY-MM-dd}"
    }
}

cd /soft/src/logstash/bin
./logstash > logstash.log

安装es和kibana（因资源不够所以es和kibana使用服务器安装，其他资源使用虚拟机安装）

docker run -p 9200:9200 -p 9330:9300 -itd -e "discovery.type=single-node" --name es \
	-v /es_data:/usr/share/elasticsearch/data \
     docker.elastic.co/elasticsearch/elasticsearch:7.9.2
     
这里可能es容器会起不来，报错 `AccessDeniedException[/usr/share/elasticsearch/data/nodes];`
解决：chmod 777 /es_data

docker run -p 5601:5601 -it -d  --link es -e ELASTICSEARCH_URL=http://localhost:9200 \
         --name kibana kibana:7.9.2
docker exec -it kibana /bin/bash
vi config/kibana.yml
elasticsearch.hosts: [ "http://42.193.12.10:9200/" ]

docker restart kibana

kibana添加索引展示

访问：http://42.193.12.10:5601/
在这里插入图片描述

这里就是logstash里面定义的index

这里展示的就是filebeat收集的nginx日志了

帅大大的架构之路

关注

3
点赞
踩
6

收藏

觉得还不错? 一键收藏
2
评论
efk+logstash+kafka（简化版）

efk（仅记录，如有问题可留言）cat >> /etc/hosts << EOF192.168.2.128 node-1192.168.2.129 node-2192.168.2.130 node-3EOF安装java环境yum install java-1.8.0-openjdk -yjava -versionopenjdk version "1.8.0_262"安装kafka和zookeepermkdir -p /soft/srccd /soft/sr
复制链接

扫一扫