【filebeat】配置抽取nginx数据到 kafka

1.kafka环境准备 

tar xvf  kafka_2.12-2.7.0.tgz
[esadmin@oracle1 soft]$ ln -sv kafka_2.12-2.7.0 kafka
‘kafka’ -> ‘kafka_2.12-2.7.0’

2.检查Logstash 插件
[esadmin@oracle1 app]$ ./bin/logstash-plugin list --verbose |grep kafka
2024-06-30T17:19:54.605+08:00 [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=org.jruby.dist --add-opens java.base/java.io=org.jruby.dist' to enable.
logstash-integration-kafka (10.4.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka

3.kafka配置 
mkdir -p /esdb/soft/kafka/logs
vi /esdb/soft/kafka/config/server.properties

2.zookeeper安装

直接使用kafka自带的zk集群。

mkdir -p /esdb/soft/kafka/zookeeper

vi /esdb/soft/kafka/config/zookeeper.properties
dataDir /esdb/soft/kafka/zookeeper

--启动zookeeper 
nohup /esdb/soft/kafka/bin/zookeeper-server-start.sh /esdb/soft/kafka/config/zookeeper.properties &

--启动kafka;
nohup /esdb/soft/kafka/bin/kafka-server-start.sh /esdb/soft/kafka/config/server.properties &

3.配置nginx-to-kafka

vi /esdb/logstash/app/config/pipelines/nginx-to-kakfa.conf

vi /esdb/logstash/app/config/pipelines.yml
  path.config: "/esdb/logstash/app/config/pipelines/nginx-to-kakfa.conf"

--启动Logstash;
nohup /esdb/logstash/app/bin/logstash &

--7.检查kafka数据
--目前kafka还没有主题。
[root@fgesdb71 ~]# /esdb/soft/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.7:2181

4.配置filebeat抽取nginx的数据到logstash;

vi /esdb/filebeat-nginx/sspu-nginx.yml


8.启动filebeat; 
启动filebeat; 
nohup /esdb/filebeat-nginx/filebeat -e -c /esdb/filebeat-nginx/sspu-nginx.yml &


[root@oracle1 ~]# jps
72865 Kafka 
45909 Elasticsearch
72486 QuorumPeerMain (kafka-zookeeper)
88248 Logstash
46155 Elasticsearch
88669 Jps
62557 Bootstrap (tomcat)
46351 Elasticsearch


/esdb/soft/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.7:2181
__consumer_offsets

5.nginx写入数据 

[root@oracle1 config]# ab -n10000 -c2 http://192.168.1.7/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.7 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.20.2
Server Hostname:        192.168.1.7
Server Port:            80

Document Path:          /
Document Length:        153 bytes

Concurrency Level:      2
Time taken for tests:   3.527 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Total transferred:      3030000 bytes
HTML transferred:       1530000 bytes
Requests per second:    2834.93 [#/sec] (mean)
Time per request:       0.705 [ms] (mean)
Time per request:       0.353 [ms] (mean, across all concurrent requests)
Transfer rate:          838.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:     0    1   1.1      0      65
Waiting:        0    0   1.1      0      65
Total:          0    1   1.2      0      66

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      1
  80%      1
  90%      1
  95%      2
  98%      2
  99%      3
 100%     66 (longest request)

6.kafka查看数据 

[root@oracle1 config]# /esdb/soft/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.7:2181
__consumer_offsets
sspu-nginx

--有16个分区。
[root@oracle1 kafka-0-logs]# pwd
/esdb/soft/kafka/logs/kafka-0-logs
[root@oracle1 kafka-0-logs]# ll |grep sspu-nginx
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-0
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-1
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-10
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-11
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-12
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-13
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-14
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-15
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-2
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-3
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-4
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-5
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-6
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-7
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-8
drwxrwxr-x 2 esadmin esadmin  141 Jul  1 11:12 sspu-nginx-9