【filebeat】filebeat采集tomcat的数据发送到logstash,logstash发送到ES数据库集群

1.安装filebeat 

 tar xvf filebeat-7.9.2-linux-x86_64.tar.gz
[esadmin@oracle1 soft]$ mkdir -p /esdb/filebeat
[esadmin@oracle1 soft]$ cp -r filebeat-7.9.2-linux-x86_64/* /esdb/filebeat
*/

2.配置filebeat 采集tomcat到Logstash

input {
	beats {
	port => "5044"
	codec => "json"
	client_inactivity_timeout => 36000
	}
}
filter {
	mutate {
	gsub => ["message","\\x","\\\x"]
	}
	json {
	source => "message"
	}
	mutate {
	add_field => { "tomcat_ip" => "%{[@metadata][ip_address]}" }
	convert => ["[geoip][coordinates]", "float"]
	remove_field => [ "host" ]
	}
}
output {
	elasticsearch {
	hosts => ["192.168.1.7:9201", "192.168.1.7:9202","192.168.1.7:9203"]
	index => "sspu-filebeat-%{+YYYY.MM.dd}"
	user => elastic
	password => esadmin
	}
}


--重启logstash 
/esdb/logstash/app/bin/logstash -f /esdb/logstash/app/config/sspu-tomcat.conf &

3.tomcat日志格式

{"clientip":"192.168.1.7","ClientUser":"-","authenticated":"-","AccessTime":"[30/Jun/2024:11:23:28 +0800]","method":"GET / HTTP/1.0","status":"200","SendBytes":"11219","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.1.7","ClientUser":"-","authenticated":"-","AccessTime":"[30/Jun/2024:11:23:28 +0800]","method":"GET / HTTP/1.0","status":"200","SendBytes":"11219","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}

4.logstash端口查看:9600

[esadmin@oracle1 config]$ netstat -anplt |grep 9600
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      60529/java          
[esadmin@oracle1 config]$ ps -ef |grep logstash
esadmin   60529      1 11 11:12 pts/6    00:04:52 /esdb/jdk-17.0.11/bin/java -server -Xms512M -Xmx512M -XX:+DisableExplicitGC -XX:+UseG1GC -XX:G1ReservePercent=25 
-XX:InitiatingHeapOccupancyPercent=30 -Djava.awt.headless=true 
-Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true 
-Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true 
-XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom 
-Dlog4j2.isThreadContextMapInheritable=true 
--add-opens java.base/sun.nio.ch=ALL-UNNAMED 
--add-opens java.base/java.io=ALL-UNNAMED 
--add-opens java.base/java.security=ALL-UNNAMED 
-cp /esdb/logstash/app/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:
/esdb/logstash/app/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:
/esdb/logstash/applogstash-core/lib/jars/commons-codec-1.14.jar:
/esdb/logstash/app/logstash-core/lib/jars/commons-compiler-3.1.0.jar:
/esdb/logstash/app/logstash-core/lib/jars/commons-logging-1.2.jar:
/esdb/logstash/app/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:
/esdb/logstash/app/logstash-core/lib/jars/google-java-format-1.1.jar:
/esdb/logstash/app/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:
/esdb/logstash/app/logstash-core/lib/jars/guava-24.1.1-jre.jar:
/esdb/logstash/app/logstash-core/lib/jars/j2objc-annotations-1.1.jar:
/esdb/logstash/app/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:
/esdb/logstash/app/logstash-core/lib/jars/jackson-core-2.9.10.jar:
/esdb/logstash/app/logstash-core/lib/jars/jackson-databind-2.9.10.4.jar:
/esdb/logstash/app/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/esdb/logstash/app/logstash-core/lib/jars/janino-3.1.0.jar:/esdb/logstash/app/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/esdb/logstash/app/logstash-core/lib/jars/jruby-complete-9.2.13.0.jar:/esdb/logstash/app/logstash-core/lib/jars/jsr305-1.3.9.jar:/esdb/logstash/app/logstash-core/lib/jars/log4j-api-2.13.3.jar:/esdb/logstash/app/logstash-core/lib/jars/log4j-core-2.13.3.jar:/esdb/logstash/app/logstash-core/lib/jars/log4j-jcl-2.13.3.jar:/esdb/logstash/app/logstash-core/lib/jars/log4j-slf4j-impl-2.13.3.jar:/esdb/logstash/app/logstash-core/lib/jars/logstash-core.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/esdb/logstash/app/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/esdb/logstash/app/logstash-core/lib/jars/reflections-0.9.11.jar:/esdb/logstash/app/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f /esdb/logstash/app/config/sspu-tomcat.conf
esadmin   61211  61105  0 11:54 pts/6    00:00:00 grep --color=auto logstash

5.启动 filebeat 

su - esadmin
chmod 755 /esdb/filebeat/sspu-tomcat.yml

nohup /esdb/filebeat/filebeat -e -c /esdb/filebeat/sspu-tomcat.yml &

6.向tomcat写入数据

--重新部署一个tomcat:8081;
ab -n100000 -c10 http://192.168.1.7:8081/
[esadmin@oracle1 bin]$ ab -n100000 -c10 http://192.168.1.7:8081/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.7 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        192.168.1.7
Server Port:            8081

Document Path:          /
Document Length:        11219 bytes

Concurrency Level:      10
Time taken for tests:   40.344 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      1133100000 bytes
HTML transferred:       1121900000 bytes
Requests per second:    2478.68 [#/sec] (mean)
Time per request:       4.034 [ms] (mean)
Time per request:       0.403 [ms] (mean, across all concurrent requests)
Transfer rate:          27427.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      18
Processing:     0    3   8.4      3    1874
Waiting:        0    3   8.3      2    1873
Total:          0    4   8.4      3    1874

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      5
  90%      6
  95%      7
  98%     11
  99%     15
 100%   1874 (longest request)

7.查看数据

[esadmin@oracle1 config]$ jps
60529 Logstash
45909 Elasticsearch
60743 Bootstrap
46155 Elasticsearch
61339 Jps
46351 Elasticsearch


sspu-filebeat-2024.06.30 green open 3 1 9779 9.6mb