ELK实战之Tomcat的json日志收集

最新推荐文章于 2024-04-15 04:08:29 发布

weixin_33695082

最新推荐文章于 2024-04-15 04:08:29 发布

阅读量259

点赞数

文章标签： java json 大数据

原文链接：https://yq.aliyun.com/articles/485279

版权

一、安装tomcat

1、下载tomcat

[root@linux-node2 ~]# wget http://apache.fayea.com/tomcat/tomcat-8/v8.5.20/bin/apache-tomcat-8.5.20.tar.gz
[root@linux-node2 ~]# tar -zxf apache-tomcat-8.5.24.tar.gz
[root@linux-node2 ~]# mv apache-tomcat-8.5.24 /usr/local/tomcat

2、修改tomcat日志格式

[root@linux-node2 ~]# cd /usr/local/tomcat/conf
[root@linux-node2 conf ]# cp server.xml{,.bak}
[root@linux-node2 conf ]# vim server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_log" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

3、增加tomcat网页

[root@linux-node2 conf ]# cd ../webapps/
[root@linux-node2 webapps ]# mkdir webdir && cd webdir
[root@linux-node2 webdir ]# echo "<h1>welcome to use tomcat</h1>" > index.html
[root@linux-node2 conf ]# ../bin/catalina.sh start
[root@linux-node2 conf ]# netstat -tulnp |grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      2362/java

4、压测页面，生成tomcat的访问日志

[root@linux-node2 conf ]# ab -n1000 -c100 http://192.168.56.12:8080/webdir/index.html 
[root@linux-node2 ~]# tailf /usr/local/tomcat/logs/tomcat_access_log.2017-12-28.log 
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:56 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}
{"clientip":"192.168.56.12","ClientUser":"-","authenticated":"-","AccessTime":"[28/Dec/2017:09:52:57 +0800]","method":"GET /webdir2/index.html HTTP/1.0","status":"200","SendBytes":"32","Query?string":"","partner":"-","AgentVersion":"ApacheBench/2.3"}

二、配置logstash

1、配置logstash

[root@linux-node2 ~]# vim /etc/logstash/conf.d/tomcat_accesslog.conf 
input {
        file {
                path => "/usr/local/tomcat/logs/tomcat_access_log.*.log"
                type => "tomcat-accesslog"
                start_position => "beginning"
                stat_interval => "2"
        }
}

output {
        elasticsearch {
                hosts => ["192.168.56.11:9200"]
                index => "logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
        }
        file {
                path => "/tmp/logstash-tomcat5612-accesslog-%{+YYYY.MM.dd}"
        }
}

2、检查logstash的配置语法并重启logstash

[root@linux-node2 tomcat]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat_accesslog.conf -t
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@linux-node2 tomcat]# systemctl restart logstash
[root@linux-node2 tomcat]# ps -ef |grep logstash
logstash  2527     1 98 09:33 ?        00:00:28 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root      2572 15939  0 09:34 pts/3    00:00:00 grep --color=auto logstash

3、elasticsearch的head插件查看

ELK实战之Tomcat的json日志收集
数据浏览：

注：此处如果无法出现tomcat的数据索引，查看一下是否有权限访问日志。

[root@linux-node2 logs]# ll /usr/local/tomcat/logs/ -d
drwxr-x--- 2 root root 4096 12月 28 09:29 /usr/local/tomcat/logs/
[root@linux-node2 logs]# chmod 755 /usr/local/tomcat/logs
[root@linux-node2 logs]# ll /usr/local/tomcat/logs/
总用量 512
-rw-r----- 1 root root   7140 12月 28 09:29 catalina.2017-12-28.log
-rw-r----- 1 root root   7140 12月 28 09:29 catalina.out
-rw-r----- 1 root root      0 12月 28 09:29 host-manager.2017-12-28.log
-rw-r----- 1 root root    284 12月 28 09:29 localhost.2017-12-28.log
-rw-r----- 1 root root      0 12月 28 09:29 manager.2017-12-28.log
-rw-r----- 1 root root 502039 12月 28 09:47 tomcat_access_log.2017-12-28.log
[root@linux-node2 logs]# chmod 644 /usr/local/tomcat/logs/