Logstash收集多类型日志、if判断使用、Tomcat访问日志转json及收集(小节3)

Soul_2016

于 2023-05-10 01:42:40 发布

阅读量466

点赞数

文章标签： tomcat json firefox chrome java

本文链接：https://blog.csdn.net/Soul_2016/article/details/132971677

版权

highlight: a11y-dark

theme: juejin

三、通过 logstash 收集日志

3.1. 通过 logstash 收集单个日志、if判断使用

前提需要logstash用户对被收集的日志文件有读的权限并对写入的文件有写权限。

web1(106)

安装openjdk apt install openjdk-8-jdk -y

下载

安装包：nginx、logstash cd /usr/local/src/ wget http://nginx.org/download/nginx-1.16.1.tar.gz 安装logstash dpkg -i logstash-6.8.3.deb 安装Nginx(编译安装出错可参考) tar xvf nginx-1.16.1.tar.gz cd nginx-1.16.1/ ./configure --prefix=/apps/nginx make make install 启动nginx /apps/nginx/sbin/nginx

可以访问

图片.png

配置logstash ``` cat /etc/logstash/conf.d/log-to-es.conf

输入

input {

系统日志

file { path => "/var/log/syslog"

start_position => "beginning"

stat_interval => "3"
type => "syslog"

}

Nginx日志

file { path => "/apps/nginx/logs/access.log" startposition => "beginning" statinterval => "3" type => "nginx-accesslog" } }

输出

output {

判断

if [type] == "syslog" { stdout { codec => "rubydebug" } }

if [type] == "nginx-accesslog" { stdout { codec => "rubydebug" }

要保证此目录有logstash权限，因为此目录是用logstash起来的。

file {
  path => "/tmp/nginx.log"
}

} } 修改logstash配置权限 vim /etc/systemd/system/logstash.service [Service] User=root Group=root 检查配置文件 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf -t ...省略中间部分 [INFO ] 2020-05-10 01:14:55.355 [LogStash::Runner] runner - Using config.testandexit mode. Config Validation Result: OK. Exiting Logstash <--正常 ```

启动 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf

host1(101)

显示头部信息 curl --head http://192.168.37.106/index.html HTTP/1.1 200 OK Server: nginx/1.16.1 Date: Tue, 09 May 2020 17:20:29 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 09 May 2020 16:53:40 GMT Connection: keep-alive ETag: "645a7a94-264" Accept-Ranges: bytes

3.2. 通过 logstash 收集多类型日志、if判断使用

web1(106)

复制后、粘贴的json在线翻译 vim /tmp/nginx.log ...最后结尾部分(粘贴下行) {"message":"192.168.37.101 - - [10/May/2020:01:34:19 +0800] \"HEAD /index.html HTTP/1.1\" 200 0 \"-\" \"curl/7.58.0\"","@timestamp":"2020-05-09T17:34:34.738Z","type":"nginx-accesslog","path":"/apps/nginx/logs/access.log","@version":"1","host":"ubuntu-6"}

图片.png

修改文件 ``` vim /etc/logstash/conf.d/log-to-es.conf

输入

input {

系统日志

file { path => "/var/log/syslog"

start_position => "beginning"

stat_interval => "3"
type => "syslog"

}

Nginx日志

file { path => "/apps/nginx/logs/access.log" startposition => "beginning" statinterval => "3" type => "nginx-accesslog" } }

输出

output {

判断

if [type] == "syslog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "syslog-37-106-%{+YYYY.ww}" }}

if [type] == "nginx-accesslog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "nginx-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```

检查 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf -t 设为开机启动、重启服务 systemctl enable logstash systemctl restart logstash 查看日志 tail -f /var/log/logstash/logstash-plain.log ...省略中间部分(下面提示成功) [2020-05-10T13:28:02,911][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

图片.png

创建索引(syslog-37-106)

图片.png

时间戳

图片.png

创建索引(nginx-accesslog-37-106)

图片.png

时间戳

图片.png

web1(106)

写入一个值后、刷新页面可以看到 echo "333" >> /var/log/syslog

图片.png

3.3. 通过 logtsash 收集tomcat日志和java日志

3.3.1 通过 logtsash 收集tomcat日志

收集Tomcat服务器的访问日志以及Tomcat错误日志进行实时统计，在kibana页面进行搜索展现，每台Tomcat服务器要安装logstash负责收集日志，然后将日志转发给elasticsearch进行分析，在通过kibana在前端展现，配置过程如下：

web1(106)

tomcat官方地址:https://archive.apache.org/dist/tomcat/

下载包

``` cd /usr/local/src

wget https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.47/bin/apache-tomcat-8.5.47.tar.gz

解压缩

tar xvf apache-tomcat-8.5.47.tar.gz

创建目录

cd apache-tomcat-8.5.47/ mkdir webapps/app

创建页面

echo "linux01" > webapps/app/index.html 修改文件

当前所在目录

pwd /usr/local/src/apache-tomcat-8.5.47

vim conf/server.xml ... ```

启动tomcat ./bin/catalina.sh start

图片.png

查看网页信息

图片.png

软链接 ln -sv /usr/local/src/apache-tomcat-8.5.47 /apps/tomcat 查看日志 tail -f /apps/tomcat/logs/tomcat-37-106_access_log.2023-05-10.log 192.168.37.1 - - [10/May/2023:14:21:45 +0800] "GET /app/ HTTP/1.1" 200 8 192.168.37.1 - - [10/May/2023:14:21:46 +0800] "GET /favicon.ico HTTP/1.1" 200 21630 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET / HTTP/1.1" 200 11215 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /tomcat.css HTTP/1.1" 200 5581 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /bg-nav.png HTTP/1.1" 200 1401 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /asf-logo-wide.svg HTTP/1.1" 200 27235 ^C 编辑文件 ``` vim /etc/logstash/conf.d/tomcat-log-to-es.conf

输入

input {

tomcat日志

file { path => "/apps/tomcat/logs/tomcat-37-106accesslog.*.log"

start_position => "beginning"

stat_interval => "3"
type => "tomcat-access-log"

} }

输出

output {

判断

if [type] == "tomcat-access-log" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tomcat-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```

检查文件

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat-log-to-es.conf -t

重启服务

systemctl restart logstash

图片.png

刷新页面可以看到

图片.png

添加索引

图片.png

时间戳

图片.png

tomcat日志改json格式(参数介绍)

``` vim /usr/local/src/apache-tomcat-8.5.47/conf/server.xml ...省略中间内容

将下行改成如下信息

pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

重启tomcat cd /usr/local/src/apache-tomcat-8.5.47/

停服务

./bin/catalina.sh stop

启服务

./bin/catalina.sh start ```

刷新页面：http://192.168.37.106:8080/后

图片.png

可以在监控中看到json信息、 tail -f logs/tomcat-37-106_access_log.2023-05-11.log ...省略中间部分 {"clientip":"192.168.37.1","ClientUser":"-","authenticated":"-","AccessTime":"[11/May/2023:00:53:34 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"11215","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"} 看复制上面信息、到网站中校验

图片.png

文件中添加codec、对日志的格式修改 ``` cat /etc/logstash/conf.d/tomcat-log-to-es.conf

输入

input {

tomcat日志

file { path => "/apps/tomcat/logs/tomcat-37-106accesslog.*.log"

start_position => "beginning"

stat_interval => "3"
type => "tomcat-access-log"
codec => "json"

} }

输出

output {

判断

if [type] == "tomcat-access-log" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tomcat-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```

重启服务

systemctl restart logstash

host1(101)

curl --head http://192.168.37.106:8080/app/index.html

强制刷新、刷新前

图片.png

刷新后

图片.png

再查看

图片.png

创建可视化

图片.png

3.3.2 通过 logtsash 收集java日志

使用codec的multiline插件实现多行匹配，这是一个可以将多行进行合并的插件，而且可以使用what指定将匹配到的行与前面的行合并还是和后面的行合并，https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

web1(106)

配置信息 ``` cd /etc/logstash/conf.d/

cat java-log-to-es.conf input { file {

日志文件位置

path => "/var/log/java.log"
type => "javalog"
start_position => "beginning"
codec => multiline {

当遇到"["开头的行时候将多行进行合并

pattern => "^\["

"true"为匹配成功进行操作，"false"为不成功进行操作

negate => true

与以前的行合并，如果是下面的行合并就是"next"

what => "previous"

}} }

日志过滤，如果所有的日志都过滤就写这里，如果只针对某一个过滤就写在"input"里面的日志输入里面

filter { }

output { if [type] == "javalog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "javalog-7-106-%{+YYYY-MM.dd}" }} } 检查 /usr/share/logstash/bin/logstash -f java-log-to-es.conf -t 启动 /usr/share/logstash/bin/logstash -f java-log-to-es.conf ``` 到kibana中添加"http://192.168.37.101:5601/app/kibana#/home?_g=()" ,【步骤：管理-(kibana)索引模式-创建索引模式-索引模式(javalog-37.106-*)-时间戳-完成】即可

将日志类型改为json格式、编辑nginx文件、添加日志类型(可添加多个) ``` vim /apps/nginx/conf/nginx.conf http { ...

在http中添加如下信息(注1)

log_format access_json '{"@timestamp":"$time_iso8601",'
    '"host":"$server_addr",'
    '"clientip":"$remote_addr",'
    '"size":$body_bytes_sent,'
    '"responsetime":$request_time,'
    '"upstreamtime":"$upstream_response_time",'
    '"upstreamhost":"$upstream_addr",'
    '"http_host":"$host",'
    '"uri":"$uri",'
    '"domain":"$host",'
    '"xff":"$http_x_forwarded_for",'
    '"referer":"$http_referer",'
    '"status":"$status"}';

日志存放目录、类型(access_json和上面'注1'一致)

access_log  /var/log/access.log  access_json;

查看 ll /var/log/access.log 备份 mv /var/log/access.log /var/log/access.log.bak 检查语法、重新加载nginx /apps/nginx/sbin/nginx -t

/apps/nginx/sbin/nginx -s reload 再次查看`生成新的日志`即可`(注意时间)` ll /var/log/access.log ```

3.3.3 通过 logtsash 收集TCP/UDP日志

通过logstash的tcp/udp插件收集日志，通常用于在向elasticsearch日志补录丢失的部分日志，可以将丢失的日志写到一个文件，然后通过TCP日志收集方式直接发送给logstash然后再写入到elasticsearch服务器。

https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html

web1(106)

```

查看当前目录

pwd /etc/logstash/conf.d

配置信息

cat tcp.conf input { tcp { #端口=>9601 port => 9889 type => "tcplog" mode => "server"
} }

output { stdout { codec => rubydebug } } 停止服务 systemctl stop logstash 检查 /usr/share/logstash/bin/logstash -f tcp.conf -t 启动后、进入等待模式 /usr/share/logstash/bin/logstash -f tcp.conf ```

在其他服务器安装nc命令：

NetCat简称nc，在网络工具中有“瑞士军刀”美誉，其功能实用，是一个简单、可靠的网络工具，可通过TCP或UDP协议传输读写数据，另外还具有很多其他功能。

web2(107)

写一个值、发送到主机'106:9889'端口 echo "123" | nc 192.168.37.106 9889 ^C <--'Ctrl+c'退出 web1(106)

查看已经收到数据 ...省略中间部分 { "@timestamp" => 2023-05-11T05:23:21.751Z, "@version" => "1", "port" => 43568, "message" => "123", "host" => "192.168.37.107", "type" => "tcplog" }

通过nc命令发送一个文件

web2(107)

nc 192.168.37.106 9889 < /etc/passwd ^C

web1(106)

再次验证 ``` ...省略中间部分

内容很多只复制了部分内容

{ "@timestamp" => 2023-05-11T05:31:17.397Z, "@version" => "1", "port" => 43570, "message" => "sshd:x:109:65534::/run/sshd:/usr/sbin/nologin", "host" => "192.168.37.107", "type" => "tcplog" } { "@timestamp" => 2023-05-11T05:31:17.401Z, "@version" => "1", "port" => 43570, "message" => "wang:x:1000:1000:wang,,,:/home/wang:/bin/bash", "host" => "192.168.37.107", "type" => "tcplog" } ```

通过伪设备的方式发送消息

在类Unix操作系统中，块设备有硬盘、内存的硬件，但是还有设备节点并不一定要对应物理设备，我们把没有这种对应关系的设备是伪设备，比如/dev/null，/dev/zero，/dev/random以及/dev/tcp和/dev/upd等，Linux操作系统使用这些伪设备提供了多种不同的功能，tcp通信只是dev下面众多伪设备当中的一种设备。

web2(107)

echo "伪设备" > /dev/tcp/192.168.37.106/9889 ^C

web1(106)

...省略中间部分 { "@timestamp" => 2023-05-11T05:39:56.661Z, "@version" => "1", "port" => 43572, "message" => "伪设备", "host" => "192.168.37.107", "type" => "tcplog" }

web2(107)

cat "/var/log/syslog" > /dev/tcp/192.168.37.106/9889

web1(106)

``` ...省略中间部分

内容很多只复制了部分内容

{ "@timestamp" => 2023-05-11T05:43:21.619Z, "@version" => "1", "port" => 43574, "message" => "May 11 12:54:42 etcd3 systemd[1]: Started Message of the Day.", "host" => "192.168.37.107", "type" => "tcplog" } { "@timestamp" => 2023-05-11T05:43:21.619Z, "@version" => "1", "port" => 43574, "message" => "May 11 13:17:01 etcd3 CRON[2091]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)", "host" => "192.168.37.107", "type" => "tcplog" } ```

将输出改为elasticsearch：

web1(106)

```

当前位置

pwd /etc/logstash/conf.d

配置内容

cat tcp.conf input { tcp { port => 9889 type => "tcplog" mode => "server"
} }

output { if [type] == "tcplog" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tcplog-37-%{+YYYY.MM.dd}" } } } 检查语法 /usr/share/logstash/bin/logstash -f tcp.conf -t 启动 /usr/share/logstash/bin/logstash -f tcp.conf ``` web2(107)

cat "/var/log/syslog" > /dev/tcp/192.168.37.106/9889

图片.png

添加到kibana即可

图片.png

通过rsyslog收集haproxy日志

rsyslog提供高性能，高安全性功能和模块化设计。虽然它最初是作为常规系统日志开发的，但是rsyslog已经发展成为一种瑞士军刀，可以接受来自各种来源的输入，转换它们，并将结果输出到不同的目的地。

当应用有限的处理时，RSYSLOG每秒可以向本地目的地传送超过一百万条消息。即使有远程目的地和更复杂的处理，性能通常被认为是“惊人的”。

在centos 6及之前的版本叫做syslog，centos 7开始叫做rsyslog，根据官方的介绍，rsyslog(2013年版本)可以达到每秒转发百万条日志的级别，官方网址：http://www.rsyslog.com/，确认系统安装的版本命令如下

图片.png

ha1(108)

安装haproxy apt install haproxy -y 修改配置文件 ``` vim /etc/rsyslog.d/49-haproxy.conf

添加下面信息

if $programname startswith 'haproxy' then @@192.168.37.106:1514

locat2.* @@192.168.37.106:1514 <--CentOS这样配置,@@代表tcp、@代表udp

设置开机启动、并重启服务 systemctl enable haproxy rsyslog systemctl restart haproxy rsyslog ```

web1(106)

``` cd /etc/logstash/conf.d/

vim rsyslog-to-es.conf

input { syslog { port => "1514" type => "syslog" host => "192.168.37.106" } }

output { stdout { codec => "rubydebug" } } 重启服务 systemctl restart logstash 检查 /usr/share/logstash/bin/logstash -f rsyslog-to-es.conf -t 启动 /usr/share/logstash/bin/logstash -f rsyslog-to-es.conf ```

ha1(108)

写入一个值 echo "111" > /var/log/haproxy.log 重启haproxy服务 systemctl restart haproxy

web1(106)

收到数据 { "@timestamp" => 2023-05-11T09:33:49.000Z, "logsource" => "ha1", "facility" => 3, "timestamp" => "May 11 17:33:49", "type" => "syslog", "severity" => 6, "severity_label" => "Informational", "priority" => 30, "facility_label" => "system", "@version" => "1", "host" => "192.168.37.108", "pid" => "14029", "message" => "[WARNING] 130/173319 (14029) : All workers exited. Exiting... (143)\n", "program" => "haproxy" }

``` cat rsyslog-to-es.conf input { syslog { port => "1514" type => "syslog" host => "192.168.37.106" } }

output { if [type] == "syslog" { elasticsearch { hosts => ["192.168.37.102:9200"] index => "syslog-haproxy-37-108-%{+YYYY.MM.dd}" }} } 检查 /usr/share/logstash/bin/logstash -f rsyslog-to-es.conf -t 重启 systemctl restart logstash ```

ha1(108)

重启haproxy服务、生成一些新的日志 systemctl restart haproxy

图片.png

在kibana添加

图片.png

问题总结：

1.elasticsearch的版本和kibana的版本兼容问题
2.logstash的日志处理能力问题
3.redis和logstash的版本兼容问题
4.elasticsearch的index大写问题
5.logstash的配置文件语法问题
6.防火墙、selinux、最大文件数
7.elasticsearch数据目录的权限问题
8.json格式日志不能有key为type的日志

Soul_2016

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Logstash收集多类型日志、if判断使用、Tomcat访问日志转json及收集(小节3)

highlight: a11y-darktheme: juejin三、通过 logstash 收集日志3.1. 通过 logstash 收集单个日志、if判断使用前提需要logstash用户对被收集的日志文件有读的权限并对写入的文件有写权限。web1(106)安装openjdkapt install openjdk-8-jdk -y下载安装包：nginx、...
复制链接

扫一扫