highlight: a11y-dark
theme: juejin
三、通过 logstash 收集日志
3.1. 通过 logstash 收集单个日志、if判断使用
前提需要logstash用户对被收集的日志文件有读的权限并对写入的文件有写权限。
web1(106)
安装openjdk apt install openjdk-8-jdk -y
下载
安装包:nginx、logstash cd /usr/local/src/ wget http://nginx.org/download/nginx-1.16.1.tar.gz
安装logstash dpkg -i logstash-6.8.3.deb
安装Nginx(编译安装出错可参考) tar xvf nginx-1.16.1.tar.gz cd nginx-1.16.1/ ./configure --prefix=/apps/nginx make make install
启动nginx /apps/nginx/sbin/nginx
可以访问
配置logstash ``` cat /etc/logstash/conf.d/log-to-es.conf
输入
input {
系统日志
file { path => "/var/log/syslog"
start_position => "beginning"
stat_interval => "3"
type => "syslog"
}
Nginx日志
file { path => "/apps/nginx/logs/access.log" startposition => "beginning" statinterval => "3" type => "nginx-accesslog" } }
输出
output {
判断
if [type] == "syslog" { stdout { codec => "rubydebug" } }
if [type] == "nginx-accesslog" { stdout { codec => "rubydebug" }
要保证此目录有logstash权限,因为此目录是用logstash起来的。
file {
path => "/tmp/nginx.log"
}
} } 修改logstash配置权限
vim /etc/systemd/system/logstash.service [Service] User=root Group=root 检查配置文件
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf -t ...省略中间部分 [INFO ] 2020-05-10 01:14:55.355 [LogStash::Runner] runner - Using config.testandexit mode. Config Validation Result: OK. Exiting Logstash <--正常 ```
启动 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf
host1(101)
显示头部信息 curl --head http://192.168.37.106/index.html HTTP/1.1 200 OK Server: nginx/1.16.1 Date: Tue, 09 May 2020 17:20:29 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 09 May 2020 16:53:40 GMT Connection: keep-alive ETag: "645a7a94-264" Accept-Ranges: bytes
3.2. 通过 logstash 收集多类型日志、if判断使用
web1(106)
复制后、粘贴的json在线翻译 vim /tmp/nginx.log ...最后结尾部分(粘贴下行) {"message":"192.168.37.101 - - [10/May/2020:01:34:19 +0800] \"HEAD /index.html HTTP/1.1\" 200 0 \"-\" \"curl/7.58.0\"","@timestamp":"2020-05-09T17:34:34.738Z","type":"nginx-accesslog","path":"/apps/nginx/logs/access.log","@version":"1","host":"ubuntu-6"}
修改文件 ``` vim /etc/logstash/conf.d/log-to-es.conf
输入
input {
系统日志
file { path => "/var/log/syslog"
start_position => "beginning"
stat_interval => "3"
type => "syslog"
}
Nginx日志
file { path => "/apps/nginx/logs/access.log" startposition => "beginning" statinterval => "3" type => "nginx-accesslog" } }
输出
output {
判断
if [type] == "syslog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "syslog-37-106-%{+YYYY.ww}" }}
if [type] == "nginx-accesslog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "nginx-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```
检查 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/log-to-es.conf -t
设为开机启动、重启服务 systemctl enable logstash systemctl restart logstash
查看日志 tail -f /var/log/logstash/logstash-plain.log ...省略中间部分(下面提示成功) [2020-05-10T13:28:02,911][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
创建索引(syslog-37-106)
时间戳
创建索引(nginx-accesslog-37-106)
时间戳
web1(106)
写入一个值后、刷新页面可以看到 echo "333" >> /var/log/syslog
3.3. 通过 logtsash 收集tomcat日志和java日志
3.3.1 通过 logtsash 收集tomcat日志
收集Tomcat服务器的访问日志以及Tomcat错误日志进行实时统计,在kibana页面进行搜索展现,每台Tomcat服务器要安装logstash负责收集日志,然后将日志转发给elasticsearch进行分析,在通过kibana在前端展现,配置过程如下:
web1(106)
tomcat官方地址:https://archive.apache.org/dist/tomcat/
下载包
``` cd /usr/local/src
wget https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.47/bin/apache-tomcat-8.5.47.tar.gz
解压缩
tar xvf apache-tomcat-8.5.47.tar.gz
创建目录
cd apache-tomcat-8.5.47/ mkdir webapps/app
创建页面
echo "linux01" > webapps/app/index.html 修改文件
当前所在目录
pwd /usr/local/src/apache-tomcat-8.5.47
vim conf/server.xml ... ```
启动tomcat ./bin/catalina.sh start
查看网页信息
软链接 ln -sv /usr/local/src/apache-tomcat-8.5.47 /apps/tomcat
查看日志 tail -f /apps/tomcat/logs/tomcat-37-106_access_log.2023-05-10.log 192.168.37.1 - - [10/May/2023:14:21:45 +0800] "GET /app/ HTTP/1.1" 200 8 192.168.37.1 - - [10/May/2023:14:21:46 +0800] "GET /favicon.ico HTTP/1.1" 200 21630 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET / HTTP/1.1" 200 11215 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /tomcat.css HTTP/1.1" 200 5581 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /bg-nav.png HTTP/1.1" 200 1401 192.168.37.1 - - [10/May/2023:14:21:54 +0800] "GET /asf-logo-wide.svg HTTP/1.1" 200 27235 ^C
编辑文件 ``` vim /etc/logstash/conf.d/tomcat-log-to-es.conf
输入
input {
tomcat日志
file { path => "/apps/tomcat/logs/tomcat-37-106accesslog.*.log"
start_position => "beginning"
stat_interval => "3"
type => "tomcat-access-log"
} }
输出
output {
判断
if [type] == "tomcat-access-log" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tomcat-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```
检查文件
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat-log-to-es.conf -t
重启服务
systemctl restart logstash
刷新页面可以看到
添加索引
时间戳
tomcat日志改json格式(参数介绍)
``` vim /usr/local/src/apache-tomcat-8.5.47/conf/server.xml ...省略中间内容
将下行改成如下信息
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
重启tomcat
cd /usr/local/src/apache-tomcat-8.5.47/
停服务
./bin/catalina.sh stop
启服务
./bin/catalina.sh start ```
刷新页面:http://192.168.37.106:8080/后
可以在监控中看到json信息、 tail -f logs/tomcat-37-106_access_log.2023-05-11.log ...省略中间部分 {"clientip":"192.168.37.1","ClientUser":"-","authenticated":"-","AccessTime":"[11/May/2023:00:53:34 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"11215","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"}
看复制上面信息、到网站中校验
文件中添加codec、对日志的格式修改 ``` cat /etc/logstash/conf.d/tomcat-log-to-es.conf
输入
input {
tomcat日志
file { path => "/apps/tomcat/logs/tomcat-37-106accesslog.*.log"
start_position => "beginning"
stat_interval => "3"
type => "tomcat-access-log"
codec => "json"
} }
输出
output {
判断
if [type] == "tomcat-access-log" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tomcat-accesslog-37-106-%{+YYYY.MM.dd}" }} } ```
重启服务
systemctl restart logstash
host1(101)
curl --head http://192.168.37.106:8080/app/index.html
强制刷新、刷新前
刷新后
再查看
创建可视化
3.3.2 通过 logtsash 收集java日志
使用codec的multiline插件实现多行匹配,这是一个可以将多行进行合并的插件,而且可以使用what指定将匹配到的行与前面的行合并还是和后面的行合并,https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html
web1(106)
配置信息 ``` cd /etc/logstash/conf.d/
cat java-log-to-es.conf input { file {
日志文件位置
path => "/var/log/java.log"
type => "javalog"
start_position => "beginning"
codec => multiline {
当遇到"["开头的行时候将多行进行合并
pattern => "^\["
"true"为匹配成功进行操作,"false"为不成功进行操作
negate => true
与以前的行合并,如果是下面的行合并就是"next"
what => "previous"
}} }
日志过滤,如果所有的日志都过滤就写这里,如果只针对某一个过滤就写在"input"里面的日志输入里面
filter { }
output { if [type] == "javalog" { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "javalog-7-106-%{+YYYY-MM.dd}" }} } 检查
/usr/share/logstash/bin/logstash -f java-log-to-es.conf -t 启动
/usr/share/logstash/bin/logstash -f java-log-to-es.conf ``` 到kibana中添加"http://192.168.37.101:5601/app/kibana#/home?_g=()" ,【步骤:管理-(kibana)索引模式-创建索引模式-索引模式(javalog-37.106-*)-时间戳-完成】即可
将日志类型改为json格式、
编辑nginx文件、添加日志类型(可添加多个) ``` vim /apps/nginx/conf/nginx.conf http { ...
在http中添加如下信息(注1)
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"uri":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
日志存放目录、类型(access_json和上面'注1'一致)
access_log /var/log/access.log access_json;
查看
ll /var/log/access.log 备份
mv /var/log/access.log /var/log/access.log.bak 检查语法、重新加载nginx
/apps/nginx/sbin/nginx -t
/apps/nginx/sbin/nginx -s reload 再次查看`生成新的日志`即可`(注意时间)`
ll /var/log/access.log ```
3.3.3 通过 logtsash 收集TCP/UDP日志
通过logstash的tcp/udp插件收集日志,通常用于在向elasticsearch日志补录丢失的部分日志,可以将丢失的日志写到一个文件,然后通过TCP日志收集方式直接发送给logstash然后再写入到elasticsearch服务器。
https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html
web1(106)
```
查看当前目录
pwd /etc/logstash/conf.d
配置信息
cat tcp.conf input { tcp { #端口=>9601 port => 9889 type => "tcplog" mode => "server"
} }
output { stdout { codec => rubydebug } } 停止服务
systemctl stop logstash 检查
/usr/share/logstash/bin/logstash -f tcp.conf -t 启动后、进入等待模式
/usr/share/logstash/bin/logstash -f tcp.conf ```
在其他服务器安装nc命令:
NetCat简称nc,在网络工具中有“瑞士军刀”美誉,其功能实用,是一个简单、可靠的网络工具,可通过TCP或UDP协议传输读写数据,另外还具有很多其他功能。
web2(107)
写一个值、发送到主机'106:9889'端口 echo "123" | nc 192.168.37.106 9889 ^C <--'Ctrl+c'退出
web1(106)
查看已经收到数据 ...省略中间部分 { "@timestamp" => 2023-05-11T05:23:21.751Z, "@version" => "1", "port" => 43568, "message" => "123", "host" => "192.168.37.107", "type" => "tcplog" }
通过nc命令发送一个文件
web2(107)
nc 192.168.37.106 9889 < /etc/passwd ^C
web1(106)
再次验证 ``` ...省略中间部分
内容很多只复制了部分内容
{ "@timestamp" => 2023-05-11T05:31:17.397Z, "@version" => "1", "port" => 43570, "message" => "sshd:x:109:65534::/run/sshd:/usr/sbin/nologin", "host" => "192.168.37.107", "type" => "tcplog" } { "@timestamp" => 2023-05-11T05:31:17.401Z, "@version" => "1", "port" => 43570, "message" => "wang:x:1000:1000:wang,,,:/home/wang:/bin/bash", "host" => "192.168.37.107", "type" => "tcplog" } ```
通过伪设备的方式发送消息
在类Unix操作系统中,块设备有硬盘、内存的硬件,但是还有设备节点并不一定要对应物理设备,我们把没有这种对应关系的设备是伪设备,比如/dev/null,/dev/zero,/dev/random以及/dev/tcp和/dev/upd等,Linux操作系统使用这些伪设备提供了多种不同的功能,tcp通信只是dev下面众多伪设备当中的一种设备。
web2(107)
echo "伪设备" > /dev/tcp/192.168.37.106/9889 ^C
web1(106)
...省略中间部分 { "@timestamp" => 2023-05-11T05:39:56.661Z, "@version" => "1", "port" => 43572, "message" => "伪设备", "host" => "192.168.37.107", "type" => "tcplog" }
web2(107)
cat "/var/log/syslog" > /dev/tcp/192.168.37.106/9889
web1(106)
``` ...省略中间部分
内容很多只复制了部分内容
{ "@timestamp" => 2023-05-11T05:43:21.619Z, "@version" => "1", "port" => 43574, "message" => "May 11 12:54:42 etcd3 systemd[1]: Started Message of the Day.", "host" => "192.168.37.107", "type" => "tcplog" } { "@timestamp" => 2023-05-11T05:43:21.619Z, "@version" => "1", "port" => 43574, "message" => "May 11 13:17:01 etcd3 CRON[2091]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)", "host" => "192.168.37.107", "type" => "tcplog" } ```
将输出改为elasticsearch:
web1(106)
```
当前位置
pwd /etc/logstash/conf.d
配置内容
cat tcp.conf input { tcp { port => 9889 type => "tcplog" mode => "server"
} }
output { if [type] == "tcplog" { elasticsearch { hosts => ["http://192.168.37.102:9200"] index => "tcplog-37-%{+YYYY.MM.dd}" } } } 检查语法
/usr/share/logstash/bin/logstash -f tcp.conf -t 启动
/usr/share/logstash/bin/logstash -f tcp.conf ``` web2(107)
cat "/var/log/syslog" > /dev/tcp/192.168.37.106/9889
添加到kibana即可
通过rsyslog收集haproxy日志
rsyslog提供高性能,高安全性功能和模块化设计。 虽然它最初是作为常规系统日志开发的,但是rsyslog已经发展成为一种瑞士军刀,可以接受来自各种来源的输入,转换它们,并将结果输出到不同的目的地。
当应用有限的处理时,RSYSLOG每秒可以向本地目的地传送超过一百万条消息。 即使有远程目的地和更复杂的处理,性能通常被认为是“惊人的”。
在centos 6及之前的版本叫做syslog,centos 7开始叫做rsyslog,根据官方的介绍,rsyslog(2013年版本)可以达到每秒转发百万条日志的级别,官方网址:http://www.rsyslog.com/,确认系统安装的版本命令如下
ha1(108)
安装haproxy apt install haproxy -y
修改配置文件 ``` vim /etc/rsyslog.d/49-haproxy.conf
添加下面信息
if $programname startswith 'haproxy' then @@192.168.37.106:1514
locat2.* @@192.168.37.106:1514 <--CentOS这样配置,@@代表tcp、@代表udp
设置开机启动、并重启服务
systemctl enable haproxy rsyslog systemctl restart haproxy rsyslog ```
web1(106)
``` cd /etc/logstash/conf.d/
vim rsyslog-to-es.conf
input { syslog { port => "1514" type => "syslog" host => "192.168.37.106" } }
output { stdout { codec => "rubydebug" } } 重启服务
systemctl restart logstash 检查
/usr/share/logstash/bin/logstash -f rsyslog-to-es.conf -t 启动
/usr/share/logstash/bin/logstash -f rsyslog-to-es.conf ```
ha1(108)
写入一个值 echo "111" > /var/log/haproxy.log
重启haproxy服务 systemctl restart haproxy
web1(106)
收到数据 { "@timestamp" => 2023-05-11T09:33:49.000Z, "logsource" => "ha1", "facility" => 3, "timestamp" => "May 11 17:33:49", "type" => "syslog", "severity" => 6, "severity_label" => "Informational", "priority" => 30, "facility_label" => "system", "@version" => "1", "host" => "192.168.37.108", "pid" => "14029", "message" => "[WARNING] 130/173319 (14029) : All workers exited. Exiting... (143)\n", "program" => "haproxy" }
``` cat rsyslog-to-es.conf input { syslog { port => "1514" type => "syslog" host => "192.168.37.106" } }
output { if [type] == "syslog" { elasticsearch { hosts => ["192.168.37.102:9200"] index => "syslog-haproxy-37-108-%{+YYYY.MM.dd}" }} } 检查
/usr/share/logstash/bin/logstash -f rsyslog-to-es.conf -t 重启
systemctl restart logstash ```
ha1(108)
重启haproxy服务、生成一些新的日志 systemctl restart haproxy
在kibana添加
问题总结:
1.elasticsearch的版本和kibana的版本兼容问题
2.logstash的日志处理能力问题
3.redis和logstash的版本兼容问题
4.elasticsearch的index大写问题
5.logstash的配置文件语法问题
6.防火墙、selinux、最大文件数
7.elasticsearch数据目录的权限问题
8.json格式日志不能有key为type的日志