背景
分析--->报警 Kibana
展示 Kibana
存储 ElasticSerach
过滤 Logstash
收集 Filebeat, Logstash
logstash：因为是由java写的，所以在它收集日志时非常占用业务系统的资源，从而影响线上业务，所以我们将其替换为filebeat.
filebeat：为轻量的日志收集组件，会让业务系统的运行更加的稳定.
由于痛点：
1.出现故障,要排查的日志非常的多,没有办法很快定位;
2.统计TOP,PV,UV,IP信息; awk无法满足需求; | elkstack | [ 在线活跃用户, 订单交易金额; ]
所以出现了日志收集: logstash 或者 filebeat
收集日志到es(elasticsearch)存储上有两种方法：
1.直接用filebeat将日志格式转为json格式
2.上面方法不可用，用filebeat收集日志，接入logstash进行处理，再收集到es存储
PS：
app日志中，若是出现了如下类型的，我们的filebeat不能够直接修改日志格式将其修改为json格式，下面会解决.
例如：
[商品名] [用户] [订单数] [交易金额]
Nginx篇
虚拟主机收集+Nginx错误日志收集
#nginx主配置文件
[root@web01 ~]# cat /etc/nginx/nginx.conf
user www;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 10000;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
'}';
access_log /var/log/nginx/access.log json;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
#网站配置文件
[root@web01 ~]# cat /etc/nginx/conf.d/elk.qian.com.conf
server {
listen 80;
server_name elk.qian.com;
root /data;
access_log /var/log/nginx/elk.log json;
location / {
index index.html;
}
}
[root@web01 ~]# cat /etc/nginx/conf.d/blog.ming.com.conf
server {
listen 80;
server_name blog.ming.com;
root /code;
access_log /var/log/nginx/blog.log json;
location / {
index index.html;
}
}
#filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/nginx.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/elk.log
json.keys_under_root: true #Flase会将json解析的格式存储至messages，改为true则不存储至message
json.overwrite_keys: true #覆盖默认message字段，使用自定义json格式的key
tags: nginx_elk
- type: log
enabled: true
paths:
- /var/log/nginx/blog.log
json.keys_under_root: true #Flase会将json解析的格式存储至messages，改为true则不存储至message
json.overwrite_keys: true #覆盖默认message字段，使用自定义json格式的key
tags: nginx_blog
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
json.keys_under_root: true #Flase会将json解析的格式存储至messages，改为true则不存储至message
json.overwrite_keys: true #覆盖默认message字段，使用自定义json格式的key
tags: nginx_error
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
indices:
- index: "nginx-elk-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx_elk"
- index: "nginx-blog-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx_blog"
- index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "nginx_error"
setup.ilm.enabled: false
setup.template.name: "nginx" #定义模板名称
setup.template.pattern: "nginx-*" #定义模板的匹配索引名称
#setup.template.settings: #开启自定义分片数量，不建议开启
# index.number_of_shards: 3 #主分片数为：3
# index.number_of_replicas: 1 #副本数为：1
Tomcat篇+错误日志收集
#Tomcat主配置文件
[root@web01 ~]# cat /soft/tomcat/conf/server.xml
........此处省略配置信息
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<!-- SingleSignOn valve, share authentication between web applications
Documentation at: /docs/config/valve.html -->
<!--
<Valve className="org.apache.catalina.authenticator.SingleSignOn" />
-->
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
<Host name="tomcat.blog.com" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="blog_tomcat_access_log" suffix=".txt"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}" />
</Host>
</Engine>
</Service>
</Server>
Filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /soft/tomcat/logs/blog_tomcat_access_log.*.txt
json.keys_under_root: true #Flase会将json解析的格式存储至messages，改为true则不存储至message
json.overwrite_keys: true #覆盖默认message字段，使用自定义json格式的key
tags: tomcat_blog
- type: log
enabled: true
paths:
- /soft/tomcat/logs/catalina.out
tags: tomcat_error
multiline.pattern: '^\d{2}' #此处应注意，因为Tomcat的错误日志不同于网站日志，不能将其改为json格式，可用此配置，具体可以 https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html 查看官网
multiline.negate: true
multiline.match: after
multiline.max_lines: 1000
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
indices:
- index: "tomcat_blog-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "tomcat_blog"
- index: "tomcat_error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "tomcat_error"
setup.ilm.enabled: false
setup.template.name: "tomcat" #定义模板名称
setup.template.pattern: "tomcat-*" #定义模板的匹配索引名称
系统篇
#用rsyslog工具将系统配置集中到一个日志中去，由filebeat收集
[root@web01 ~]# yum install rsyslog -y
[root@web01 ~]# vim /etc/rsyslog.conf #修改rsyslog配置文件
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
*.* /var/log/ming.log #添加
[root@web01 ~]# systemctl start rsyslog
#filebeat配置文件
#基本配置
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/text.log
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
#增强版配置
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/text.log
include_lines: ['^ERR', '^WARN', 'sshd'] # 仅包含,错误信息,警告信息,sshd的相关配置,其他的都会过滤掉
output.elasticsearch:
hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
续：请看下面关于ELK的文章
2449
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
