ELK-filbeate收集tomcat日志

filebeat作为代理安装在服务器上,监视指定的日志文件或位置,收集日志事件,并将他们转发到logstash,elasticsearch,kafka等

input 我们要采集的日志文件路径, 收割机 harvester 监听文件的变化 -->
splooer程序 --> 转发 es | logstash | kafka | redis
在这里插入图片描述

filebeat.inputs:
 - type: stdin #标准输入
   enabled: true #启用
 output.console: #标准输出
 pretty: true
 enable: true
将文件最新发生变化的内容,存入ES
[root@web01 ~]# cat /etc/filebeat/file.yml
filebeat.inputs:
- type: log
  paths: /var/log/nginx/access.log
  enabled: true
  output.elasticsearch:
  hosts:["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]

收集系统日志

特别分散–> syslog --> file.txt
1.减少无用的数据
2.调整索引名称
3.测试调整模板,设定分片

[root@web01 filebeat]# cat filebeat_system.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/oldxu.log
  include_lines: ['^ERR','^WARN','sshd'] #只看指定的日志

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  index: "system-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.ilm.enabled: false
setup.template.name: "system"
setup.template.pattern: "system-*"

setup.template.settings:            #定义索引分片数和副本
  index.number_of_shards: 3
  index.number_of_replicas: 1

1.修改system模板   ---> 添加 shards 分片数数量,replicas的数量
2.删除模板关联的索引
3.删除filebeat自行指定的分片数和副本数
4.重启filebeat
5.产生新的日志

收集Nginx日志

配置filebeat

[root@web01 filebeat]# cat filebeat_nginx.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true    #默认Flase,还会将json解析的日志存储至messages字段
  json.overwrite_keys: true     #覆盖默认的key,使用自定义json格式的key

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  index: "nginx-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.ilm.enabled: false
setup.template.name: nginx   #索引关联的模板名称
setup.template.pattern: nginx-*

收集nginx访问日志和错误日志

[root@web01 filebeat]# cat filebeat_access.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true    #默认Flase,还会将json解析的日志存储至messages字段
  json.overwrite_keys: true     #覆盖默认的key,使用自定义json格式的key
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  indices:
    - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"

    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "error"

setup.ilm.enabled: false
setup.template.name: nginx   #索引关联的模板名称
setup.template.pattern: nginx-*

收集nginx多个虚拟主机的日志

在这里插入图片描述
1.虚拟主机

[root@web01 conf.d]# cat elk.oldxu.com.conf 
server {
    listen 80;
    server_name elk.oldxu.com;
    root /code/elk;
        access_log /var/log/nginx/elk.oldxu.com.log json;

    location / {
        index index.html;
    }
}
[root@web01 conf.d]# cat bk.oldxu.com.conf 
server {
    listen 80;
    server_name bk.oldxu.com;
    root /code/bk;
        access_log /var/log/nginx/bk.oldxu.com.log json;

    location / {
        index index.html;
    }
}
[root@web01 conf.d]# cat bs.oldxu.com.conf 
server {
    listen 80;
    server_name bs.oldxu.com;
    root /code/bs;
        access_log /var/log/nginx/bs.oldxu.com.log json;

    location / {
        index index.html;
    }
}

2.测试,模拟产生日志

[root@web01 conf.d]# curl -H Host:elk.oldxu.com http://10.0.0.7
elk.oldux.com
[root@web01 conf.d]# curl -H Host:bs.oldxu.com http://10.0.0.7
bs.oldux.com
[root@web01 conf.d]# curl -H Host:bk.oldxu.com http://10.0.0.7
bk.oldux.com

3.配置filebeat

[root@web01 filebeat]# cat filebeat-vhosts.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/elk.oldxu.com.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx-elk-host"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/bs.oldxu.com.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx-bs-host"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/bk.oldxu.com.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx-bk-host"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["nginx-error"]

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  indices:
    - index: "nginx-elk-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx-elk-host"

    - index: "nginx-bs-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx-bs-host"

    - index: "nginx-bk-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx-bk-host"

    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx-error"

setup.ilm.enabled: false
setup.template.name: nginx   #索引关联的模板名称
setup.template.pattern: nginx-*

Tomcat日志

访问日志 —> json格式

1.修改tomcat日志格式
 [root@web02 soft]# yum install java -y
 [root@web02 soft]# vim tomcat/conf/server.xml
     <Host name="tomcat.oldxu.com" appBase="webapps"
           unpackWARs="true" autoDeploy="true">
       <Valve
className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="tomcat.oldxu.com.log"
suffix=".txt"
               pattern="
{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&q
uot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&
quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;metho
d&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot
;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?
string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%
{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User�Agent}i&quot;}" />
     </Host>

配置filebeat
[root@web01 filebeat]# cat filebeat-tomcat-mutilline.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /soft/tomcat/logs/tomcat.oldxu.com.log.*.txt
  json.keys_under_root: true    #默认Flase,还会将json解析的日志存储至messages字段
  json.overwrite_keys: true     #覆盖默认的key,使用自定义json格式的key
  tags: ["tomcat-access"]

- type: log
  enabled: true
  paths:
    - /soft/tomcat/logs/catalina.out
  multiline.pattern: '^\d{2}'   #匹配以2个数字开头的
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 10000    #默认最大合并行为500,可根据实际情况调整。
  tags: ["tomcat-error"]

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200"]
  indices:
    - index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "tomcat-access"

    - index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "tomcat-error"

setup.ilm.enabled: false
setup.template.name: tomcat   #索引关联的模板名称
setup.template.pattern: tomcat-*

参考链接 :
day105 ELK-filbeate收集日志 : https://www.jianshu.com/p/fffee6c527de

©️2020 CSDN 皮肤主题: 大白 设计师: CSDN官方博客 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值