Day05-filebeat常用的输出组件，logstash的输入输出组件及date，grok，geoip过滤插件实战案例

我心中有一片海

于 2024-07-15 12:00:00 发布

阅读量592

点赞数 5

分类专栏： # ELK 文章标签： Linux filebeat ELK 运维日志分析

本文链接：https://blog.csdn.net/dws123654/article/details/140274628

版权

ELK 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

Day05-filebeat常用的输出组件，logstash的输入输出组件及date，grok，geoip过滤插件实战案例

1、使用filebeat采集docker日志
2、filebeat的input类型之filestream实战案例
3、logstash

多行匹配案例补充:

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/10-log_multiline-to-console.yaml 
filebeat.inputs:
- type: log
  paths:
    - /tmp/oldboyedu-linux85/linux85.log
      multiline:
        # 指定多行匹配的类型为count
        type: count
        # 将3行数据合并为1行数据
        count_lines: 3

output.console:
  pretty: true
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#

1、使用filebeat采集docker日志

(1)安装docker

wget http://192.168.15.253/ElasticStack/day05-/softwares/oldboyedu-docker-ce-23_0_1.tar.gz
tar xf oldboyedu-docker-ce-23_0_1.tar.gz 
yum -y localinstall oldboyedu-docker-ce-23_0_1/*.rpm

(2)配置docker的镜像加速

[root@elk103.oldboyedu.com ~]# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker",
   "registry-mirrors": ["https://tuv7rqqq.mirror.aliyuncs.com","https://hub-mirror.c.1com/","https://docker.mirrors.ustc.edu.cn","https://reg-mirror.qiniu.com"]
}

[root@elk103.oldboyedu.com ~]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

(3)下载nginx镜像

docker run -dp 88:80 --name mynginx --restart always nginx:1.22.1-alpine
docker run -dp 89:8080 --name mytomcat --restart always  tomcat:jre8-alpine

(3)使用filebeat采集容器日志

# 通过docker
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/11-docker-to-console.yaml
filebeat.inputs:
  # 指定输入类型为docker类型
- type: docker
  # 指定容器的ID
  containers.ids: 
    - '*'

output.console:
  pretty: true 
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/11-docker-to-console.yaml

# 通过container
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/12-container-to-console.yaml 
filebeat.inputs:
- type: container
  paths: 
    - '/var/lib/docker/containers/*/*.log'

# output.console:
#   pretty: true

output.elasticsearch:
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/12-container-to-console.yaml

2、filebeat的input类型之filestream实战案例

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/13-filestream-to-console.yaml 
filebeat.inputs:
  # 指定类型为filestream，在7.16版本中已经弃用log类型
- type: filestream
  enabled: false
  paths:
    - /tmp/oldboyedu-linux85/linux85.log

- type: filestream
  enabled: false
  paths:
    - /tmp/oldboyedu-linux85/docker.json
  # 配置解析
  parsers:
    # 配置json格式解析
    - ndjson:
       # 将错误消息记录到error字段中
       add_error_key: true
       # 如果解析的json格式字段和filebeat内置的顶级字段冲突，则覆盖，默认是不覆盖的。
       overwrite_keys: true
       # 将message解析的字段放入一个自定义的字段下。若不指定该字段，则默认解析的键值对会在顶级字段.
       #target: oldboyedu-linux85

- type: filestream
  enabled: false
  paths:
    - /tmp/oldboyedu-linux85/linux85.log
  # 配置解析
  parsers:
    - multiline:
        type: count
        count_lines: 3


- type: filestream
  enabled: true
  paths:
    - /tmp/oldboyedu-linux85/demo.log
  parsers:
    - multiline:
        type: count
        count_lines: 4
    - ndjson:
       add_error_key: true
       overwrite_keys: true
       target: oldboyedu-linux85-demo

output.console:
  pretty: true

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#

2.1 课堂练习案例

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/14-ketanglianxi.yaml 
filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /tmp/oldboyedu-linux85/shopping.json
      parsers:
    - multiline:
      type: count
      count_lines: 7
    - ndjson:
      add_error_key: true
      overwrite_keys: true

output.elasticsearch:
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# ./filebeat -e -c config/14-ketanglianxi.yaml

2.2 将数据写入到本地文件案例

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/15-stdin-to-file.yaml
filebeat.inputs:
- type: stdin

# 指定输出的类型为本地文件
output.file:
  # 指定文件存储的路径
  path: "/tmp/oldboyedu-linux85"
  # 指定文件的名称
  filename: stdin.log
  
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# ./filebeat -e -c config/15-stdin-to-file.yaml

2.3 写入数据到ES集群

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/16-log-to-es.yaml 
filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /tmp/oldboyedu-linux85/shopping.json
      parsers:
    - multiline:
      type: count
      count_lines: 7
    - ndjson:
      add_error_key: true
      overwrite_keys: true
      
# 将日志输出到ES集群
output.elasticsearch:
  # 指定ES集群地址
  hosts: 
  - "http://10.0.0.101:9200"
  - "http://10.0.0.102:9200"
  - "http://10.0.0.103:9200"
  # 指定索引
  index: "oldboyedu-linux85-shopping-%{+yyyy.MM.dd}"
# 禁用索引声明管理周期，若不禁用则自动忽略自定义索引名称
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux85-shopping"
# 指定索引模板的匹配模式
setup.template.pattern: "oldboyedu-linux85-shopping-*"
# 是否覆盖原有的索引模板
setup.template.overwrite: true
# 设置索引模板
setup.template.settings:
  # 指定分片数量为8
  index.number_of_shards: 8
  # 指定副本数量为0
  index.number_of_replicas: 0
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#

2.4 将多个数据源写入到ES集群不同索引

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/17-filestream-to-es.yaml 
filebeat.inputs:
- type: filestream
  enabled: true
  tags: "docker"
  paths:
    - /tmp/oldboyedu-linux85/docker.json
  parsers:
    - ndjson:
       add_error_key: true
       #overwrite_keys: true


- type: filestream
  enabled: true
  tags: "linux85"
  paths:
    - /tmp/oldboyedu-linux85/linux85.log
  parsers:
    - multiline:
        type: count
        count_lines: 3


- type: filestream
  enabled: true
  tags: "demo"
  paths:
    - /tmp/oldboyedu-linux85/demo.log
  parsers:
    - multiline:
        type: count
        count_lines: 4
    - ndjson:
       add_error_key: true
       overwrite_keys: true
       target: oldboyedu-linux85-demo


output.elasticsearch:
  hosts: 
  - "http://10.0.0.101:9200"
  - "http://10.0.0.102:9200"
  - "http://10.0.0.103:9200"
  # index: "oldboyedu-linux85-shopping-%{+yyyy.MM.dd}"
  indices:
     - index: "oldboyedu-jiaoshi07-docker-%{+yyyy.MM.dd}"
       when.contains:
         tags: "docker"
     - index: "oldboyedu-jiaoshi07-linux85-%{+yyyy.MM.dd}"
       when.contains:
         tags: "linux85"
     - index: "oldboyedu-jiaoshi07-demo-%{+yyyy.MM.dd}"
       when.contains:
         tags: "demo"

setup.ilm.enabled: false
setup.template.name: "oldboyedu-jiaoshi07"
setup.template.pattern: "oldboyedu-jiaoshi07-*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/17-filestream-to-es.yaml

3、logstash

3.1 基于rpm方式安装logstash

(1)下载软件包

wget http://192.168.15.253/ElasticStack/day05-/softwares/logstash-7.17.5-x86_64.rpm

(2)安装logstash

rpm -ivh logstash-7.17.5-x86_64.rpm

(3)验证logstash版本

ln -svf /usr/share/logstash/bin/logstash /usr/local/sbin
logstash -V

(4)基于命令行启动logstash实例

logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }"

(5)测试logstash

自行输入数据即可。

3.2 基于二进制方式安装logstash

(1)下载软件包

wget http://192.168.15.253/ElasticStack/day05-/softwares/logstash-7.17.5-linux-x86_64.tar.gz

(2)解压软件包

tar xf logstash-7.17.5-linux-x86_64.tar.gz -C /oldboyedu/softwares/

(3)验证logstash版本

ln -svf /oldboyedu/softwares/logstash-7.17.5/bin/logstash /usr/local/sbin/
logstash -V

(4)基于命令行启动logstash实例

logstash -e "input { stdin { type => stdin } } output { stdout {} }"

(5)测试logstash

自行输入数据即可。

3.3 编写第一个logstash配置文件

(1)编写配置文件

[root@elk101.oldboyedu.com ~]# cat config/01-stdin-to-stdout.conf
input { 
  stdin { type => stdin } 
} 

output { 
  stdout {} 
}
[root@elk101.oldboyedu.com ~]#

(2)启动logstash实例

[root@elk101.oldboyedu.com ~]# logstash -f /config/01-stdin-to-stdout.conf

3.4 logstash搭配filebeat实战案例

(1)编写logstash配置文件并启动

[root@elk101.oldboyedu.com ~]# cat /config/02-beats-to-stdout.conf 
input { 
  # 指定输入的类型是一个beats
  beats {
    # 指定监听的端口号
    port => 8888
  }
} 

output { 
  # 将数据在标准输出显示
  stdout {} 

  # 将数据写入ES集群
  elasticsearch {
    # 指定ES主机地址
    hosts => ["http://localhost:9200"]
    # 指定索引名称
    index => "oldboyedu-linux85-logstash"
  }
}
 
[root@elk101.oldboyedu.com ~]# logstash -rf config/02-beats-to-stdout.conf

(2)启动filebeat实例并写入数据

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/18-nginx-to-logstash.yaml
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log*
# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["10.0.0.101:8888"]
 
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/18-nginx-to-logstash.yaml

3.5 logstash的过滤插件之geoip实战案例

(1)logstash配置文件

[root@elk101.oldboyedu.com ~]# cat config/03-beats-geoip-es.conf 
input { 
  # 指定输入的类型是一个beats
  beats {
    # 指定监听的端口号
    port => 8888
  }
} 

filter {
  # 根据IP地址分析客户端的经纬度，国家，城市信息等。
  geoip {
     source => "clientip"
     remove_field => [ "agent","log","input","host","ecs","tags" ]
  }

}

output { 
  # 将数据在标准输出显示
  stdout {} 
  # 将数据写入ES集群
  elasticsearch {
    # 指定ES主机地址
    hosts => ["http://localhost:9200"]
    # 指定索引名称
    index => "oldboyedu-linux85-logstash"
  }
}

[root@elk101.oldboyedu.com ~]# logstash -rf config/03-beats-geoip-es.conf

(2)filebeat采集数据到logstash

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/18-nginx-to-logstash.yaml 
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log*
      json.keys_under_root: true
      json.add_error_key: true

# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["10.0.0.101:8888"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# 
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/18-nginx-to-logstash.yaml

(3)日志格式

[root@elk103.oldboyedu.com oldboyedu-linux85]# cat /var/log/nginx/access.log 
{"@timestamp":"2023-04-06T16:17:43+08:00","host":"10.0.0.103","clientip":"110.110.110.110","SendBytes":615,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.29.0","status":"200"}
{"@timestamp":"2023-04-06T18:18:18+08:00","host":"10.0.0.103","clientip":"101.231.54.100","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T08:18:32+08:00","host":"10.0.0.103","clientip":"219.141.136.10","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T10:18:52+08:00","host":"10.0.0.103","clientip":"221.118.208.184","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T12:19:07+08:00","host":"10.0.0.103","clientip":"21.118.208.84","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (Linux; Android 10; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36","status":"404"}
[root@elk103.oldboyedu.com oldboyedu-linux85]#

3.6 logstash解析nginx原生日志并分析IP地址实战

(1)logstash配置文件编写

[root@elk101.oldboyedu.com ~]# cat config/04-beats-grok_geoip-es.conf 
input { 
  beats {
    port => 8888
  }
} 

filter {
   grok {
      match => { "message" => "%{HTTPD_COMBINEDLOG}" }
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

  geoip {
     source => "clientip"
  }

}

output { 
 #  stdout {} 

  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "oldboyedu-linux85-logstash-nginx"
  }
}
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# logstash -rf config/04-beats-grok_geoip-es.conf

(2)filbeat采集日志

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/19-nginx-to-logstash.yaml
filebeat.inputs:
- type: log
  paths:
    - /tmp/oldboyedu-linux85/access.log

# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["10.0.0.101:8888"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# 
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/19-nginx-to-logstash.yaml

3.7 logstash解析将实际写入时间更正案例

(1)logstash配置文件

[root@elk101.oldboyedu.com ~]# cat config/05-beats-grok_geoip_date-es.conf
input { 
  beats {
    port => 8888
  }
} 

filter {
   grok {
      match => { "message" => "%{HTTPD_COMBINEDLOG}" }
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }

  geoip {
     source => "clientip"
  }

  date {
      # 匹配时间字符串字段并格式化
      # "22/Nov/2015:11:57:34 +0800"
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      # 匹配时区
      timezone => "Asia/Shanghai"
      # 将转后的日期替换为指定字段，若不指定，则默认值为"@timestamp"
      target => "oldboyedu-linux85-date"
  }

}

output { 
 #stdout {} 

 elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "oldboyedu-linux85-logstash-nginx-date"
 }
}

[root@elk101.oldboyedu.com ~]# logstash -rf config/05-beats-grok_geoip_date-es.conf

(2)filebeat配置文件

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/19-nginx-to-logstash.yaml 
filebeat.inputs:
- type: log
  paths:
    - /tmp/oldboyedu-linux85/access.log

# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["10.0.0.101:8888"]

[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/19-nginx-to-logstash.yaml

周六作业:
(1)完成课堂的所有练习并整理思维导图;
(2)使用shell脚本编写一个nginx原生日志格式，要求生成100w条记录，并使用ElasticStack技术栈分析并能查询数据;

扩展作业:
config/05-beats-grok_geoip_date-es.conf

我心中有一片海

关注

5
点赞
踩
10

收藏

觉得还不错? 一键收藏
0
评论
Day05-filebeat常用的输出组件，logstash的输入输出组件及date，grok，geoip过滤插件实战案例

(2)filebeat采集数据到logstash。(4)基于命令行启动logstash实例。(4)基于命令行启动logstash实例。(1)编写logstash配置文件并启动。(2)启动filebeat实例并写入数据。(1)logstash配置文件编写。(3)验证logstash版本。(2)启动logstash实例。(1)logstash配置文件。(1)logstash配置文件。(2)filebeat配置文件。(2)安装logstash。(5)测试logstash。(5)测试logstash。
复制链接

扫一扫