FileBeat部署及基础使用

一、FileBeat部署及基础使用

• 软件包下载地址：
  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.6-x86_64.rpm

1.部署FileBeat环境

yum -y localinstall filebeat-7.17.6-x86_64.rpm 

2.修改FileBeat的配置⽂件

(1)编写测试的配置⽂件
mkdir /etc/filebeat/config
cat > /etc/filebeat/config/01-stdin-to-console.yml <<'EOF'
# 指定输⼊的类型
filebeat.inputs:
# 指定输⼊的类型为"stdin",表示标准输⼊
- type: stdin
# 指定输出的类型
output.console:
  # 打印漂亮的格式
  pretty: true
EOF

(2)运⾏filebeat实例
filebeat -e -c /etc/filebeat/config/01-stdin-to-console.yml 

(3)测试
⻅视频。

3.FileBeat的Log类型

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log

output.console:
  pretty: true

4.input的通配符案例

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log
    - /tmp/*.txt

output.console:
  pretty: true

5.input的通用字段案例

filebeat.inputs:
- type: log
  # 是否启动当前的输⼊类型，默认值为true
  enabled: true
  # 指定数据路径
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  # 给当前的输⼊类型搭上标签
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]
  # ⾃定义字段
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  # 将⾃定义字段的key-value放到顶级字段.
  # 默认值为false，会将数据放在⼀个叫"fields"字段的下⾯.
  fields_under_root: true

output.console:
  pretty: true

6.日志过滤案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test/*.log
  # 注意，⿊⽩名单均⽀持通配符,⽣产环节中不建议同时使⽤
  # 指定⽩名单，包含指定的内容才会采集，且区分⼤⼩写!
  include_lines: ['^ERR', '^WARN','oldboyedu']
  # 指定⿊名单，排除指定的内容
  exclude_lines: ['^DBG',"linux","oldboyedu"]

output.console:
  pretty: true

7.将数据写入ES案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]

8.自定义es索引名称

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

9.多个索引写入案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"

10.自定义分片和副本案例

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.log
    - /tmp/*.txt
  tags: ["oldboyedu-linux80","容器运维","DBA运维","SRE运维⼯程师"]  
  fields:
    school: "北京昌平区沙河镇"
    class: "linux80"

- type: log
  enabled: true
  paths:
    - /tmp/test/*/*.log
  tags: ["oldboyedu-python","云原⽣开发"]
  fields:
    name: "oldboy" 
    hobby: "linux,抖⾳" 
  fields_under_root: true

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] 
  # index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
  indices:
    - index: "oldboyedu-linux-elk-%{+yyyy.MM.dd}"
      # 匹配指定字段包含的内容
      when.contains:
        tags: "oldboyedu-linux80"
    - index: "oldboyedu-linux-python-%{+yyyy.MM.dd}"
      when.contains:
        tags: "oldboyedu-python"
# 禁⽤索引⽣命周期管理
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux"
# 设置索引模板的匹配模式 
setup.template.pattern: "oldboyedu-linux*"
# 覆盖已有的索引模板
setup.template.overwrite: false
# 配置索引模板
setup.template.settings:
  # 设置分⽚数量
  index.number_of_shards: 3
  # 设置副本数量，要求⼩于集群的数量
  index.number_of_replicas: 2

11.FileBeat实现日志聚合到本地

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"

output.file:
  path: "/tmp/filebeat"
  filename: oldboyedu-linux80
  # 指定⽂件的滚动⼤⼩，默认值为20MB
  rotate_every_kb: 102400
  # 指定保存的⽂件个数，默认是7个，有效值为2-1024个
  number_of_files: 300
  # 指定⽂件的权限，默认权限是0600
  permissions: 0600

12.FileBeat实现日志聚合到ES集群

filebeat.inputs:
- type: tcp
  host: "0.0.0.0:9000"
  tags: ["aaa"]

- type: tcp
  host: "0.0.0.0:8000"
  tags: ["bbb"]

output.elasticsearch:
  enabled: true
  hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
  indices:
    - index: "oldboyedu-linux80-elk-aaa-%{+yyyy.MM.dd}"
      when.contains:
        tags: "aaa"

    - index: "oldboyedu-linux80-elk-bbb-%{+yyyy.MM.dd}"
      when.contains:
        tags: "bbb"

setup.ilm.enabled: false
setup.template.name: "oldboyedu-linux80-elk"
setup.template.pattern: "oldboyedu-linux80-elk*"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0

13.FileBeat的Log类型删除不必要的字段

filebeat.inputs:
- type: log
  paths:
    - /tmp/test.log

output.console:
  pretty: true

processors:
  - drop_fields:
      fields: ['agent','ecs', 'beat', 'input_type', 'tags', 'count', '@version', 'log', 'offset', 'type', 'host','@metadata','input']