elk7.5 之 filebeat部署

原理篇请查看https://blog.csdn.net/xiaokangtongxue410/article/details/82838872
这里我就不copy了

部署、配置过程：
安装略。。。
配置目录放在：/etc/filebeat

$ cd /etc/filebeat
$ ll
-rw-r--r-- 1 root root  938 Oct 14 10:46 filebeat-htjfweb.yml
drwxr-xr-x 2 root root 4096 Nov  8 14:39 prospectors.d
$ cat filebeat-htjfweb.yml
path.config: /etc/filebeat
path.logs: /var/log/filebeat-kafka-htjfweb
path.data: /var/lib/filebeat/data-kafka-htjfweb
filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb
filebeat.shutdown_timeout: 0
logging.level: info
logging.metrics.enabled: false
logging.files.rotateeverybytes: 104857600
logging.files.keepfiles: 10
logging.files.permissions: 0600

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
filebeat.config:
    prospectors:
        enabled: true
        path: ${path.config}/prospectors.d/product-htjfweb*.yml ##热加载配置
        reload.enabled: true
        reload.period: 10s

output.kafka:
      enabled: true
      hosts: ["10.100.202.177:9092","10.100.202.191:9092","10.100.202.192:9092"]
      topic: logstash-htjfweb
      partition.round_robin: # 集群是否开启kafka的partition分区
        reachable_only: false
      compression: gzip
      max_message_bytes: 10000000
      required_acks: 1

我的配置文件放在prospectors.d目录中

$ cat prospectors.d/product-htjfweb-account-web.yml
- type: log
  enabled: true
  paths:
      - /data/WEBLOG/product-htjfweb-account-web/*/*.log
  scan_frequency: 10s
  fields_under_root: true
  multiline.pattern: '^\d{4}-\d{1,2}-\d{1,2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500
  multiline.timeout: 5s
  exclude_lines: ['WARN']

  fields:
      env: product
      index: logstash-htjf
      logtype: product-htjfweb-account-web
      topic: logstash-htjf
      k8s_host: node
       
  tail_files: false
  close_inactive: 2h
  close_eof: false
  close_removed: true
  clean_removed: true
  close_renamed: false

启动filebeat

$ /usr/local/bin/filebeat -c /etc/filebeat/filebeat-htjfweb.yml

如有多个配置文件 需要将以下配置变更成不同的文件，不然启动报错。

path.config: /etc/filebeat
path.logs: /var/log/filebeat-kafka-htjfweb
path.data: /var/lib/filebeat/data-kafka-htjfweb
filebeat.registry_file: /var/lib/filebeat/registry-kafka-htjfweb

通过查看kafka 的topic里是否有数据来判断收集是否成功。

$ cd /usr/local/kafka/bin/
$ ./kafka-console-consumer.sh --bootstrap-server 10.100.202.191:9092 --topic logstash-htjfweb --from-beginning ##从头查看收集到的数据
$  ./kafka-consumer-groups.sh --group logstash-htjfweb --describe --bootstrap-server 10.100.202.177:9092     ##查看消费情况    

GROUP            TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
logstash-htjfweb logstash-htjfweb 0          13965817        13965822        5               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0
logstash-htjfweb logstash-htjfweb 1          13965595        13965599        4               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0
logstash-htjfweb logstash-htjfweb 2          13965756        13965761        5               logstash-0-e16abafa-1976-4677-a520-ec95c9ce38b8 /10.100.202.47   logstash-0