基于Centos7.6安装Filebeat7.3

前言

该篇为EFK日志系统搭建的姊妹篇，第一篇为基于Centos7.6安装ElasticSearch7.3,主要就是记录一下Filebeat的安装配置以及抓取日志落地到ElasticSearch.

下载解压

$ wget -P /data/app/ https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.1-linux-x86_64.tar.gz
$ cd /data/app 
$ tar -xvf filebeat-7.3.1-linux-x86_64.tar.gz
# filebeat-7.3.1/filebeat -e

配置Filebeat

进入filebeat-7.3.1安装目录，可以看见其配置文件filebeat.yml。

Inputs

input 配置，告诉crawler抓取哪些文件变化，为了抓取的数据便于后面可视化分析，这里配置为json数据，在应用日志输出格式定义一下即可。

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /app/xxx/logs/*_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

Outputs

为beat配置输出环境，默认为ElasticSearch,是当前Elastic官方推荐的默认输出。

#================================ Outputs =====================================
# 解决Index Pattern不生效
setup.ilm.enabled: false
setup.template:
  name: "log"
    # pattern与output的index保持一致
  pattern: "log-access-*"
  settings:
  # 本地日志不需要多副本和分片，节省一点ES的资源
     index.number_of_shards: 1
     index.number_of_replicas: 0

#-------------------------- Elasticsearch output ------------------------------
# ES 地址及索引
output.elasticsearch:
  hosts: ["10.231.xx.xxx:8200"]
  # 自定义索引,需要配置上面的模板
  index: "log-access-%{[agent.version]}-%{+yyyy.MM.dd}"
# 权限认证配置
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

启动测试

终端键入:

$ ./filebeat -e -c filebeat.yml
...
2019-09-15T15:41:42.205+0800    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.3.1
2019-09-15T15:41:42.205+0800    INFO    elasticsearch/client.go:170     Elasticsearch url: http://10.231.xxx.xxx:8200
2019-09-15T15:41:42.206+0800    INFO    [publisher]     pipeline/module.go:97   Beat name: 1-app-web1.localdomain
2019-09-15T15:41:42.206+0800    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-09-15T15:41:42.206+0800    INFO    instance/beat.go:421    filebeat start running.
2019-09-15T15:41:42.207+0800    INFO    registrar/registrar.go:145      Loading registrar data from /app/filebeat/data/registry/filebeat/data.json
2019-09-15T15:41:42.207+0800    INFO    registrar/registrar.go:152      States Loaded from registrar: 1
2019-09-15T15:41:42.207+0800    INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-09-15T15:41:42.207+0800    INFO    log/input.go:148        Configured paths: [/app/xxx/logs/one_json.log]
2019-09-15T15:41:42.207+0800    INFO    input/input.go:114      Starting input of type: log; ID: 16861026505510540300
2019-09-15T15:41:42.207+0800    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-09-15T15:41:42.207+0800    INFO    cfgfile/reload.go:171   Config reloader started
2019-09-15T15:41:42.208+0800    INFO    log/harvester.go:253    Harvester started for file: /app/xxx/logs/one_json.log
2019-09-15T15:41:42.208+0800    INFO    cfgfile/reload.go:226   Loading of config files completed.
2019-09-15T15:41:45.200+0800    INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2019-09-15T15:41:46.201+0800    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://10.231.xxx.xxx:8200))
2019-09-15T15:41:46.208+0800    INFO    elasticsearch/client.go:743     Attempting to connect to Elasticsearch version 7.3.1
2019-09-15T15:41:46.239+0800    INFO    template/load.go:88     Template log already exists and will not be overwritten.
2019-09-15T15:41:46.239+0800    INFO    [index-management]      idxmgmt/std.go:289      Loaded index template.
2019-09-15T15:41:46.242+0800    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://10.231.xxx.xxx:8200)) established

由于还没安装Kibana,所以暂时通过elasticsearch-head进行查看，打开可视化页面看到以log-开头的相关index,如图:

如需切到后台运行:

$ nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

后记

选择Filebeat而非Logstash的原因,有以下两点:

• 相比Logstash之下，它是比较轻量级的数据采集器，尤其是资源开销(主要因素)。
• 业务简单，就是分布式纯日志收集

后面应该还会进行一次业务技术升级，使用消息队列Kafka作为连接Filebeat和ES的中间层，因为随着QPS的增大，ES可能被压垮，需要通过Kafka来削峰填谷。

参考文章

• filebeat下载地址
• filebeat7.3官方文档
• filebeat7.3配置文档