EFK 收集 httpd 日志

环境：centos7.6、filebeat7.9.2、 elasticsearch7.9.2、kibana7.9.2，Apache/2.4.6

kibana 未配置账号密码，需要注意网络安全

1、安装 ElasticSearch

官网下载 elasticsearch-7.9.2-x86_64.rpm

rpm -ivh elasticsearch-7.9.2-x86_64.rpm

启动 ElasticSearch

systemctl start elaticsearch

2、安装 kibana

官网下载 kibana-7.9.2-linux-x86_64.tar.gz, 解压

tar -zxvf kibana-7.9.2-linux-x86_64.tar.gz
cd kibana-7.9.2-linux-x86_64

修改 kibana 配置文件 config/kibana.yml 以下几个位置

server.host: "0.0.0.0"
elasticsearch.hosts: ["
http://localhost:9200"
;]
kibana.defaultAppId: "discover"

启动 kibana（后面操作需要另开一个终端）

bin/kibana --allow-root

配置防火墙，开启 5601 端口

3、安装 httpd

yum install httpd
systemctl start httpd

配置防火墙，开启 80 端口

4、安装 filebeat

官网下载 filebeat-7.9.2-x86_64.rpm

rpm -ivh filebeat-7.9.2-x86_64.rpm

修改 filebeat 配置文件，收集 httpd 日志文件 /var/log/httpd/access_log 下面的日志

[root@ansible002 EFK]# cat /etc/filebeat/filebeat.yml
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  # Change to true to enable this input configuration.
  enabled: true   
  paths:      
    - /var/log/httpd/access_log      #apache日志路径

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]

setup.kibana:
  host: "localhost:5601"

启动 filebeat

systemctl start filebeat

访问 kibana ，kibana:5601, Create index pattern filebeat*

5、修改 httpd 日志格式（第 5、6、7 步修改 httpd 日志格式，方便 kibana 绘图，可以省略），参考文章 https://blog.51cto.com/10950710/2330594

增加 LogFormat “{ … }” apache_json

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    LogFormat "{ \
                 \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \
                 \"@version\": \"1\", \
                 \"tags\":[\"apache\"], \
                 \"message\": \"%h %l %u %t \\\"%r\\\" %>s %b\", \
                 \"clientip\": \"%a\", \
                 \"duration\": %D, \
                 \"status\": %>s, \
                 \"request\": \"%U%q\", \
                 \"urlpath\": \"%U\", \
                 \"urlquery\": \"%q\", \
                 \"bytes\": %B, \
                 \"method\": \"%m\", \
                 \"site\": \"%{Host}i\", \
                 \"referer\": \"%{Referer}i\", \
                 \"useragent\": \"%{User-agent}i\" \
                 }" apache_json

修改 CustomLog 部分 combined 为 apache_json

CustomLog "logs/access_log" apache_json

清空原来的日志

echo "" > /var/log/httpd/access_log

重启 httpd

systemctl restart httpd

6、修改 filebeat 配置文件，增加后面两行

[root@ansible002 EFK]# cat /etc/filebeat/filebeat.yml
...
  paths:      
    - /var/log/httpd/access_log      #apache日志路径
  json.keys_under_root: true         #收集 json 格式的日志
  json.overwrite_keys: true
...

重启 filebeat

systemctl restart filebeat

7、清除 kibana 和 ElasticSearch 里面的 filebeat 索引数据

删除 kibana 界面创建的 filebeat-* 索引

删除 ElasticSearch 里面的 filebeat 索引

[root@ansible002 EFK]# curl localhost:9200/_cat/indices
green  open .kibana-event-log-7.9.2-000001   DZ36w3CZSeikyYnmwHpemA 1 0      5    0  21.7kb  21.7kb
yellow open bank                             vdOh8e0LTvCjb4LdI5iOBA 1 1   1000    0 382.3kb 382.3kb
green  open .apm-custom-link                 UO7UdcAZQQO28H5blUgoqg 1 0      0    0    208b    208b
green  open .kibana_task_manager_1           O6CYMxBDTxqMqRlUrLWYnw 1 0      6 6804 781.2kb 781.2kb
yellow open filebeat-7.9.2-2020.10.14-000001 P_25CSkOQh6fmc4ofbMkHw 1 1     44    0  50.6kb  50.6kb
green  open .apm-agent-configuration         W-fm1FbWQIiVhMf0m5bxJg 1 0      0    0    208b    208b
yellow open shakespeare                      2sDJrDCuTNy5rmp15mF49A 1 1 111396    0  19.4mb  19.4mb
yellow open logstash-2015.05.20              K8xfj2_BROyIKeHVOzLfDw 1 1   4750    0  17.6mb  17.6mb
green  open .async-search                    U4andTshSUmYHTAHiYBUzw 1 0    103   10 296.7mb 296.7mb
green  open .kibana_1                        jPZUqiP6TFGHzYiBkB2hMg 1 0     70  930  13.7mb  13.7mb
yellow open logstash-2015.05.18              iP8xo2uzQaCDcq8NedgwkA 1 1   4631    0  16.6mb  16.6mb
yellow open logstash-2015.05.19              SxOUJ9OASF-z33P1IfdyAQ 1 1   4624    0  16.1mb  16.1mb

[root@ansible002 EFK]# curl -XDELETE localhost:9200/filebeat-7.9.2-2020.10.14-000001
{"acknowledged":true

8、kibana 界面创建的 filebeat-* 索引

9、kibana 使用参考官网https://www.elastic.co/guide/cn/kibana/current/tutorial-load-dataset.html