我的EFK机构如下,一共39台VM,fluented从各个服务(图中标记为APP)中以tail的形式读取日志,提取日志可以使用http://fluentular.herokuapp.com/网站对正则进行测试,然后发送到ElasticSearch,然后使用Kibana来展示,EFK均运行在单独的docker容器中
搭建步骤如下
一 Fluented的搭建
1 创建docker-compose.yml,挂载两个挂载点,一个是fluentd的配置文件,一个是日志所在的文件夹
version: '2'
services:
fluentd:
build: .
expose:
- 24224
ports:
- "24224:24224"
volumes:
- /data/conf/fluent.conf:/fluentd/etc/fluent.conf
- /data/logs/nginx:/data/logs/nginx
restart: always
2 创建Dockerfile,增加elasticsearch插件和forward插件
FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install","fluent-plugin-elasticsearch","--no-document", "--version", "4.3.3"]
RUN ["gem", "install","fluent-plugin-forest","--no-document"]
USER root
3 配置fluent.conf
<source>
@type tail
path /data/logs/nginx/https-access.log
pos_file /data/logs/nginx/https-access.log.pos
<parse>
@type nginx
</parse>
tag *
</source>
<match *.**>
@type forest
subtype copy
<template>
<store>
@type elasticsearch
host XX.XX.XX.XX
port 9200
</store>
</template>
</match>
2 ElasticSearch和Kibana的配置
配置docker-compose.xml
version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
environment:
- "discovery.type=single-node"
expose:
- 9200
ports:
- "9200:9200"
kibana:
image: kibana:7.10.1
links:
- "elasticsearch"
ports:
- "80:5601"
3 启动EFK
分别在flentd和EK下运行 docker-compose up --d
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4941a0198a4f fluentd_fluentd "tini -- /bin/entr..." 4 days ago Up 4 days 5140/tcp, 0.0.0.0:24224->24224/tcp fluentd_fluentd_1
a33dee4a2bdb kibana:7.10.1 "/usr/local/bin/du..." 7 days ago Up 6 days 0.0.0.0:80->5601/tcp efk_kibana_1
dd05f58e2cbe docker.elastic.co/elasticsearch/elasticsearch:7.10.2 "/tini -- /usr/loc..." 7 days ago Up 6 days 0.0.0.0:9200->9200/tcp, 9300/tcp efk_elasticsearch_1
4 访问Kibana所在服务器设置index
Discover中进行kibana的日志查询即可