概述
ELK stack是一套日志分析软件集合,包括beats, logstash, Elasticsearch, Kibana等;
- Beats 搜集数据的集合, 包括:
- Audit data Auditbeat 搜集linux审计信息,以保证文件完整性
- Log files Filebeat 搜集各种文件类日志
- Availability Heartbeat 分析服务状态,如定时心跳包请求url等
- Metrics Metricbeat 分析系统状态如cpu, redix, memory等
- Network traffic Packetbeat
- Windows event logs Winlogbeat
- Logstash 搜集并对搜集来的数据进行分析转化, 如接收FileBeat的数据,过滤分析后,发送Elasticsearch
- Elasticsearch 全文搜索引擎, 底层是Lucene;
- Kibana 图表画分析显示
- X-Pack 报表,报警,临控,图表等扩展的套件.
参考
安装
规划
方案
节点上安装FileBeat
基于 debian:9.5-slim (22M) 构建
filebeat不依赖java
# Dockerfile
from debian:9.5-slim
ARG version=6.3.2
RUN mkdir -p /tmp; \
cd /tmp; \
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${version}-amd64.deb; \
dpkg -i filebeat-${version}-amd64.deb; \
apt-get install --fix-missing; \
dpkg -i filebeat-${version}-amd64.deb; \
apt-get autoremove -y ; \
apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
ADD root /
CMD [ "filebeat", "-e" ]
配置文件 /root/etc/filebeat/filebeat.yml;
参考: 官方说明(英文)
示例:
filebeat.inputs:
- type: log #可以为Log, Stdin, Redis, UDP, Docker, TCP, Syslog
enabled: true
paths:
- /data/container_vols/nginx/logs/*.log
- /data/container_vols/php7/logs/*.log
output.logstash:
enabled: true
hosts: ["10.0.0.2:5044"]
#ssl:
# certificate_authorities: ["/etc/pki/root/ca.pem"]
# certificate: "/etc/pki/client/cert.pem"
# key: "/etc/pki/client/cert.key"
安全起见,最好配置ssl; 测试时,因为是专用内网,所以未设置。
如果不用logstash, 则可以在filebeat.yml中配置module; filebeat的module指文件集中的特定配置,所谓文件集,指由logstash, filebeat, elasticsearch , kibana等相关配置文件的集合。 比如nginx 文件 集;
nginx module时filebeat配置
filebeat_modules:
- module: nginx
access:
enabled: true
var.paths: ["/path/to/log/nginx/access.log*"]
error:
enabled: true
var.paths: ["/path/to/log/nginx/error.log*"]
管理节点
管理节点安装
docker pull sebp/elk
修改 02-beats-input.conf为:
input {
beats {
port => 5044
ssl => false
}
}
注意事项
- vm.max_map_count; 需要主机修改为>262144; 修改方式:
sysctl -w vm.max_map_count=262144
运行
docker run -p 5601:5601 \
-p 9200:9200 \
-p 5044:5044 \
-e TZ=Asia/Shanghai \
-v ${PWD}/root/etc/logstash/conf.d/02-beats-input.conf:/etc/logstash/conf.d/02-beats-input.conf \
-it --name elk sebp/elk
* Starting Elasticsearch Server [ OK ]
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
...
[2018-08-12T21:48:32,166][INFO ][o.e.n.Node ] [] initializing ...
[2018-08-12T21:48:32,372][INFO ][o.e.e.NodeEnvironment ] [F3qOo7i] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/vdb1)]], net usable_space [12.5gb], net total_space [19.5gb], types [ext4]
[2018-08-12T21:48:32,381][INFO ][o.e.e.NodeEnvironment ] [F3qOo7i] heap size [1015.6mb], compressed ordinary object pointers [true]
[2018-08-12T21:48:32,382][INFO ][o.e.n.Node ] [F3qOo7i] node name derived from node ID [F3qOo7iNRIW14qY5ogFppg]; set [node.name] to override
...