elk 提供了完备且成熟的日志存储和分析的解决方案,本文主要介绍 kibana + logstash + filebeat集群的环境搭建
在上篇文章中介绍了elasticsearch 集群的环境搭建。本文章为大家介绍如何实现kibana + logstash + filebeat集群的环境搭建。
elasticsearch+ kibana+ logstash+ filebeat构建高可用分布式日志集群系统(一):elasticsearch集群的安装
elasticsearch+ kibana+ logstash+ filebeat构建高可用分布式日志集群系统(二):kibana + logstash + filebeat集群的安装(本文)
下载 kibana(120节点操作)
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.1.0-linux-x86_64.tar.gz
下载 logstash(120/130/140节点操作)
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.0.tar.gz
下载 filebeat(120/130/140节点操作)
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.0-linux-x86_64.tar.gz
解压logstash/filebeat至安装目录(120/130/140节点操作)
tar -zxvf kibana-7.1.0-linux-x86_64.tar.gz -C /usr/local #仅120节点操作
tar -zxvf logstash-7.1.0.tar.gz -C /usr/local
tar -zxvf filebeat-7.1.0-linux-x86_64.tar.gz -C /usr/local
cd /usr/local
重命名安装目录(120/130/140节点操作)
mv kibana-7.1.0 kibana #仅120节点操作
mv logstash-7.1.0 logstash
mv filebeat-7.1.0 filebeat
创建kibana日志存放路径(120节点操作)
mkdir /opt/ELK/kibana
修改kibana配置文件(120节点操作)
vim /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.77.120:9200","http://192.168.77.130:9200"]
logging.dest: /opt/ELk/kibana/kibana.log
修改logstash配置文件(120/130/140节点操作)
vim /usr/local/logstash/config/logstash-sample.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
}
}
filter {
if [fields][logtype] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}else{
multiline {
pattern => '^\s*({")'
negate => true
what => "previous"
}
grok {
match => { "message" => "%{DATA:timestamp}\|%{IP:springHost}\|\|"}
}
date {
match => [ "timestamp","yyyy-MM-dd-HH:mm:ss"]
locale => "cn"
}
geoip {
source => "springHost"
}
json {
source => "message"
target => "content"
remove_field=>["tags", "beat"]
}
}
}
output {
if [fields][logtype] == "account-log" {
elasticsearch {
hosts => ["http://192.168.77.120:9200","http://192.168.77.130:9200"]
index => "account-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
if [fields][logtype] == "product-log" {
elasticsearch {
hosts => ["http://192.168.77.120:9200","http://192.168.77.130:9200"]
index => "product-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
if [fields][logtype] == "insurance-log" {
elasticsearch {
hosts => ["http://192.168.77.120:9200","http://192.168.77.130:9200"]
index => "insurance-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
if [fields][logtype] == "syslog"{
elasticsearch {
hosts => ["http://192.168.77.120:9200","http://192.168.77.130:9200"]
index => "filebeat-%{+YYYY.MM.dd}"
}
}
}
修改filebeat配置文件(120/130/140节点操作)
filebeat.inputs:
- type: tcp
max_message_size: 10MiB
host: "0.0.0.0:45001"
ienabled: true
document_type: "account-log"
fields:
logtype: account-log
- type: tcp
max_message_size: 10MiB
host: "0.0.0.0:45002"
ienabled: true
document_type: "product-log"
fields:
logtype: product-log
- type: tcp
max_message_size: 10MiB
host: "0.0.0.0:45003"
ienabled: true
document_type: "insurance-log"
fields:
logtype: insurance-log
- type: log
paths:
- /var/log/messages
document_type: "syslog"
fields:
logtype: syslog
output.logstash:
# The Logstash hosts
hosts: ["192.168.77.120:5044","192.168.77.130:5044","192.168.77.140:5044"]
启动安装程序(120/130/140节点操作)
1.启动kibana
nohup ./usr/local/kibana/bin/kibana -d &
2.启动logstash
nohup ./usr/local/logstash/bin/logstash -f ../config/logstash-sample.conf &
3.启动filebeat
nohup ./usr/local/filebeat/filebeat &
系统管理界面/客户端配置
1.kibana可视化管理地址
http://192.168.77.120:5601/app/infra
2.filebeat客户端连接ip端口
192.168.77.120:45001,192.168.77.130:45001,192.168.77.140:45001
192.168.77.120:45002,192.168.77.130:45002,192.168.77.140:45002
192.168.77.120:45003,192.168.77.130:45003,192.168.77.140:45003
到此,关于kibana + logstash + filebeat集群的环境搭建完毕