简介
近期公司安排部署一套ELK日志系统,分配给我6台服务器(8C 16G 500G),按照自己的知识分配了如下架构,图画的有些烂,莫怪。
准备工作:
- elastic官网:www.elastic.co | kfka官网:kafka.apache.org
-
下载:elasticsearch、logstash、kibaba、filebeat、kafka
- 服务器配置ip分别为192.168.1.1、192.168.1.2、 192.168.1.3、192.168.1.4、192.168.1.5、192.168.1.6
解压步骤啥的就不说了,直接贴出配置文件了
Elasticsearch集群部署
elasticsearch常见报错处理:https://www.cnblogs.com/zhi-leaf/p/8484337.html
主节点配置文件(3台)
cluster.name: elk-cluster
node.name: elk-master1 # 其余两台分别改成elk-master2 elk-master3
node.master: true
node.data: false
path.data: /app/data/elasticsearch2/data
path.logs: /app/data/elasticsearch2/logs
path.repo: /app/data/elasticsearch2/backup
bootstrap.memory_lock: false
http.port: 10200
transport.tcp.port: 10300
network.host: 192.168.1.1 # 其余两台分别改成 192.168.1.2 192.168.1.3
discovery.zen.ping.unicast.hosts: ["192.168.1.1:10300","192.168.1.2:10300","192.168.1.3:10300"]
cluster.initial_master_nodes: ["192.168.1.1:10300","192.168.1.2:10300","192.168.1.3:10300"]
数据节点配置文件(6台)
cluster.name: elk-cluster
node.name: elk-data1 # 其余五台分别改成elk-data2 elk-data3 elk-data4 elk-data5 elk-data6
node.master: false
node.data: true
path.data: /app/data/elasticsearch/data
path.logs: /app/data/elasticsearch/logs
path.repo: /app/data/elasticsearch/backup
bootstrap.memory_lock: false
http.port: 9200
network.host: 192.168.1.1 # 其余五台分别改成192.168.1.2 192.168.1.3 192.168.1.4 192.168.1.5 192.168.1.6
discovery.zen.ping.unicast.hosts: ["192.168.1.1:10300","192.168.1.2:10300","192.168.1.3:10300"]
cluster.initial_master_nodes: ["192.168.1.1:10300","192.168.1.2:10300","192.168.1.3:10300"]
logstash部署
logstash启动命令: /app/logstash-7.6.0/bin/logstash -f /app/logstash-7.6.0/config/smartproperty-manage-log.yml --config.reload.automatic --path.data /app/data/logstash/data_1
多个logstash启动修改"–path.data"路径
input {
kafka{
bootstrap_servers => ["192.168.1.1:10300","192.168.1.2:10300","192.168.1.3:10300"]
group_id => "logstash-app-dage" # kafka 消费组id,两台logstash配置相同group id实现高可用
auto_offset_reset => earliest
consumer_threads => 3 # 消费线程,和kafka分区一致,多余kafka分区也是闲着
decorate_events => true
topics => ["log-app-dage"] # filebeat输出到kafka的topic name
codec => json
}
}
filter {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:log_time}"} # 匹配生产日志中的时间戳
}
date {
match => ["log_time", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"] # 匹配logtime时间戳
target => "@timestamp" # 将匹配的生产日志中的时间戳替换为@timestamp
}
mutate {
add_field => { "appname" => "dage"} # 添加字段
remove_field => ["agent", "ecs", "log_time"] # 移除字段
}
}
output {
# 判断不同日志级别输出到不同的索引
if [loglevel] == "info" {
elasticsearch {
hosts => ["192.168.1.1:9200"]
index => "log-app-dage-info-%{+YYYY.MM.dd}"
}
}
if [loglevel] == "error" {
elasticsearch {
hosts => ["192.168.1.1:9200"]
index => "log-app-dage-error-%{+YYYY.MM.dd}"
}
}
if [loglevel] == "debug" {
elasticsearch {
hosts => ["192.168.1.1:9200"]
index => "log-app-dage-debug-%{+YYYY.MM.dd}"
}
}
if [loglevel] == "warn" {
elasticsearch {
hosts => ["192.168.1.1:9200"]
index => "log-app-dage-warn-%{+YYYY.MM.dd}"
}
}
}
filebeat配置文件
filebeat.inputs:
- type: log
enabled: true
fields_under_root: true
paths:
- /app/app-dage/logs/app-error.log
fields:
hostIP: '192.168.1.7'
loglevel: 'error'
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- type: log
enabled: true
fields_under_root: true
paths:
- /app/app-dage/logs/app-info.log
fields:
hostIP: '192.168.1.7'
loglevel: 'info'
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- type: log
enabled: true
fields_under_root: true
paths:
- /app/app-dage/logs/app-debug.log
fields:
hostIP: '192.168.1.7'
loglevel: 'debug'
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- type: log
enabled: true
fields_under_root: true
paths:
- /app/app-dage/logs/app-warn.log
fields:
hostIP: '192.168.1.7'
loglevel: 'warn'
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
output.kafka:
enabled: true
hosts: ["192.168.1.4:9092", "192.168.1.5:9092", "192.168.1.6:9092"]
topic: 'log-app-dage'
kafka和zookeeper配置文件
kafka
broker.id=1
advertised.listeners=PLAINTEXT://192.168.1.4:9092 # 其他两台分别修改为 192.168.1.5:9092 192.168.1.6:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
zookeeper.connect=192.168.1.4:2181,192.168.1.5:2181,192.168.1.6:2181
log.dirs=/app/data/kafka/log/kafka
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.roll.hours=168
zookeeper
dataDir=/app/data/kafka/zookeeper
dataLogDir=/app/data/kafka/log/zookeeper
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=5
server.0=172.91.10.17:2888:3888
server.1=172.91.10.26:2888:3888
server.2=172.91.10.27:2888:3888
kafka常用命令
- bin/kafka-topics.sh --bootstrap-server 192.168.1.4:9092 --list # 查看所有topic
- bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.4:9092,192.168.1.5:9092,192.168.1.6:9092 --topic log-app-dage–from-beginning # 查看指定topic中的消息
所有进程统一使用supervisor管理,配置文件参数如下
[program:log-pmtlog]
command=/app/logstash-7.6.0/bin/logstash -f /app/logstash-7.6.0/config/p-m-task.yml --config.reload.automatic --path.data /app/data/logstash/data_3
directory=/app/logstash-7.6.0
numprocs=1
priority=1
autostart=true
satrtretries=3
autorestart=true
stopasgroup=true
killasgroup=true
redirect_stderr=true
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
stdout_logfile=/var/log/supervisor/supervisor_log_pmtlog.log# 进程的日志
environment=JAVA_HOME=/app/jdk1.8.0_221 # JAVA_HOME
下图是我supervisor管理的进程截图
logstash配置备份
input {
kafka {
bootstrap_servers => ["172.87.200.40:9092,172.87.200.50:9092,172.87.200.60:9092"]
group_id => "aml"
auto_offset_reset => earliest
consumer_threads => 3
decorate_events => true
topics => ["aml_log"]
codec => json
}
}
filter{
if [appname] == "aml_web" {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:logtime}, %{LOGLEVEL:loglevel}, %{GREEDYDATA:msg}"}
}
} else if [appname] == "aml_task" {
grok {
match => {"message" => "%{AML_TIME:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:msg}"}
pattern_definitions => {
"AML_TIME" => "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}-%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"
}
}
} else {
grok {
match => {"message" => "%{DATA:logtime} %{DATA:loglevel} %{GREEDYDATA:msg}"}
}
}
mutate {
remove_field => ["agent", "ecs", "log_time", "@version", "endTime", "startTime", "[input][type]", "message"]
lowercase => ["[@metadata][kafka][topic]"]
}
}
output {
elasticsearch {
hosts => ["172.87.200.30:9200"]
index => "%{[@metadata][kafka][topic]}-%{+YYYY.MM.dd}"
}
}