ELK日志服务搭建和配置-详解

一、下载Elasticsearch 、Kibana、Logstash

https://www.elastic.co/cn/downloads/elasticsearch
https://www.elastic.co/cn/downloads/kibana
https://www.elastic.co/cn/downloads/logstash

二、Elasticsearch安装和配置（占用端口9400）,默认配置为9200

1、解压到制定路径
tar -zxvf elasticsearch-7.16.1-linux-x86_64.tar.gz -C /home/work

2、修改配置文件-elasticsearch.yml
cd /home/work/elasticsearch-7.16.1/config

node.name: node-1
path.data: /usr/local/elasticsearch-7.10.2/data
path.logs: /usr/local/elasticsearch-7.10.2/logs
network.host: 127.0.0.1
http.host: 0.0.0.0
http.port: 9400
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["node-1"]

3、启动-后台
cd /home/work/elasticsearch-7.16.1/bin
./elasticsearch -d

4、减少内存占用
cd /home/work/elasticsearch-7.16.1/config
vim jvm.options
-Xms512m
-Xmx512m


安装插件


三、Logstash安装和配置

1、解压到制定路径
tar -zxvf logstash-7.16.1-linux-x86_64.tar.gz -C /home/work

2、配置文件
cd /usr/local/logstash-7.10.2/bin
新建配置文件
vim logstash-elasticsearch.conf
内容为：
input {
stdin {}
}
output {
elasticsearch {
hosts => '120.78.129.95:9200'
}
stdout {
codec => rubydebug
}
}

#参考配置：
input {
file {
path => "/home/project/logs/sys-*.log"
start_position => beginning
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"
negate => true
auto_flush_interval => 3
what => previous
}
}
}
#配置一个项目下不同类型日志文件，存放到不同type（表）中
filter {
if [path] =~ "info" {
mutate { replace => { type => "sys-info" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
} else if [path] =~ "error" {
mutate { replace => { type => "sys-error" } }
} else {
mutate { replace => { type => "random_logs" } }
}
}

output {
elasticsearch {
hosts => '120.78.129.95:9200'
}
stdout { codec => rubydebug }
}

#配置多个项目路径下日志，存放不同index（库）中
input {
file {
path => "/home/work/project/*.log"
start_position => beginning
type => "log"
# 每次启动从头开始读取日志（如果规则改变）
sincedb_path => "/dev/null"
}

filter {
# grok {
# match => { "message" => "%{COMBINEDAPACHELOG}" }
# }
# date {
# match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
# target => "@timestamp"
# }
}



3、启动
cd /home/work/logstash-7.16.1/bin
./logstash -f logstash-logs.conf > nohup.out 2>&1 &
./logstash -f logstash-apache.conf &
./logstash -f logstash-elasticsearch.conf &

4、修改配置文件减少内存占用
/home/work/logstash-7.16.1/config
vim jvm.options
-Xms256m
-Xmx256m

四、Kibana安装和配置（占用端口5601）
1、解压到制定路径
tar -zxvf kibana-7.16.1-linux-x86_64.tar.gz -C /home/work

2、配置文件
cd /usr/local/kibana-7.10.2/config
vim kibana.yml

修改内容为：
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9400"]
kibana.index: ".kibana"

3、启动
cd /home/work/kibana-7.16.1/bin
./kibana &
nohup ./kibana > nohup.out 2>&1 &