文章目录
目录
下载 ELK-7.13.2
mkdir -p /opt/software/elk
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.2-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/logstash/logstash-7.13.2-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/kibana/kibana-7.13.2-linux-x86_64.tar.gz
ELK其他历史版本:https://www.elastic.co/cn/downloads/past-releases
解压
tar -zxvf elasticsearch-7.13.2-linux-x86_64.tar.gz
tar -zxvf kibana-7.13.2-linux-x86_64.tar.gz
tar -zxvf logstash-7.13.2-linux-x86_64.tar.gz
mv elasticsearch-7.13.2 /usr/local/
mv logstash-7.13.2 /usr/local/
mv kibana-7.13.2-linux-x86_64 /usr/local/
修改ELK配置文件
ES相关
Logstash相关
修改配置
cd /usr/local/logstash-7.13.2
vim logstash.conf
input {
# stdin { }
tcp {
host => "172.18.141.1" port => 5044 mode => "server" tags => ["tags"] codec => json_lines
}
}
output {
elasticsearch {
hosts => ["172.18.141.1:9200"]
index => "springboot-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
jvm.options 修改内存大小
如果多个配置启动就需要修改 pipelines.yml
vim pipelines.yml
- pipeline.id: a
path.config: "/usr/local/logstash-7.13.2/config/logstash-a-prod.conf"
- pipeline.id: b
path.config: "/usr/local/logstash-7.13.2/config/logstash-b-test.conf"
- pipeline.id: c
path.config: "/usr/local/logstash-7.13.2/config/logstash-c-prod.conf"
启动 Logstash
如若单个启动
bin/logstash -f config/logstash-c-prod.conf
多个配置需要这样后台启动,添加--path.data:
nohup bin/logstash -f config/logstash-c-prod.conf --path.data /data/elk/logstash/logstash-c-prod &
Kibana相关
修改配置
vim /usr/local/kibana-7.13.2-linux-x86_64/config/kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "eddie"
elasticsearch.password: "abcdee"
启动 Kibana
cd /usr/local/kibana-7.13.2-linux-x86_64
nohup ./bin/kibana --allow-root &
Java
pom.xml
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>5.2</version>
</dependency>
logback.xml
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>172.18.141.1:5044</destination>
<!-- 日志输出编码 -->
<encoder
class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<timeZone>UTC</timeZone>
</timestamp>
<pattern>
<pattern>
{
"severity": "%level",
"service": "${springAppName:-}",
"trace": "%X{X-B3-TraceId:-}",
"span": "%X{X-B3-SpanId:-}",
"exportable": "%X{X-Span-Export:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
<root level="info">
<appender-ref ref="logstash"/>
</root>
访问 http://ip:5601
ES定时删除脚本
deleteEsData.sh
#!/bin/bash
today=`date +%Y.%m.%d`;
echo "今天是${today}"
# 获得要删除的日期
# 不指定参数时,默认删除30天前以aaa-开头的数据(因为是凌晨删除,所以不含当天)
daynum=5
# 当参数个数大于1时,提示参数错误
if [ $# -gt 1 ] ;then
echo "要么不传参数,要么只传1个参数!"
exit 101;
fi
# 当参数个数为1时,获取指定的参数
if [ $# == 1 ] ;then
daynum=$1
fi
esday=`date -d '-'"${daynum}"' day' +%Y.%m.%d`;
echo "${daynum}天前是${esday}"
curl -XDELETE http://localhost:9200/test-${esday}
echo "${today}执行完成"
# echo curl -XDELETE http://localhost:9200/test-2021-${esday}
脚本原创地址:https://my.oschina.net/ylchou/blog/507075
Delete Before
Delete After
[root@gfs_v_test_001 elk]# sh deleteEsData.sh
今天是2021.07.13
5天前是2021.07.08
{"acknowledged":true}2021.07.13执行完成
Crontab 定时任务
每晚两点进行删除操作,并且记录ES操作
crontab -e
0 2 * * * sh /data/elk/deleteEsData.sh >> /data/elk/run.log 2>&1