ELK原生单节点部署

目录

下载 ELK-7.13.2

mkdir -p /opt/software/elk

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.2-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/logstash/logstash-7.13.2-linux-x86_64.tar.gz
https://artifacts.elastic.co/downloads/kibana/kibana-7.13.2-linux-x86_64.tar.gz

ELK其他历史版本：https://www.elastic.co/cn/downloads/past-releases

解压

tar -zxvf elasticsearch-7.13.2-linux-x86_64.tar.gz
tar -zxvf kibana-7.13.2-linux-x86_64.tar.gz
tar -zxvf logstash-7.13.2-linux-x86_64.tar.gz

mv elasticsearch-7.13.2 /usr/local/
mv logstash-7.13.2 /usr/local/
mv kibana-7.13.2-linux-x86_64 /usr/local/

修改ELK配置文件

ES相关

Elasticsearch 原生安装

Logstash相关

修改配置

cd /usr/local/logstash-7.13.2

vim logstash.conf

input {
  # stdin { }
  tcp {
    host => "172.18.141.1" port => 5044 mode => "server" tags => ["tags"] codec => json_lines
  }
}
output {
  elasticsearch {
        hosts => ["172.18.141.1:9200"]
        index => "springboot-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

jvm.options 修改内存大小

如果多个配置启动就需要修改 pipelines.yml

vim pipelines.yml

- pipeline.id: a
  path.config: "/usr/local/logstash-7.13.2/config/logstash-a-prod.conf"
- pipeline.id: b
  path.config: "/usr/local/logstash-7.13.2/config/logstash-b-test.conf"
- pipeline.id: c
  path.config: "/usr/local/logstash-7.13.2/config/logstash-c-prod.conf"

启动 Logstash

如若单个启动 
bin/logstash -f config/logstash-c-prod.conf 

多个配置需要这样后台启动，添加--path.data：
nohup bin/logstash -f config/logstash-c-prod.conf --path.data /data/elk/logstash/logstash-c-prod & 

Kibana相关

修改配置

vim /usr/local/kibana-7.13.2-linux-x86_64/config/kibana.yml

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "eddie"
elasticsearch.password: "abcdee"

启动 Kibana

cd /usr/local/kibana-7.13.2-linux-x86_64
nohup ./bin/kibana --allow-root &

Java

pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>5.2</version>
</dependency>

logback.xml

<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
	<destination>172.18.141.1:5044</destination>
	<!-- 日志输出编码 -->
	<encoder
			class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
		<providers>
			<timestamp>
				<timeZone>UTC</timeZone>
			</timestamp>
			<pattern>
				<pattern>
					{
					"severity": "%level",
					"service": "${springAppName:-}",
					"trace": "%X{X-B3-TraceId:-}",
					"span": "%X{X-B3-SpanId:-}",
					"exportable": "%X{X-Span-Export:-}",
					"pid": "${PID:-}",
					"thread": "%thread",
					"class": "%logger{40}",
					"rest": "%message"
					}
				</pattern>
			</pattern>
		</providers>
	</encoder>
</appender>

<root level="info">
	<appender-ref ref="logstash"/>
</root>

访问 http://ip:5601

ES定时删除脚本

deleteEsData.sh

#!/bin/bash
today=`date +%Y.%m.%d`;
echo "今天是${today}"
# 获得要删除的日期
# 不指定参数时，默认删除30天前以aaa-开头的数据（因为是凌晨删除，所以不含当天）
daynum=5
# 当参数个数大于1时，提示参数错误
if [ $# -gt 1 ] ;then
        echo "要么不传参数，要么只传1个参数!"
        exit 101;
fi
# 当参数个数为1时，获取指定的参数
if [ $# == 1 ] ;then
        daynum=$1
fi
esday=`date -d '-'"${daynum}"' day' +%Y.%m.%d`;
echo "${daynum}天前是${esday}"
curl -XDELETE http://localhost:9200/test-${esday}
echo "${today}执行完成"
# echo curl -XDELETE http://localhost:9200/test-2021-${esday}

脚本原创地址：https://my.oschina.net/ylchou/blog/507075

Delete Before

Delete After

[root@gfs_v_test_001 elk]# sh deleteEsData.sh 
今天是2021.07.13
5天前是2021.07.08
{"acknowledged":true}2021.07.13执行完成

Crontab 定时任务

每晚两点进行删除操作，并且记录ES操作

crontab -e
0 2 * * * sh /data/elk/deleteEsData.sh  >> /data/elk/run.log 2>&1