1、python脚本(write_log.py)
#coding=utf-8
import os
import time
import sys,glob
def write(log):
path = "/data/logs"
filedir = path + "/log.log"
if os.path.exists(filedir):
if os.path.getsize(filedir) > 1024000:
#将log.log重命名
num = len(glob.glob(path+'/log.log*'))
#print(num)
newname = "log.log."+str(num)
os.chdir(path)
os.rename("log.log",newname)
with open(filedir,'a+') as f:
txt = f.read()
f.write(log+'\n')
f.close()
def sleeptime(hour,min,sec):
return hour*3600 + min*60 + sec;
if __name__ == '__main__':
i=0
while 1==1:
i += 1
t = str(time.time())
con = str(i)+'##'+t+'进一步的,如果 url_args 中有过多字段,可能导致 Elasticsearch 集群因为频繁 update mapping 或者消耗太多内存在 cluster state 上而宕机。所以,更优的选择,是只保留明确有用的 url_args 内容,其他部分舍去。END'
write(con)
执行python脚本
python write_log.py
2、logstash配置
1)、导出到file配置
# Beats -> Logstash -> file pipeline.
input {
beats {
port => 5044
}
}
filter {
mutate {
remove_field => ["os","log","tags","@version","ecs","agent","host"]
}
}
output {
file {
path => "/aa/log.log"
}
}
2、导出到es配置
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
}
}
filter {
mutate {
remove_field => ["os","log","tags","@version","ecs","agent","host"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
启动logstash
bin/logstash -f config/log-txt.conf
3、filebeat配置
1)、主机1配置
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /data/logs/*
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
2)、主机2配置
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /data/logs/*
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.157.128:5044"]
启动filebeat
./filebeat -e -c filebeat.yml