filebeat+elk集群配置

 

 

1、python脚本（write_log.py）

#coding=utf-8

import os
import time
import sys,glob


def write(log):
   path = "/data/logs"
   filedir = path + "/log.log"
   if os.path.exists(filedir):
      if os.path.getsize(filedir) > 1024000:
         #将log.log重命名
         num = len(glob.glob(path+'/log.log*'))
         #print(num)
         
         newname = "log.log."+str(num)
         os.chdir(path)
         os.rename("log.log",newname)

   with open(filedir,'a+') as f:
      txt = f.read()
      f.write(log+'\n')
   f.close()

def sleeptime(hour,min,sec):
   return hour*3600 + min*60 + sec;

if __name__ == '__main__':
   
   i=0
   while 1==1:
      i += 1
      t = str(time.time())
      con = str(i)+'##'+t+'进一步的，如果 url_args 中有过多字段，可能导致 Elasticsearch 集群因为频繁 update mapping 或者消耗太多内存在 cluster state 上而宕机。所以，更优的选择，是只保留明确有用的 url_args 内容，其他部分舍去。END'
      write(con)

执行python脚本

python write_log.py

2、logstash配置

1)、导出到file配置

# Beats -> Logstash -> file pipeline.

input {
  beats {
    port => 5044
  }
}
filter {
  mutate {
    remove_field => ["os","log","tags","@version","ecs","agent","host"]
  }
}
output {
  file {
    path => "/aa/log.log"
  }
}

2、导出到es配置

# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

filter {
  mutate {
    remove_field => ["os","log","tags","@version","ecs","agent","host"]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

启动logstash

bin/logstash -f config/log-txt.conf

3、filebeat配置

1）、主机1配置

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /data/logs/*
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

2）、主机2配置

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /data/logs/*
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.157.128:5044"]

启动filebeat

./filebeat -e -c filebeat.yml