Filebeat + Logstash + Eleaticsearch收集日志

使用ELK方案收集日志

一、部署Filebeat

组织结构

.
├── bin
│   └── filebeat-ctl
├── conf
│   └── filebeat.yml
├── current -> /opt/filebeat/release/filebeat-7.4.2-linux-x86_64
├── data
│   ├── meta.json
│   └── registry
├── logs
└── release
    └── filebeat-7.4.2-linux-x86_64

注意：filebeat的下载请到官网下载，这里的版本是7.4.2

编辑配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/applogs/trace-es7/trace-log-cluter-es7.log
  max_bytes: 1048576
  ignore_older: 168h
  multiline.pattern: '^\[\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500
  multiline.timeout: 5
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  reload.period: 10s
output.logstash:
  hosts: ["172.17.105.101:5046"]
  bulk_max_size: 100

# filebeat logs setting
logging.level: info
logging.to_files: true
logging.files:
  path: /home/user/applogs/souche-filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644
  rotateeverybytesedit: 209715200

编辑启动脚本

#!/bin/bash

FILEBEAT_USER="root"
BASE_DIR="$(cd $(dirname $0)/..; pwd)"
AGENT="${BASE_DIR}/current/filebeat"
ARGS="-c ${BASE_DIR}/conf/filebeat.yml --path.home ${BASE_DIR}/current --path.config ${BASE_DIR}/current --path.data ${BASE_DIR}/data"
TEST_ARGS="test config ${BASE_DIR}/conf/filebeat.yml"

test() {
    su - ${FILEBEAT_USER} -c "$AGENT $ARGS $TEST_ARGS"
}

start() {
    pid="$(ps -ef |grep ${BASE_DIR}/data |grep -v grep |awk '{print $2}')"
    if [ ! "$pid" ];then
        echo "Starting filebeat: "
        test
        if [ $? -ne 0 ]; then
            echo
            exit 1
        fi
        /bin/su - ${FILEBEAT_USER} -c "${AGENT} ${ARGS} &"
        if [ $? -eq 0 ];then
            echo "Start filebeat ok"
        else
            echo "Start filebeat failed"
        fi
    else
        echo "filebeat is still running!"
        exit
    fi
}
stop() {
    echo -n "Stopping filebeat: "
    pid="$(ps -ef |grep ${BASE_DIR}/data |grep -v grep |awk '{print $2}')"
    if [ ! "$pid" ];then
        echo "filebeat is not running"
    else
        kill $pid
        echo "Stopping ..." && sleep 2
        echo "Stop filebeat ok"
    fi
}
restart() {
    stop
    start
}
status(){
    pid="$(ps -ef |grep ${BASE_DIR}/data |grep -v grep |awk '{print $2}')"
    if [ ! "$pid" ];then
        echo "filebeat is not running"
    else
        echo "filebeat is running"
    fi
}
case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    status)
        status
    ;;
    *)
        echo "Usage: ${BASE_DIR} {start|stop|restart|status}"
        exit 1
esac

二、部署logstash

软件包去官方下载，然后在解压目录创建conf.d存放配置文件，可以存放多个配置文件，只需要启动时，指定这个配置目录即可，需要注意的是，每个配置文件都不是独立的配置文件，需要做标识，否则最终启动会报错

参考配置一：

input {
    beats {
        port => 5045
        type => "hadoop"
    }
}

output {
    if [type] == "hadoop" {
        elasticsearch {
            hosts => ["172.16.35.91:10201","172.16.35.90:10201"]
            index => "hadoop-yarn-log-%{+YYYY-MM-dd}"
        }
        #stdout { codec => rubydebug }
    }
}

参考配置二：

input {
  beats {
    add_field => {"beatType" => "tracelog"}
    port => "5045"
  }

  beats {
    add_field => {"beatType" => "applog"}
    port => "5044"
  }
}

filter {
    if [type] == "tracelog" {
        json {
            source => "message"
            remove_field => [ "message" ]
        }
    }
}

output {
  if [beatType] == "tracelog" {
    kafka {
        codec => plain { format => "%{msg}"}
        topic_id => "Topic_trace_log"
        max_request_size => 10485848
        batch_size => 5242880
        retries => 3
        bootstrap_servers => "10.0.0.1:9020,10.0.0.2:9092,10.0.0.3:9092"
    }
    #stdout {
    #  codec => rubydebug
    #}
  }

  if [beatType] == "applog" {
    kafka {
        codec => plain { format => "%{msg}"}
        topic_id => "kafka-applogs-collect"
        max_request_size => 10485848
        batch_size => 5242880
        retries => 3
        bootstrap_servers => "10.0.0.1:9020,10.0.0.2:9092,10.0.0.3:9092"
    }
  }
}

• 使用supervisor应用管理工具

ubuntu系统下载

apt update && apt install -y supervisor

supervisor配置文件/etc/supervisor/conf.d/logstash.conf

[program:logstash]
directory=/home/user/projects/logstash-5.5.3
command=/home/user/projects/logstash-5.5.3/bin/logstash -f /home/user/projects/logstash-5.5.3/conf.d
user=souche
autostart=true
autorestart=false
redirect_stderr=true
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=3
stdout_logfile=/home/user/projects/logstash-5.5.3/logs/logstash.log

启动

systemctl enable supervisor
systemctl start supervisor