AWS上的服务器日志收集系统架构ELFK+Redis

AWS上的服务器日志收集系统架构ELFK+Redis

架构说明

在服务器端使用filebeat搜集日志信息，不使用logstash搜集而使用filebeat因为logstash的资源占用是filebeat的10倍。在filebeat将日志传输至单独的redis机器172-30-3-5,在该机器上使用docker安装logstash处理日志传输至AWS ES，因为ES只有内网才可以传输，采用单独的内网机器传输日志。

安装filebeat

windows版本

下载url

https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.1-windows-x86_64.msi

直接解压在硬盘里面

配置文件如下

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - D:\WPAYDP\WPAYDP\WPAYDP_PayMachine\logs\*
    - D:\WPAYDP\WPAYDP\WPAYDP_PayMachine\logs\*\*
  fields:
    log_source: wpaydp_paymachine
  fields_under_root: true

  multiline.pattern: '^\d+-\d+-\d+ \d+:\d+:\d+'
  multiline.negate: true
  multiline.match: after

    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------

#-------------------------- redis output ------------------------------
output.redis:
 hosts: ["redis地址:6379"]
 password: xxx
 key: xxx
 db: 0

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

#----------------------------- redis output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== X-Pack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

注意事项

• paths后面跟日志位置
• paths后面的的multiline是日志合并，避免多次换行的行为
• output.redis输出位置为xxxx:6379有key和密码
• .\filebeat -e -c filebeat.yml 执行软件
• fields:
  log_source: wpaydp_paymachine 这个wpaydp_paymahine是区分每个项目的标识符

配置redis服务

修改redis的安全配置,修改端口，修改启动用户，增加密码，以及放在后台运行

port 9999
daemonize yes
requirepass xxxx

使用
sudo -u redis /usr/local/redis/bin/redis-server /usr/local/redis/redis.conf
启动服务

使用logstash处理redis信息

docker安装logstash

docker pull docker.elastic.co/logstash/logstash:7.7.1

设置配置文件

touch /usr/local/logstash/wuerp-wechatmall.conf

文件内容如下

input {
  redis {
    data_type => "list"
    host => "xxxxx"
    db => "0"
    port => 6379
    password => "xxxx"
    key => "xxxxx"
  }
}
 
output {
   if 'wechat-mall' in [kubernetes][namespace]  {
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "wuerp-wechatmall-%{+YYYY.MM.dd}"
  }
}
   if [log_source] == 'wpaydp_paymachine' {
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "wpaydp_paymachine-%{+YYYY.MM.dd}"
  }
}
   if [log_source] == 'wpaydp_paymachine_test' {
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "wpaydp_paymachine_test-%{+YYYY.MM.dd}"
  }
}

   if [log_source] == 'wechatmall-tmp' {
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "wechatmall-tmp-%{+YYYY.MM.dd}"
  }
}
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "k8s-%{+YYYY.MM.dd}"
}
  if [log_source] == 'wechatmall-kq' {
  elasticsearch {
    hosts => ["https://search-wuerp-elk-cluster-gz7ibpo2hmevmchupkd6sa6owy.cn-northwest-1.es.amazonaws.com.cn:443"]
    index => "wechatmall-kq-%{+YYYY.MM.dd}"
}
}
}

这里是在filebeat设置了标识符

启动logstash

docker run -itd -v /usr/local/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml -v /usr/local/logstash/wuerp-wechatmall.conf:/usr/share/logstash/pipeline/logstash.conf  --name logstash.t1 docker.elastic.co/logstash/logstash:7.7.1

ES服务

登录kibana设置刚才设置过的项目

选择Create index pattern

输入刚才设置的信息信息

就可以在kibana看到刚才设置的内容了