搭建ELK elasticsearch+kibana+filebeat

1、启动elasticsearch

docker run -d --name elasticsearch -p 9209:9200 -p 9309:9300 -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms256m -Xmx256m"   docker.io/elasticsearch
其中 -e ES_JAVA_OPTS="-Xms256m -Xmx256m"为设置elasticsearch的使用内存大小为256m，默认为2G
临时启动一个elastic容器以后用docker cp 命令把相应的文件复制到宿主机
docker run -d --name elasticsearch -p 9209:9200 -p 9309:9300 -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -v /zhangm/docker/config/es/config:/usr/share/elasticsearch/config -v /zhangm/docker/data/es/data:/usr/share/elasticsearch/data docker.io/elasticsearch
问题：挂在路径以后elasticsearch无法启动，提示目录没有权限，所以需要弄明白相应的文件夹权限
ps：elastic服务不能用root权限启动

2、启动kibana

docker run --name kibana -p 5609:5601  --link elasticsearch:elasticsearch  -d kibana:latest 

3、安装filebeat
下载

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-x86_64.rpm

安装filebeat

rpm -ivh filebeat-5.5.1-x86_64.rpm

配置文件路径
/etc/filebeat/filebeat.yml
日志文件路径
/var/log/filebeat
注意 每次新启动都会生成一个新的filebeat，上次启动的会被mv为filebeat.1
启动命令
systemctl restart filebeat

编辑filebeat的配置文件filebeat.yml,增加"pipeline= "yum"

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/yum.log
    #- c:\programdata\elasticsearch\logs\*
  document_type: "yum"
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.42:9209"]
  pipeline: "yum"
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

将pipeline加入到elastic
方法1：
创建一个json文件yum-pipeline.json

{
"description": "grok yum-pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{SYSLOGTIMESTAMP:date} %{WORD:method} ?: %{USERNAME:name}"]
      }
    },
    {
      "remove": {
        "field": "method"
      }
    }
  ]
  }

导入
curl -H 'Content-Type:application/json' -XPUT 'http://192.168.1.42:9209/_ingest/pipeline/yum' -d@yum-pipeline.json

方法2：
访问kibana服务，找到Dev Tools运行

PUT _ingest/pipeline/yum
{
    "description": "...",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{SYSLOGTIMESTAMP:date} %{WORD:method} ?: %{USERNAME:name}"]
        }
      },
      {
        "remove": {
          "field": "message"
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "yyyy-MM-dd HH:mm:ss"
          ]
        }
      }
    ]
}

 

参考：http://blog.51cto.com/dressame/2166174?source=dra