集中式日志分析系统ELK安装部署

ELK安装

安装Elasticsearch

安装Logstash

安装Kibana

安装Filebeat

（步骤略，参考官网即可https://www.elastic.co/guide/en/elasticsearch/reference/6.0/getting-started.html）

ELK集群配置

elasticsearch node-1

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-application

# ------------------------------------ Node ------------------------------------
node.name: node-1

# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200

# --------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["192.168.20.132","192.168.20.131"]
discovery.zen.minimum_master_nodes: 1


#form www.elastic.co  ->  Docs  ->  X-Pack -> install
#action.auto_create_index: .security,.security-6,monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*

#install ssl 
#xpack.ssl.key: /opt/elasticsearch-6.1.0/config/certs/mynetstation/mynetstation.key 
#xpack.ssl.certificate: /opt/elasticsearch-6.1.0/config/certs/mynetstation/mynetstation.crt 
#xpack.ssl.certificate_authorities: /opt/elasticsearch-6.1.0/config/certs/ca/ca.crt 
#xpack.security.transport.ssl.enabled: false

elasticsearch node-2

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
cluster.name: my-application

# ------------------------------------ Node ------------------------------------
node.name: node-2

# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200

# --------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["192.168.20.132", "192.168.20.131"]
discovery.zen.minimum_master_nodes: 1

logstash

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    id => "esUser"
    hosts => ["192.168.20.132:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

filebeat

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
filebeat:
- idle_timeout : "5s" 
#=========================== Filebeat prospectors ============================= 
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3

#============================== Kibana =====================================
setup.kibana:
  host: "192.168.20.132:5601"

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["192.168.20.132:5044"]

kibana

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.20.132:9200"

注意事项

es不推荐用root用户启动，需要创建一个用户并授予启动es权限

es启动会自动检查启动用户配置，还需要用户设置Lock内存大小

vi /etc/security/limits.conf

esUser - nofile 65536
esUser soft memlock unlimited
esUser hard memlock unlimited