ELK+Filebeat日志收集集群搭建

ELK安装教程

作者:半江瑟瑟

•   ELK安装教程
• Elasticsearch 6.3.1 安装

     1.1创建elk域

                               groupadd elk

                               useradd -g elk -m elk

                               usermod -a -G elk elk

                               groups elk

                               passwd elk

                               密码：elk123

    1.2. 修改配置文件，防止安装时出现问题(root 目录操作)

        1. 2.1修改limit信息

             vi /etc/security/limits.conf 

* soft nofile 65536 * hard nofile 131072 * soft nproc 4096 * hard nproc 4096 * soft memlock unlimited * hard memlock unlimited

              执行：source /etc/ security/limits.conf 

           1.2.2 修改集群配置信息

$ vim /etc/hosts 11.181.14.181 node-1 11.181.14.182 node-2 11.181.14.183 node-3

 

          1.2.3修改sysctl 文件

vi /etc/sysctl.conf 

添加下面配置： vm.max_map_count=655360 并执行命令： sysctl -p

     1.3 安装elasticsearch

          1.3.1 上传elasticsearch 文件到elk组下

                    在/home/elk/ap/ifsp/下上传 elasticsearch-6.3.1.zip/ elasticsearch-6.3.1.tz 文件

                    su elk 进入elk用户

                    Zip:

                     Unzip elasticsearch-6.3.1.zip 命令解压

                    Tz

                    tar -zxvf elasticsearch-6.3.1.tz 命令解压

          1.3.2修改配置文件

                  [elk@gaapospapp31l-14181 config]$ cd /home/elk/ap/ifsp/elasticsearch-6.3.1/config

                  cd /home/elk/ap/ifsp/elasticsearch-6.3.1/conf/

                  vi elasticsearch.yml

# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. #       Before you set out to tweak and tune the configuration, make sure you #       understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #集群名称 cluster.name: application # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: #节点名同一集群下节点名称不同 node.name: node-1 #主节点（父节点） node.master: true node.data: false # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): #路径修改为自己设置的路径 path.data: /home/elk/ap/ifsp/data # # Path to log files: #路径修改为自己设置的路径 path.logs: /home/elk/ap/ifsp/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: #设置内存锁定，防止数据swap bootstrap.memory_lock: true bootstrap.system_call_filter: false # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): #设置绑定地址，这里取本机地址 network.host: 11.18.14.181 # # Set a custom port for HTTP: #本机端口 http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] #集群中其他节点ip discovery.zen.ping.unicast.hosts: ["11.18.14.181", "11.18.14.182","11.18.14.183"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true

                        这里三台机器进行集群，每台的配置和这个类似只是稍微有些去别，配置的集群名在所有参加集群的机器中都是一样的，只是节点的地方有改动例如下面时配置：

# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. #       Before you set out to tweak and tune the configuration, make sure you #       understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: #集群名，所有主节点和子节点都是使用这个名字 cluster.name: application # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name fo