ELK集群安装教程

一、安装elasticsearch

1. 从官网下载安装包elasticsearch、logstash、filebeat、kibana，版本尽量保持一致

   elasticsearch：https://www.elastic.co/cn/downloads/past-releases#elasticsearch
   logstash：https://www.elastic.co/cn/downloads/past-releases#logstash
   filebeat：https://www.elastic.co/cn/downloads/past-releases#filebeat
   kibana：https://www.elastic.co/cn/downloads/past-releases#kibana
   

2. 创建用户

   useradd es
   passwd es
   

3. 进入目录：cd /home/es

4. 上传elasticsearch-7.6.2-linux-x86_64.tar.gz并且解压

5. 创建目录

   mkdir /home/es/elasticsearch-7.6.2/logs
   mkdir /home/es/elasticsearch-7.6.2/data   
   

6. 修改配置文件：vi /home/es/elasticsearch-7.6.2/config/elasticsearch.yml

   cluster.name: es-application
   node.name: master
   path.data: /home/es/elasticsearch-7.6.2/logs
   path.logs: /home/es/elasticsearch-7.6.2/data   
   network.host: 192.168.248.10
   discovery.seed_hosts: ["192.168.248.10","192.168.248.11","192.168.248.12"]
   cluster.initial_master_nodes: ["master"]
   node.master: true
   http.port: 9200
   http.cors.enabled: true
   http.cors.allow-origin: "*"
   

7. 配置资源使用：vi /etc/security/limits.conf，在文件末尾增加

   * soft nofile 65536
   * hard nofile 131072
   * soft nproc 65535
   * hard nproc 65535 
   End of file
   

8. 配置虚拟内存大小：vi /etc/sysctl.conf

   vm.max_map_count=655360
   

9. 刷新配置：sysctl -p

10. 赋权给es用户：chown -R es:es /home/es

11. 由于我本地装的是jdk8，而es运行需要jdk11，所以修改配置：vi /home/es/elasticsearch-7.6.2/bin/elasticsearch-env，删除判断
    

12. 将虚拟机拷贝两份，作为集群，然后各自修改elasticsearch.yml配置文件即可

    node.name: master
    network.host: 192.168.248.10
    node.master: true
    

13. 切换es用户，然后三台都启动：/home/es/elasticsearch-7.6.2/bin/elasticsearch

    cd /home/es/elasticsearch-7.6.2/bin
    nohup ./elasticsearch &
    

14. 出现相关报错及解决办法

    1）max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

    cat /proc/sys/vm/max_map_count
    sudo sysctl -w vm.max_map_count=262144
    cat /proc/sys/vm/max_map_count
    

    2）the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

    cluster.initial_master_nodes: ["master"]
    

二、安装logstash

1. 上传logstash-7.6.2.tar.gz并且解压

2. 修改配置文件：vi /home/es/logstash-7.6.2/config/logstash-sample.conf

   # Sample Logstash configuration for creating a simple
   # Beats -> Logstash -> Elasticsearch pipeline.
   
   input {
    # 这里可以通过端口传输日志到es
     beats {
       port => 5044
     }
    # 读取本地的日志到es
    # file {
    #   path => ['/home/es/logdata/*.log']
    # }
   }
   filter {
     mutate {
       remove_field => [ "host" ]
     }
   }
   output {
     if [fields][filetype] == "testlog-log" {
      elasticsearch {
       hosts => ["http://192.168.248.10:9200","http://192.168.248.11:9200","http://192.168.248.12:9200"]
       index => "testlog-%{+YYYY.MM.dd}"
      }
     } else if [fields][filetype] == "jar-log"{
      elasticsearch {
       hosts => ["http://192.168.248.10:9200","http://192.168.248.11:9200","http://192.168.248.12:9200"]
       index => "jar-%{+YYYY.MM.dd}"
       #user => "elastic"
       #password => "changeme"
      }
     }else {
      elasticsearch {
       hosts => ["http://192.168.248.10:9200","http://192.168.248.11:9200","http://192.168.248.12:9200"]
       index => "hdfs-%{+YYYY.MM.dd}"
       #user => "elastic"
       #password => "changeme"
      }
     }
   }
   

3. 默认的启动堆栈是4g，如果系统配置不高可以适当减少（可忽略）：vi /home/es/logstash-7.6.2/config/jvm.options.conf

   -Xms400M
   -Xmx400M
   

4. 启动logstash：nohup /home/es/logstash-7.6.2/bin/logstash -f /home/es/logstash-7.6.2/config/logstash-sample.conf &

三、安装kibana

1. 上传kibana-7.6.2-linux-x86_64.tar.gz并且解压

2. 修改配置：vi /home/es/kibana-7.6.2-linux-x86_64/config/kibana.yml

   server.port: 5601
   server.host: "192.168.248.10"
   elasticsearch.hosts: ["http://192.168.248.10:9200","http://192.168.248.11:9200","http://192.168.248.12:9200"]
   i18n.locale: "zh-CN"
   

3. 启动kibana：nohup /home/es/kibana-7.6.2-linux-x86_64/bin/kibana &

4. 测试es集群的健康状态：get _cat/allocation?v
   

5. 测试

   1）修改logstash的logstash-sample.conf配置，修改input，然后重启

   input {
    # 这里可以通过端口传输日志到es
     beats {
       port => 5044
     }
    # 读取本地的日志到es
    # file {
    #   path => ['/home/es/logdata/*.log']
    # }
   }
   

   2）手动修改/home/es/logdata底下的log日志

   3）页面查看，说明日志读取成功
   

   4）创建索引模式
   
   
   

   5）查看内容

   

   6）测试结束，将logstash的logstash-sample.conf配置还原，重启

四、安装filebeat

1. 上传kibana-7.6.2-linux-x86_64.tar.gz并且解压

2. 修改配置文件：vi /home/es/filebeat-7.6.2-linux-x86_64/filebeat.yml

   #输出到logstash
   output.logstash:
     hosts: ["192.168.248.10:5044"]
   
   注释以下这段
   #-------------------------- Elasticsearch output ------------------------------
   #output.elasticsearch:
     # Array of hosts to connect to.
     #hosts: ["localhost:9200"]
   
     # Protocol - either `http` (default) or `https`.
     #protocol: "https"
   
     # Authentication credentials - either API key or username/password.
     #api_key: "id:api_key"
     #username: "elastic"
     #password: "changeme"
      
   # 配置需要采集的日志，一般采集不同应用的日志，分开采集，统一上传到5044
   # 可以跨服务，但是都需要配置filebeat
   # 这里可以自定义filetype,传给logstash,对日志进行分类
   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /home/es/testlog-log/*.log
     fields:     
       filetype: testlog-log
   - type: log
     enabled: true
     paths:
       - /home/es/jar-log/*.log
     fields:     
       filetype: jar-log
   

   ps：需要保证logstash-sample.conf的配置

   input {
     beats {
       path => 5044
     }
   }
   

3. 启动：nohup /home/es/filebeat-7.6.2-linux-x86_64/filebeat -e -c /home/es/filebeat-7.6.2-linux-x86_64/filebeat.yml &

五、安装elasticsearch-analysis-ik

1. 创建文件夹：mkdir /home/es/elasticsearch-7.6.2/plugins/analysis-ik

2. 本地解压elasticsearch-analysis-ik-7.6.2，上传至 /home/es/elasticsearch-7.6.2/plugins/analysis-ik下

3. 重启elasticsearch

4. 测试

   POST _analyze
   {
    "analyzer":"ik_max_word",
       
       
    "text":["杭州市长春药店"]
   }
   

   解析成功