ELK日志分析系统

Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤，并将其存储供以后使用（如，搜索）。

Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

ELK工作流程

在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。

配置基础环境

1.修改主机名

修改3台虚拟主机的名字

[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
[root@localhost ~]# setenforce 0
setenforce: SELinux is disabled

[root@localhost ~]# hostnamectl set-hostname elk-1
[root@localhost ~]# bash

[root@localhost ~]# hostnamectl set-hostname elk-2
[root@localhost ~]# bash

[root@localhost ~]# hostnamectl set-hostname elk-3
[root@localhost ~]# bash

配置hosts文件

3台节点配置相同

[root@elk-1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
[root@elk-2 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
[root@elk-3 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
[root@elk-1 ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

2.elasticserach安装

三台机器都需要配置

[root@elk-1 ~]# rpm -ivh elasticsearch-6.0.0.rpm 
[root@elk-1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
cluster.name: ELK
node.name: elk-1
node.master: true
node.data: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.100.10
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elk-1","elk-2","elk-3"]

启动服务

通过命令启动es服务，启动后使用ps命令查看进程是否存在或者使用netstat命令查看是否端口启动。

[root@elk-3 ~]# systemctl start elasticsearch
[root@elk-3 ~]# ps -ef |grep elasticsearch
elastic+   2080      1 60 05:50 ?        00:00:13 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root       2149   1853  0 05:50 pts/0    00:00:00 grep --color=auto elasticsearch
[root@elk-3 ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      972/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1202/master         
tcp6       0      0 192.168.100.30:9200     :::*                    LISTEN      2080/java           
tcp6       0      0 192.168.100.30:9300     :::*                    LISTEN      2080/java           
tcp6       0      0 :::22                   :::*                    LISTEN      972/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1202/master         

检测集群状态

通过   来检查集群状态，命令如下

[root@elk-1 ~]# curl '192.168.100.10:9200/_cluster/health?pretty'
{
  "cluster_name" : "ELK",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

部署kibana

  安装kibana

[root@elk-1 ~]# rpm -ivh kibana-6.0.0-x86_64.rpm
warning: kibana-6.0.0-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-6.0.0-1                   ################################# [100%]

配置kibana

[root@elk-1 ~]# vim /etc/kibana/kibana.yml 
server.port: 5601
server.host: 192.168.16.10
elasticsearch.url: "http://192.168.16.10:9200"

启动kibana

[root@elk-1 ~]# systemctl start kibana
[root@elk-1 ~]# ps -ef |grep kibana
kibana     3292      1 21 22:23 ?        00:00:03 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root       3307   2353  0 22:23 pts/1    00:00:00 grep --color=auto kibana
[root@elk-1 ~]# netstat -lntp | grep node
tcp        0      0 192.168.100.10:5601     0.0.0.0:*               LISTEN      3292/node           

启动成功后网页访问，可以访问到如下界面。

Logstash部署

安装logstash

[root@elk-2 ~]# rpm -ivh logstash-6.0.0.rpm 
warning: logstash-6.0.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.0.0-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

配置logstash

 配置/etc/logstash/logstash.yml，修改增加如下

[root@elk-2 ~]# vim /etc/logstash/logstash.yml 
190 http.host: "192.168.100.20"

配置logstan收集syslog日志

[root@elk-2 ~]# vim /etc/logstash/conf.d/syslog.conf
input {  
    file {
        path => "/var/log/messages"
        type => "systemlog"
        start_position => "beginning"
        stat_interval => "3"
    }
}
output {
    if [type] == "systemlog" {
        elasticsearch {
            hosts => ["192.168.100.10:9200"]    //elk-1的IP地址
            index => "system-log-%{+YYYY.MM.dd}"
        }
    }
}
[root@elk-2 ~]# chmod 644 /var/log/messages 

检测配置文件是否错误：

[root@elk-2 ~]# ln -s /usr/share/logstash/bin/logstash /usr/bin/     //创建软连接，方便使用logstash命令

[root@elk-2 ~]# logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK   //ok就没问题啦

启动logstash

[root@elk-2 ~]# vim /etc/rsyslog.conf
#### RULES ####   //在下面添加
*.*@@192.168.100.20:10514
[root@elk-2 ~]# systemctl start logstash

 查看进程

[root@elk-2 ~]# yum install -y policycoreutils-python
[root@elk-2 ~]# semanage port-l |grep syslog
[root@elk-2 ~]# ps -ef |grep logstash
logstash   3792      1 99 22:48 ?        00:00:09 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root       3825   2368  0 22:49 pts/2    00:00:00 grep --color=auto logstash

查看端口

[root@elk-2 ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      969/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1224/master         
tcp6       0      0 192.168.100.20:9600     :::*                    LISTEN      4256/java           
tcp6       0      0 192.168.100.20:9200     :::*                    LISTEN      3054/java           
tcp6       0      0 192.168.100.20:9300     :::*                    LISTEN      3054/java           
tcp6       0      0 :::22                   :::*                    LISTEN      969/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1224/master         

出不来端口

[root@elk-2 ~]# cat /var/log/logstash/logstash-plain.log
[2021-10-31T22:44:08,971][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2021-10-31T22:44:08,974][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2021-10-31T22:44:08,989][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2021-10-31T22:44:08,990][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2021-10-31T22:44:09,157][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-10-31T22:44:09,646][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@elk-2 ~]#  ll /var/lib/logstash/  
total 0
drwxr-xr-x 2 root root 6 Oct 31 22:44 dead_letter_queue
drwxr-xr-x 2 root root 6 Oct 31 22:44 queue
[root@elk-2 ~]#  chown -R logstash /var/lib/logstash/
[root@elk-2 ~]#  systemctl restart logstash 
[root@elk-2 ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      969/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1224/master         
tcp6       0      0 192.168.100.20:9200     :::*                    LISTEN      3054/java           
tcp6       0      0 192.168.100.20:9300     :::*                    LISTEN      3054/java           
tcp6       0      0 :::22                   :::*                    LISTEN      969/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1224/master         

kibana上查看日志

之前部署kibana完成后，还没有检索日志。现在logstash部署完成，我们回到kibana服务器上查看日志索引

2/node           
[root@elk-1 ~]# curl '192.168.100.10:9200/_cat/indices?v'
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana q0OxmWkGQo6hLdhjEb6Dzw   1   1          1            0      7.3kb          3.6kb

获取/删除指定索引详细信息

[root@elk-1 ~]# curl '192.168.100.10:9200/_cat/indices?v'                          health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana q0OxmWkGQo6hLdhjEb6Dzw   1   1          1            0      7.3kb          3.6kb
[root@elk-1 ~]#  curl -XGET/DELETE '192.168.100.10:9200/system-log-2021.11.01?pretty'
{
  "system-log-2021.11.01" : {
    "aliases" : { },
    "mappings" : {
      "systemlog" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "host" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "path" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "type" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1635735662867",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "uJ0xJJGaTGS9mReLdcVTjQ",
        "version" : {
          "created" : "6000099"
        },
        "provided_name" : "system-log-2021.11.01"
      }
    }
  }
}
[root@elk-1 ~]# 

web界面配置