ELK日志分析Logstash

elasticsearch的安装和集群配置
https://blog.csdn.net/zhangyu_sing/article/details/99854860

安装logstash

yum install ./logstash-6.6.1.rpm

编写配置文件进行测试

input {
        stdin {}
}
filter {
  grok {
    match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
  }
}

output {
         stdout {}
}

指定测试配置文件运行logstash服务

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

输入指定内容测试其分析能力

172.25.33.250 GET /index.html 15824 0.043

安装apache，编辑其默认页面并启动

主机通过ab命令进行访问测试

ab -c 1 -n 100 http://172.25.42.2/index.html

编写es.conf进行测试

input {
#         stdin {}
#          file {
#                  path => "/var/log/elasticsearch/my-es.log"
#                  start_position => "beginning"
#                  codec => multiline {
#                           pattern => "^\["
#                           negate => "true"
#                           what => "previous"
#                }
#          }
#
#           syslog {
#                     port => 514
#            }
             file {
                      path => "/var/log/httpd/access_log"
                      start_position => "beginning"
             }
}

filter {
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG" }
  }
}

output {
         stdout {}

         elasticsearch {
                hosts => ["172.25.33.1:9200"]
                index => "apachelog-%{+YYYY.MM.dd}"
        }

}

指定配置文件再次运行logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf

通过UI界面索引查看其相应的信息

加入数据库储存日志信息
安装redis

yum install -y make gcc
make && make install         

初始化redis服务

cd utils
./install_server.sh

修改配置文件并开启redis服务

cat /etc/redis/6379.conf |grep ^bind

重新启动服务使得配置生效

/etc/init.d/redis_6379 restart

修改logstash配置文件中的接收数据的主机为redis所在主机

redis {
                host => ["172.25.33.3:6379"]
                data_type => "list"
                key => logstashtoredis
        }

在server3上同样安装logstash

修改配置文件，将server2的日志数据作为输入.输出至主节点

vim /etc/logstash/conf.d/redis.conf

文件内容如下

input {
        redis {
              host => ["172.25.33.3"]
              data_type => "list"
              key => "logstashtoredis"
        }
}


output {
        stdout {}

        elasticsearch {
             hosts => ["172.25.33.1:9200"]
             index => "apachelog-%{+YYYY.MM.dd}"
         }
}

启动server2的logstash服务

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf 

启动server3的logstash服务

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf

web界面查看信息