centos7--部署EFK服务(2)--创建nginx日志视图

在EFK部署–zookeeper+kafka服务环境下

1，在elasticsearch服务的主机B下，安装nginx，并启动

[root@kafka02 ~] yum -y install gcc pcre-devel zlib-devel
[root@kafka02 ~] tar -zxf nginx-1.12.2.tar.gz 
[root@kafka02 ~] cd nginx-1.12.2/
[root@kafka02 ~] ./configure 
[root@kafka02 ~] make && make install

[root@kafka02 ~] /usr/local/nginx/sbin/nginx
[root@kafka02 ~] curl -I http://127.0.0.1  #测试一下

#可以在主机A上安装httpd-tools进行压测，增加nginx的日志量
[root@kafka01 ~] yum -y install httpd-tools
[root@kafka01 ~] ab -n 100 -c 100 http://192.168.59.111/index.html

2，在主机B上修改filebeat配置文件

[root@kafka02 ~] vim /etc/filebeat/filebeat.yml
[root@kafka02 ~] cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:					#新增字段
    log_topic: msg      #新增索引
#新增nginx日志收集项
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  fields:
    log_topics: nginx

output.kafka:
  enabled: true
  hosts: ["192.168.59.110:9092","192.168.59.111:9092","192.168.59.112:9092"]
#  topic: msg
  topic: '%{[fields][log_topics]}' #新增引入字段

3，查看主机B的当前topic，和创建nginx的topic，并启动filebeat

[root@kafka02 ~] /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.59.111:2181
__consumer_offsets
msg

[root@kafka02 ~] /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.59.111:2181 --replication-factor 2 --partitions 3 --topic nginx
Created topic nginx.

[root@kafka02 ~] systemctl restart filebeat
[root@kafka02 ~] tailf /var/log/filebeat/filebeat  #查看有误报错

#模拟消费
[root@kafka02 ~] /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.59.111:9092 --topic nginx --from-beginning

4，在logstash服务主机A下，修改配置nginx.conf，并重启logstash

[root@kafka01 ~] vim /etc/logstash/conf.d/nginx.conf
[root@kafka01 ~] cat /etc/logstash/conf.d/nginx.conf
input {
	kafka {
		bootstrap_servers => ["192.168.59.110:9092,192.168.59.111:9092,192.168.59.112:9092"]
	group_id => "logstash"
	topics => "nginx"
	consumer_threads => 5
	}
}
filter {
        json {
                source => "message" #过滤
        }
        mutate { #删除多余字段，因为日志字段量大而导致启动失败
                remove_field => ["@version","fields","prospector","source","host","beat","input","offset","log"] 
        }
        grok { #按照正则分将索引片
                match => { "message" => "%{NGINXACCESS}" }
        }
}
output {
	elasticsearch {
		hosts => "192.168.59.111:9200"
		index => "nginx-%{+YYYY.MM.dd}"
	}
}

#添加正则
[root@kafka01 ~] cd /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/

[root@kafka01 patterns] vim nginx_access
[root@kafka01 patterns] cat nginx_access 
NGINXACCESS %{IPORHOST:client_ip} (%{USER:ident}|- ) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{NUMBER:status} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"

#增加日志收集管道
[root@kafka01 ~] vim /etc/logstash/pipelines.yml

- pipeline.id: msg
  path.config: "/etc/logstash/conf.d/messages.conf"
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
#重启logstash
[root@kafka01 ~] systemctl restart logstash
#可以压测一下，再查看日志
[root@kafka01 ~] ab -n 100 -c 100 http://192.168.59.111/index.html
[root@kafka01 ~] tailf /var/log/logstash/logstash-plain.log

5，在浏览器上测试，创建索引





6，创建nginx日志视图








7,添加访问监控




8， 添加ip访问量前10的ip




9，访问量趋势




10，创建仪表盘




没有nginx 索引需注意以下几点：
1：用talif 查看日志，可以看出服务是否生效
2: pipiline.yml 文件
3：nginx log 里要有内容 可以用ab压测用具生成一些日志