Linux——EFLk+Redis+Nginx日志收集

案例

EFLk+Redis+Nginx日志收集

环境

主机IP	主机名
192.168.1.19	EFk

架构服务

Nginx、Filebeat、Redis、Logstash、Elasticsearch、Kibana

1.部署EFK

以下只是下载服务包，具体修改参考：

https://blog.csdn.net/weixin_45191791/article/details/107769529

Elasticsearch：

[root@efk ~]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/elasticsearch-6.6.0.rpm

Filebeat：

[root@efk ~]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/filebeat-6.6.0-x86_64.rpm 

Kibana：

[root@efk ~]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/kibana-6.6.0-x86_64.rpm 

EFK效果：

2.部署Redis

（1）准备安装和数据目录

[root@efk ~]# mkdir -p /data/soft     
[root@efk ~]# mkdir -p /opt/redis_cluster/redis_6379/{conf,logs,pid}  

（2）下载Redis包 并加压到安装目录

[root@efk ~]# cd /data/soft/
[root@efk soft]# wget http://download.redis.io/releases/redis-5.0.7.tar.gz
[root@efk soft]# tar xf redis-5.0.7.tar.gz -C /opt/redis_cluster/    
[root@efk soft]# ln -s /opt/redis_cluster/redis-5.0.7  /opt/redis_cluster/redis

（3）编译安装Redis

[root@efk soft]# cd /opt/redis_cluster/redis
[root@efk redis]# make && make install

（4）编写配置文件，启动服务

[root@efk ~]# vim /opt/redis_cluster/redis_6379/conf/6379.conf
bind 127.0.0.1 192.168.1.19
port 6379
daemonize yes
pidfile /opt/redis_cluster/redis_6379/pid/redis_6379.pid
logfile /opt/redis_cluster/redis_6379/logs/redis_6379.log
databases 16
dbfilename redis.rdb
dir /opt/redis_cluster/redis_6379
[root@efk ~]# redis-server /opt/redis_cluster/redis_6379/conf/6379.conf

3.修改Filebeat配置文件

（1）将output由原来的指向elasticsearch改为指向Redis

[root@efk ~]# vim /etc/filebeat/filebeat.yml 
修改为：
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:

output.redis:
  hosts: ["192.168.1.19"]
  key: "filebeat"
  db: 0
  timeout: 5
[root@efk ~]# systemctl restart filebeat.service 

（2）测试访问网站，登录redis，查看指向是否设置成功

[root@Client ~]# ab -n 1000 -c 20 http://192.168.1.19/


[root@efk ~]# redis-cli    # 登录
127.0.0.1:6379> KEYS *   #列出所有键值
1) "filebeat"
127.0.0.1:6379> TYPE filebeat  # filebeat为键值名
list
127.0.0.1:6379> LLEN filebeat   # 查看list长度
(integer) 1000 
127.0.0.1:6379> LRANGE filebeat 0 -1  # 查看list所有内容

4.部署Logstash

（1）下载Logstash包

[root@efk ~]# cd /data/soft/
[root@efk soft]# ls
redis-5.0.7.tar.gz
[root@efk soft]# yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/6.x/yum/6.6.0/logstash-6.6.0.rpm

（2）配置Logstash，启动服务

[root@efk ~]# cat /etc/logstash/conf.d/redis.conf
input {
  redis {
    host => "192.168.1.19"
    port => "6379"
    db => "0"
    key => "filebeat"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time","float"]
    convert => ["request_time","float"]
  }
}

output {
  stdout {}
  elasticsearch {
    hosts => ["http://192.168.1.19:9200"]
    index => "nginx_access-%{+YYYY.MM.dd}"
    manage_template => false
  }
}
[root@efk ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf  &  # 词条命令在后台执行
[1] 27716

（3）查看是否设置成功

[1]在客户机访问网站

[root@Client ~]# ab -n 100 -c 20 http://192.168.1.19/

[2]刷新es页面

5.分离日志

（1）修改Logstash配置文件

[root@efk ~]# cat /etc/logstash/conf.d/redis.conf
input {
  redis {
    host => "192.168.1.19"
    port => "6379"
    db => "0"
    key => "filebeat"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time","float"]
    convert => ["request_time","float"]
  }
}

output {
  stdout {}
   if "access" in [tags] {
    elasticsearch {
      hosts => ["http://192.168.1.19:9200"]
      index => "nginx_access-%{+YYYY.MM.dd}"
      manage_template => false
    }
   }
   if "error" in [tags] {
    elasticsearch {
      hosts => ["http://192.168.1.19:9200"]
      index => "nginx_error-%{+YYYY.MM.dd}"
      manage_template => false
    }
   }
}
[root@efk ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf  &

（2）自己随意修改nginx配置文件模拟故障 ，但是不要修改的太离谱。

[root@efk ~]# vim /etc/nginx/nginx.conf
......
[root@efk ~]# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
[root@efk ~]# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

（3）es页面刷新

（4）把nginx配置更改回去

[root@efk ~]# vim /etc/nginx/nginx.conf
[root@efk ~]# systemctl restart nginx
[root@efk ~]# systemctl restart nginx

（5）在kibana页面创建索引

6.Kibana图表（可视化观察nginx访问量）

7.Kibana监控（x-pack）