文章目录
1、Logstash利用redis做缓存从filebeat收集nginx日志和系统日志。
环境：dpkg 安装好es集群、kibana和filebeat
efk版本：8.9.0
系统： ubuntu20.04
| IP | 软件 |
|---|---|
| 192.168.1.136 | elasticsearch + redis |
| 192.168.1.137 | elasticsearch |
| 192.168.1.138 | elasticsearch |
| 192.168.1.140 | kibana |
| 192.168.1.141 | filebeat+nginx |
| 192.168.1.142 | filebeat+nginx |
| 192.168.1.143 | logstash |
日志收集流程：
filebeat收集nginx的访问和错误日志、rsyslog日志后发送到redis，logstash从Redis中获取数据，并将其处理后发送到Elasticsearch集群进行索引和存储，Kibana是通过与Elasticsearch交互来取回数据，并将其用于查询、分析和可视化
nginx配置
#nginx 修改配置，添加json日志格式
cat /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"uri":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"tcp_xff":"$proxy_protocol_addr",'
'"http_user_agent":"$http_user_agent",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json ;
#access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#filebeat配置
Filebeat：Filebeat 是一个轻量级的日志文件收集器，用于实时收集和发送日志文件到 Elasticsearch 或者 Logstash 以进行存储和分析。它可以监视指定的日志文件或者目录，并将新写入的日志数据发送到后端。它可以按行读取日志文件，并将其转发到 Elasticsearch 或者 Logstash。
141和142filebeat配置相同
cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access_json.log
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: true #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["nginx-access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
- type: log
enabled: true
paths:
- /var/log/syslog
tags: ["syslog"]
output.redis:
hosts: ["192.168.1.136:6379"]
key: "filebeat"
password: "123456"
#db: 0
setup.ilm.enabled: false
redis
Redis可以作为Logstash的输出插件，将数据缓存到Redis中。这可以提高性能，减少对Elasticsearch的直接访问负载。
redis修改配置并启动
vi /etc/redis/redis.conf
bind 0.0.0.0
requirepass 123456
登录redis，可以查看有syslog的日志
127.0.0.1:6379> llen filebeat
(integer) 23024
127.0.0.1:6379> lpop filebeat
logstash配置
Logstash：Logstash 是一个数据收集和处理管道工具，用于从多个来源（如日志文件、消息队列）收集、处理和转换数据，并将其发送到 Elasticsearch 作为日志存储。Logstash 支持丰富的过滤器和插件，用于处理和转换数据。
cat filebeat_redis_logstash_es.conf
input {
redis {
host => '192.168.1.136'
port => "6379"
password => "123456"
db => "0"
data_type => 'list'
key => "filebeat"
}
}
output {
if "syslog" in [tags] {
elasticsearch {
hosts =>["192.168.1.136:9200","192.168.1.137:9200","192.168.1.138:9200"]
index => "syslog-%{+YYYY.MM.dd}"
template_overwrite => true
}
}
if "nginx-access" in [tags] {
elasticsearch {
hosts => ["192.168.1.136:9200"]
index => "ngin2-accesslog-%{+YYYY.MM.dd}"
template_overwrite => true
}
}
if "nginx-error" in [tags] {
elasticsearch {
hosts => ["192.168.1.136:9200"]
index => "nginx-errorlog-%{+YYYY.MM.dd}"
template_overwrite => true #
}
}
}
重启logstash，插件可以看到生成的索引
Kibana：Kibana 是一个可视化和分析平台，用于在 Elasticsearch 上的数据进行搜索、查看、分析和可视化。它提供了一个友好的 Web 界面，使用户能够轻松地探索和理解存储在 Elasticsearch 中的数据。Kibana 支持创建仪表板、图表和地图等可视化组件，以可视化和呈现数据。
访问kibana，分别创建nginx-access访问日志，nginx-error错误日志，rsyslog系统日志的数据视图
查看
2、分别部署RabbitMQ单机版及RabbitMQ集群
2.1 rabbitMQ 单机 安装
官方网站https://www.rabbitmq.com/install-debian.html#apt-quick-start-cloudsmith
官方安装脚本，ubuntu22.04系统
root@server01:~/sh# cat install_rabbitmq.sh
#!/bin/sh
sudo apt-get install curl gnupg apt-transport-https -y
## Team RabbitMQ's main signing key
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
## Community mirror of Cloudsmith: modern Erlang repository
curl -1sLf https://ppa.novemberain.com/gpg.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
## Community mirror of Cloudsmith: RabbitMQ repository
curl -1sLf https://ppa.novemberain.com/gpg.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null
## Add apt repositories maintained by Team RabbitMQ
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
## Provides modern Erlang/OTP releases
##
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
# another mirror for redundancy
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
## Provides RabbitMQ
##
deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
# another mirror for redundancy
deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF
## Update package indices
sudo apt-get update -y
## Install Erlang packages
sudo apt-get install -y erlang-base \
erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
erlang-runtime-tools erlang-snmp erlang-ssl \
erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl
## Install rabbitmq-server and its dependencies
sudo apt-get install rabbitmq-server -y --fix-missing
列出rabbitmq-server软件的所有来源
apt-cache madison rabbitmq-server
查看rabbitmq版本
root@server01:~/sh# rabbitmqctl status | grep “RabbitMQ version”
RabbitMQ version: 3.12.3
创建用户
# 1、创建用户并设置密码
[root@rabbit01 ~]# rabbitmqctl add_user admin 123456
#2、赋予其administrator角色
[root@rabbit01 ~]# rabbitmqctl set_user_tags admin administrator
#3、设置权限
[root@rabbit01 ~]# rabbitmqctl set_permissions -p "/" admin '.*' '.*' '.*'
修改密码
root@server02:~# rabbitmqctl change_password n80 12345
访问
http://120.79.60.104:15672/#/
root@server02:~# rabbitmqctl add_vhost n80 #是用来创建一个名为n80的虚拟主机
root@server02:~# rabbitmqctl list_vhosts #用来列出当前 RabbitMQ 服务器上的所有虚拟主机。
root@server02:~# rabbitmqctl list_queues #列出当前 RabbitMQ 服务器上的所有队列及其相关信息。
添加用户，配置权限
2.2 rabbitMQ集群部署
集群介绍
rabbitmq集群分为二中方式：普通模式和镜像模式
集群中有二种节点类型：
磁盘节点：将数据保存到内存和磁盘
内存节点：将数据保存到内存
内存节点虽然不写入磁盘，但是它执行比磁盘节点要好，集群中，只需要一个磁
盘节点来保存数据就足够了如果集群中只有内存节点，那么不能全部停止它们，
否则所有数据消息在服务器全部停机之后都会丢失
三个节点使用官方脚本安装rabbitmq，配置主机名和hosts解析
vi /etc/hosts
192.168.74.70 node1.rabbit.org node1
192.168.74.71 node2.rabbit.org node2
192.168.74.72 node3.rabbit.org node3
创建rabbitMQ集群
各服务器关闭 RabbitMQ：
rabbitmq的集群是依赖与erlang的集群来工作的，所以必须先构建erlang的集群环境，而 Erlang 的集群中各节点是通过一个magic cookie来实现的，这个cookie存放在/var/lib/rabbitmq/.erlang.cookie中，文件权限400，各个节点cookie保持一致，否则节点无法通信
在 mq-node1同步.erlang.cookie 至其他两台服务器：
scp /var/lib/rabbitmq/.erlang.cookie 192.168.74.71:/var/lib/rabbitmq/
scp /var/lib/rabbitmq/.erlang.cookie 192.168.74.72:/var/lib/rabbitmq/
node1：
在 node1 作为内存节点添加到node3，并作为内存节点，在 node1执行以下命令
root@node1:/etc/rabbitmq# rabbitmqctl stop_app
Stopping rabbit application on node rabbit@node1 ...
root@node1:/etc/rabbitmq# rabbitmqctl reset
Resetting node rabbit@node1 ...
root@node1:/etc/rabbitmq# rabbitmqctl join_cluster rabbit@node3 --ram
Clustering node rabbit@node1 with rabbit@node3
root@node1:/etc/rabbitmq# rabbitmqctl start_app
Starting node rabbit@node1 ...
node2：
在 node2 作为内存节点添加到 node3，并作为内存节点，在node2执行以下命令：
root@node2:~# systemctl restart rabbitmq-server
root@node2:~# rabbitmqctl stop_app ##停止 app 服务
Stopping rabbit application on node rabbit@node2 ...
root@node2:~# rabbitmqctl reset ##清空元数据
Resetting node rabbit@node2 ...
root@node2:~# rabbitmqctl join_cluster rabbit@node3 --ram
Clustering node rabbit@node2 with rabbit@node3
root@node2:~# rabbitmqctl start_app ##启动 app 服务
Starting node rabbit@node2 ...
node3查看
root@node3:~# rabbitmqctl cluster_status
............
Disk Nodes
rabbit@node3
RAM Nodes #内存节点
rabbit@node1
rabbit@node2
Running Nodes #运行的节点
rabbit@node1
rabbit@node2
rabbit@node3
Versions
rabbit@node1: RabbitMQ 3.11.15 on Erlang 25.3.1
rabbit@node2: RabbitMQ 3.11.15 on Erlang 25.3.1
rabbit@node3: RabbitMQ 3.11.15 on Erlang 25.3.1
CPU Cores
Node: rabbit@node1, available CPU cores: 2
Node: rabbit@node2, available CPU cores: 2
Node: rabbit@node3, available CPU cores: 2
............
将集群设置为镜像模式
rabbitmqctl set_policy ha-all “^” ‘{“ha-mode”:“all”}’
验证集群状态
在任一节点创建账号
rabbitmqctl add_user admin 123456
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p "/" admin '.*' '.*' '.*'
所有节点启用插件
sudo rabbitmq-plugins enable rabbitmq_management #启用web插件
三个节点都可以登录
869
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
