centos7.9环境安装kibana-8.8.2 logstash-8.8.2 elasticsearch-8.8.2 filebeat-8.8.2版本收集nginx日志

一、环境准备

1.架构图

在这里插入图片描述

2.环境准备

主机操作系统安装软件ip
node01CentOS7.9nginx filebeat-8.8.2172.11.0.90
node02CentOS7.9logstash-8.8.2172.11.0.89
node03CentOS7.9elasticsearch-8.8.2 kibana-8.8.2172.11.0.88

二、节点node01安装nginx和filebeat-8.8.2

1.安装nginx

yum -y install epel-release
yum install -y nginx
systemctl start nginx

查看nginx 日志:

cat /var/log/nginx/access.log
192.168.10.2 - - [06/Jul/2023:13:37:30 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"

2.安装filebeat-8.8.2

官方地址:链接

  • 下载安装公开签名秘钥:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
  • 配置RPM repo
cd /etc/yum.repos.d/
vim elastic.repo  
#添加以下内容
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
  • 安装filebeat-8.8.2
yum install -y filebeat-8.8.2
cd /etc/filebeat/
mv filebeat.yml filebeat.yml.bak
vim filebeat.yml
#添加以下内容
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    nginx: true
  fields_under_root: true

output.logstash:
  hosts: ["172.11.0.89:5044"]
  • 启动filebeat-8.8.2
systemctl start filebeat
systemctl enable filebeat
查看filebeat运行是否正常:
systemctl status filebeat

三、节点node02安装logstash-8.8.2

官方地址:链接

  • 下载安装公开签名秘钥:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • 配置RPM repo
cd /etc/yum.repos.d/
vim logstash.repo
#添加如下内容
[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
  • 安装logstash-8.8.2
yum install -y logstash-8.8.2
cd /etc/logstash/

vim logstash.yml
#添加如下内容
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
api.http.host: 172.11.0.89
api.http.port: 9600-9700
path.logs: /var/log/logstash

配置nginx日志收集:
vim conf.d/logstash-nginx.conf
input {
  beats {
    host => "0.0.0.0"
    port => 5044
  }
}

filter {
  if [fields][nginx] {
    grok {
      match => { "message" => '%{IPV4:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} \"%{URIHOST:referrer}\" \"%{DATA:agent}\"' }
    }
  }
}

output {
  elasticsearch {
    hosts => ["172.11.0.88:9200"]
    index => "nginx-%{+YYYY.MM.dd}"
    cacert => "/etc/logstash/certs/http_ca.crt"
    user => "elastic"
    password => "zHRLCoPlOQXk3Fq1D8i9"
    ssl => true
    ssl_certificate_verification => true
  }
  stdout { codec => rubydebug }
}
配置证书:
将elasticsearch-8.8.2的证书拷到本地文件夹:
scp -r root@172.11.0.88:/etc/elasticsearch/certs/http_ca.crt ./
修改证书权限:
chmod 644 http_ca.crt
查看:
ls
#certs文件夹问证书文件http_ca.crt
certs  conf.d  jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options

ll -h certs
total 4.0K
-rw-r--r--. 1 root root 1.9K Jul  5 13:34 http_ca.crt
  • 启动logstash-8.8.2
systemctl start logstash
systemctl enable logstash
查看运行情况:
systemctl status logstash

四、节点node03安装elasticsearch-8.8.2

4.1 安装准备:

vim /etc/sysctl.conf
#添加如下内容:
vm.max_map_count=262144
sysctl -p /etc/sysctl.conf
vm.max_map_count = 262144

vim /etc/security/limits.conf
#添加如下内容:
 *        soft    nofile          65536
 *        hard    nofile          65536
 *        soft    nproc           4096
 *        hard    nproc           4096

4.2安装elasticsearch-8.8.2

  • 下载安装公开签名秘钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • 配置RPM repo
vim es.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
  • 安装elasticsearch-8.8.2
yum install --enablerepo=elasticsearch elasticsearch-8.8.2
#输入一下内容
Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : zHRLCoPlOQXk3Fq1D8i9

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
  • 修改配置文件
cd /etc/elasticsearch
vim elasticsearch.yml
#配置如下:
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.11.0.88
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node03"]
http.host: 0.0.0.0
  • 运行elasticsearch-8.8.2
systemctl daemon-reload   
systemctl start elasticsearch    //启动服务
systemctl enable elasticsearch  //设置服务开机自启
systemctl status elasticsearch  //查看服务运行状态
  • 检查elasticsearch是否运行情况:
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://172.11.0.88:9200
Enter host password for user 'elastic':
{
  "name" : "node03",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "CGyyt3xhTTOvAnywgMuEUQ",
  "version" : {
    "number" : "8.8.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "98e1271edf932a480e4262a471281f1ee295ce6b",
    "build_date" : "2023-06-26T05:16:16.196344851Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

五.节点node03安装kibana-8.8.2

  • 下载安装公开签名秘钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • 配置RPM repo
vim kibana.repo
#添加如下内容:
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
  • 安装kibana-8.8.2
yum install kibana -y
  • 配置kibana-8.8.2
cd /etc/kibana/
vim kibana.yml
#修改如下内容:
server.port: 5601
server.host: "0.0.0.0"
  • 运行kibana-8.8.2
systemctl enable kibana    //服务开机自启
systemctl start kibana     //开启服务
systemctl status kibana    //查看服务状态

在网页上输入http://172.11.0.88:5601/,刚开始会提示输入Enrollment token:
在这里插入图片描述
token获取:
安装elasticsearch其实有提示;

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxNzIuMTEuMC44ODo5MjAwIl0sImZnciI6IjJiYWYwNTg2NDQxOWJkMTlmZTA0OTZlNzI0NTdhYmYzMDU3ZmI2YjUwOTcxZTAyMjE3NzMzNmFjOGNjMWFjNmIiLCJrZXkiOiI3ZlBQUG9rQkx6WndScURsMHdLWDpQV285WHk0b1NqaUpLcHN4WWJib1h3In0=

拷贝token到网页,点击Configure Elastic ,然后会提示输入Verification required code。
在这里插入图片描述
获取code:

/usr/share/kibana/bin/kibana-verification-code

输入后就可以进入kibana页面了,就可以导入sample数据进行探索了:
在这里插入图片描述
在这里插入图片描述
配置nginx数据视图:
在这里插入图片描述
在这里插入图片描述

在这里插入图片描述

六.调试:

调试logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-messages.conf
  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值