centos7.9环境安装kibana-8.8.2 logstash-8.8.2 elasticsearch-8.8.2 filebeat-8.8.2版本收集nginx日志

闫利朋

已于 2024-04-26 11:17:30 修改

阅读量604

点赞数

分类专栏： elk 文章标签： elasticsearch nginx 大数据

于 2023-07-10 17:17:45 首次发布

本文链接：https://blog.csdn.net/BY_xiaopeng/article/details/131576135

版权

elk 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

一、环境准备

1.架构图

在这里插入图片描述

2.环境准备

主机	操作系统	安装软件	ip
node01	CentOS7.9	nginx filebeat-8.8.2	172.11.0.90
node02	CentOS7.9	logstash-8.8.2	172.11.0.89
node03	CentOS7.9	elasticsearch-8.8.2 kibana-8.8.2	172.11.0.88

二、节点node01安装nginx和filebeat-8.8.2

1.安装nginx

yum -y install epel-release
yum install -y nginx
systemctl start nginx

查看nginx 日志：

cat /var/log/nginx/access.log
192.168.10.2 - - [06/Jul/2023:13:37:30 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"

2.安装filebeat-8.8.2

官方地址:链接

下载安装公开签名秘钥：

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

配置RPM repo

cd /etc/yum.repos.d/
vim elastic.repo  
#添加以下内容
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装filebeat-8.8.2

yum install -y filebeat-8.8.2
cd /etc/filebeat/
mv filebeat.yml filebeat.yml.bak
vim filebeat.yml
#添加以下内容
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    nginx: true
  fields_under_root: true

output.logstash:
  hosts: ["172.11.0.89:5044"]

启动filebeat-8.8.2

systemctl start filebeat
systemctl enable filebeat
查看filebeat运行是否正常：
systemctl status filebeat

三、节点node02安装logstash-8.8.2

官方地址:链接

下载安装公开签名秘钥：

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

配置RPM repo

cd /etc/yum.repos.d/
vim logstash.repo
#添加如下内容
[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装logstash-8.8.2

yum install -y logstash-8.8.2
cd /etc/logstash/

vim logstash.yml
#添加如下内容
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
api.http.host: 172.11.0.89
api.http.port: 9600-9700
path.logs: /var/log/logstash

配置nginx日志收集：
vim conf.d/logstash-nginx.conf
input {
  beats {
    host => "0.0.0.0"
    port => 5044
  }
}

filter {
  if [fields][nginx] {
    grok {
      match => { "message" => '%{IPV4:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} \"%{URIHOST:referrer}\" \"%{DATA:agent}\"' }
    }
  }
}

output {
  elasticsearch {
    hosts => ["172.11.0.88:9200"]
    index => "nginx-%{+YYYY.MM.dd}"
    cacert => "/etc/logstash/certs/http_ca.crt"
    user => "elastic"
    password => "zHRLCoPlOQXk3Fq1D8i9"
    ssl => true
    ssl_certificate_verification => true
  }
  stdout { codec => rubydebug }
}
配置证书：
将elasticsearch-8.8.2的证书拷到本地文件夹：
scp -r root@172.11.0.88:/etc/elasticsearch/certs/http_ca.crt ./
修改证书权限：
chmod 644 http_ca.crt
查看：
ls
#certs文件夹问证书文件http_ca.crt
certs  conf.d  jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options

ll -h certs
total 4.0K
-rw-r--r--. 1 root root 1.9K Jul  5 13:34 http_ca.crt

启动logstash-8.8.2

systemctl start logstash
systemctl enable logstash
查看运行情况：
systemctl status logstash

四、节点node03安装elasticsearch-8.8.2

4.1 安装准备：

vim /etc/sysctl.conf
#添加如下内容：
vm.max_map_count=262144
sysctl -p /etc/sysctl.conf
vm.max_map_count = 262144

vim /etc/security/limits.conf
#添加如下内容：
 *        soft    nofile          65536
 *        hard    nofile          65536
 *        soft    nproc           4096
 *        hard    nproc           4096

4.2安装elasticsearch-8.8.2

下载安装公开签名秘钥

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

配置RPM repo

vim es.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

安装elasticsearch-8.8.2

yum install --enablerepo=elasticsearch elasticsearch-8.8.2
#输入一下内容
Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : zHRLCoPlOQXk3Fq1D8i9

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload

修改配置文件

cd /etc/elasticsearch
vim elasticsearch.yml
#配置如下：
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.11.0.88
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node03"]
http.host: 0.0.0.0

运行elasticsearch-8.8.2

systemctl daemon-reload   
systemctl start elasticsearch    //启动服务
systemctl enable elasticsearch  //设置服务开机自启
systemctl status elasticsearch  //查看服务运行状态

检查elasticsearch是否运行情况：

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://172.11.0.88:9200
Enter host password for user 'elastic':
{
  "name" : "node03",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "CGyyt3xhTTOvAnywgMuEUQ",
  "version" : {
    "number" : "8.8.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "98e1271edf932a480e4262a471281f1ee295ce6b",
    "build_date" : "2023-06-26T05:16:16.196344851Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

五.节点node03安装kibana-8.8.2

下载安装公开签名秘钥

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

配置RPM repo

vim kibana.repo
#添加如下内容：
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装kibana-8.8.2

yum install kibana -y

配置kibana-8.8.2

cd /etc/kibana/
vim kibana.yml
#修改如下内容：
server.port: 5601
server.host: "0.0.0.0"

运行kibana-8.8.2

systemctl enable kibana    //服务开机自启
systemctl start kibana     //开启服务
systemctl status kibana    //查看服务状态

在网页上输入http://172.11.0.88:5601/，刚开始会提示输入Enrollment token:
在这里插入图片描述
token获取：
安装elasticsearch其实有提示;

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxNzIuMTEuMC44ODo5MjAwIl0sImZnciI6IjJiYWYwNTg2NDQxOWJkMTlmZTA0OTZlNzI0NTdhYmYzMDU3ZmI2YjUwOTcxZTAyMjE3NzMzNmFjOGNjMWFjNmIiLCJrZXkiOiI3ZlBQUG9rQkx6WndScURsMHdLWDpQV285WHk0b1NqaUpLcHN4WWJib1h3In0=

拷贝token到网页，点击Configure Elastic ，然后会提示输入Verification required code。
在这里插入图片描述
获取code:

/usr/share/kibana/bin/kibana-verification-code

输入后就可以进入kibana页面了，就可以导入sample数据进行探索了：
在这里插入图片描述

配置nginx数据视图：

在这里插入图片描述

六.调试：

调试logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-messages.conf

闫利朋

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
centos7.9环境安装kibana-8.8.2 logstash-8.8.2 elasticsearch-8.8.2 filebeat-8.8.2版本收集nginx日志

elasticsearch filebeat logstash kibana
复制链接

扫一扫

专栏目录