一、环境准备
1.架构图
2.环境准备
主机 | 操作系统 | 安装软件 | ip |
---|---|---|---|
node01 | CentOS7.9 | nginx filebeat-8.8.2 | 172.11.0.90 |
node02 | CentOS7.9 | logstash-8.8.2 | 172.11.0.89 |
node03 | CentOS7.9 | elasticsearch-8.8.2 kibana-8.8.2 | 172.11.0.88 |
二、节点node01安装nginx和filebeat-8.8.2
1.安装nginx
yum -y install epel-release
yum install -y nginx
systemctl start nginx
查看nginx 日志:
cat /var/log/nginx/access.log
192.168.10.2 - - [06/Jul/2023:13:37:30 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
192.168.10.2 - - [06/Jul/2023:13:37:31 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58" "-"
2.安装filebeat-8.8.2
官方地址:链接
- 下载安装公开签名秘钥:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
- 配置RPM repo
cd /etc/yum.repos.d/
vim elastic.repo
#添加以下内容
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
- 安装filebeat-8.8.2
yum install -y filebeat-8.8.2
cd /etc/filebeat/
mv filebeat.yml filebeat.yml.bak
vim filebeat.yml
#添加以下内容
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
nginx: true
fields_under_root: true
output.logstash:
hosts: ["172.11.0.89:5044"]
- 启动filebeat-8.8.2
systemctl start filebeat
systemctl enable filebeat
查看filebeat运行是否正常:
systemctl status filebeat
三、节点node02安装logstash-8.8.2
官方地址:链接
- 下载安装公开签名秘钥:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
- 配置RPM repo
cd /etc/yum.repos.d/
vim logstash.repo
#添加如下内容
[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
- 安装logstash-8.8.2
yum install -y logstash-8.8.2
cd /etc/logstash/
vim logstash.yml
#添加如下内容
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
api.http.host: 172.11.0.89
api.http.port: 9600-9700
path.logs: /var/log/logstash
配置nginx日志收集:
vim conf.d/logstash-nginx.conf
input {
beats {
host => "0.0.0.0"
port => 5044
}
}
filter {
if [fields][nginx] {
grok {
match => { "message" => '%{IPV4:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} \"%{URIHOST:referrer}\" \"%{DATA:agent}\"' }
}
}
}
output {
elasticsearch {
hosts => ["172.11.0.88:9200"]
index => "nginx-%{+YYYY.MM.dd}"
cacert => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "zHRLCoPlOQXk3Fq1D8i9"
ssl => true
ssl_certificate_verification => true
}
stdout { codec => rubydebug }
}
配置证书:
将elasticsearch-8.8.2的证书拷到本地文件夹:
scp -r root@172.11.0.88:/etc/elasticsearch/certs/http_ca.crt ./
修改证书权限:
chmod 644 http_ca.crt
查看:
ls
#certs文件夹问证书文件http_ca.crt
certs conf.d jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options
ll -h certs
total 4.0K
-rw-r--r--. 1 root root 1.9K Jul 5 13:34 http_ca.crt
- 启动logstash-8.8.2
systemctl start logstash
systemctl enable logstash
查看运行情况:
systemctl status logstash
四、节点node03安装elasticsearch-8.8.2
4.1 安装准备:
vim /etc/sysctl.conf
#添加如下内容:
vm.max_map_count=262144
sysctl -p /etc/sysctl.conf
vm.max_map_count = 262144
vim /etc/security/limits.conf
#添加如下内容:
* soft nofile 65536
* hard nofile 65536
* soft nproc 4096
* hard nproc 4096
4.2安装elasticsearch-8.8.2
- 下载安装公开签名秘钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
- 配置RPM repo
vim es.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
- 安装elasticsearch-8.8.2
yum install --enablerepo=elasticsearch elasticsearch-8.8.2
#输入一下内容
Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.
The generated password for the elastic built-in superuser is : zHRLCoPlOQXk3Fq1D8i9
If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.
You can complete the following actions at any time:
Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.
Generate an enrollment token for Kibana instances with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.
-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
- 修改配置文件
cd /etc/elasticsearch
vim elasticsearch.yml
#配置如下:
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.11.0.88
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node03"]
http.host: 0.0.0.0
- 运行elasticsearch-8.8.2
systemctl daemon-reload
systemctl start elasticsearch //启动服务
systemctl enable elasticsearch //设置服务开机自启
systemctl status elasticsearch //查看服务运行状态
- 检查elasticsearch是否运行情况:
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://172.11.0.88:9200
Enter host password for user 'elastic':
{
"name" : "node03",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "CGyyt3xhTTOvAnywgMuEUQ",
"version" : {
"number" : "8.8.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "98e1271edf932a480e4262a471281f1ee295ce6b",
"build_date" : "2023-06-26T05:16:16.196344851Z",
"build_snapshot" : false,
"lucene_version" : "9.6.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
五.节点node03安装kibana-8.8.2
- 下载安装公开签名秘钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
- 配置RPM repo
vim kibana.repo
#添加如下内容:
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
- 安装kibana-8.8.2
yum install kibana -y
- 配置kibana-8.8.2
cd /etc/kibana/
vim kibana.yml
#修改如下内容:
server.port: 5601
server.host: "0.0.0.0"
- 运行kibana-8.8.2
systemctl enable kibana //服务开机自启
systemctl start kibana //开启服务
systemctl status kibana //查看服务状态
在网页上输入http://172.11.0.88:5601/,刚开始会提示输入Enrollment token:
token获取:
安装elasticsearch其实有提示;
Generate an enrollment token for Kibana instances with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjguMiIsImFkciI6WyIxNzIuMTEuMC44ODo5MjAwIl0sImZnciI6IjJiYWYwNTg2NDQxOWJkMTlmZTA0OTZlNzI0NTdhYmYzMDU3ZmI2YjUwOTcxZTAyMjE3NzMzNmFjOGNjMWFjNmIiLCJrZXkiOiI3ZlBQUG9rQkx6WndScURsMHdLWDpQV285WHk0b1NqaUpLcHN4WWJib1h3In0=
拷贝token到网页,点击Configure Elastic ,然后会提示输入Verification required code。
获取code:
/usr/share/kibana/bin/kibana-verification-code
输入后就可以进入kibana页面了,就可以导入sample数据进行探索了:
配置nginx数据视图:
六.调试:
调试logstash
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-messages.conf