Linux--kibana练习_linux安装了ik分词器kibana怎么测试-CSDN博客

本文链接：https://blog.csdn.net/m0_73705628/article/details/137403204

他日故里定逢春，你我山巅自相逢

## kibana安装

主机	ES启动用户	kibana启动用户
192.168.182.131	elsearch	超级用户
192.168.182.153	elsearch	未下载
192.168.182.154	elsearch	未下载

目的：实现搭建webUI界面，实现将数据展示在webUI界面上

kibana是Elastic Stack成员之一，设计用和Elasticsearch协作，可以使用kibana对Elasticsearch索引中的数据进行搜索、查看、交互操作。可以很方便的利用图表、表格及地图对数据进行多元化的分析和呈现。

前提：需要将ES集群中的主机启动

单个节点：将不同服务器的日志放到一台采集器上去采集

多台：给每个es服务器都运行一个logstash，分别采集

注：kibana只需要启动一个服务，只是一个UI界面，而ES做存储引擎，logstash作集群或单个节点

配置安装：

# 准备安装包放置的目录
mkdir -p /opt/kibana/
# 解压安装包
tar -xvf kibana-6.5.4-linux-x86_64.tar.gz -C /opt/kibana/
# 修改配置文件
cd /opt/kibana/kibana-6.5.4-linux-x86_64
vim config/kibana.yml
## 添加信息为：
### 对外暴露服务的地址
server.host: "192.168.182.131"
### 配置Elasticsearch
elasticsearch.url: "http://192.168.182.131:9200"
# 启动kibana
./bin/kibana
## 后台启动则使用nohup ./bin/kibana &
# 通过浏览器进行访问
http://192.168.182.153:5601/opt/kibana

kibana好处：

比mysql数据库好管理，字段可以随意添加或删除，且不需要关注数据类型，只作数据存储，只能插入没有关联关系的数据存储和查询，只插入日志和监控的数据

数据展示：

## kibana6.5.4版本汉化

①下载kibana中文汉化包

wget https://mirrors.yangxingzhen.com/kibana/Kibana_Hanization.tar.gz

②执行汉化

mkdir /opt/kibana_hanhua/

tar zxf Kibana_Hanization.tar.gz -C /opt/kibana_hanhua/

cd /opt/kibana_hanhua/Kibana_Hanization/old/

# /opt/kibana/kibana-6.5.4-linux-x86_64/ 为Kibana 安装目录

python main.py /opt/kibana/kibana-6.5.4-linux-x86_64

③重启kibana

# 查看端口

fuser -n tcp 5601
kill -9 端口

cd /opt/kibana/kibana-6.5.4-linux-x86_64/
前台启动 ./kibana
或后台启动：nohup ./bin/kibana &

## ELK综合实战--动态采集示例：

# logstash进行动态采集--写配置文件
cd /opt/logstash/logstash-6.5.4
vim config/nginx_collect_system.conf
## 内容如下：
input {
file {
path => "/var/log/messages"
type => "system_log"
start_position => "beginning"
}
file {
path => "/var/log/secure"
type => "secure_log"
start_position => "beginning"
}
}
output {
if [type] == "system_log" {
elasticsearch {
hosts => ["192.168.182.131:9200"]
index => "system-log-%{+YYYY.MM.dd}"
}
}
if [type] == "secure_log" {
elasticsearch {
hosts => ["192.168.182.131:9200"]
index => "secure-log-%{+YYYY.MM.dd}"
}
}
}
# 追加错误信息进入日志
echo "Apr 4 09:40:33 k8s-master systemd: Error elasticsearch.service faild." >> /var/log/messages
# 进行动态采集
./bin/logstash -f config/nginx_collect_system.conf

添加索引后查看信息：

## kibana实战--通过地图统计客户IP地址所在城市

①下载官网的地图软件，并将软件存放到logstash目录下，对压缩包文件进行解压

地图官方下载地址： https://dev.maxmind.com/geoip/geoip2/geolite2/　

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

tar xvf GeoLite2-City_20190723.tar.gz -C /opt/logstash/logstash-6.5.4/

# 需要引用的配置文件： GeoLite2-City.mmdb

②配置logstash主机配置文件，将日志传递到elasticsearch主机

1>修改Logstash的config目录下的配置文件，加入一个加载地址地图的模块

vim logstash-to-es.conf
## 内容如下--此处只是配置文件说明，若要使用，使用说明下的部分
input {
file {
path => [ "/var/log/nginx/access.log" ]
start_position => "beginning"
ignore_older => 0 
### 忽略早于24小时（默认值86400）的日志，设为0，即关闭该功能，以防止文件中的事件由于是早期的被logstash所忽略。
codec => json
}
}
filter {
date {
match => [ "timestamp" ,"dd/MMM/YYYY:HH:mm:ss Z" ] 
### 匹配到timestamp字段后，修改格式为dd/MMM/yyyy:HH:mm:ss Z
}
geoip { 
### 该过滤器从geoip中匹配ip字段，显示该ip的地理位置
source => "client" 
### ip来源字段，这里我们选择的是日志文件中的最后一个字段，如果你的是默认的nginx日志，选择第一个字段即可(注：这里写的字段是/opt/logstash/patterns/nginx 里面定义转换后的)
target => "geoip" 
### 指定插入的logstash字断目标存储为geoip
database => "/opt/logstash/logstash-
6.5.4/GeoLite2-City.mmdb" 
### 数据库的存放路径,修改你的指定的地址位置
add_field => [ "[geoip][coordinates]", "%{[geoip]
[longitude]}" ] 
### 增加的字段，坐标经度
add_field => [ "[geoip][coordinates]", "%{[geoip]
[latitude]}" ] 
### 增加的字段，坐标纬度
}
mutate { 
### 数据的修改、删除、类型转换
convert => [ "[geoip][coordinates]", "float" ] 
### 将坐标转为float类型
remove_field => "timestamp" 
### 移除timestamp的内容，因为数据已经过滤了一份，这里不必在用到该字段了。不然会相当于存两份
}
if "_geoip_lookup_failure" in [tags] { drop { } }
### 如果解析的地址是内网IP geoip解析将会失败，会生成_geoip_lookup_failure字段，这段话的意思是如果内网地址drop掉这个字段。
}
output {
elasticsearch {
hosts => ["192.168.182.131:9200"]
index => "logstash-nginx-accesslog-geoip-%{+YYYY.MM.dd}" 
### 为日志创建索引logstash-nginx-accesslog-geoip＊，这里也就是kibana那里添加索引时的名称
}
}

配置文件具体配置：

input {
file {
    path => [ "/var/log/nginx/access.log" ]
    start_position => "beginning"
    ignore_older => 0
    codec => json
}
}
filter {
date {
    match => [ "timestamp" ,"dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
    source => "client"
    target => "geoip"
    database => "/opt/logstash/logstash-6.5.4/GeoLite2-City.mmdb"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
    convert => [ "[geoip][coordinates]", "float" ]
    remove_field => "timestamp"
}
if "_geoip_lookup_failure" in [tags] { drop { } }
}
output {
elasticsearch {
    hosts => ["192.168.182.131:9200"]
    index => "logstash-nginx-accesslog-geoip-%{+YYYY.MM.dd}"
}
}

2>测试logstash配置文件

./bin/logstash -f config/logstash-to-es.conf -t

3>重启logstash服务

./bin/logstash -f config/logstash-to-es.conf

③在nginx服务器上进行测试

先编辑配置文件，加入新的日志格式，将之前的日志格式注释掉

vim /etc/nginx/nginx.conf
## 内容如下：
 log_format main '{"@timestamp":"$time_iso8601",'
                        '"@source":"$server_addr",'
                        '"hostname":"$hostname",'
                        '"ip":"$remote_addr",'
                        '"client":"$remote_addr",'
                        '"request_method":"$request_method",'
                        '"scheme":"$scheme",'
                        '"domain":"$server_name",'
                        '"referer":"$http_referer",'
                        '"request":"$request_uri",'
                        '"args":"$args",'
                        '"size":$body_bytes_sent,'
                        '"status": $status,'
                        '"responsetime":$request_time,'
                        '"upstreamtime":"$upstream_response_time",'
                        '"upstreamaddr":"$upstream_addr",'
                        '"http_user_agent":"$http_user_agent",'
                        '"https":"$https"'
                        '}';

1>在nginx服务器上导入部分log

因为是单机环境，日志里面没有公网 IP ，所以我们需要自己往 nginx 日志里输入公网IP

# 北京公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2024-04-04T09:20:56+08:00","@source":"222.28.0.112","hostname":"k8s-master","ip":"222.28.0.112","client":"222.28.0.112","request_method":"GET","scheme":"http","domain":"_","referer":"-","request":"/","args":"-","size":0,"status":304,"responsetime":0.000,"upstreamtime":"-","upstreamaddr":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36","https":""}' >>/var/log/nginx/access.log
# 海南公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2024-04-04T17:20:56+08:00","@source":"124.225.0.13","hostname":"k8s-master","ip":"124.225.0.13","client":"124.225.0.13","request_method":"GET","scheme":"http","domain":"_","referer":"-","request":"/","args":"-","size":0,"status":304,"responsetime":0.000,"upstreamtime":"-","upstreamaddr":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36","https":""}' >>/var/log/nginx/access.log
# 黑龙江公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2024-04-04T17:20:56+08:00","@source":"123.164.0.18","hostname":"k8s-master","ip":"123.164.0.18","client":"123.164.0.18","request_method":"GET","scheme":"http","domain":"_","referer":"-","request":"/","args":"-","size":0,"status":304,"responsetime":0.000,"upstreamtime":"-","upstreamaddr":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36","https":""}' >>/var/log/nginx/access.log
#吉林公网IP
[root@linux-node1 conf.d]# echo '{"@timestamp":"2024-04-04T09:20:56+08:00","@source":"124.234.0.12","hostname":"k8s-master","ip":"124.234.0.12","client":"124.225.0.13","request_method":"GET","scheme":"http","domain":"_","referer":"-","request":"/","args":"-","size":0,"status":304，"responsetime":0.000,"upstreamtime":"-","upstreamaddr":"-","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36","https":""}' >>/var/log/nginx/access.log