NSD ARCHITECTURE DAY04
1 案例1:安装Kibana
1.1 问题
本案例要求:
- 安装Kibana
- 配置启动服务查看5601端口是否正常
- 通过web页面访问Kibana
1.2 步骤
实现此案例需要按照如下步骤进行
步骤一:安装kibana
1)在另一台主机,配置ip为192.168.1.56,配置yum源,更改主机名
2)安装kibana
- [root@kibana ~]# yum -y install kibana
- [root@kibana ~]# rpm -qc kibana
- /opt/kibana/config/kibana.yml
- [root@kibana ~]# vim /opt/kibana/config/kibana.yml
- 2 server.port: 5601
- //若把端口改为80,可以成功启动kibana,但ss时没有端口,没有监听80端口,服务里面写死了,不能用80端口,只能是5601这个端口
- 5 server.host: "0.0.0.0" //服务器监听地址
- 15 elasticsearch.url: http://192.168.1.51:9200
- //声明地址,从哪里查,集群里面随便选一个
- 23 kibana.index: ".kibana" //kibana自己创建的索引
- 26 kibana.defaultAppId: "discover" //打开kibana页面时,默认打开的页面discover
- 53 elasticsearch.pingTimeout: 1500 //ping检测超时时间
- 57 elasticsearch.requestTimeout: 30000 //请求超时
- 64 elasticsearch.startupTimeout: 5000 //启动超时
- [root@kibana ~]# systemctl restart kibana
- [root@kibana ~]# systemctl enable kibana
- Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
- [root@kibana ~]# ss -antup | grep 5601 //查看监听端口
3)浏览器访问kibana,如图-1所示:
- [student@room9pc01 ~]$ firefox 192.168.1.56:5601
图-1
4)点击Status,查看是否安装成功,全部是绿色的对钩,说明安装成功,如图-2所示:
图-2
5)用head插件访问会有.kibana的索引信息,如图-3所示:
- [student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head
图-3
步骤二:使用kibana查看数据是否导入成功
1)数据导入以后查看logs是否导入成功,如图-4所示:
- [student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head
图-4
2)kibana导入数据,如图-5所示:
- [student@room9pc01 ~]$ firefox http://192.168.1.56:5601
图-5
3)成功创建会有logstash-*,如图-6所示:
图-6
4)导入成功之后选择Discover,如图-7所示:
图-7
注意: 这里没有数据的原因是导入日志的时间段不对,默认配置是最近15分钟,在这可以修改一下时间来显示
5)kibana修改时间,选择Lsat 15 miuntes,如图-8所示:
图-8
6)选择Absolute,如图-9所示:
图-9
7)选择时间2015-5-15到2015-5-22,如图-10所示:
图-10
8)查看结果,如图-11所示:
图-11
9)除了柱状图,Kibana还支持很多种展示方式 ,如图-12所示:
图-12
10)做一个饼图,选择Pie chart,如图-13所示:
图-14
11)选择from a new serach,如图-11所示:
图-15
12)选择Spilt Slices,如图-16所示:
图-16
13)选择Trems,Memary(也可以选择其他的,这个不固定),如图-17所示:
图-17
14)结果,如图-18所示:
图-18
15)保存后可以在Dashboard查看,如图-19所示:
图-19
2 案例2:综合练习
2.1 问题
本案例要求:
- 安装配置 beats插件
- 安装一台Apache服务并配置
- 使用filebeat收集Apache服务器的日志
- 使用grok处理filebeat发送过来的日志
- 存入elasticsearch
- 使用 kibana 做图形展示
2.2 步骤
实现此案例需要按照如下步骤进行。
步骤一:安装logstash
1)配置主机名,ip和yum源,配置/etc/hosts(请把es1-es5、kibana主机配置和logstash一样的/etc/hosts)
- [root@logstash ~]# vim /etc/hosts
- 192.168.1.51 es1
- 192.168.1.52 es2
- 192.168.1.53 es3
- 192.168.1.54 es4
- 192.168.1.55 es5
- 192.168.1.56 kibana
- 192.168.1.57 logstash
2)安装java-1.8.0-openjdk和logstash
- [root@logstash ~]# yum -y install java-1.8.0-openjdk
- [root@logstash ~]# yum -y install logstash
- [root@logstash ~]# java -version
- openjdk version "1.8.0_161"
- OpenJDK Runtime Environment (build 1.8.0_161-b14)
- OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
- [root@logstash ~]# touch /etc/logstash/logstash.conf
- [root@logstash ~]# /opt/logstash/bin/logstash --version
- logstash 2.3.4
- [root@logstash ~]# /opt/logstash/bin/logstash-plugin list //查看插件
- ...
- logstash-input-stdin //标准输入插件
- logstash-output-stdout //标准输出插件
- ...
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{
- }
- }
- filter{
- }
- output{
- stdout{
- }
- }
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- //启动并测试
- Settings: Default pipeline workers: 2
- Pipeline main started
- aa //logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
- 2018-09-15T06:19:28.724Z logstash aa
备注:若不会写配置文件可以找帮助,插件文档的位置:
https://github.com/logstash-plugins
3)codec类插件
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{
- codec => "json" //输入设置为编码json
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug" //输出设置为rubydebug
- }
- }
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {"a":1}
- {
- "a" => 1,
- "@version" => "1",
- "@timestamp" => "2019-03-12T03:25:58.778Z",
- "host" => "logstash"
- }
4)file模块插件
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- file {
- path => [ "/tmp/a.log", "/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb" //记录读取文件的位置
- start_position => "beginning" //配置第一次读取文件从什么地方开始
- type => "testlog" //类型名称
- }
- }
- filter{
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
- [root@logstash ~]# touch /tmp/a.log
- [root@logstash ~]# touch /tmp/b.log
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
另开一个终端:写入数据
- [root@logstash ~]# echo a1 > /tmp/a.log
- [root@logstash ~]# echo b1 > /var/tmp/b.log
之前终端查看:
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "a1",
- "@version" => "1",
- "@timestamp" => "2019-03-12T03:40:24.111Z",
- "path" => "/tmp/a.log",
- "host" => "logstash",
- "type" => "testlog"
- }
- {
- "message" => "b1",
- "@version" => "1",
- "@timestamp" => "2019-03-12T03:40:49.167Z",
- "path" => "/tmp/b.log",
- "host" => "logstash",
- "type" => "testlog"
- }
7)filter grok插件
grok插件:
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配,正则表达式需要根据具体数据结构编写
虽然编写困难,但适用性极广
解析Apache的日志
- [root@es5 ~]# yum -y install httpd
- [root@es5 ~]# systemctl restart httpd
浏览器访问网页,在/var/log/httpd/access_log有日志出现
- [root@es5 ~]# cat /var/log/httpd/access_log
- 192.168.1.254 - - [12/Mar/2019:11:51:31 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- file {
- path => [ "/tmp/a.log", "/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb"
- start_position => "beginning"
- type => "testlog"
- }
- }
- filter{
- grok{
- match => [ "message", "(?<key>reg)" ]
- }
- }
- output{
- stdout{
- codec => "rubydebug"
- }
- }
复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log
- [root@logstash ~]# vim /tmp/a.log
- 192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- //出现message的日志,但是没有解析是什么意思
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => ".168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
- "@version" => "1",
- "@timestamp" => "2018-09-15T10:26:51.335Z",
- "path" => "/tmp/a.log",
- "host" => "logstash",
- "type" => "testlog",
- "tags" => [
- [0] "_grokparsefailure"
- ]
- }
若要解决没有解析的问题,同样的方法把日志复制到/tmp/a.log,logstash.conf配置文件里面修改grok
查找正则宏路径
- [root@logstash ~]# cd /opt/logstash/vendor/bundle/ \
- jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
- [root@logstash ~]# vim grok-patterns //查找COMBINEDAPACHELOG
- COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- ...
- filter{
- grok{
- match => ["message", "%{COMBINEDAPACHELOG}"]
- }
- }
- ...
解析出的结果
- [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
- {
- "message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
- "@version" => "1",
- "@timestamp" => "2018-09-15T10:55:57.743Z",
- "path" => "/tmp/a.log",
- ZZ "host" => "logstash",
- "type" => "testlog",
- "clientip" => "192.168.1.254",
- "ident" => "-",
- "auth" => "-",
- "timestamp" => "15/Sep/2019:18:25:46 +0800",
- "verb" => "GET",
- "request" => "/noindex/css/open-sans.css",
- "httpversion" => "1.1",
- "response" => "200",
- "bytes" => "5081",
- "referrer" => "\"http://192.168.1.65/\"",
- "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
- }
- ...
步骤二: 安装Apache服务,用filebeat收集Apache服务器的日志,并存入elasticsearch
1)在之前安装了Apache的主机上面安装filebeat
- [root@se5 ~]# yum -y install filebeat
- [root@se5 ~]# vim/etc/filebeat/filebeat.yml
- paths:
- - /var/log/httpd/access_log //日志的路径,短横线加空格代表yml格式
- document_type: apachelog //文档类型
- elasticsearch: //加上注释
- hosts: ["localhost:9200"] //加上注释
- logstash: //去掉注释
- hosts: ["192.168.1.57:5044"] //去掉注释,logstash那台主机的ip
- [root@se5 ~]# systemctl start filebeat
- [root@logstash ~]# vim /etc/logstash/logstash.conf
- input{
- stdin{ codec => "json" }
- beats{
- port => 5044
- }
- file {
- path => [ "/tmp/a.log", "/tmp/b.log" ]
- sincedb_path => "/var/lib/logstash/sincedb"
- start_position => "beginning"
- type => "testlog"
- }
- filter{
- if [type] == "apachelog"{
- grok{
- match => ["message", "%{COMBINEDAPACHELOG}"]
- }}
- }
- output{
- stdout{ codec => "rubydebug" }
- if [type] == "filelog"{
- elasticsearch {
- hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
- index => "filelog"
- flush_size => 2000
- idle_flush_time => 10
- }}
- }
- [root@logstash logstash]# /opt/logstash/bin/logstash \
- -f /etc/logstash/logstash.conf
打开另一终端查看5044是否成功启动
- [root@logstash ~]# netstat -antup | grep 5044
- tcp6 0 0 :::5044 :::* LISTEN 23776/java
- [root@se5 ~]# firefox 192.168.1.55 //ip为安装filebeat的那台机器
回到原来的终端,有数据
2)修改logstash.conf文件
- [root@logstash logstash]# vim logstash.conf
- ...
- output{
- stdout{ codec => "rubydebug" }
- if [type] == "apachelog"{
- elasticsearch {
- hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
- index => "apachelog"
- flush_size => 2000
- idle_flush_time => 10
- }}
- }
浏览器访问Elasticsearch,有apachelog,如图-20所示:
- [student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head
图-20