[elk]elk学习笔记（2）
一、kibana和logstash学习笔记（1）
在简单学习了es的基本原理和简单用法之后: link.，现在我们通过学习elk的经典应用日志管理，实战应用复习一下elasticsearch并进一步学习kibana和logstash的用法。
什么是kibana、logstash、filebeat
- kibana
官方解释
Kibana gives shape to your data and is the extensible user interface.
其实就是提供了一种可视化的界面，让你存到es中的抽象文档数据，直观地展示出来。
- logstash
官方解释
Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem.
是有一套插件生态的数据收集管道。可以对数据进行业务需求的格式化，并导入es中。
- filebeat
官方解释
当您要面对成百上千、甚至成千上万的服务器、虚拟机和容器生成的日志时，请告别 SSH 吧。Filebeat 将为您提供一种轻量型方法，用于转发和汇总日志与文件，让简单的事情不再繁杂。
搭建流程
本文整理的日志搭建思路为通过filebeat监听日志文件（以nginx日志为例），通过logstash对数据进行简单的格式化，并灌进es中，通过kibana查询到所有的日志，并且做一些简单的分析功能。除了监听文件的方式之外，还有tcp等方式，个人认为监听文件的方式让应用服务与elk之间更加解耦，不会因为elk的停止导致应用服务的不可用。
1）kibana、logstash、filebeat、nginx的安装
略。单机版安装较为简单友好，在此不做赘述，网上随便一搜有一万种安装的方式。
2）nginx自定义日志
// nginx http配置
http {
***
log_format main '$remote_addr [$time_local] "$request" [$request_time] $http_host' "$http_referer";
access_log logs/access.log main;
***
}
nginx启动访问，日志格式如下：
127.0.0.1 [08/Sep/2019:11:10:02 +0800] "GET /docs/maven-jars.html HTTP/1.1" [0.009] localhost:8090 "http://localhost:8090/docs/extras.html"
后面logstash部分我们会对日志进行解析。
通过log_format自定义输出日志内容。可以从中挑一些比较重要的信息，如下表：
| 日志类型 | 配置项 |
|---|---|
| 访问ip | $remote_addr |
| 访问url | $http_referer |
| 访问时间 | $time_local |
| 访问的ip:port | $http_host |
| 请求request信息 | $request |
| user_agent | $http_user_agent |
| 请求时间 | $request_time |
| 响应时间 | $upstream_response_time |
| 状态码 | $status |
| content_type | $upstream_http_content_type |
| 发送大小 | $body_bytes_sent |
注意取不到的配置项，打印出来是短横杠 “-” 的形式
3）filebeat配置
filebeat.inputs:
- type: log
enabled: true # 务必改为true
paths:
- D:\softwares\nginx-1.14.2\logs\access.log # 日志文件路径
output.logstash:
hosts: ["127.0.0.1:5044"]
filebeat目录下 .\filebeat -e -c filebeat.yml 命令启动
2019-09-08T09:23:33.108+0800 INFO instance/beat.go:606 Home path: [D:\softwares\elk\filebeat-7.2.0-windows-x86_64] Config path: [D:\softwares\elk\filebeat-7.2.0-windows-x86_64] Data path: [D:\softwares\elk\filebeat-7.2.0-windows-x86_64\data] Logs path: [D:\softwares\elk\filebeat-7.2.0-windows-x86_64\logs]
2019-09-08T09:23:33.223+0800 INFO instance/beat.go:614 Beat ID: 8bf8d617-8079-4564-93f6-080833f63e92
2019-09-08T09:23:33.728+0800 INFO [beat] instance/beat.go:902 Beat info {"system_info": {"beat": {"path": {"config": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64", "data": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64\\data", "home": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64", "logs": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "8bf8d617-8079-4564-93f6-080833f63e92"}}}
2019-09-08T09:23:33.767+0800 INFO [beat] instance/beat.go:911 Build info {"system_info": {"build": {"commit": "9ba65d864ca37cd32c25b980dbb4020975288fc0", "libbeat": "7.2.0", "time": "2019-06-20T15:05:29.000Z", "version": "7.2.0"}}}
2019-09-08T09:23:33.769+0800 INFO [beat] instance/beat.go:914 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.12.4"}}}
2019-09-08T09:23:34.219+0800 INFO [beat] instance/beat.go:918 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-08-31T16:12:31.41+08:00","name":"desktop","ip":["fe80::c05:2b1b:b4d3:fd7e/64","169.254.253.126/16","fe80::a86b:6185:7b76:9ba1/64","169.254.155.161/16","fe80::9c32:c9e2:56fe:b86c/64","169.254.184.108/16","fe80::5835:6ead:55da:41ed/64","169.254.65.237/16","fe80::b052:a19d:84a8:4846/64","169.254.72.70/16","fe80::a435:3126:1e50:73bc/64","169.254.115.188/16","fe80::60c6:1872:8964:8671/64","192.168.31.209/24","fe80::94ad:e8c8:f953:9b59/64","172.17.211.1/16","192.168.137.1/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.295 (WinBuild.160101.0800)","mac":["80:fa:5b:2f:ac:5b","00:db:df:c4:24:e0","00:ff:2a:94:00:06","00:50:56:c0:00:01","00:50:56:c0:00:08","00:ff:16:8d:25:a8","00:db:df:c4:24:df","02:db:df:c4:24:df"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Home","version":"10.0","major":10,"minor":0,"patch":0,"build":"18362.295"},"timezone":"CST","timezone_offset_sec":28800,"id":"d977aa0b-c4ba-4f7c-b86f-15840f748ec2"}}}
2019-09-08T09:23:34.240+0800 INFO [beat] instance/beat.go:947 Process info {"system_info": {"process": {"cwd": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64", "exe": "D:\\softwares\\elk\\filebeat-7.2.0-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 3536, "ppid": 7676, "start_time": "2019-09-08T09:23:31.925+0800"}}}
2019-09-08T09:23:34.242+0800 INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.2.0
2019-09-08T09:23:34.292+0800 INFO [publisher] pipeline/module.go:97 Beat name: wangjiahe
2019-09-08T09:23:34.375+0800 INFO instance/beat.go:421 filebeat start running.
2019-09-08T09:23:34.368+0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-09-08T09:23:34.385+0800 INFO registrar/registrar.go:145 Loading registrar data from D:\softwares\elk\filebeat-7.2.0-windows-x86_64\data\registry\filebeat\data.json
2019-09-08T09:23:34.397+0800 INFO registrar/registrar.go:152 States Loaded from registrar: 1
2019-09-08T09:23:34.398+0800 WARN beater/filebeat.go:358 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-09-08T09:23:34.407+0800 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-09-08T09:23:34.430+0800 INFO log/input.go:148 Configured paths: [D:\softwares\nginx-1.14.2\logs\access.log]
2019-09-08T09:23:34.431+0800 INFO input/input.go:114 Starting input of type: log; ID: 440809478459449177
2019-09-08T09:23:34.432+0800 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-09-08T09:23:34.433+0800 INFO cfgfile/reload.go:172 Config reloader started
2019-09-08T09:23:34.498+0800 INFO cfgfile/reload.go:227 Loading of config files completed.
2019-09-08T09:23:34.531+0800 INFO log/harvester.go:253 Harvester started for file: D:\softwares\nginx-1.14.2\logs\access.log
2019-09-08T09:23:36.774+0800 INFO add_cloud_metadata/add_cloud_metadata.go:347 add_cloud_metadata: hosting provider type not detected.
2019-09-08T09:23:37.807+0800 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://127.0.0.1:5044))
2019-09-08T09:23:41.654+0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://127.0.0.1:5044)): dial tcp 127.0.0.1:5044: connectex: No connection could be made because the target machine actively refused it.
2019-09-08T09:23:41.655+0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://127.0.0.1:5044)) with 1 reconnect attempt(s)
可以看到启动成功了，来源数据文件也读取到了，但是由于我的logstash还没有启动，所以一直在尝试连接logstash的5044端口。
注意只能有一个output，filebeat默认是的output是es，由于咱们要用logstash将日志进行格式化，所以将output默认改为logstash。5044端口是后面logstash需要开启的beats端口。
4）logstash配置
input {
beats {
port => 5044 # 开启给beats用的5044端口
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" \[%{BASE16FLOAT:request_time}\] %{HOSTPORT:host} %{QS:referrer}" }
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
manage_template => false
index => "nginx_logs"
}
}
这里我们logstash的filter用grok表达是对进行格式化，已有的一些grok表达式可以参考github
日志与grok表达式的匹配，可以在官网推荐的网站，测试也可以在kibana中的dev_tools中的Grok Debugger中测试。
通过kibana自带的Grok工具测试我们的grok表达式我们可以解析到自己想要的数据。
日志格式
127.0.0.1 [08/Sep/2019:11:46:06 +0800] "GET /favicon.ico HTTP/1.1" [0.007] localhost:8090 "http://localhost:8090/docs/apr.html"
grok格式
%{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" \[%{BASE16FLOAT:request_time}\] %{HOSTPORT:host} %{QS:referrer}
- grok的简单用法
基本格式为 %{type:alias}
其中type可以在参考github，也可以自定义。alias是匹配成功后的别名信息。
5）使用kibana查看
-
可以在设置中看到我们刚刚生成的索引
-
在dev_tool中可以通过命令查询刚刚生成的索引，可以发现用grok表达式解析的字段已经存在了
-
创建索引模式，以查看所有日志
- 这里可以匹配到一个索引，如果有多个索引的话，kibana也可以集成，进行聚合查询统计等
- 这里可以匹配到一个索引，如果有多个索引的话，kibana也可以集成，进行聚合查询统计等
-
在Discover查看所有索引,在这里可以根据各种条件聚合查询已有的索引
避免了用tail -f xxxs.log查看日志不方便的尴尬局面。
-
用kibana自带的可视化、仪表盘等按自己的需求自定义查看
扫一扫
1148
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
