1 管道配置
在kibana上面通过Dev Tools 定义管道
举例如下:
PUT _ingest/pipeline/pipeline-nginx-access
{
"description" : "nginx access log",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{IP:clientip} - - \\[%{DATA:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<url>[^\"]+)\" \"(?<browser>[^\"]+)\""]
}
},{
"remove": {
"field": "message"
}
}
]
}
出现如下界面表示配置成功
查看和删除定义管道
GET _ingest/pipeline/pipeline-nginx-access
DELETE _ingest/pipeline/pipeline-nginx-access
2 配置filebeat.yml
* 配置nginx日志路径
* 配置索引和管道名称及es各项字段
* 配置kibana字段
配置代码如下
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /usr/local/nginx/logs/access.log
fields:
type : "access-log"
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
indices:
- index : "filebeat-nginx-access-%{+yyyy.MM.dd}"
when.equals:
fields.type : "access-log"
pipelines:
- pipeline : "pipeline-nginx-access"
when.equals:
fields.type : "access-log"
username : "elastic"
password : "changeme"
setup.kibana:
host: "localhost:5601"
username : "elastic"
password : "elastic"
* 启动filebeat
./filebeat -e
3 在kibana查看结果
错误总结:
错误1 :
该正则在grok debugger可以匹配到数据,但在定义管道时候报错,暂未解决
PUT _ingest/pipeline/pipeline-nginx-access
{
"description" : "nginx access log",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{IP:clientip} - - \\[%{DATA:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<http_referer>\S+)\" \"(?<http_user_agent>(\S+\s+)*\S+)\"" ]
}
},{
"remove": {
"field": "message"
}
}
]
}
错误2:
在启动filebeat出现如下错误
通过 ./filebeat -e -d "*" 进入debug模式,查看到报错如下:
通过报错查看是正则匹配没有匹配到数据,修改定义管道解决该问题。
参考资料:
filebeat使用ingest节点解析日志并导入elasticsearch
grok debugger解析nginx日志
https://www.cnblogs.com/lc226/p/11228215.html
logstash grok 内置正则
https://blog.csdn.net/crmcaedbpd247410/article/details/100496804
filebeat 配置说明
https://www.jianshu.com/p/0a5acf831409
fieebeat官网
https://www.elastic.co/guide/en/beats/filebeat/current/index.html