下载filebeat的rpm包安装filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.0-x86_64.rpm
安装
filebeat-6.3.0-x86_64.rpm
配置文件/etc/filebeat/filebeat.yml
写一个配置文件
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/messages
exclude_lines: ['^DBG','^$']
document_type: system-log-5611
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.file:
path: "/tmp"
name: "filebeat.txt"
默认不带type这里自定义type为document_type: system-log-5611
排除空行exclude_lines: ['^DBG','^$']
这里不写入到elasticsearch而是先写入到一个文件
启动
systemctl start filebeat
PS:在/tmp下面生成了文件filebeat但是没有txt(原因未知)
修改配置文件把输出改成redis
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/messages
tags: ["system-log-5611"]
exclude_lines: ['^DBG','^$']
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.redis:
hosts: ["192.168.56.11"]
db: "3"
port: "6379"
password: "123456"
key: "system-log-5611"
PS:tags才能生效 redis里面的key不能输出对应的key值(filebeat版本为6.3)
redis必须设置密码,否则启动filebeat报错,报错日志文件为/var/log/filebeat/filebeat
重启filebeat
systemctl restart filebeat
使用echo的方式往/var/log/messages插入几条数据然后使用客户端连接redis查看
配置使用logstash取出redis里面的数据
input{
redis {
host => "192.168.56.11"
port => "6379"
password => "123456"
db => "3"
data_type => "list"
key => "system-log-5611"
}
}
output{
if "system-log-5611" in [tags] {
elasticsearch {
hosts => ["192.168.56.11:9200"]
index => "system-log-5611-%{+YYYY.MM.dd}"
}
stdout{
codec => rubydebug
}
}
}
启动logstash输出
同时elasticsearch也收到了