方法1:一实例多topic:
https://discuss.elastic.co/t/filebeat-5-0-output-to-kafka-multiple-topics/67934
The document_type per prospector becomes the event field type. That's why the filter won't match. Instead of conditionals consider using the format string like:
filebeat.prospectors: - ... document_type: myapp_applog - ... document_type: myapp_applog_stats - ... document_type: myapp_elblog output.kafka: topic: '%{[type]}' # use document_type to set topic
btw. the topic field in conditionals also supports format strings.
cat filebeat.yml
output.kafka: enabled: true hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"] topic: '%{[type]}' filebeat.prospectors: #------------------------------ Log prospector -------------------------------- - paths: - /data/logs/financial-manager/server0.log fields: app_id: financial-manager multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.' multiline.negate: true multiline.match: after document_type: log-log4j - paths: - /data/logs/user-service/*.log fields: app_id: user-service multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.' multiline.negate: true multiline.match: after document_type: log-iis - paths: - /data/logs/financial-manager/access_log.log fields: app_id: financial-manager document_type: log-undertow
方法二:多filebeat实例,每个filebeat实例设置一个topic
运行多个filebeat实例,每个实例对应一个配置文件
cat filebeat.yml output.kafka: enabled: true hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"] topic: log-log4j filebeat.prospectors: #------------------------------ Log prospector -------------------------------- - paths: - /data/logs/subscribe-consumer/server0.log fields: app_id: subscribe-consumer multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.' multiline.negate: true multiline.match: after
cat filebeat2.yml output.kafka: enabled: true hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"] topic: log-tomcat filebeat.prospectors: #------------------------------ Log prospector -------------------------------- - paths: - /data/logs/subscribe-consumer/server0.log fields: app_id: subscribe-consumer multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.' multiline.negate: true multiline.match: after