flume-ng收集windows日志笔记

最新推荐文章于 2024-07-23 08:03:31 发布

我的天空永远蔚蓝

最新推荐文章于 2024-07-23 08:03:31 发布

阅读量6.6k

点赞数 2

分类专栏： Flume-NG

本文链接：https://blog.csdn.net/lilizhibing/article/details/48240081

版权

本文介绍了如何使用flume-ng收集Windows系统的日志，并通过SPILLABLEMEMORY通道和spoolDirTailFile源，将日志实时发送到RabbitMQ。配置包括下载必要的jar包，设置flume-ng配置文件，如设置通道、源和接收器的详细参数，以及在Windows环境下启动flume-ng的bat脚本。此外，文章还分享了关于flume源、通道和接收器的使用心得。

摘要由CSDN通过智能技术生成

flume-ng收集windows下的日志并且使用SPILLABLEMEMORY 方式，日志实时收集使用spoolDirTailFile并且写入到rabbitmq

1.下载spoolDirTailFile 用到的jar 地址https://github.com/ningg/flume-ng-extends-source

2.下载rabbitmq sink用到的 jar 地址https://github.com/aweber/rabbitmq-flume-plugin

3.配置 flume-ng-config

agent.channels = c1
agent.sources = spoolDirTailFile

agent.channels = c1
agent.channels.c1.type = SPILLABLEMEMORY
agent.channels.c1.memoryCapacity = 10000
agent.channels.c1.overflowCapacity = 1000000
agent.channels.c1.byteCapacity = 800000
agent.channels.c1.checkpointDir =C:\\log\\checkpoint
agent.channels.c1.dataDirs = C:\\log\\data

agent.sinks.k1.channel = c1
agent.sinks.k1.type =com.aweber.flume.sink.rabbitmq.RabbitMQSink
agent.sinks.k1.host = 主机地址
agent.sinks.k1.port = 5672
agent.sinks.k1.virtual-host =mq-host
agent.sinks.k1.username = 用户名
agent.sinks.k1.password = 密码
agent.sinks.k1.exchange = mq-队列名称
agent.sinks.k1.routing-key = mq-key
agent.sinks.k1.publisher-confirms = true

agent.sinks=k1

# Spooling dir and tail file Source
agent.sources.spoolDirTailFile.type = com.github.ningg.flume.source.SpoolDirectoryTailFileSource
# on WIN plantform spoolDir should be format like: E:/program files/spoolDir
# Note: the value of spoolDir MUST NOT be surrounded by quotation marks.
agent.sources.spoolDirTailFile.spoolDir = C:\\log\\file
agent.sources.spoolDirTailFile.fileSuffix = .COMPLETED
agent.sources.spoolDirTailFile.deletePolicy = never
agent.sources.spoolDirTailFile.ignorePattern = ^$
agent.sources.spoolDirTailFile.targetPattern = .*(\\d){10}.*
agent.sources.spoolDirTailFile.targetFilename = yyyyMMddhh
agent.sources.spoolDirTailFile.trackerDir = .flumespooltail
agent.sources.spoolDirTailFile.consumeOrder = oldest
agent.sources.spoolDirTailFile.batchSize = 100
agent.sources.spoolDirTailFile.inputCharset = UTF-8
agent.sources.spoolDirTailFile.decodeErrorPolicy = REPLACE
agent.sources.spoolDirTailFile.deserializer = LINE
agent.sources.spoolDirTailFile.channels= c1
agent.sources.spoolDirTailFile.fileHeader= true

以下是配置多个agent 供大家参考
agent2.channels = c2
agent