GitHub上有唯一一个实现好了的插件 fluent-plugin-opentsdb
https://github.com/emurphy/fluent-plugin-opentsdb
经过一天的研究,没使用起来,或者说没有得到我们想要的数据,只有打算自己写一个插件了
官网也有如何自定义插件的方法:http://docs.fluentd.org/v0.12/articles/plugin-development#writing-buffered-output-plugins
由于使用的是 td-agent 所以插件放置路径改为:.../td-agent/plugin
config配置文件如下:
<source>
# in_tail 输入插件允许fluentd从文本文件的尾部读事件。它的行为类似于tail-f 命令。
@type tail
#read_from_head true
format json
path /test/logs/nginx-logs/test-*.log
pos_file /var/log/td-agent/nginx.log.pos
tag es.nginx
</source>
<match es.nginx>
@type opentsdb1
host 132.12.1.23
port 4242
</match>
使用tail 输入插件,日志文件改动以后,将新增加的数据反馈到自己实现的输出插件
将官网的输出插件拷贝过来,填充自己的内容
require 'fluent/output' #添加基础模块和自己需要用到的模块
module Fluent
class SomeOutput < BufferedOutput #自己写类继承输出模块,我是继承自 Output
# First, register the plugin. NAME is the name of this plugin
# and identifies the plugin in the configuration file.
Fluent::Plugin.register_output('NAME', self) # 写入自己的插件名
# config_param defines a parameter. You can refer a parameter via @path instance variable
# Without :default, a parameter is required.
config_param :path, :string 定义一些自己需要的输出插件参数
# This method is called before starting.
# 'conf' is a Hash that includes configuration parameters.
# If the configuration is invalid, raise Fluent::ConfigError.
def configure(conf) #启动前自动调用
super
# You can also refer raw parameter via conf[name].
@path = conf['path'] #将配置文件的参数获取出来
...
end
# This method is called when starting.
# Open sockets or files here.
def start #启动时调用,可写一些自己需要的实现在这里,笔者是这里写入测试与opentsdb的连接状态
super
...
end
# This method is called when shutting down.
# Shutdown the thread and close sockets or files here.
def shutdown #关闭是调用,笔者在这里写的关闭所有socket连接
super
...
end
# This method is called when an event reaches to Fluentd.
# Convert the event to a raw string.
def format(tag, time, record) #事件触发函数,将事件转换成原始字符串,笔者没有使用这个,用 emit()代替
[tag, time, record].to_json + "\n"
## Alternatively, use msgpack to serialize the object.
# [tag, time, record].to_msgpack
end
def emit(tag, es, chain)
es.each do |time,record|
取出事件的事件和记录
record 为hash数据,利用tail输入插件,这里的record记录了日志新增的内容
通过自己定义函数解析这里的内容,获取opentsdb需要的数据
笔者是用put 方法向 opentsdb写入数据
@socket = TCPSocket.new(@host, @port)
@socket.puts(message)
@socket.close
message 格式
message = ['put', name, time, value, tag].join(' ')
通过解析上面的 record 获取或组装成 message 里面的 name,time,value,tag
到这里,一条日志数据就用put方式写入opentsdb了,如果解析比较繁琐,也能自定义解析函数调用,使代码更清晰
end
# This method is called every flush interval. Write the buffer chunk
# to files or databases here.
# 'chunk' is a buffer chunk that includes multiple formatted
# events. You can use 'data = chunk.read' to get all events and
# 'chunk.open {|io| ... }' to get IO objects.
#
# NOTE! This method is called by internal thread, not Fluentd's main thread. So IO wait doesn't affect other plugins.
def write(chunk) #未使用
data = chunk.read
print data
end
## Optionally, you can use chunk.msgpack_each to deserialize objects.
#def write(chunk) #未使用
# chunk.msgpack_each {|(tag,time,record)|
# }
#end
end
end
笔者自己写好的一个输出到opentsdb的插件 :http://download.csdn.net/detail/qq_21398167/9858721