logstash tcp multihost output(多目标主机输出,保证TCP输出链路的稳定性)

在清洗日志时，有一个应用场景，就是TCP输出时，需要在一个主机挂了的情况下，自已切换到下一个可用入口，而原tcp output仅支持单个目标主机设定。故本人在原tcp的基础上，开发出tcp_multihost输出插件，来满足此场景。

插件在一开始的时候会随机选择一个链路，而在链路出错连续超过３(默认)次后会尝试数组中下一个主机

github: http://github.com/xiaohelong2005

Logstash版本:1.4.2

文件位置:

# encoding: utf-8
require "logstash/outputs/base"
require "logstash/namespace"
require "thread"
#@auhor:xiaohelong 
#@date:2014-10-24
#@email:xiaohelong2005@gmail.com
# Write events over a TCP socket.
#Auto select the host from iplist to tolerate the link
# Each event json is separated by a newline.
#
# Can  connect to a server,
class LogStash::Outputs::Tcp_Multihost < LogStash::Outputs::Base
  config_name "tcp_multihost"
  milestone  0
  default :codec, "json"
  # the address to connect to.
  #hosts config example
  config :hosts, :validate => :array, :required => true,:default=>{}
    config :reconnect_times, :validate => :number,:default=>3
  # When connect failed,retry interval in sec.
  config :reconnect_interval, :validate => :number, :default => 10
 #last available host
 @hosts_size=0
 #last available host ip
@host="0.0.0.0"
#last available port
@port=9200
#retry action count,if retry_count<=0,we need to update the host and port , also include retry_count itself
@retry_count=0
#get the desgined index data
@initloc=0

  public
  def register
    require "stud/try"
  #here we use the recorded host,if recorded host is not available, we need update it
  @retry_count=@reconnect_times
   @hosts_size=@hosts.length
    @logger.info("length:#@hosts_size; hosts:#@hosts")
   @initloc=Random.rand(@hosts_size)#generate 0-(hosts_size-1) int
    @logger.info("initloc:#@initloc")
    icount=0;
      @hosts.each do |hosthash|               
            @logger.info("hosthash info",hosthash)
       end#do
        @host=@hosts[@initloc].keys[0]        
       @port=@hosts[@initloc][@host]
       
      client_socket = nil
      @codec.on_event do |payload|
        begin
            @retry_count=@reconnect_times#here we need to init retry mark
          client_socket = connect unless client_socket
          r,w,e = IO.select([client_socket], [client_socket], [client_socket], nil)          
          # don't expect any reads, but a readable socket might
          # mean the remote end closed, so read it and throw it away.
          # we'll get an EOFError if it happens.
          client_socket.sysread(16384) if r.any?
          # Now send the payload
          client_socket.syswrite(payload) if w.any?
           @logger.info("tcp output info:", :host => @host, :port => @port,
                       :exception => e, :backtrace => e.backtrace)
        rescue => e
          @logger.warn("tcp output exception", :host => @host, :port => @port,
                       :exception => e, :backtrace => e.backtrace)
          client_socket.close rescue nil
          client_socket = nil
          @retry_count-=1
            @logger.info("retry_count:#@retry_count")
          if    @retry_count<=0
                                        @initloc+=1
                              @initloc=@initloc%@hosts_size #update  init location
                                           @host=@hosts[@initloc].keys[0]        
                                            @port=@hosts[@initloc][@host]
                   @retry_count=@reconnect_times #update retry_count
                      @logger.info("retry_count <=0,initloc:#@initloc,retry_count=#@retry_count:", :host => @host, :port => @port, :exception => e, :backtrace => e.backtrace)
          end
          sleep @reconnect_interval
          retry
        end
      end
  end # def register

  private
  def connect
    Stud::try do
      return TCPSocket.new(@host,@port)
    end
  end # def connect
  public
  def receive(event)
    return unless output?(event)
    @codec.encode(event)
  end # def receive
end # class LogStash::Outputs::Tcp_multihost

配置说明(我放在LOGSTASH_HOME/config)：

output{

tcp_multihost{

   hosts=>[
            {"127.0.0.1"=>"9202"},
       {"localhost"=>"9201"},
   {"127.0.0.1"=>"9203"},
            {"127.0.0.1"=>"9204"}
         ] #主机列表
    workers =>16　#线程,默认1

reconnect_times=>3 # 默认3次, 尝试多少次数后切换

reconnect_interval=>3 #默认10秒，失败重连间隔

}

}

调用执行:

 "LOGSTASH_HOME/bin/logstash" agent --debug -f "LOGSTASH_HOME/config/shipper.config"  --pluginpath "LOGSTASH_HOME"

NC接收端可以尝试:

nc -lkv 9201 之类的