Log Monitor - 把分散在各台server上的log集中起来monitor

公司的系统很分散，monitor log很费劲。写了一个ruby程序，让所有的log集中在一个地方。试用以后，系统维护的工夫可以少很多，还可以在用户报告之前发现问题。效果不错。

主要原理就是用net/ssh登录各台server，tail -f那些log，捕捉sysout（channel.on_data），然后加一些辅助信息（比如[module]-[file]-orignal log）输出。

以前也用过chainsaw等工具，不方便的地方在于：只能monitor log4j；不能集成在一个output里。

这个ruby程序只有几十行，如果以后要加一些auto mail，垂直log等功能也很灵活。还有些说明与窍门都一一分享在code comments里。

### license: http://creativecommons.org/licenses/by-sa/2.5/deed.zh
### by: caoweiyuan#gmail.com
require 'net/ssh'

require 'yaml'
modules = YAML.load(File.open('modules.conf'))
### sample config
# engine-1:
#   username: admin
#   password: ******
#   logs:
#     /opt/apps/production/tomcat/logs/app.log:
#       name: [module-1]
#       greps:
#         error.log: 
#           - ERROR
#         info.log: 
#           - INFO

def do_grep(name, line, greps)
  log = "[#{name}] - #{line}"

  # default output to screen
  puts log

  # grep patterns to sepecific files
  for log_file, patterns in greps
    pattern = (patterns.class == Array) ? patterns.join("|") : patterns
    File.open('log/' + log_file, 'a') {|f| f.puts log} if line =~ /#{pattern}/
    # TODO: auto mail for errors
    # TODO: horizontal log for different products, and delete after each process over.
  end
end

def do_tail( session, log_file, log_conf )
  session.open_channel do |channel|
    channel.on_data do |ch, data|
      data.each_line do |line|
        do_grep log_conf['name'], line, log_conf['greps']
      end
    end
    channel.exec "tail -1f #{log_file}"
  end
end

for key, value in modules
  t = Thread.new(key, value) do |host, host_conf|
    Net::SSH.start( host, host_conf['username'], host_conf['password'] ) do |session|
      puts "#{host} logged in."
      for log_file, log_conf in host_conf['logs']
        puts "start watching #{log_file}"
        # open channels for multiple log files
        do_tail session, log_file, log_conf
      end # log
      session.loop # important: loop to keep reading channel outputs
    end # session
  end # thread
end # host

# TODO: implement a formal deamon
loop do
  # deamonize
  sleep 1
end

最后，还有一点要分享的，这种方式不支持windows openssh，如果log在windows机器上，要装cygwin。还有，vmware的windows装了cygwin也不行，sshd开不了，如果有高手知道原因和方案，请不吝赐教。