之前的logstash都是单机模式，把几个组件都安装在一台机器上，这一次把每个组件都安装到独立的server上，下面是架构图。

logstash
日志通过syslog-ng传输到logstash的agent端，agent主要就是负责接收日志，然后把日志放到redis，redis在这里的作用就是一个队列，主要是日志的缓存，redis是以内存做缓存的，一段时间存到磁盘上。在redis另一端是logstash index，这端主要是从redis取出日志，然后进行filter和output，filter就是对日志进行切割，匹配，过滤，logstash index这里可以是集群的，都从redis拿就行。取出来之后output可以到elasticsearch，elasticsearch也可以是集群，专门做索引。在最前端就是kibana3，再js里面配置从elasticsearch的端口读取数据，呈现到前端。

logstash agent ip 198.15.145.213
logstash index ip 198.15.145.214
kibana3 elasticsearch ip 198.15.145.215

agent和index基础安装需要：logstash 
yum -y install unzip java-1.6.0*
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.2.2-flatjar.jar
mkdir -p /usr/local/bin/logstash/
mv logstash-1.2.2-flatjar.jar /usr/local/bin/logstash/
mkdir -p /etc/logstash/
cd /usr/local/bin/logstash/ && ln -s ./logstash-1.2.2-flatjar.jar ./logstash.jar

logstash agent conf文件配置 ip 198.15.145.213


vim /etc/logstash/agent.conf 
input {
 udp {
     type => "linux-syslog"
     port => 515
  }
}

output {
   redis {
        host => "198.15.145.214"
        type => "linux-syslog"
        data_type => "list"
        key => "logstash"
   }

启动logstash agent 
java -jar /usr/local/bin/logstash/logstash.jar agent -f /etc/logstash/agent.conf

logstash index redis安装 ip 198.15.145.214


安装需要：redis 参考  http://www.firefoxbug.com/?p=2191

logstash index conf文件配置 ip 198.15.145.214


vim /etc/logstash/index.conf 
input {
   redis {
        host => "127.0.0.1"
        type => "linux-syslog"
        data_type => "list"
        key => "logstash"
   }
}

filter {
grok {
        type => "linux-syslog"
        pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{QS:x_forword} %{QS:upstream_cache_status}&@&(%{HOST:domain}|-)"
}

#mutate {
#        remove => [ "message", "@version"]
#}
geoip {
   source => "source_ip"
   type => "linux-syslog"
   add_tag => [ "geoip" ]
}
}
output {
#  stdout {
#       debug => true
#       debug_format => json
#  }
   elasticsearch {
      host => "198.15.145.215"
      embedded => true
   }
}

启动logstash index 
java -jar /usr/local/bin/logstash/logstash.jar agent -f /etc/logstash/index.conf

kibana3 elasticsearch 安装ip 198.15.145.215


mkdir -p /opt/elasticsearch && cd /opt/elasticsearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.7.tar.gz
tar -zxvf elasticsearch-0.90.7.tar.gz
cd elasticsearch-0.90.7
bin/elasticsearch -f

kibana3安装查看 http://www.firefoxbug.com/?p=2128