logstash与kafka

1.编译logstash-kafka-master包

从github上将其签下: https://github.com/joekiller/logstash-kafka

[root@xzw logstash-kafka-master]# pwd
/root/logstash-kafka-master
[root@xzw logstash-kafka-master]# ls
build CHANGELOG extract_services.rb gembag.rb jruby-kafka-0.2.1-java.gem lib LICENSE.txt  logstash-kafka.gemspec Makefile pkg README.md spec vendor
[root@xzw logstash-kafka-master]# make tarball
这个make tarball会从网络上下载几个包, 其中有一个地址指向了s3.amazonaws.com, 这个地址貌似很不稳定, 可能要多试几次才会成功.

编译完成后, 会在build目录下生成一个logstash-1.4.2.tar.gz, 这个包并不是logstash官网的, 而是在刚才的编译中被添加了kafka的支持.

2. 解压build/logstash-1.4.2.tar.gz即可使用了.

创建logstash.conf配置文件. 这个文件名是任意的.

[root@xzw logstash-1.4.2]# vim logstash.conf /* 创建一个logstash.conf配置文件. 内容如下: */
	input {
               kafka {
                    zk_connect => "192.168.1.117:2181" /* zookeeper 地址. */
                    group_id => "logstash" /* 消费者组名. */
                    topic_id => "logstash" /* 消费的 topic 名称. */
               }
        }
       output {
                stdout {
                          codec => rubydebug /* 这里只是将日志吐出到了标准输出 . */
                }
       }

3. 启动logstash.

[root@xzw logstash-1.4.2]# bin/logstash -f logstash.conf Using milestone 1 input plugin 'kafka'. This plugin should work, but would benefit from use by folks like you. Please let us know 
if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see 
http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
{
    "message" => "infox",
        "@version" => "1",
      "@timestamp" => "2014-11-18T03:04:58.160Z"

}

4. 在kafka上生产一个消息.

[root@xzw kafka_2.10-0.8.1.1]# bin/kafka-topics.sh --create --zookeeper 192.168.1.117:2181 --replication-factor 1 --partitions 1 --topic logstash
Created topic "logstash".    /* 创建一个kafka的topic, 命名为: logstash. */
[root@xzw kafka_2.10-0.8.1.1]# bin/kafka-console-producer.sh --broker-list 192.168.1.117:9092 --topic logstash
infox /* 在topic logstash上生产一个消息, "infox", 即可看到上面logstrash的输出了. */

5. 上面的logstash扮演的是一个kafka的消费者. 下面的配置则是将它置成生产者.

将input段置为mongodb的日志输出文件.

input{
	file{
      		path=> "/usr/local/mongodb-linux-x86_64-2.6.5/gmlog/log/*.log"
       		type=> "string"
	}
}

将output段置为:

output {
	kafka{
		broker_list => "192.168.1.117:9092"
		topic_id => "logstash"
	}
}

重启logstash.

开启kafka消费者.
bin/kafka-console-consumer.sh --zookeeper 192.168.1.117:2181 --topic logstash –from-beginning

尝试启动一次mongodb, 即可看到消费的消息了.

mongod --dbpath=/usr/local/mongodb-linux-x86_64-2.6.5/gmlog/dat –logpath=/usr/local/mongodb-linux-x86_64-2.6.5/gmlog/log/log.log –logappend