kafka的介绍以及使用安装请查看博文http://blog.csdn.net/hmsiwtv/article/details/46960053
讲的很详细,基本一看就会,这边主要介绍一下nginx记录日志kafka扫描
一 nginx主要配置
log_format ht-video '$request|$msec|$http_x_forwarded_for|$http_user_agent|$request_body';
server{
server_name 192.168.50.42;
location / {
root html;
index index.html;
}
location /ht{
#root html;
#index index.html index.htm;
if ( $request ~ "GET" ) {
#access_log logs/test.log tes;
access_log /htlog/test.log test;
}
client_max_body_size 8000M;
#error_page 405 =200 $1;
}
}
然后可以发送get请求 http://192.168.50.42:8080/ht/p?data=11211111111
注意在nginx的html目录下边需要有ht目录 然后目录下边需要建一个p文件,里面随便输点内容,是为了能请求到对应的文件
二 修改Kafka文件
vi /usr/local/kafka_2.11-0.9.0.1/config/server.properties
这个文件里要改下面5点:
broker.id=0
port=9092
host.name=192.168.50.42
log.dirs=/log
zookeeper.connect=192.168.50.42:2181
三、启动zookeeper和kafka
/usr/local/kafka_2.11-0.9.0.1/bin/zookeeper-server-start.sh -daemon /usr/local/kafka_2.11-0.9.0.1/config/zookeeper.properties
/usr/local/kafka_2.11-0.9.0.1/bin/kafka-server-start.sh -daemon /usr/local/kafka_2.11-0.9.0.1/config/server.properties
四,创建topic
输入命令创建一个叫test的topic:bin/kafka-console-producer.sh --broker-list 192.168.50.42:9092 --topic test
消费topic的命令:bin/kafka-console-consumer.sh --zookeeper 192.168.50.42:2181 --topic test --from-beginning
五,扫描Nginx日志发送到kafka
tail -n 0 -f /htlog/ht-video.log | /usr/local/kafka_2.10-0.9.0.1/bin/kafka-console-producer.sh --broker-list 192.168.50.42:9092 --topic test
六,java客户端消费日志
public static List<String> getKafkaData(){
List<String> list=new ArrayList<String>();
Properties props = new Properties();
props.put("zookeeper.connect", "192.168.50.42:2181");
props.put("group.id", "group");
props.put("zookeeper.session.timeout.ms", "40000");
props.put("zookeeper.sync.time.ms", "200");
props.put("auto.commit.interval.ms", "1000");
ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put("test", new Integer(1));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
KafkaStream<byte[], byte[]> stream = consumerMap.get(Kafkaproperties.topic).get(0);
ConsumerIterator<byte[], byte[]> it = stream.iterator();
String message="GGGG";
while (it.hasNext()) {
message=new String(it.next().message());
System.out.println("receive:" + message);
list.add(message);
}
return list;
}