1. Web server 集群数据采集采用的架构
2. 在每台web server上启动一个flume agent ( Flume1.3.1 : http://flume.apache.org/download.html ),启动命令为:./bin/flume-ng agent --conf-file ./conf/flume.conf --name a1 -Dflume.root.logger=INFO,console
flume-conf文件如下:
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 41414
# Describe the sink
a1.sinks.k1.type = logger
# avro sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = consolidationServerHost
a1.sinks.k1.port = 41414
# Use a channel which buffers events in file
a1.channels = c1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
a1.channels.c1.dataDirs = /mnt/flume/data
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3. 日志合并服务器上的启动一个flume agent,用于合并来自于web server上的flume agent传送过来的日志。flume-conf文件如下:
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 41414
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = timestamp
#hdfs sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://hostname:9000/path/to/log/dir /%Y-%m-%d/%H
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.filePrefix = appName-
a1.sinks.k1.hdfs.rollInterval = 3600
# Use a channel which buffers events in file
a1.channels = c1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
a1.channels.c1.dataDirs = /mnt/flume/data
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
4. 每台web server上的log4j.properties配置文件中加入以下内容:
log4j.rootLogger=INFO,flume
log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname=localhost
log4j.appender.flume.Port=41414
5. 每台web server上的app的lib库里面添加以下来自于flume的jar包:
avro-1.7.2.jar
avro-ipc-1.7.2.jar
flume-ng-log4jappender-1.3.1.jar
flume-ng-sdk-1.3.1.jar
jackson-core-asl-1.9.3.jar
jackson-mapper-asl-1.9.3.jar
log4j-1.2.16.jar
netty-3.4.0.Final.jar
slf4j-api-1.6.1.jar
6. 测试用的程序如下:
package flume.log4j.test;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.Charset;
import org.apache.log4j.Logger;
public class LogTestApp {
public static void main(String[] args) throws IOException {
Logger logger = Logger.getLogger(LogTestApp.class);
BufferedReader in = new BufferedReader(
new InputStreamReader(System.in, Charset.forName("UTF-8")));
String line;
System.out.println("Initializing Flume log4j appender test.");
System.out.println("Each line entered will be sent to Flume.");
// send this line to Flume
logger.info("LogTestApp initialized");
while ((line = in.readLine()) != null) {
System.out.println("Sending to log4j: " + line);
logger.info(line);
}
}
}