flume日志配置
背景:http接受参数,使用mongodb将数据插入数据哭
使用网上java开发的代码进行修改
https://github.com/leonlee/flume-ng-mongodb-sink
agent2.sources = source2
agent2.sinks = sink2
agent2.channels = channel2
agent2.sources.source2.type = http
agent2.sources.source2.bind = 192.168.1.118
agent2.sources.source2.port = 4444
agent2.sources.source2.channels = channel2
agent2.sinks.sink2.type = org.riderzen.flume.sink.MongoSink
agent2.sinks.sink2.host = 192.168.1.120
agent2.sinks.sink2.port = 27017
agent2.sinks.sink2.model = SINGLE
agent2.sinks.sink2.collection = events
agent2.sinks.sink2.batch = 100
agent2.sinks.sink2.channel = channel2
agent2.channels.channel2.type = memory
agent2.channels.channel2.capacity = 1000
agent2.channels.channel2.transactionCapacity = 100
出现2个问题
1:无法处理body里面传的json
2:中文乱码:
解决策略
1:修改代码
String bodys = new String(body,"utf-8");
logger.debug("WWWWWWWWWWWWWWW" + bodys);
String[] strArray = bodys.split(",");
String strAll = "";
DBObject math = new BasicDBObject();
for(int i = 0;i < strArray.length;i++){
String[] list = strArray[i].split(":");
strAll = strAll +list[0] + ":'" + list[1] + "'";
if(i < strArray.length-1){
strAll = strAll + ",";
}
math.put(list[0], list[1]);
}
eventJson = math;
2:unicode编码传数据
name:中文,age:2
linux命令行下测试代码
curl -H "Content-Type: application/json" -XPOST http://192.168.1.118:4444 -d '[{"body":"\u006e\u0061\u006d\u0065\u003a\u4e2d\u6587\u002c\u0061\u0067\u0065\u003a\u0032"}]'
php测试代码:
$str_name = "name:小明,age:12";
$body = json_encode($str_name);
$array = array(array("headers" => array("collection" => "log"),"body" => $str_name));
var_dump(json_encode($array));
$ch = curl_init('http://192.168.1.118:44448');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS,$array);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . 1)
);
$result = curl_exec($ch);
补充flume执行命令:
在bin录下执行./flume-ng agent -c /home/hadoop/apache-flume-1.6.0-bin/conf -f /home/hadoop/apache-flume-1.6.0-bin/conf/netcat.conf -n agent2 -Dflume.root.logger=DUBUG,console
如果启动报错可查看flume-env.sh 文件中如下配置export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141.x86_64/jre