我想将WEB服务器日志通过管道处理然后写入数据库
- $ cat match.d
- import std.regex;
- import std.stdio;
- import std.string;
- import std.array;
- void main()
- {
- auto r = regex(`^(\S+) (\S+) (\S+) \[(.+)\] "([^"]+)" ([0-9]{3}) ([0-9]+) "([^"]+)" "([^"]+)" "([^"]+)"`);
- foreach(line; stdin.byLine)
- {
- foreach(m; match(line, r)){
- auto c = m.captures;
- c.popFront();
- auto value = join(c, "\",\"");
- auto sql = format("insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value(\"%s\");", value );
- writeln(sql);
- }
- }
- }
编译
- $ dmd match.d
- 编译优化(减肥)
- $ strip match
测试
- $ cat access.log | ./match
- 输出类似
- insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value("192.168.2.76","-","-","19/Mar/2013:15:24:20 +0800","GET /favicon.ico HTTP/1.1","404","193","-","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","-");
用法
- $ cat access.log | match | mysql -hlocalhost -ulog -p123456 logging
高级用法, 实时处理日志
- 首先创建一个管道
- 寻该日志文件写入管道中
- cat 管道名 | match | mysql -hlocalhost -ulog -p123456 logging 这样就可以实现实时日志插入。
上面程序稍加修改即可实现Hbase, Hypertable 插入