ELK之LogStash读取JSON日志分类型建立索引

为什么80%的码农都做不了架构师？>>>   

测试目的是，用ELK处理在业务中用户定义的json log日志,则试PHP脚本如下:

<?php
for( $i=0;$i<100;$i++)
{
        $reg = array(
                'method' => 'login',
                'user_id' => rand(1000,3000),
                'user_name' => "name_".rand(1,3000 ),
                'level' => 1,
                'register_time' => time(),
        );
        $str = json_encode( $reg );
        file_put_contents( "testlog" , $str."\n" , FILE_APPEND );
        $reg = array(
                'method' => 'register',
                'user_id' => rand(1000,3000),
                'user_name' => "name_".rand(1,3000 ),
                'level' => rand(1,30),
                'login_time' => time(),
        );
        $str = json_encode( $reg );
        file_put_contents( "testlog" , $str."\n" , FILE_APPEND );
}

循环生成注册log和登录log保存到testlog文件中,结果如下:

{"method":"register","user_id":2933,"user_name":"name_91","level":27,"login_time":1470179550}
{"method":"login","user_id":1247,"user_name":"name_979","level":1,"register_time":1470179550}
{"method":"register","user_id":2896,"user_name":"name_1972","level":17,"login_time":1470179550}
{"method":"login","user_id":2411,"user_name":"name_2719","level":1,"register_time":1470179550}
{"method":"register","user_id":1588,"user_name":"name_1484","level":4,"login_time":1470179550}
{"method":"login","user_id":2507,"user_name":"name_1190","level":1,"register_time":1470179550}
{"method":"register","user_id":2382,"user_name":"name_234","level":21,"login_time":1470179550}
{"method":"login","user_id":1208,"user_name":"name_443","level":1,"register_time":1470179550}
{"method":"register","user_id":1331,"user_name":"name_1297","level":3,"login_time":1470179550}
{"method":"login","user_id":2809,"user_name":"name_743","level":1,"register_time":1470179550}

 

logstash目录下建立配置文件

vim config/json.conf

input {
    file {
        path => "/home/bona/logstash-2.3.4/testlog"
        start_position => "beginning"
        codec => "json"
    }
}


output {
   elasticsearch {
        hosts => ["192.168.68.135:9200"]
        index => "data_%{method}"
   }
}

重点是index中，%{method} 来匹配log中的method字段.

以上log就会分别建立data_login   data_register两个索引, 要注意的是索引名称必须全部小写

ES中已经成功以method建立了索引

elasticsearch-sql查询

 

参考资料:

http://udn.yyuap.com/doc/logstash-best-practice-cn/output/elasticsearch.html

https://github.com/NLPchina/elasticsearch-sql

转载于:https://my.oschina.net/appnet/blog/726275