我的logstash是从kafka中读取数据,取出来的数据格式为"zhangsan_1_25_student_15064573848";一个人的姓名_性别_年龄_职业_时间戳,我的需求是将每个字段分别存到elasticsearch中,所以首先需要对取出来的字符串拆分,以便模板能映射各个字段
filter{ mutate{ split => ["message","_"] //使用split以下划线进行拆分 add_field => { //由于拆分完,只有值,没有字段名,所以还要使用add_field添加字段 "name" => "%{[message][0]}" //姓名 } add_field => { "gender" => "%{[message][1]}" //性别 } add_field => { "age" => "%{[message][2]}" //年龄 } add_field => { "job" => "%{[message][3]}" //职业 } add_field => { "time" => "%{[message][4]}" //时间戳 } convert => { "gender" => "integer"} //由于拆分完每个都是字符串,而对于age等字段我想存储为整型,所以要 convert => { "age" => "integer"} //对字段进行转换,这里虽然写的是integer,但是模板会映射为long类型 convert => { "time" => "integer"} //但是转换并没生效,后面会解释 remove_field => ["message"] } }
由于字段转换没生效,我开始思考是不是由于字段名是从add_field方法添加来的,所以才没生效;后来我再添加一个mutate就成功了,当时怎么转不过弯,又没规定一个filter里面只能写一个mutate。
filter{ mutate{ split=>["message","_"] add_field => { "name" => "%{[message][0]}" } add_field => { "gender" => "%{[message][1]}" } add_field => { "age" => "%{[message][2]}" } add_field => { "job" => "%{[message][3]}" } add_field => { "time" => "%{[message][4]}" } remove_field => ["message"] } mutate{ //要转换字段,请重新再添加一个mutate convert => { "gender" => "integer"} convert => { "age" => "integer"} convert => { "time" => "integer"} } }