elk7.7.1【系列九】logstash filter先处理字符串，在json格式化

1、数据源

notice - - - {"kill_chain":"remote-control","dst_ip_geoloc":"34.7725,113.7266","sub_category":"APT_htran (apt );APT_others (apt );APT_others(apt )","src_ip_city":"Beijing","hostip":"10.1.0.37","reliability":10,"classtype":"threat-intelligence-alarm","increment":0,"dst_ip_country_code":"CN","event_source":"NDE","rid":"10180001","interface":"zc:99@2","uuid":"2ecf0bff-6eb1-42a7-8f0b-ff6eb1a2a786","dst_ip":"114.114.114.114","src_ip":"10.1.0.198","dst_service":"dns","hostname":"ATD","commid":"ECtmCxaV8nDaCsMKjHknBj41QEo=","sensor_ip":"10.1.0.37","timestamp":"1550799035951","severity":"5","dns":{"qtype_name":"A","query":"bbs.india-videoer.com"},"dst_ip_country":"China","message":"DNS resolves malicious domain names","app_proto":"dns","target":"bbs.india-videoer.com","src_port":"51095","engine_type":"intel","src_ip_country_code":"CN","proto":"udp","dst_port":"53","src_ip_geoloc":"39.9047,116.4072","src_ip_country":"China","family":"APT_htran (apt );APT_others (apt );APT_others(apt )","category":"malicious-domain-dns-query","desc":"family:APT_htran (apt )APT_others (apt )APT_others(apt );query:bbs.india-videoer.com;answers:;qtype_name:A;","sensor_name":"ATD"}

需要将以上数据源格式化为json存入es中，首先需要将

notice - - - 

替换为空字符串，然后在filter中配置自动解析json格式

2、logstash配置文件

input {
    stdin {}
}
filter{
    mutate {
        gsub =>[
            "message", "notice - - - ", ""
        ]
    }
    json{
        source => "message"
    }
}
output {
    stdout {}
}

3、输出效果

logstash会自动见json中的键解析为es中的字段进行存储