logstash使用csv插件解析格式化日志

csv插件官方手册：https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-columns

需求示例：用filebeat收集原始日志文件样例如下：

filebeat 收集后进入logstash，利用logstash中的csv插件，将上述日志内容解析到指定的字段名中，如字段名依次为：“date_time”, “username”, “ssh_connection”, “pwd”, “pid”, “action”

logstash中filter部分配置如下：

filter {
    if [log_type] == "bashhistory" {
        mutate {
            replace => [
                "agent_id","%{[agent][id]}",
                "agent_version", "diagent-%{[agent][version]}",
                "hostname","%{[agent][hostname]}",
                "os_name","%{[host][os][name]} %{[host][os][version]}",
                "os_mac","%{[host][mac]}",
                "os_kernel","%{[host][os][kernel]}",
                "os_platform","%{[host][os][platform]}",
                "log_path","%{[log][file][path]}"
            ]
            remove_field =>["host","ecs","input","log","@version","tags","agent"]
        }

        csv {
            separator => " ### " # 日志中的字段间的分隔符，默认为","
            columns => ["date_time", "username", "ssh_connection", "pwd", "pid", "action"]
        }
    }
}

解析后的结果如下：

注意：message中不能有双引号，否则会导致csv插件解析失败