logstash使用“|”切分日志报错：Exception caught while applying mutate filter {:exception=>"Invalid FieldReferen...

最新推荐文章于 2022-08-23 09:36:38 发布

不想下火车的人

最新推荐文章于 2022-08-23 09:36:38 发布

阅读量569

点赞数 1

本文链接：https://blog.csdn.net/wuxun1997/article/details/115527286

版权

　　如题，通过FileBeat输入的日志里有“|”分割，希望拿到“|”切分后的字段，在logstash配置文件中配置：

input{
 beats {
    port => "5044"
  }
}

filter{
    mutate {
      split => {"message"=>"|"}
    }
    
    mutate {
        add_field => {
            "helloObject" => "%{message[1]}"
        }
    }        
          
}
output {
  stdout { codec => rubydebug }
}

　　跑logstash和filebeat后，报错：

D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T07:50:21,150][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T07:50:28,752][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
[2020-09-04T07:50:30,253][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
[2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T07:50:30,341][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T07:50:30,568][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
[2020-09-04T07:50:31,325][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate  ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}
D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T07:50:21,150][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T07:50:28,752][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
[2020-09-04T07:50:30,253][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
[2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T07:50:30,341][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T07:50:30,568][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
[2020-09-04T07:50:31,325][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate  ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}

　　虽然切是切了，但想要的日志字段没有加出来：

{
           "log" => {
          "file" => {
            "path" => "D:\\wlf\\logs\\hello-2020-09-04.0.log"
        },
        "offset" => 111785
    },
           "ecs" => {
        "version" => "1.5.0"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_mutate_error"
    ],
         "input" => {
        "type" => "log"
    },
    "@timestamp" => 2020-09-03T23:48:37.101Z,
         "agent" => {
        "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
                "type" => "filebeat",
             "version" => "7.9.0",
            "hostname" => "wulf00",
                "name" => "wulf00",
                  "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242"
    },
          "host" => {
        "name" => "wulf00"
    },
       "message" => [
        [0] "07:48:36.452 [scheduling-1] [] [] INFO  com.wlf.elasticsearchstatictis.Begin - hello",
        [1] "world."
    ],
      "@version" => "1"
}

　　配置文件没问题，但添加字段那一个mutate需要给message套一层中括号：

mutate {
        add_field => {
            "helloObject" => "%{[message][1]}"
        }
}

　　重新启动logstash，报错没了，message[1]也能取到了：

D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T08:20:10,122][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T08:20:10,584][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T08:20:13,684][INFO ][org.reflections.Reflections] Reflections took 45 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T08:20:13,937][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T08:20:16,047][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x3bb3812e run>"}
[2020-09-04T08:20:17,233][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.16}
[2020-09-04T08:20:17,268][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T08:20:17,326][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T08:20:17,536][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T08:20:17,607][INFO ][org.logstash.beats.Server][main][be332a0e557bc16ff2fcdf24fcbd3b24c16b1141269fff75b0e835137ee7698f] Starting server on port: 5044
[2020-09-04T08:20:18,205][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
           "host" => {
        "name" => "wulf00"
    },
            "ecs" => {
        "version" => "1.5.0"
    },
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "helloObject" => "world.",
     "@timestamp" => 2020-09-04T00:20:14.782Z,
          "agent" => {
            "hostname" => "wulf00",
        "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
             "version" => "7.9.0",
                  "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242",
                "type" => "filebeat",
                "name" => "wulf00"
    },
        "message" => [
        [0] "08:20:11.653 [scheduling-1] [] [] INFO  com.wlf.elasticsearchstatictis.Begin - hello",
        [1] "world."
    ],
       "@version" => "1",
            "log" => {
        "offset" => 131366,
          "file" => {
            "path" => "D:\\wlf\\logs\\hello-2020-09-04.0.log"
        }
    },
          "input" => {
        "type" => "log"
    }
}

不想下火车的人

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
logstash使用“|”切分日志报错：Exception caught while applying mutate filter {:exception=>"Invalid FieldReferen...

　　如题，通过FileBeat输入的日志里有“|”分割，希望拿到“|”切分后的字段，在logstash配置文件中配置：input{ beats { port => "5044" }}filter{ mutate { split => {"message"=>"|"} } mutate { ad...
复制链接

扫一扫