如题,通过FileBeat输入的日志里有“|”分割,希望拿到“|”切分后的字段,在logstash配置文件中配置:
input{
beats {
port => "5044"
}
}
filter{
mutate {
split => {"message"=>"|"}
}
mutate {
add_field => {
"helloObject" => "%{message[1]}"
}
}
}
output {
stdout { codec => rubydebug }
}
跑logstash和filebeat后,报错:
D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T07:50:21,150][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T07:50:28,752][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
[2020-09-04T07:50:30,253][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
[2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T07:50:30,341][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T07:50:30,568][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
[2020-09-04T07:50:31,325][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}
D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T07:50:21,150][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T07:50:28,752][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
[2020-09-04T07:50:30,253][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
[2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T07:50:30,341][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T07:50:30,568][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
[2020-09-04T07:50:31,325][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}
虽然切是切了,但想要的日志字段没有加出来:
{
"log" => {
"file" => {
"path" => "D:\\wlf\\logs\\hello-2020-09-04.0.log"
},
"offset" => 111785
},
"ecs" => {
"version" => "1.5.0"
},
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_mutate_error"
],
"input" => {
"type" => "log"
},
"@timestamp" => 2020-09-03T23:48:37.101Z,
"agent" => {
"ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
"type" => "filebeat",
"version" => "7.9.0",
"hostname" => "wulf00",
"name" => "wulf00",
"id" => "ae375dc0-d6e2-488c-be87-2544c05b1242"
},
"host" => {
"name" => "wulf00"
},
"message" => [
[0] "07:48:36.452 [scheduling-1] [] [] INFO com.wlf.elasticsearchstatictis.Begin - hello",
[1] "world."
],
"@version" => "1"
}
配置文件没问题,但添加字段那一个mutate需要给message套一层中括号:
mutate {
add_field => {
"helloObject" => "%{[message][1]}"
}
}
重新启动logstash,报错没了,message[1]也能取到了:
D:\elk\logstash-7.9.0\bin>.\logstash -f ..\config\logstash-simple.conf
Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
[2020-09-04T08:20:10,122][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
[2020-09-04T08:20:10,584][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-09-04T08:20:13,684][INFO ][org.reflections.Reflections] Reflections took 45 ms to scan 1 urls, producing 22 keys and 45 values
[2020-09-04T08:20:13,937][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
[2020-09-04T08:20:16,047][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x3bb3812e run>"}
[2020-09-04T08:20:17,233][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.16}
[2020-09-04T08:20:17,268][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-04T08:20:17,326][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-04T08:20:17,536][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-04T08:20:17,607][INFO ][org.logstash.beats.Server][main][be332a0e557bc16ff2fcdf24fcbd3b24c16b1141269fff75b0e835137ee7698f] Starting server on port: 5044
[2020-09-04T08:20:18,205][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"host" => {
"name" => "wulf00"
},
"ecs" => {
"version" => "1.5.0"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"helloObject" => "world.",
"@timestamp" => 2020-09-04T00:20:14.782Z,
"agent" => {
"hostname" => "wulf00",
"ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
"version" => "7.9.0",
"id" => "ae375dc0-d6e2-488c-be87-2544c05b1242",
"type" => "filebeat",
"name" => "wulf00"
},
"message" => [
[0] "08:20:11.653 [scheduling-1] [] [] INFO com.wlf.elasticsearchstatictis.Begin - hello",
[1] "world."
],
"@version" => "1",
"log" => {
"offset" => 131366,
"file" => {
"path" => "D:\\wlf\\logs\\hello-2020-09-04.0.log"
}
},
"input" => {
"type" => "log"
}
}